The foreign ministers of the countries, George Papandreou and Ismail Cem, described the deal as a major step towards solving what they say is an enormous problem.
Correspondents say that each year, up to 250,000 people from Africa, the Middle East and Asia try to enter Europe through the border between Greece and Turkey.
Under the new agreement Greece, can return all illegal migrants within 14 days of their arrival, although Athens says it will not apply it for political refugees.
The agreement was signed during a visit to Athens by Mr Cem, during which the ministers also agreed back United Nations efforts to restart talks on unifying the divided island of Cyprus.