Machine summary:
The provinces were governed by governors or 'amils appointed by the caliph himself as they were very important officers of the state and controlled Egypt, Syria and 'Iraq, which were far-flung from the centre.
As a matter of fact it was the viziers, secretaries, chamberlains, judges, postmasters-general, captains of the guards and the provincial governors who were all-important and performed the duties of the state in their respective departments and spheres.
In this connection Ibn Khaldun, tracing the origin of this office, remarks that the Umayyads and the' Abbasids gave this title of hajib to an officer who acted as the chamberlain or the person in charge of embassies and foreign affairs connected with the court of a caliph, giving admission to those who wanted to see him at the suitable time and keeping them out when necessary .
1 First there was the vizier, then the hajib, then the chief qadi, next the captain of the guard or sahib al-shurtalz, not to mension the muhtasib and the postmaster-general or sahib al-barid the last two of whom were perhaps not much in esteem at the court and last of all was the office of the secretary) several of whom acted as heads of departments or various diwans under the supervision of the grand vizier or oieir-i-i azam.
e. the bureau of taxes or the finance department and its presiding officer as the sahib al-kharaj particularly in a province where he worked under the governor or was directly responsible to the caliph.