There are no specific guidelines in the Constitution on what the Governor should do or whom he or she should invite in case there is a face/off between parties or coalition allies. In this matter, the Sarkaria Commission recommendations, which were affirmed by a Constitution Bench of the SC in Rameshwar Prasad vs Union of India in 2005 are often the guiding principle and are the main reference point. The mandate is outlined in the Commission recommendation. 

The Commission report details the options before the governor in the situation where no single party has obtained an absolute majority. 

The order of preference for the Governor in such case is 

(1) An alliance of parties that was formed prior to the elections

(2) The single largest party staking a claim to form the government with the support of others, including independents

(3) A post-electoral coalition of parties, with all the partners in the coalition joining the government

(4) A post-electoral alliance of parties, with some of the parties in the alliance forming a government and the remaining parties, including independents, supporting the government from outside.

Sarkaria Commission was set up in June 1983 to examine the relationship and balance of power between state and central governments and suggest changes within the framework of the Constitution.

However, the discretionary powers of the Governor offer him or her vast liberty in actions. According to the Constitution, the decision of the Governor is final and the validity of anything done by him as a matter of his discretion cannot be questioned.

Source: http://ddnews.gov.in/national/what-are-sarkaria-commission-recommendations-case-hung-assembly