Leaving Oman

Pls sir, I really need your help and advise quickly I have read a similar cases to mine but still need further explanations, I have work for six month and I want to cancel my contract with my employer but he took me to the agent that brought me to Oman, for more than a month now I have been with the agent because they also request for money. My brother in Dubai got me a visit visa which will expire this month but the agent was insisting on leaving Oman to my country Nigeria before going to Dubai, now am helpless and confused on what to do because my sponsor have already given them my passport but they refused to let me go. Pls kindly advise me on what to do, thank you