The language

Very hard.

If you ever have learned French, I can tell you that the grammar is quite similar to the French one (hard, with lots of tricky rules). But for myself, the worse part is to understand, they don't pronounce all the letters as in spanish and the rules of pronuncation are difficult to get. Are you learning the Brasilian Portugaise or the Portugal one? Cause the Portuguese of Portugal is harder to get.

It's an oblique language - I've a '30s all-language phrasebook. 'These eggs are bad' translates directly in Spanish, but in Portuguese becomes 'these eggs are not good'. Anyway, try being hungry on a wet winter's night with five euros in your pocket, that'll break the fear barrier for you.  Read, practice, dare, love it.