Bill,
There are plenty of upsides to expat life in VN, but I won't go into them because you specifically asked for the downsides. After 5.5 years in Saigon and 4.5 years in KL, here goes:
1) -- Food. Generally lovely and fresh with lots of healthy greens etc, but ignore the potential traces of mercury, arsenic, PCBs and dioxin (agent orange) in the groundwater. I've never been game to touch river fish and am wary about coastal seafood. On top of that, after a while you begin to realise there are only about ten dishes or so; everyone does them differently but they do get a bit same-same.
2) -- Air pollution. There are worse places in the world, and it's worse in Hanoi than Saigon, but still...
3) -- Traffic. It seems chaotic, but get a motorbike, learn to go with the flow (forget any official 'rules') and you'll be right. Just remember that no-one gives way; carefully insist on merging (or crossing the road as a pedestrian) and they'll simply swerve around you in a general give-and-take. I've only ever seen what you could call "road rage" once.
4) -- Medical care. VN is trying to sell itself as a destination for first-class dental treatments that are dirt-cheap by first-world standards, and by most accounts it's succeeding. However, for anything else, get the most expensive insurance you can afford and you'll get preferential treatment for what it's worth -- which isn't much. I slipped on our wet marble porch while holding a glass of coke, fell with my bum straight onto the glass, resulting in a wide cut two finger-joints deep through my pants. Made my way with heavy bleeding (towel on the taxi seat) to the FV hospital in Saigon's D7, reputed to be one of the best hospitals in the country (French management). After a four-hour wait they removed some glass shards and put me in a shared room for two days. Nobody gave me any consistent updates, but after two days I was let go with some Betadine to bathe my bum twice a day, and told to come back after a week. Fortunately, a friend of my wife visited from Australia when I came home. She's an experienced trauma nurse, took one look at the wound and said it looked ugly and needed stitching urgently. So, I contacted my insurance and booked a flight to Bangkok the next day, where they put in 15 external stitches and 14 internal. Lesson learned. For anything more than a superficial wound, go to S'pore or Bangkok.
5) -- Diseases. If you stick to the main urban areas, there aren't many you need to worry about. Just make sure your DTP jab is up to date, just like you should back home. Malaria is not really an issue in the cities but dengue is, as I experienced with close friends in Jakarta and Saigon. There's no vaccination or prophylactic. Talk to your doctor or tropical-diseases specialist about mosquitoes.
6) -- Officialdom. Where to start with make-work procedures in a top-down bureaucracy? Keen to promote tourism? Take a look at their neighbours' smooth visa procedures on arrival. Ditto residency and work permits. Local 'agents' will help for a massive fee (which might include underhand payments), even for something that should be straightforward like transferring a foreign rider's/driver's licence.
7) -- Language. I've mentioned it before on this forum. I like to think I have a head for languages and speak several fluently, but I gave up on the tonal aspects of Vietnamese, where a slight inflection in a three-letter word like "nam" can mean totally different things. I had no trouble picking up enough to get by in Malaysia, but Vietnam required lots of smiles and sign language. Fortunately the Vietnamese I encountered always had a great sense of humour and were happy to help.
8) -- Variety. Someone said here that there isn't much of it in the country, but I beg to disagree. The cultural differences between the north and south are massive -- more so than between northern and southern Italy which also has a north-south geographic stretch with different seasonal patterns. Political inclinations aside, just look at language, where they often have trouble understanding each other (in the centre, in Hue, that applies too). And when it comes to landscapes, the country has everything from stunning coastlines to mountain rainforests and even snowy peaks in winter. It doesn't have mighty deserts but everything else is catered for. Viet vets, much as they hated being here, often commented on the country's beauty.
9) -- Other downsides? Sure, there are more, just like there are more downsides anywhere else in the world. But often they're more than balanced by upsides depending on your priorities -- and they will shift when you shift :-)