The time you spend in the queue depends on how do you get to Gib:
1. There is no queue for pedestrians, just walk through without stopping (in / out)
2. There is no queue if you go in by bicycle but usually you spend around 5-15 minutes in the queue when you go back. There is a separate lane for bicycles.
3. If you have a bike / moped then it's the same quick to get in, but there could be a queue for getting out (10-20 min to several hours)
4. Taking car is the worst idea. It's not a problem to get in (15 minutes in average) but it's unpredictable how much time you spend in the queue in the evening. It could be 20, 40, 50 minutes or sometimes several hours.
My choice is bicycle 
Usually people chose La linea because:
- it's close to Gib
- there is quite a lot of places to go for a beer or diner
- you aren't obligated to have a car
Personally I live in Santa Margarita. It's 10 minutes drive to the border but you can find better quality apartment or house for relatively the same price in La Linea. And it's much quite and beautiful comparing to La Linea