I still don't get when I win or lose at Caravan

I thought I'd gotten the hang of this rather interesting game, but every now and then I seem to win or loose eventhough neither I or the opponent have three complete caravans. My last match, he even had an empty caravan and I suddenly lost. I thought that you win when you have three caravans that all outnumber the opponent but things don't seem to be that simple.

Have I misunderstood the winning conditions?