I know that this is an old post, but I know what the problem is.

The 640x480 is interlaced and when you're reducing it to 320x240 you're either using one field instead of two or two fields instead of one, or your field dominance (even vs odd) is wrong.