HansensUniverse,
Given you're using
Avisynth, try
CropResize. If you follow the link there's pictures showing how to use it.
5:4 is the storage aspect ratio or resolution. Your capture is anamorphic so it has to be resized to the correct display aspect ratio (or dimensions) on playback. Or in your case, while upscaling.
The purpose of CropResize is to crop and resize while preventing aspect error, although it's calculations are based on the input display aspect ratio you tell it to use. A VHS capture should have a 15:11 display aspect ratio. It's slightly wider than 4:3 as only 704 pixels of the width should contribute to a 4:3 aspect ratio. The extra 8 pixels each side are probably black or crud.
This is an example for resizing 4:3 PAL to non-anamorphic dimensions (square pixels). Let's assume you need to crop 8 pixels from the left, 10 from the right and 4 from both the top and bottom to remove all the crud (of course cropping is optional). The first two numbers are the output width and height. If you specify both, the function will crop more picture if necessary to prevent distortion when it resizes, so it's possible to specify only one and zero for the other, which the script will then calculate for you.
CropResize(768,0, 8,4,-10,-4, InDAR=15.0/11.0)
If you want an output that's exactly 4:3, specify any exact 4:3 dimensions and the script will crop extra picture if necessary to make it 4:3 before resizing (based on the specified cropping below, it'd only need to crop a few extra pixels for 4:3).
CropResize(960,720, 8,4,-10,-4, InDAR=15.0/11.0)
Personally I wouldn't upscale unless there's a need to, but if you do need to, I'd consider converting the colors to the HD standard, although for VHS it probably won't make a noticeable difference.
CropResize(960,720, 8,4,-10,-4, InDAR=15.0/11.0, ColorMode="601-709")
If you're mixing the VHS content with 16:9 content, you can also upscale to your desired 16:9 resolution while adding borders to the sides. When adding borders, the script generally won't have to crop much extra picture, if any, in order to resize without aspect error. To add borders you must specify both an output width and height, and Info=true will tell you what CropResize is doing.
CropResize(1920,1080, 8,4,-10,-4, InDAR=15.0/11.0, ColorMode="601-709", Borders=true, Info=true)
Of course you need to de-interlace before CropResize in the script. How are you de-interlacing? With QTGMC?
Oh, and if you think the correct display aspect ratio for the capture is 4:3 rather than 15:11, use 4.0/3.0 for the InDAR instead.
CropResize(1920,1080, 8,4,-10,-4, InDAR=4.0/3.0, ColorMode="601-709", Borders=true)