Getting larger files than expected using Xvid to compress AVI

[pronco]

Hi everyone, I am reposting to this forum on the suggestion of an Afterdawn reader. If anyone can help me with my problem it would be greatly appreciated. Thanks,

-Paul




16 Aug 2011 @ 18:28
pronco Newbie

Hi all, I am having a problem and was hoping someone might be able to help me out.

Using Virtualdub, I have ripped an old, 90-minute interlaced VHS movie to uncompressed AVI. The file is 144 GB in size. I want to be able to reduce the size to a compressed AVI format that is under 1 GB but still retains the original video quality.

Using Virtualdub as my editor, I have used the XVid codec and its included Minicalc program to find a bitrate that will provide results for under 1 GB in size, but test runs always produce very noticeable artifacts. To retain similar quality to the original, the smallest file I have been able to create is about 4.7 GB in size. I suspect it can be made smaller, though, because I have seen 2-hr DVD movies in HD that were encoded with XVid that were under 1 GB in AVI format, and these movies retain a high level of their original quality. What is the trick that these ripped DVDs are using to get such good compression/quality ratios? When I look at the DVDs' file information in Virtualdub, they have unfamiliar framerates such as 23.97 or 25, whereas the VHS movie I ripped has 29.97. I am afraid to tinker with the framerate as I understand it could de-synch my audio. I also notice that the DVD rips have odd resolutions such as 624x272, 720x304, 608x256 and the like. My rip is standard 4:3, 640x480. Do I need to drop my framerate or adjust my resolutions? I can't imagine that this should be necessary. I suspect I am missing something else.

Thanks,

-Paul



17 Aug 2011 @ 9:33
attar AfterDawn Addict

The size of any compressed file will always be the product of the running time and the bitrate (time in seconds x rate).
Neither the resolution (frame size) nor frame rate affects the size of the output.
(North America) NTSC video (for playback on a TV) is 29.97fps.

Newer compression methods (with smaller file size) generally give better results than the older codecs like XviD (.mp4 using x264 codec?) but the quality can't be better than the original and playback on a standalone DVD player is out.


I would try the Videohelp forum for more expert advice.

https://forum.videohelp.com/forums/10-Capturing




18 Aug 2011 @ 2:27
pronco Newbie
Thank you. If you don't mind, I will repost my question (with your reply) to the other forum, to see what others have to say.

Good to know about the bitrate * seconds rule. That makes sense. Still, some of the DVD rips were encoded in XVid and they seem to be a LOT better quality than what I was able to get from the tape. I wonder if that had anything to do with my source being from VHS and interlaced. The uncompressed tape rip, on the other hand, was true to the quality of the tape, which was also pretty good.

I have no idea how some of these DVD rips got down to bitrates of 112 kb/s while still preserving so much quality.

-Paul

[DB83]

These 'odd' sizes are the result of encoding a wide-screen (16:9 or 2.35:1) dvd. If your source is 4:3 then you must keep to a 4:3 ratio.

25 fps is the PAL speed. 23.98fps is a pull-down variant of NTSC.

You will NEVER get the quality that you see from a dvd rip. Thse will typically be produced from the original ripped VOBs but if you do have an uncompressed source you should get a reasonable final quality.

I am no vdub expert but I do wonder if your final avi had retained the audio compression of your original capture(your rip). The calculator only deals will the video bitrate and your original audio is also uncompressed whereas in the final rip it shoule be compressed mp3.

Instead of vdub, I would suggest avidemux where you can experiment easier with audio and video codecs.

[hech54]

I want to be able to reduce the size to a compressed AVI format that is under 1 GB but still retains the original video quality.

Never gonna happen.

[dragonkeeper]

I want to be able to reduce the size to a compressed AVI format that is under 1 GB but still retains the original video quality.

Never gonna happen.

I agree especially using VirtualDub alone, VHS usually requires some hefty lifting (avisynth to increase the compressibility and perceived quality), since this is VHS we're speaking of you de-interlacing\decombing the video will help improve look of the video a great deal.
There are many methods that can be used to increase the perceived quality of the video but it will never look as good as a DVD. By the way how are you compressing your audio mp3, vorbis, ac3? This will have a huge bearing on final file size.

[edDV]

VHS is noisy and thus won't compress cleanly without heavy (and lossy) noise reduction filtering before compression. If this is film source (movies or TV series) the VHS recording will be telecine. Most noise reduction filters will perform better if you inverse telecine first.

In other words, restoring and compressing VHS recordings is hard work and requires high skill. DVD versions of most movies are very cheap online. What is your time worth?

[pronco]

Thanks everyone. These responses have been a big help. All of the reasoning makes sense. Yes, I was compressing audio to 41000 162 CBR MP3. So I take it that, if I decide to do this, I should invert telecine and deinterlace, and accept a somewhat larger file size in the end due to the inherent VHS noise. Apparently I am going to have to re-rip the source. Can I invert telecine, deinterlace, and use noise reduction filters all at the same time? What VHS noise reduction filter do you recommend for Vdub? This movie was never transferred to DVD, which is why I'm willing to spend the time on this.

Thanks again.

[Bjs]

You could try avidemux using xvid via average bitrate ... use the first pass as null at around 4500kbps and the second pass as indicated by the inbuilt xvid calculator 1000mb output ... you should use some filters to help retain and cleanup the vhs source before compression takes place ... Ive run a few tests lately and found the average rather than fixed encoding retains a lot more quality ... and drop audio down to mp3 at 22,000khz / 6kbps ... more space for video in final size output.

[edDV]

If the source is normal interlace (e.g. camcorder or live TV) best to keep it interlace to the DVD unless extreme compression is the goal.

If film source (movies or most TV series), then inverse telecine before noise reduction.

You don't deinterlace and inverse telecine.

[pronco]

This is an old rental video. Is that considered film source?

[edDV]

This is an old rental video. Is that considered film source?

If it is a cinematic movie, yes. If sports or SNL, no.

[pronco]

Thanks everyone. From the info you've provided I now I have a good idea of why I'm running into these problems... inherent dirtiness of VHS complicated further by interlace and telecine. For the sake of remaining true to the source, I'm not going to do the additional edits suggested. With the exception of removing the distorted overscan lines with Virtualdub's "resize" filter, I'm leaving the video as-is. Interlaced, using XVid compression. The result is a 90 minute 640x480 movie approximately 4.7 GB in size that is almost identical in perceived quality to the uncompressed AVI rip.

One last question however, I'd like to clean the pops out of the audio. I have tinkered with Cool Edit's hiss/pop removal but that seems more geared towards removing noise generated by vinyl records. I will continue to try to work with Cool Edit, as that is the sound editing application I have experience with, but does anyone know of a good way to clean the audio from a cinematic release VHS tape?

[pronco]

One clarification... I never suggested that I wanted to be able to make the VHS rip look as good as a DVD. I know that is impossible. I was comparing the compression/quality ratios and wondering why I couldn't get <1GB files similar in quality to the original 90-minute VHS when <1GB files have been made of DVD releases that are even longer.

[DB83]

Post the mediainfo of your last avi - the 4.7 gig one. If you read the above posts there is no reason why an avi should be so big and still have a percieved good quality. 'Perceived' is the all important word here as, whatever its size, it will not have the actual quality of your uncompressed rip.

In my own opinion it is a big mistake to crop the video to remove VHS overscan artifacts. When you replay your vid on a tv you would not see them anyway. The cropping invloves extra re-encoding and that also has an adverse affect on final quality.

[edDV]

In my own opinion it is a big mistake to crop the video to remove VHS overscan artifacts. When you replay your vid on a tv you would not see them anyway. The cropping invloves extra re-encoding and that also has an adverse affect on final quality.

I fully agree and was going to post a similar comment. Resizing interlace video creates serious artifacts. Deinterlace or inverse telecine followed by resize and recode further lowers quality. I would do this only for video intended for internet distribution.

When I do a similar VHS project, I target DVD spec MPeg2. I usually keep interlace source interlace even telecine source. I let the players and TV sets do the deinterlace or inverse telecine. Now that hard disks are cheap, I see no reason to compress tighter than ~4 Mb/s MPeg2.

[pronco]

The intended audience is online, primarily, but I'm keeping the interlace to preserve original quality. Also in the event anyone wants to download and burn to DVD for television watching. I find that having to look at the overscan throughout the movie becomes a nuisance. I don't see how cropping the edges should involve additional re-encoding. It just lops some of the signal off. Vdub will automatically center the cropped video in a letterbox, so original aspect ratio is preserved. I perform the cropping in a cascade, just prior to compression, in one pass. When I scroll through the uncompressed AVI, Vdub's output pane shows the cropping instantly. There is no lag. So I don't think additional encoding is taking place.

I know the compressed version cannot ever be technically identical in quality to the original. I tried several compression ratios, however, and settled at the point where I couldn't tell the difference.

Here is the output: http://www.youtube.com/watch?v=nLjdD-PGGu0

For VHS I think it's pretty good. Maybe a little washed-out, but I noticed that making it darker lost detail.

[DB83]

My own opinion about the re-encoding does not matter but you obviously know better than one who has been on here muuch longer than both of us.

It happens. End of.

I checked that clip. I believe I noticed several interlace artifacts in the 360p version. The 480p version was even worse.

[pronco]

Well, I gave my reasons for why I believe what I believe. I didn't say you were wrong, 'end of.' If you want to explain why cropping the sides of the video requires additional encoding, and why my reasoning is wrong, please do.

Of course you are going to notice interlace artifacts... I kept it interlaced. There are trade-offs no matter which route I had taken.

What do you mean, 360p version and 480p? Are you referring to the 1 of 7 clip as opposed to the 7 of 7 clip? Even still, they should reflect the same type of compression... I thought I had used exactly the same settings. The only difference between the two should be that a different copy of the tape was used.

[poisondeathray]

If you want to explain why cropping the sides of the video requires additional encoding, and why my reasoning is wrong, please do.

I don't think it does in your case, because you're going to be encoding to some distribution format anyways. It's like a "sunk cost"; you're encoding eitherway. The cropping transformation is applied before the encoding , not after

Of course you are going to notice interlace artifacts... I kept it interlaced. There are trade-offs no matter which route I had taken.

The problem is this is a film source. ie. progressive. By not IVTCing, you leave combing artifacts. Not only does this look bad on youtube, a progressive display, it's negatively impacts the encoding efficiency. You have have extra frames to encode (29.97fps instead of 23.976fps), the combing acts as "noise" and is more difficult for the encoder, both the one you used to encode, and then again when youtube re-encodes it.

If end users want to make a dvd, they take the progressive film source , progressive resize it to NTSC DVD 720x480, and encode with MPEG2 using soft pulldown flags. Much better quality




Also , I know you think you addressed this, but I think the black level is way too high. That is causing the "washed out" look

If you were concerned about filesize, then consider using x264. You could probably get it down to 1/2 to 2/3 of the size and still better quality than using xvid at these bitrates that you are using. Or better quality at the same filesize . Of course, when youtube butchers it , you might not see the big difference. Youtube is the weakest link here.

What do you mean, 360p version and 480p? Are you referring to the 1 of 7 clip as opposed to the 7 of 7 clip? Even still, they should reflect the same type of compression... I thought I had used exactly the same settings. The only difference between the two should be that a different copy of the tape was used.

He means youtube provides several versions. If you look at the drop down menu ther is a 360p version and a 480p version for each video

This is an old rental video. Is that considered film source?

I hate to be the "narc" here, but isn't this a problem uploading rental content ?

[DB83]

I will happily withdraw the no additional re-encoding remark if the OP answers the request for the mediainfo report of the Xvid as created - the one AFTER the overscan line were cropped away.

After all the topic revolves around the stated fact that the final encode is somewhat larger than expected. I would like to see the mediainfo report to pass comment on this and also satisfy myself that no additional re-encoding is taking place.

[pronco]

I will get to all of this. Thanks. To answer the "narc" question, however, this would not be the first Hollywood movie uploaded to Youtube in its entirety (e.g. The Neverending Story, Conrack, the list goes on). If Paramount wanted it gone, they would have flagged it by now. This movie was never transferred to DVD, the (aging) VHS rental copies are no longer in production, and as far as I can tell no one else on the Internet has done the rip. As such I am archiving it as fair use-- like I said, if Paramount wants it gone, Youtube will ensure that it is removed. I had to mute the first two minutes of the film, in fact, because Sony complained that it used too much of Jefferson Airplane's 1967 track, "White Rabbit." The 1966 track "Wild Thing" by the Troggs, however, which is played in its entirety in the end credits and is arguably just as famous as "White Rabbit," has received no flags.

[pronco]

Alright, I've just confirmed that the "washing out" of the black is caused by Youtube's butchering process. So, there's nothing I can do about that. My source video looks fine. Here is the mediainfo of the cropped xvid file:

General
Complete name : C:\Documents and Settings\Owner\Desktop\New Folder\wt7.avi
Format : AVI
Format/Info : Audio Video Interleave
File size : 781 MiB
Duration : 14mn 44s
Overall bit rate : 7 405 Kbps
Writing library : VirtualDub build 32842/release

Video
ID : 0
Format : MPEG-4 Visual
Format profile : Advanced Simple@L5
Format settings, BVOP : 1
Format settings, QPel : No
Format settings, GMC : No warppoints
Format settings, Matrix : Default (H.263)
Muxing mode : Packed bitstream
Codec ID : XVID
Codec ID/Hint : XviD
Duration : 14mn 44s
Bit rate : 4 321 Kbps
Width : 640 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate : 29.970 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Bottom Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.469
Stream size : 456 MiB (58%)
Writing library : XviD 64

Audio
ID : 1
Format : PCM
Format settings, Endianness : Little
Format settings, Sign : Signed
Codec ID : 1
Codec ID/Hint : Microsoft
Duration : 14mn 44s
Bit rate mode : Constant
Bit rate : 3 072 Kbps
Channel(s) : 2 channels
Sampling rate : 96.0 KHz
Bit depth : 16 bits
Stream size : 324 MiB (41%)
Interleave, duration : 33 ms (1.00 video frame)
Interleave, preload duration : 500 ms

I am looking into Inverse Telecining now. From what I understand from reading a doom9 post, there is a subtle difference between IVTC and deinterlacing, a difference that, I certainly admit, I am not exactly clear on. I tried a preliminary deinterlace test using Virtualdub's "deinterlace" feature at Yadif algorithm basic settings, but that produced a lot of snow. Gunnar Thalin's "deinterlace-smooth" filter stretched my video to 640x960 for some reason and I could not figure out why. Perhaps from looking at this mediainfo you could tell whether it should be IVTC'd or deinterlaced, as I am not sure what to look for. Honestly I am starting to wonder whether it would be worth it. As DB83 pointed out, yes, I see some combing artifacts at 360p, and 480p is intolerable. But at 360p they are not too noticeable and the video is nice and crisp. I remember reading a Lordsmurf article that recommended to always stay true to the source... e.g. if it's interlaced, leave it interlaced. Then again, that's what he said about deinterlacing. I don't know if IVTC falls under the same category.

I understand that this tape was never meant to be viewed on a progressive output, which is basically what I am trying to do. I also know that keeping the video interlaced will produce larger filesizes upon compression. If anyone can recommend whether I should ITVC or deinterlace, and what filter they would use, I will definitely try it.

I will try x264 instead of XVid for compression and see how that goes. Still trying to figure out how to get the x264 filter into Vdub...

[manono]

Perhaps from looking at this mediainfo you could tell whether it should be IVTC'd or deinterlaced, as I am not sure what to look for.

You said it's a movie, right? And movies aren't 29.97fps. It should be IVTC'd. Preferably the source should be IVTC'd, and not your cropped and letterboxed version. Especially for upload it should be made progressive, and the best way to do that is with a full-on AviSynth IVTC of the source. By deinterlacing it you ruin it by either creating jerky playback due to every fifth frame being a duplicate, or blurry strobing playback with 2 of ever five frames being blended, depending on the deinterlacer used.

Samples would be welcome, especially a short 10 seconds from the source, one showing steady movement.

Alright, I've just confirmed that the "washing out" of the black is caused by Youtube's butchering process.

The black levels in my 1000+ videos on YouTube look fine.

For VHS I think it's pretty good. Maybe a little washed-out, but I noticed that making it darker lost detail.

It could have been a lot better. It's still interlaced, with 2 of every five frames being interlaced and, as you know, the black levels are very bad. They can be properly restored without losing detail. By IVTCing the thing you gain a lot of compressibility. That is, you'll get much better quality for a much smaller file size.

I thought people whose accounts are in good standing on YouTube could now upload videos as long as they wanted? That they're no longer bound by that 15 minute limitation? Someone correct me if I'm wrong.

[pronco]

The black levels in my 1000+ videos on YouTube look fine.

Not sure why that is, then. I compared my original file with what it turned into on Youtube and there is a clear and obvious difference in the black levels. The saturation was changed during Youtube's processing, of that there is no doubt. Whether it may have happened because of the way my file was encoded is another question, however.

You said it's a movie, right? And movies aren't 29.97fps. It should be IVTC'd. Preferably the source should be IVTC'd, and not your cropped and letterboxed version. Especially for upload it should be made progressive, and the best way to do that is with a full-on AviSynth IVTC of the source.

This might be what I have to do, then. It might even resolve the wash-out issue. Would it be possible to IVTC my rip, however? (I am talking about the raw 122 GB source file-- prior to the crop, letterbox, and compression phase). I have made some time-consuming edits, and starting over entirely from scratch to re-rip the VHS tape would be a real pain. The rip was uncompressed AVI, so I am assuming that if the rip could be IVTC'd, it's probably already in the best format possible for doing so.

By deinterlacing it you ruin it by either creating jerky playback due to every fifth frame being a duplicate, or blurry strobing playback with 2 of ever five frames being blended, depending on the deinterlacer used.

My test runs weren't jerky, but "blurry strobing" might be another way to describe what looked to me like snow. Not really "snow" as in television static, but lots of little blurry blotches all over the screen. Does IVTC remove the interlacing?

I thought people whose accounts are in good standing on YouTube could now upload videos as long as they wanted? That they're no longer bound by that 15 minute limitation? Someone correct me if I'm wrong.

I have looked into this. You have to apply for a "partner" account, which I wouldn't be able to qualify for for various reasons.

http://www.youtube.com/t/partnerships_benefits

Regular users have always been restricted first to 10, and now to 15 minutes. It doesn't have to do with good standing. Bad standing would lose you access to the partner account, however.

[DB83]

Since your vid is now resized to 640*480 then I believe that answers the no additional re-encoding puzzle. I was specifically referring to a 720*480 source that remains at 720*480 ie a crop and a resize of the original source.

Now to answer the question of the size of the encoded clip. You have indeed re-encoded the video from uncompressed to xVID - the bitrate appears a little high for xVID compression - 1200 - 1500kbps should give you decent visual quality. However, the audio has not been re-encoded. It still remains an uncompressed PCM and not compressed mp3. File size is all about bitrate and you should now see that your video/audio bitrates are way to high. Having said that it is a matter of choice and if you are happy with the result then so be it.

I have no idea what youtube does to the video in its encoding process. But to get the best quality you should follow their guidelines.

[manono]

My test runs weren't jerky, but "blurry strobing" might be another way to describe what looked to me like snow.

The best way to tell what the result of deinterlacing it is by first pausing it and then advancing by a frame at a time. Either every fifth frame will be a dupe or 2 of every 5 frames will be ghosted/blended/have a double image. To repeat, deinterlacing a film source is about the worst possible thing you can do to it. It's almost as bad as leaving in the interlacing, as it is with your video on YouTube now.

Would it be possible to IVTC my rip, however?

The uncompressed source file? Yes, unless it's already been resized in the vertical direction, which I don't think is was. Yes, just use a standard AviSynth IVTC, one such as TIVTC. VDub also has an IVTC, but it's not very good.

I have looked into this. You have to apply for a "partner" account, which I wouldn't be able to qualify for for various reasons.

Nah, plenty of people have (or had, not sure) the ability to upload videos longer than 15 minutes. Plenty of 'normal' people have uploaded entire movies, that is, people who aren't partners. YouTube is a little unclear about just who can and who cannot upload those longer videos, but this thread explains how to check:

http://www.google.com/support/forum/p/youtube/thread?tid=7a24fa07803bd71f&hl=en

[pronco]

Wow. This has been an eye opener. I didn't know my audio had remained uncompressed. Clearly a big problem right there, size-wise. I will check out the 1200 - 1500kbps suggestion, too. Thanks for suggesting the avisynth for the IVTC, and for letting me know that it can be performed on my source file. Lastly, I will see if Youtube allows me to go over the 15 minute limit.

This is a lot to tackle. I will let you know how things turn out.

[jagabo]

The black levels in my 1000+ videos on YouTube look fine.

Not sure why that is, then. I compared my original file with what it turned into on Youtube and there is a clear and obvious difference in the black levels. The saturation was changed during Youtube's processing, of that there is no doubt. Whether it may have happened because of the way my file was encoded is another question, however.

This is happening to you because the graphic settings on your computer are wrong. When a video is playing in your web browser the flash player is converting the YUV video to RGB with the standard rec.601 matrix and sending that RGB to the graphics card, were it is adjusted by the Desktop proc amp settings before being sent to the monitor. When you play a video locally in a media player the YUV video is sent directly to the graphics card. The graphics card converts the video to RGB using a rec.601 matrix and the video proc amp settings (not the Desktop proc amp settings). Finally the adjusted video is sent to the monitor.

You can verify this by downloading one of your videos from Youtube and playing it in a media player. You will see that the downloaded video looks the same as the source you uploaded. Don't compare two videos playing side by side on the screen in two media players. Often one is using the graphics card's hardware to convert YUV to RGB, and the other is performing the conversion in software ("video overlay" is sometimes limited to one instance).

You need to calibrate your system so that both software and hardware are converting YUV video to RGB video properly. You may also need to calibrate your monitor so that it displays RGB video properly. Start by using a video levels chart like the video in this post:

https://forum.videohelp.com/threads/326496-file-in-Virtualdub-has-strange-colors-when-o...=1#post2022085

[pronco]

Alright, then is it safe to say that I cannot trust the quality of my original source video now, which looks fine on MY monitor? When I performed the original rip, I tuned the color levels of the source video using my (uncalibrated) monitor and software settings. Should I recalibrate my monitor and settings, and then re-rip the video, or at least then re-adjust the color levels when I encode the final output file, so that I can be sure that the video is correct, and that if other people have saturation problems, it can be safely blamed on their own monitors and settings?

I'd like to know this answer before I go ahead and work on this (yet again) new project. While I could go ahead and calibrate my monitor and settings, I don't really know how to do that yet, and I'm suspecting that once I've done it, I will realize that my source video is off anyway.

[jagabo]

Alright, then is it safe to say that I cannot trust the quality of my original source video now, which looks fine on MY monitor?

Yes. Your graphics card and monitor probably need calibration.

Should I recalibrate my monitor and settings, and then re-rip the video, or at least then re-adjust the color levels when I encode the final output file, so that I can be sure that the video is correct

Yes.

if other people have saturation problems, it can be safely blamed on their own monitors and settings?

Yes.

While I could go ahead and calibrate my monitor and settings, I don't really know how to do that yet, and I'm suspecting that once I've done it, I will realize that my source video is off anyway.

The most important thing to get right is the black and white levels -- basically, the contrast. Start with the desktop, something like this:

http://www.lagom.nl/lcd-test/all_tests.php
http://www.imaging-resource.com/ARTS/MONCAL/CALIBRATE.HTM

A google search will turn up many more.

Then move on to video test patterns like the one I linked to.

[pronco]

Okay, I just did the tests, and my Sony LCD (SDM-X93) monitor displays all of the different white and black blocks. However, I can clearly see the black and white lines inside the 50% grey box at the end of the test on http://www.imaging-resource.com/ARTS/MONCAL/CALIBRATE.HTM

I have a hard time believing that this should look like a uniformly grey box. Am I misinterpreting something?