Understanding conversion to lossless

[moxiecat]

Hello everyone,

I've been reading many threads about the lossless codecs and have tried my hand at converting via Lagarith, Huffyuv, and UT video (and regular DV via Cedocida). I don't have a question so much as I'd like confirmation that my understanding of the conversion process is correct.

Is the following statement true?: When converting a file to lossless, just make sure that the color space and chroma subsampling match the original.

Or are there other parameters that need to be set to make the conversion lossless?

For example, let's take a VOB file. The one I have is YUV, 4:2:0, 8 bits, and BT.601 NTSC (color primaries), per MediaInfo. And, this YUV is technically YV12, because of the chroma value, correct?

I converted this file to lossless using both UT Video and Lagarith. The UT video codec I used is "YUV420 BT.601 VCM (ULY0)". This is the only UT codec that uses 4:2:0 and 601/standard def. In Lagarith, I set the "Mode" to YV12. Both exported files are listed in MediaInfo as YUV 4:2:0. The UT file is also listed as ULY0 and "lossless."

I did not convert using Huffyuv because it does not output to YV12. So Huffyuv should not be used for this particular conversion, correct? (When I tried it, I got an RGB file.)

And then finally, I did convert the file to regular DV using Cedocida. It stayed YUV but the chroma was converted to 4:1:1. I know this is what DV uses but is there a way to force 4:2:0 (and should that even be done?)

This is probably old-hat stuff to many of you, but I'm just trying to understand the basics involved in converting a file to lossless, as well as the differences in color spaces and chroma subsampling. Please let me know if I'm on the right track or if there are other issues I need to watch for. Thanks so much!

Jen

[hech54]

A VOB file is MPEG2.....you know that you DON'T get any quality back by converting to lossless right? These aren't ZIP files.
Just asking.

[Bernix]

Hi,
I think 4:2:0 is Pal DV thing, ntsc DV has 4:1:1,
also i read that some lossless codecs do internal colorspace conversion so shouldn't be called lossless. I read this on comparison by MSU. So MSU doesn't change internal colorspace conversion for sure.
That is what I know. Don't know if that helps you.

Bernix

[moxiecat]

Yes, I was just using the VOB as a test file....just to understand the process.

[jagabo]

Is the following statement true?: When converting a file to lossless, just make sure that the color space and chroma subsampling match the original.

Generally, yes.

Or are there other parameters that need to be set to make the conversion lossless?

Just make sure the software you are using isn't performing other conversion. For example, some editors convert everything to RGB internally. That conversion will not be lossless and the conversion back to YUV for compression will not be lossless. And some programs treat different sources differently. For example, they may assume full range rather than limited range depending on the container, codec, chroma subsampling, etc. You have to experiment with the software you use to detect such issues.

For example, let's take a VOB file. The one I have is YUV, 4:2:0, 8 bits, and BT.601 NTSC (color primaries), per MediaInfo. And, this YUV is technically YV12, because of the chroma value, correct?

Yes.

I converted this file to lossless using both UT Video and Lagarith. The UT video codec I used is "YUV420 BT.601 VCM (ULY0)". This is the only UT codec that uses 4:2:0 and 601/standard def. In Lagarith, I set the "Mode" to YV12. Both exported files are listed in MediaInfo as YUV 4:2:0. The UT file is also listed as ULY0 and "lossless."

Note that UT's BT601 and BT709 codes are exactly the same (except for the fourcc) if there are no RGB conversions by the codecs. If the codecs are given RGB they will convert to YV12 with the indicated matrix. If a downstream program asks for RGB they will convert using the specified matrix.

There are some specific cases where you can convert colorspaces and chroma subsampling without losses. For example, YV12 can be converted to YUY2 and back to YV12 losslessly (using point resampling, most programs don't do this). And 8 bit RGB can be converted to 10 bit YV24 and back to 8 bit RGB losslessly.

And then finally, I did convert the file to regular DV using Cedocida. It stayed YUV but the chroma was converted to 4:1:1. I know this is what DV uses but is there a way to force 4:2:0 (and should that even be done?)

No... Well, there is one way you could could avoid 4:1:1 chroma it with an NTSC YV12 source. Take your 720x480 YV12 source, add 96 scan lines to the bottom (or top) of the frame, and pretend the frame rate is 25 fps. Then encode with a PAL DV encoder. Later, you'll have to remember the frame rate is really 29.97 fps and that 96 lines have to be cut off the bottom (or top) of the frame. Of course, DV compression isn't lossless.

[moxiecat]

Jagabo, thank you so much! That is exactly the kind of feedback I was looking for.

Apologies - I should have mentioned that I was running all the conversions through VirtualDubMod. I don't think that converts to RGB internally. I didn't run any filters or do any editing in it at all. Just loaded the file and converted.

Last (unanswered) question: I was correct to not use Huffyuv for this conversion because of its inability to export to YV12, right?

Thanks again!

[Bernix]

Here is list of absolutely lossless codecs. Hyffyuv is among them. It is bit old, but not truly lossless codecs was not used in test. (internal colorspace conversion) so I believe Huffyuv is truly lossless. Best balanced was FFV1. Even 10 years old, good to read.

http://www.compression.ru/video/codec_comparison/lossless_codecs_2007_en.html

Edit according to this page, in Huffyuv supported RGB24, YUY2 and YV12 too

Bernix

[jagabo]

Huffyuv is lossless given YUY2 in and YUY2 out, or RGB in and RGB out compressed as RGB. It is not lossless with RGB in and "convert to YUY2" mode.

[jagabo]

I should have mentioned that I was running all the conversions through VirtualDubMod. I don't think that converts to RGB internally. I didn't run any filters or do any editing in it at all. Just loaded the file and converted.

If you didn't use Video -> Fast Recompress mode it converted incoming YV12 to RGB, then RGB was converted back to YV12 -- not lossless. At least with the version I have, 1.5.10.2, build 2542.

[moxiecat]

OK, I need some clarification....I open the VOB file. I go to "Save as..." Under Video mode, it says "Full processing mode" (which can be changed, but I didn't) and then I can choose my codec (Uncompressed RGB is the default, but that's where I changed it to UT, Lagarith, etc.)

So are you saying that with this process, the clip is somehow being converted to RGB in VDM and back to YV12 when I export?

[Cornucopia]

"Opened the VOB file"..how? (With what decoder?)

"Full processing mode"...pretty sure this converts to RGB internally in the app, wasting your attempts to bypass colorspace conversion. What jagabo was talking about.

Scott

[moxiecat]

Scott,

I start VirtualDubMod. File --> Open Video File. I select the VOB. It opens in the VDM window. Not sure what "decoder" you're talking about....I'm just using the installed program.

Under Video, there are options for Direct Stream Copy, Fast Recompress, Normal Recompress, and Full Processing Mode (which is the default). If I change the setting there, it copies over to the save dialog, so I think it affects the exported file, not the file upon on import. I looked through the preferences but I cannot find a setting for how a file is processed upon import. I guess I'm just not getting how this works.

[raffriff42]

I haven't followed this whole thread, but just want to mention regarding Vdub 1.10 -

VirtualDub help - Processing: The pipeline

In previous versions of VirtualDub, enabling full processing mode would always force a conversion to 32-bit RGB. This is no longer the case — if no video filters are used, this conversion step is omitted and the video is directly converted to the output format as in Normal Recompress mode.

Virtualdub FilterMod is even better in this regard, BTW (more filters support YUV, the interface reminds you of the color format when saving, etc)

[moxiecat]

Thanks raffriff42, that's good to know for other projects. Unfortunately, staying with my example, I would need to use VirtualDubMod because of the VOB input file. (I was told in another thread this is the best version to use for VOB/MPEG-2 source material.)

Is there any program that would push a YUV/YV12 source like MPEG-2 through to a YV12 lossless encoder without converting to RGB upon loading the file? Or is this the way it generally works?

[KarMa]

Thanks raffriff42, that's good to know for other projects. Unfortunately, staying with my example, I would need to use VirtualDubMod because of the VOB input file. (I was told in another thread this is the best version to use for VOB/MPEG-2 source material.)

Is there any program that would push a YUV/YV12 source like MPEG-2 through to a YV12 lossless encoder without converting to RGB upon loading the file? Or is this the way it generally works?

Could create a avisynth script with DGIndex decoding your VOB (in YV12), and then open that script up in Virtualdub to be encoded to your lossless codec of choice. Just need to be sure Virtualdub is inputing and outputting YV12.

[jagabo]

VirtualDubMod is based on an old version of VirtualDub. In full processing mode it always converts incoming video to RGB.

Full Processing Mode: incoming video is decompressed and converted to RGB, filtering is enabled
Normal Recompress   : incoming video is decompressed and converted to RGB, filtering is not enabled
Fast Recompress     : incoming video is decompressed but remains in the colorspace and subsampling it comes in, filtering is disabled
Direct Steam Copy   : incoming video is not decompressed, just passed to the output file (remuxed), filtering is disabled

[moxiecat]

So there's no way to do it then. Rats. Even if I select "Direct Stream Copy" under the Video menu before opening the file, I have to change to a different mode in order to export it.

Thanks, KarMa, but that's beyond me at this point. I appreciate the tip though.

Question: Is there another program besides VirtualDubMod that WOULD keep the color space? I know you guys love VirtualDub but would another program be better suited for this task?

[raffriff42]

Virtualdub FilterMod should open .VOB files. I don't have any at hand to test, but basically if ffmpeg supports it, so does VdubFM.

[moxiecat]

Raffriff42, at least one reviewer implies that it does! I will check it out. Thank you!

[jagabo]

I already told you to set Video -> Fast Recompress mode to retain YV12 with MPEG 2 sources in VirtualDubMod. But the program isn't great for handling MPEG 2 video. It doesn't understand pulldown flags so it's prone to frame rate and A/V sync problems. Karma's recommendation to use DgIndex/Mpeg2Source() in AviSynth is the best way to handle MPEG 2 video. Also be aware that VOB files can contain multiple interleaved video streams. For those you need to use a program like VOB2MPG to extract only one video stream (and the audio streams) as an MPG file and use DgIndex/Mpeg2Source() to access it.

[moxiecat]

Jagabo - thanks, I think I misunderstood you. Thank you for clearing that up. I'll double-check those potential issues if I ever move beyond testing this out. I will also look into Karma's recommendation though I do not know (yet) how use AviSynth.

[KarMa]

In avisynth you can also create fake uncompressed AVI files, so that it can be opened up in other video editors losslessly. When I say fake I mean that it actually does not fill up your HDD drive full of uncompressed video data but instead it's really just decompressing the source file on the go when demanded. But other programs don't really seem to know the difference. Which might be more useful to you instead of converting VOB files to Huffyuv for whatever reason.

https://www.videohelp.com/software/Avisynth-Virtual-File-System

As far as VirtualdubMod converting everything to RGB in both Full and Normal processing mode, that is news to me and scared me for a few minutes. Upon looking at Virtualdub 1.10.4 (the one I use) help information, it says that "In previous versions of VirtualDub, enabling full processing mode would always force a conversion to 32-bit RGB. This is no longer the case — if no video filters are used, this conversion step is omitted and the video is directly converted to the output format as in Normal Recompress mode." So it seems like these forced RGB conversions is a problem with older versions and I guess with VirtualdubMod. Happy to know this even though I don't use Virtualdub much.

[jagabo]

The latest build of VirtualDubMod that I know of is 10 years old and based on VirtualDub 1.5.10.3 -- before the smarter colorspace handling was added.

In any case, you still need to be careful loading MPEG 2 sources directly with VirtualDub -- different source filters may give different colorspaces. So you have to check.