x264 conversion with Virtualdub/sony vegas 13 Color issues

[MajorFoley]

Hey guys, so for the last few of years i have been using Virtual dub to compress my youtube videos I make using FRAPS (I know inefficient at the moment) and have been experimenting with others such as dxtory and afterburner and will also be able to try out shadowplay soon.

My issue is kind of color based. Usually I open up the file in virtual dub or sony vegas and the color tends to get a little washed up (reds look orange for instance) I have noticed this is because it selects YUV:4:2:0 by default I think it was. However it also does save space instead of Keep/accept only RGB but is there a way to get the best of both? Keep the colors but compress a bit more? What is the option input colorspace for instance?

Also another little issue is when I actually do a compression with sony vegas, I cant reput that video in again because apparently unsupported?

[LigH.de]

You may indeed insist on "full range" encoding if you ensure that RGB/YUV conversion will not clamp to TV range (in AviSynth, there is a "coring" option in several functions). But I can't tell you how to ensure this in VirtualDub. You did not even tell us if you use x264vfw or x264.exe as external encoder. Furthermore I can't help you with expensive commercial tools; and they often expect you to pay even more for input plugins of formats they already support as output ... there is a lot of freeware which can do everything regarding technical details, it just doesn't provide the most user friendly interfaces.

[jagabo]

color tends to get a little washed up (reds look orange for instance)

A rec.601 vs rec.709 issue too?
https://forum.videohelp.com/threads/329866-incorrect-collor-display-in-video-playback?p...=1#post2045830
https://forum.videohelp.com/threads/329866-incorrect-collor-display-in-video-playback?p...=1#post2045481

[MajorFoley]

You may indeed insist on "full range" encoding if you ensure that RGB/YUV conversion will not clamp to TV range (in AviSynth, there is a "coring" option in several functions). But I can't tell you how to ensure this in VirtualDub. You did not even tell us if you use x264vfw or x264.exe as external encoder. Furthermore I can't help you with expensive commercial tools; and they often expect you to pay even more for input plugins of formats they already support as output ... there is a lot of freeware which can do everything regarding technical details, it just doesn't provide the most user friendly interfaces.

Oh ill provide a picture and yes im using x264vfw and you kinda have me lost on the avisynth part and such with the clamp to tv range
I also cant use any other YUV options because its set to keep/accept only, I guess that's why I get the error for those options because the YUV 4:2:0 is convert to it not keep.

A rec.601 vs rec.709 issue too?
https://forum.videohelp.com/threads/329866-incorrect-collor-display-in-video-playback?p...=1#post2045830
https://forum.videohelp.com/threads/329866-incorrect-collor-display-in-video-playback?p...=1#post2045481

I don't think that my problem may be that, I will try watching on my laptop and see if I get that issue but whenever I see them on my desktop or on youtube they seem washed up.

I have provided examples as well as the image of my settings
YUV 4:2:0

RGB

Colorspace

Settings

[jagabo]

YUV is a different way of representing color video. It is used by all commercial video like DVD, Blu-ray, broadcast TV, Youtube, etc. The Y (luma) channel is a greyscale image. The U and V (chroma) channels represent colors that are added or subtracted from the greyscale image to produce the final color.

The U and V channels are often encoded at half the resolution of the greyscale channel because the human eye has less color resolution than greyscale resolution. So a 1920x1080 video will have a 1920x1080 Y channel, but only 960x540 U and V channels. That is what the YUV 4:2:0 is referring to. YUV 4:4:4 (aka 1:1:1) uses the same resolution for all three channels.

In x264vfw, when you select "Convert to YUV 4:2:0" an RGB input (your screen capture) is converted to YUV 4:2:0 before the video is compressed. When you select "Keep/Accept only RGB" the video isn't really kept as RGB but it keeps all three YUV channels the same resolution, ie YUV 4:4:4. x264vfw uses a rec.601 matrix to convert from RGB to YUV though it doesn't flag what matrix was used within the file. High definition video usually uses rec.709 so most players/editors assume rec.709 when playing unflagged HD video. Youtube assumes rec.709 too.

YUV 4:4:4 isn't frequently used so many players and devices can't handle it, will handle it incorrectly, or handle it differently, than the usual YUV 4:2:0. That's what you're seeing. I'd recommend you stick with 4:2:0. You can increase the likelihood of your video being interpreted correctly by flagging the color matrix. In x264vfw you can do that by adding "--colormatrix=bt470bg" in the Extra Options box. Even better would be to convert HD video with the rec.709 rather than rec.601. I don't know if/how you can do that when using x264vfw in Fraps. You may have to convert from rec.601 to rec.709 later in your editor. For example, in AviSynth you can to it with ColorMatrix(mode="rec.601->rec.709").

[jagabo]

Some other issue:

I don't use Vegas but from what I understand it treats different containers/codecs differently when it comes to color matrices. Some it treats as full range (Y = 0 to 255) others as limited range (Y = 16 - 235, the norm). You need to figure out what it does for your sources in particular. Using the wrong range will give you crushed blacks and brights, or washed out darks and dim brights.

VirtualDub usually assumes rec.601 (limited range, 601 matrix) for all its YUV/RGB conversions. Viewing rec.709 video in the preview window will usually show the wrong colors. But results may vary depending on the DirectX settings, your graphics card and its setup, and the resolution of the video.

Quicktime uses apparently non-standard conversions. It should be banned from your computer.

[MajorFoley]

YUV is a different way of representing color video. It is used by all commercial video like DVD, Blu-ray, broadcast TV, Youtube, etc. The Y (luma) channel is a greyscale image. The U and V (chroma) channels represent colors that are added or subtracted from the greyscale image to produce the final color.

The U and V channels are often encoded at half the resolution of the greyscale channel because the human eye has less color resolution than greyscale resolution. So a 1920x1080 video will have a 1920x1080 Y channel, but only 960x540 U and V channels. That is what the YUV 4:2:0 is referring to. YUV 4:4:4 (aka 1:1:1) uses the same resolution for all three channels.

In x264vfw, when you select "Convert to YUV 4:2:0" an RGB input (your screen capture) is converted to YUV 4:2:0 before the video is compressed. When you select "Keep/Accept only RGB" the video isn't really kept as RGB but it keeps all three YUV channels the same resolution, ie YUV 4:4:4. x264vfw uses a rec.601 matrix to convert from RGB to YUV though it doesn't flag what matrix was used within the file. High definition video usually uses rec.709 so most players/editors assume rec.709 when playing unflagged HD video. Youtube assumes rec.709 too.

YUV 4:4:4 isn't frequently used so many players and devices can't handle it, will handle it incorrectly, or handle it differently, than the usual YUV 4:2:0. That's what you're seeing. I'd recommend you stick with 4:2:0. You can increase the likelihood of your video being interpreted correctly by flagging the color matrix. In x264vfw you can do that by adding "--colormatrix=bt470bg" in the Extra Options box. Even better would be to convert HD video with the rec.709 rather than rec.601. I don't know if/how you can do that when using x264vfw in Fraps. You may have to convert from rec.601 to rec.709 later in your editor. For example, in AviSynth you can to it with ColorMatrix(mode="rec.601->rec.709

Some other issue:

I don't use Vegas but from what I understand it treats different containers/codecs differently when it comes to color matrices. Some it treats as full range (Y = 0 to 255) others as limited range (Y = 16 - 235, the norm). You need to figure out what it does for your sources in particular. Using the wrong range will give you crushed blacks and brights, or washed out darks and dim brights.

VirtualDub usually assumes rec.601 (limited range, 601 matrix) for all its YUV/RGB conversions. Viewing rec.709 video in the preview window will usually show the wrong colors. But results may vary depending on the DirectX settings, your graphics card and its setup, and the resolution of the video.

Quicktime uses apparently non-standard conversions. It should be banned from your computer.

First, rec 607 and 709 are standards i guess? Kinda like 607 was DVD and 709 is blu ray? Kinda like how IDE'S went on to become SATA for hard drives? Apart from compressing and recording video i have not delve deep into advanced settings like the command line options you seem to use.

I believe fraps records completely uncompressed its why i use the x264vfw codec to compress and i just use virtual dub and vegas (was planning on vegas so i could add intros and outtros to my videos).
In vegas i create the video profile and can choose x264 settings in there just like i could with virtual dub (without the profile editing) Its also why i cant wait to be able to give shadow play a shot myself. From what i gather virtual dub only does 607 conversion then what would use 709 but still compress just as great as x264 and is accepted on youtube? Because it does turn fraps videos which reached over 100gb to 2gb and less.

Is there an introductory guide to learning this avisynth and other things? Because this sounds like it could also help rip my blu rays and put them on hard drives so i can watch them on my media player. It sucks having a 8gb movie file on your portable HDD.

On another note i havent used quick time since my days in highschool when i was told too

[LigH.de]

ITU-R Recommendations BT.601/709/2020 are colorimetry standards, defining how to convert between RGB and YCbCr a.k.a YUV (there are equation sets with slightly different parameters) and several more video attributes for SD/HD/UHD resolutions, respectively. The ITU is an international standardization commitee, also known for audio codec standards (e.g. G.723) or video codec standards (H.26#).

FRAPS uses some lossless compression, I believe; at least it is certainly not very efficient. During the recording of screen content, speed is the most important constraint. Wasting time with efficient video compression would make the recorder miss frames.

There are surely several AviSynth guides. I would recommend starting with reading the included documentation to learn the script language and basic concepts, and using the AviSynth Wiki as technical reference. People who already have experience with script languages similar to C / PHP / Perl / Python etc. in syntax should understand AviSynth easily.

There is an even more powerful video framework in Python, VapourSynth. It is quite similar to AviSynth, and many AviSynth plugins are available for VapourSynth too, even in 64 bit (useful to utilize a lot more RAM; 32 bit processes are limited to 2 GB which can be tight while filtering HD or even UHD video).

Please read at least the YCbCr page on Wikipedia, it summarizes the "Rec. #" topic quite well.

[MajorFoley]

ITU-R Recommendations BT.601/709/2020 are colorimetry standards, defining how to convert between RGB and YCbCr a.k.a YUV (there are equation sets with slightly different parameters) and several more video attributes for SD/HD/UHD resolutions, respectively. The ITU is an international standardization commitee, also known for audio codec standards (e.g. G.723) or video codec standards (H.26#).

FRAPS uses some lossless compression, I believe; at least it is certainly not very efficient. During the recording of screen content, speed is the most important constraint. Wasting time with efficient video compression would make the recorder miss frames.

There are surely several AviSynth guides. I would recommend starting with reading the included documentation to learn the script language and basic concepts, and using the AviSynth Wiki as technical reference. People who already have experience with script languages similar to C / PHP / Perl / Python etc. in syntax should understand AviSynth easily.

There is an even more powerful video framework in Python, VapourSynth. It is quite similar to AviSynth, and many AviSynth plugins are available for VapourSynth too, even in 64 bit (useful to utilize a lot more RAM; 32 bit processes are limited to 2 GB which can be tight while filtering HD or even UHD video).

Please read at least the YCbCr page on Wikipedia, it summarizes the "Rec. #" topic quite well.

Alright ill at least try to read the YCB and hope it wont confuse me. So your recommendation is to keep using convert 4:2:0 but use that extra command line to see if it helps keep more of the colour?

[LigH.de]

Well, yes, try to keep "full color range" (a.k.a. "PC range"). But I don't know if that is possible with x264vfw, I never use that (only x264.exe; VirtualDub 1.10+ would support that as "external encoder", this workflow should have been documented here somewhere already).

      --range <string>        Specify color range ["auto"]
                                  - auto, tv, pc
...

      --input-range <string>  Specify input color range ["auto"]
                                  - auto, tv, pc

I am not sure which of them is required (possibly both). And there are many more colorimentry options. And I don't know if any of the previous links in the chain of tools would clamp YUV video down to "TV range" already...

[jagabo]

I recommend you not use full range luma and chroma. Almost all commercial sources use limited range. That is what most software and hardware expects and handles correctly. And both full range and limited range give the same range of levels and colors when handled correctly -- it's just that you get more steps in between with full range (256 rather than ~220), meaning a little less posterization (banding). What you need to do is figure out a workflow that handles your sources correctly.

I have one fraps file in my collection of odds and ends. It comes out of ffdshow as full range YUV 4:2:0 (YV12) so it needs to be converted to limited range. I don't have the original RGB source of that video but judging by picture content it appears to use the 709 matrix. A web search indicates the same.

In AviSynth the correct conversion back to RGB for a full range rec.709 source is: ConvertToRGB(matrix="PC.709"). A simple script to do this would be:

AviSource("filename.avi")
ConvertToRGB(matrix="PC.709")

That should give you the same RGB levels and colors* that were on the screen when caputred.

To get limited range YUV rec.709 follow that with:

ConvertToYV12(matrix="rec709")

* The conversion from RGB to YUV and back to RGB will lose a little precision. So RGB values may differ by 1, in practice.

[LigH.de]

The sources in this topic appear to be screen recordings of PC games. They will be in PC range...

[jagabo]

The sources in this topic appear to be screen recordings of PC games. They will be in PC range...

Of course, what was on the screen originally was full range RGB. But to get those original levels/colors back you have to use the same (inverted) matrix fraps used to convert to YUV. And for the highest probability of proper playback you need to convert to limited range rec.709 YUV 4:2:0 and flag it as as such.

[MajorFoley]

Uh guys i appreciate the help but i have never used avisynth and all that and i may come around as dumb to you gus who might have done this for months or even years, but like i showed with the screenshots it does let me choose the RGB just comes out bigger, the YUV file i think ended up being 175MB the other 2 300MB or so. Now i guess i could keep asking for solutions here but its clear im probably going to need to broaden my knowledge on this subject and until then my best choices would either be deal with the washed up colors and save MB's or use KEEP RGB or KEEP COLORPSACE to keep the colors while increasing the size. Now you guys have pretty much told me that you both seem to use avisynth and it seems to work well for you guys, but i guess im gonna need to find a dummies guide for myself .

Thanks ill keep reading the links and hopefully we'll see where i can go from there. Maybe even using shadowplay will just help as well and compressing from shadowplay instead of FRAPS

[jagabo]

Keep RGB doesn't really keep the video as RGB. x264 can't work in RGB. What Keep RGB does is keep the full resolution for all three YUV channels (YUV 4:4:4). If you're seeing different colors with Keep RGB vs. Conv YUV 4:2:0 it's a decoding issue -- the software is using a different decoder or different YUV/RGB conversions for YUV 4:4:4 vs. YUV 4:2:0. On my computer I get the same colors with both.

[MajorFoley]

Keep RGB doesn't really keep the video as RGB. x264 can't work in RGB. What Keep RGB does is keep the full resolution for all three YUV channels (YUV 4:4:4). If you're seeing different colors with Keep RGB vs. Conv YUV 4:2:0 it's a decoding issue -- the software is using a different decoder or different YUV/RGB conversions for YUV 4:4:4 vs. YUV 4:2:0. On my computer I get the same colors with both.

Then I guess the suggestion would be to use this avisynth?

But then how would the decoding issue affect it? Is it because im using a different version of x264 or is it virtual dub?

[LigH.de]

You are using x264vfw. So as a "basic user", you are limited to the few options it publishes as controls in its user interface. But the x264 encoder in general (regardless of the user interface) has more options to control it. The command line version (x264.exe) handles most of the available options as command line parameters. You can enter such parameters in x264vfw too, in the "Extra command line (for advanced users)".

I believe that "full range" is similar to x.v.Color some AVCHD cameras use.

[MajorFoley]

well I tried that colour --colormatrix=bt470bg thing but it didn't work but it did kind of give me an idea.

You know I have been thinking about this little issue and I did another comparison of the screenshots of the original fraps video and just my usual compression and it seems like its not really changing the colours but perhaps the contrast or the brightness. Instead of comparing the actual red colour on the soldiers I was looking at the bar on the top right and I noticed it getting brighter as well. Would this be a different issue?

[LigH.de]

Yes, this can be the difference between full RGB scale and limited YUV scale. Some decoders or renderers don't expand the limited YUV values (suitable for display on an external TV set) back to full scale (required for display on a PC monitor) after converting YUV to RGB; if your player is using the "Hardware overlay" renderer, check your graphic card driver's video options; if any other renderer is used, blame the decoder software being set up inappropriately.

In rare cases, already the encoding procedure limited the luminance and chrominance. Trying to encode PC monitor optimized material requires to notify the encoder about "full input range". When you convert your usual movies, the encoder usually already handles input in the limited TV range of YUV values. But FRAPS records full PC scale in RGB.

[MajorFoley]

Yes, this can be the difference between full RGB scale and limited YUV scale. Some decoders or renderers don't expand the limited YUV values (suitable for display on an external TV set) back to full scale (required for display on a PC monitor) after converting YUV to RGB; if your player is using the "Hardware overlay" renderer, check your graphic card driver's video options; if any other renderer is used, blame the decoder software being set up inappropriately.

In rare cases, already the encoding procedure limited the luminance and chrominance. Trying to encode PC monitor optimized material requires to notify the encoder about "full input range". When you convert your usual movies, the encoder usually already handles input in the limited TV range of YUV values. But FRAPS records full PC scale in RGB.

Sorry its been a while. So I need to find a way for my decoder (x264 in this case right?) Doesn't change the colour range when converting to YUV12 from RGB (I think this way would just save me a lot of hard drive space compared to keep RGB)

Also in this video I watched (https://www.youtube.com/watch?v=XX1WN96XrLs) the man says that if you record straight with x264 using dxtory, you have to have the command line --keyint 1 or sony vegas wont open it. What changes with that command?

[jagabo]

--keyint 1 makes every frame a keyframe. It doesn't make sense that Vegas wouldn't open an x264 encoded video just because keyint wasn't set to 1. It would be slow seeking with long keyint values, like the default 250, but it should still be able to open the file. Setting keyint to 1 negates most of the compression improvements h.264 has over MJPEG or similar all keyframe codecs.

[MajorFoley]

--keyint 1 makes every frame a keyframe. It doesn't make sense that Vegas wouldn't open an x264 encoded video just because keyint wasn't set to 1. It would be slow seeking with long keyint values, like the default 250, but it should still be able to open the file. Setting keyint to 1 negates most of the compression improvements h.264 has over MJPEG or similar all keyframe codecs.

I think people meant that if they already recorded in x264 like with shadowplay that option has to be in or something I don't know, still trying to solve this colour issue and my damn white noise issue when I record. They are both annoying me to no end. But the color issue does come first, is there a command line that could force the brightness to remain the same? That colormatrix=bt470bg did not seem to work when i tried it out.

[dracuella]

@MajorFoley,

if you haven't found a solution to your problem yet, you could try the --range or --input-range talked about in this thread:
http://forum.doom9.org/showthread.php?t=168027 (it's generally a really good thread about colour and conversion).

I found that either of those fixed my problem which was (I found out via MediaInfo) that apparently, my encoded files were being read as "color range: limited" instead of full. I've been through pretty much all the motions you've been doing (colormatrix etc) and only these two consistently fixed my issue.

So try the --input-range pc or --range pc

"--range"
Dark Shikari: " '--range' affects the output range used by the YUV conversion process."
Lord_Mulder: "'--range' is used to indicate the range that the decoder shall use for any colorspace conversions that may be required after decompressing the video. As x264 is not involved in the decoding process, all it can do is write the desired "--range" into the VUI* and hope that the decoder will take care of that...

"--input-range"
Dark Shikari: "If you're trying to make x264 assume the input is a particular range, you might want to look at --input-range."
Lord_Mulder: "The '--input-range' is under the input/output options (not VUI) and it's used by x264 to handle colorspace conversions of the input, before the actual encoding."

As to which other side effects they introduce, you'll need to ask the more seasoned x264'ers here in the forum.

* VUI (Video Usability Info), see this post: http://forum.doom9.org/showthread.php?t=101058. The important thing to note here is "The VUI settings are not used by the encoder but are merely suggestions to the playback equipment. See doc/vui.txt for details. Use at your own risk."

Disclaimer: 1) I a complete noob; 2) I only playback on my computer.

[MajorFoley]

Been a while. Real life does unfortuantely catch up on things. Still trying to solve the issue and i might be switching to DXtory, shadow play is now out of the question to use and FRAPS hasn't been updated in years. If i record something straight with x264 on dxtory does that mean it's recording in its form instaed of compressing and say if i recoreded in fraps then compressed and converted with x264 they'd be similar in size (with the same x264 settings of course)