VDub Color Space problem processing Nexus 5x video

[RandomVHUser]

Whenever I process any Nexus 5x videos, (default settings - input color depth: auto, output color: 24 bit RGB), regardless of output codec used, the colors are washed out.

The source decodes to Planar 4:2:0 YUV full scale
The output decodes to Planar 4:2:0 YUV

So its limiting the color range, causing the image to look washed out:
Original:


Output:



I've also tried setting the decompression format in VDub to YCbCr (rec. 709) Full 4:2:0, still doesn't work
And setting the output to YCbCr (rec. 709) Full 4:2:0 gives me an error message on encode:

"Unable to initialize video compression: The selected output format is not compatible with the Windows video codec API."


Any ideas how I can get the right colorspace on output?

[jagabo]

VirtualDub doesn't deal with full range YUV well. It internally converts to RGB with a standard rec.601 matrix for filtering, crushing the extended brights and darks. You can use the Brightness/Contrast filter (it can work in YUV) to reduce to limited range before any other filtering (though the granularity of the filter means you can't get perfect results). Or use AviSynth to convert to limited range before giving the video to VirtualDub.

I never figured out how to use any of the "other" color depths working properly.

[RandomVHUser]

So, are you saying Virtualdub can not properly convert this video, that it always will have to squash the dynamic range?

[jagabo]

So, are you saying Virtualdub can not properly convert this video, that it always will have to squash the dynamic range?

Yes. Or you could convert externally to RGB with a PC.709 matrix (full range YUV to RGB), work in RGB in VirtualDub, then convert externally again to full range YUV.

[RandomVHUser]

Ok, thanks for the suggestion, I'll try experimenting with that idea - now to find a good program to lossless convert for vdub in, lossless output vdub, then encode in handbrake or something...any recommendations?

[sneaker]

It internally converts to RGB with a standard rec.601 matrix for filtering

I don't believe that's true anymore. The filter system also works in YUV nowadays.

I agree that the whole VirtualDub color system is confusing and from OP's post alone it is not possible to say what is going wrong because there are too many possible variables.
1. the video is from Nexus 5x. I don't know what that implies. RGB? YUV? Range? I can't trust OP without a sample. (screenshot is not a good sample because screenshot creation usually implies color conversion)
2. the decompressor? VfW? AviSynth? ffinputdriver? DirectShow? What decoder? Again, many possibilities.
3. what target encoder with what settings? Every encoder and decoder may have its own internal conversion.

Usually problem can be solved with right "Color Depth" setting and/or the combination of "convert format" and "alias format" filter. "alias format" filter can trick encoder to think it is getting a different (compatible) format.

[jagabo]

It internally converts to RGB with a standard rec.601 matrix for filtering

I don't believe that's true anymore. The filter system also works in YUV nowadays.

In my experience, only a few of the filters work in YUV. Turn on the "Show image formats" option in the filters dialog to see what colorspace the filters are working in.

[poisondeathray]

But what sneaker is saying is also true - there should be a way to do this in vdub alone , for example the ffinputdriver has full range/ 709 input options to override "auto" , the video=>color depth => other has many options, and the alias format can work in YUV

[RandomVHUser]

It internally converts to RGB with a standard rec.601 matrix for filtering

I don't believe that's true anymore. The filter system also works in YUV nowadays.

I agree that the whole VirtualDub color system is confusing and from OP's post alone it is not possible to say what is going wrong because there are too many possible variables.
1. the video is from Nexus 5x. I don't know what that implies. RGB? YUV? Range? I can't trust OP without a sample. (screenshot is not a good sample because screenshot creation usually implies color conversion)
2. the opening system. VfW? AviSynth? ffinputdriver? DirectShow? What decoder? Again, many possibilities.
3. what target encoder with what settings? Every encoder and decoder may have its own internal conversion.

Usually problem can be solved with right "Color Depth" setting and/or the combination of "convert format" and "alias format" filter. "alias format" filter can trick encoder to think it is getting a different (compatible) format.

1) Nexus 5x - the media information on the original file states:
H264 - MPEG-4 AVC (part 10) (avc1)
Decoded Format: Planar 4:2:0 YUV full scale

2) Standard Virtualdub x64 install with FFMPeg x64 input plugin.

3) Target encoder: tried uncompressed, x264vfw(both standard settings and "keep input colorspace). I've already tried different colorspaces as stated in my first post.

[raffriff42]

VirtualDub can't tell one form of YUV 4:2:0 (Rec607/709, full/limited range) from another unless given a "hint" -- when converting to RGB it assumes Rec.601 and TV range. However, if you use the Alias Format filter (which does not change the video data itself, only adds a "flag"), it can convert the colors correctly.

Some decoders such as the FFmpeg input driver and the Caching input driver bundled with VirtualDub FilterMod, try to flag the video properly, but even they guess wrong sometimes (just as media players sometimes do). When that happens, you need to manually apply Alias format.

[sneaker]

1) Nexus 5x - the media information on the original file states:
H264 - MPEG-4 AVC (part 10) (avc1)
Decoded Format: Planar 4:2:0 YUV full scale

An actual sample would be more helpful. Any kind of color flags aren't necessarily correct.

2) Standard Virtualdub x64 install with FFMPeg x64 input plugin.

Note that the ffmpeg input plugin has its own color conversion.
Click "File"->"Open video fie...". Select file type "ffmpeg supported files". At the bottom left of the file open dialog tick "Ask for extended options after this dialog". Then click open and the options will appear. I suggest you do not let the plugin do any kind of detection/conversion, i.e. like this:

Then set "Video"->"Fast recompress". Set "keep input colorspace" in x264vfw. Then encode and see if it worked.

[poisondeathray]

I just downloaded a native nexus 5x video and it worked fine . It's probably an issue with your video player configuration or method of screenshot taking on your end

Color range : Limited
Color primaries : BT.709
Transfer characteristics : BT.709
Matrix coefficients : BT.709

There is nothing "funky" about the video or flags out of the camera . It's standard range video content, Rec709, flagged properly 709. Some cameras have certain flags that can cause discrepancies between some decoders (e.g. some canon cameras), but this should work straight with default settings. But you might want to flag your exported video as Rec709 - that will increase the chances a certain device or software player will play it correctly

The ffinput driver has extended options to override "auto" . In the open dialog box, checkmark "ask for extended options" , make sure the "use full range" box isn't checked (maybe you inadvertently checked it for another video) - whoops I missed to post above. sneaker is too quick

[shekh]

> doesn't deal with full range YUV well. It internally converts to RGB with a standard rec.601 matrix

This is not true even with standard VD, but also I fixed many related issues in FilterMod so it should be possible to do the right thing.
Also "alias format" is a legacy, now you can override color range in "video->decode format" dialog.

> So its limiting the color range, causing the image to look washed out:
> Original:
> ...

Now what is original and what is output?
Do you realize that interpreting video as full range intends to fit more dynamic range and therefore "whashed colors" is indeed the correct (intermediate) result?

[jagabo]

RandomVHUser should post a sample video.

[RandomVHUser]

I've posted a 5 second clip here (its 4k video, 22MB):

http://www.mediafire.com/file/pqa1l1zyuguowpx/snip.mp4


The upside down orientation is due to AviDemux and other programs not detecting the orientation properly, but this still has the proper color space.

Load the snip into avidemux/VLC and notice the color range is proper (full range)

Load the snip into VDub, the preview pane looks washed out and the resulting encode looks washed out when played back in VLC (and does not show decoding to full range).

I've tried the alias format, the open "advanced options" checkmarks, and combo'ing them with many encode/decode colorspace options, but no dice.

If anyone can find what I'm missing, I'd be very appreciative, thanks

[sneaker]

Now I see why you are having such trouble. The convert format filter doesn't actually handle range even though you can select between limited and full range formats.
Maybe shek can give you a hint about his mod. I usually use AviSynth for these kind of conversions. Sorry for presenting non-working "solutions".

[shekh]

Load the snip into avidemux/VLC and notice the color range is proper (full range)

How do you notice that?

If you want to sort this, begin with something manageable. Here is a color chart encoded with same params as your sample (YUV420, 601, FR)
You can check the colors, notice they are the same as intended, in particular greys 255, 192, 129, 116, 62, 0

If you see something else at this stage - you are using wrong input driver, wrong alias format etc.
Of course I insist on using FilterMod pack.
When you open the file notice the following:
File->Information: pixel format is 601:FR
Video->Decode format: current format is YUV420-FR (clueless with official VD!)
Go to filters, add filter "crop" (does not exist in official VD)
Now use color picker (hold shift) to inspect values. R,G,B should closely match what is written on image.
Now if you want, add flip filter.
Now if you want to encode with x264, just press "save avi", select x264 8bit.
Notice pixel format is set automatically to YUV420-FR.
Pixel data is not modified between VD and codec at all!
All this encoding sugar is new (and was a minefield before).

Now to check the result, open new file and repeat all steps above.

If for a reason you still want to keep old VD, the tough part is sending YUV420-FR from VD to encoder. However I am not sure if you wanted exactly this.

[sneaker]

@shekh
Let's just assume for a second that the source is actually YUV BT.601 Full Range, we want to open using your mod and with ffinput driver and we want to output to YUV BT.709 limited range using external encoder x264. How would I do that?

Pixel data is not modified between VD and codec at all!

But we usually want YUV BT.709 limited range output.

[shekh]

> Let's just assume for a second that the source is actually YUV BT.601 Full Range

It is what reported by FFMPEG. If it is wrong, use overrides.

> output to YUV BT.709 limited range using external encoder x264

In that case you are screwed, because there is no normal method to pass YUV-709 to external encoder.
Technically you can use something like
convert format to YUV-709
alias format to YUV-601 (to hide fact it is 709, otherwise it would convert to 601 implicitly)
put VUI param in command line for 709 (dont remember the syntax)

> But we usually want YUV BT.709 limited range output

in that case if the rest is fine, use output override to make it 709.

> The convert format filter doesn't actually handle range even though you can select between limited and full range formats

Dont know, should work IMO. Convert format depends on same code used elsewhere, if it does not work then nothing should.
But there is always chance for a bug.

[sneaker]

In that case you are screwed, because there is no normal method to pass YUV-709 to external encoder.
Technically you can use something like
convert format to YUV-709

The "convert format" filter does not seem to work as expected in this regard for me. The "[C]" marking indicates some other conversion and the encoded results confirm this. No idea why. It seems strange to have all these options if they don't work so maybe I'm just missing some important detail.

in that case if the rest is fine, use output override to make it 709.

Where exactly is "output override"? With so many different dialogs it's difficult to find for me.

[poisondeathray]

Interesting that the specs are different from this other Nexus 5x video that I tested earlier. Is that 5sec clip directly from the camera or other processing ? How was it "cut" ? Or perhaps that other sample linked to in the YT video was actually processed

https://www.youtube.com/watch?v=9uV2LPftW-k&feature=youtu.be
https://docs.google.com/uc?id=0B9832-kb7o1hVDZFU0tRSGVkaGs&export=download

Color range                              : Full
Color primaries                          : BT.601 PAL
Transfer characteristics                 : BT.601
Matrix coefficients                      : BT.601

[RandomVHUser]

Load the snip into avidemux/VLC and notice the color range is proper (full range)

How do you notice that?

If you want to sort this, begin with something manageable. Here is a color chart encoded with same params as your sample (YUV420, 601, FR)
You can check the colors, notice they are the same as intended, in particular greys 255, 192, 129, 116, 62, 0

If you see something else at this stage - you are using wrong input driver, wrong alias format etc.
Of course I insist on using FilterMod pack.
When you open the file notice the following:
File->Information: pixel format is 601:FR
Video->Decode format: current format is YUV420-FR (clueless with official VD!)
Go to filters, add filter "crop" (does not exist in official VD)
Now use color picker (hold shift) to inspect values. R,G,B should closely match what is written on image.
Now if you want, add flip filter.
Now if you want to encode with x264, just press "save avi", select x264 8bit.
Notice pixel format is set automatically to YUV420-FR.
Pixel data is not modified between VD and codec at all!
All this encoding sugar is new (and was a minefield before).

Now to check the result, open new file and repeat all steps above.

If for a reason you still want to keep old VD, the tough part is sending YUV420-FR from VD to encoder. However I am not sure if you wanted exactly this.

How I notice? Original mp4 in vlc and output mp4 in vlc, switching back and forth, the light and dark contrast differences are very apparent. And not just visually, but obviously if the VLC decoder says its decoded colorspace is difference, thats another confirmation.

I d/led VirtualDub FilterMod(1.10.5), the x264vfw encoder crashes when saving. I tried your settings, I ran uncompressed output, viewed it, the range is still off.

Example GIF of the differences (the darker image is the original):

[RandomVHUser]

Interesting that the specs are different from this other Nexus 5x video that I tested earlier. Is that 5sec clip directly from the camera or other processing ? How was it "cut" ? Or perhaps that other sample linked to in the YT video was actually processed

https://www.youtube.com/watch?v=9uV2LPftW-k&feature=youtu.be
https://docs.google.com/uc?id=0B9832-kb7o1hVDZFU0tRSGVkaGs&export=download

Color range                              : Full
Color primaries                          : BT.601 PAL
Transfer characteristics                 : BT.601
Matrix coefficients                      : BT.601

Direct from the source, after being cut to 5 seconds with avidemux (no re-encoding or additional processing)

[poisondeathray]

For your 5second clip, it should look like the lighter, washed out, low contrast version in your gif when displayed by a media player. Full range YUV converted to RGB full range for display.

The darker image is technically "wrong" for that video, data is lost Y<16, Y>235. Notice the details in the tree bark etc.. are crushed and missing

Normally you want it to be in standard range, so fewer things can go wrong by various devices and software . Full range is "bad practice"

But I'm wondering if ti was cut properly ? The difference in flags, flipped image, ... maybe some error in processing ?

[RandomVHUser]

For your 5second clip, it should look like the lighter, washed out, low contrast version in your gif when displayed by a media player. Full range YUV converted to RGB full range for display.

The darker image is technically "wrong" for that video, data is lost Y<16, Y>235. Notice the details in the tree bark etc.. are crushed and missing

The darker image is how the source is shown in VLC. The output should visually match the source if no settings are changed? And why would the source be labeled "YUV full scale" and the output not state full scale? Are you saying VLC/AviDemux is improperly rendering the video as full range?

[RandomVHUser]

For your 5second clip, it should look like the lighter, washed out, low contrast version in your gif when displayed by a media player. Full range YUV converted to RGB full range for display.

The darker image is technically "wrong" for that video, data is lost Y<16, Y>235. Notice the details in the tree bark etc.. are crushed and missing

Normally you want it to be in standard range, so fewer things can go wrong by various devices and software . Full range is "bad practice"

But I'm wondering if ti was cut properly ? The difference in flags, flipped image, ... maybe some error in processing ?

What software are you using to verify the specs? I can run it against my original, but they both should be the same.

[RandomVHUser]

The darker image is technically "wrong" for that video, data is lost Y<16, Y>235. Notice the details in the tree bark etc.. are crushed and missing

I don't believe the data is lost, I believe its just lower in the spectrum. I tested this by manually adjusting the brightness up by 15 and all the low details appeared on the original image.

[shekh]

The "convert format" filter does not seem to work as expected in this regard for me. The "[C]" marking indicates some other conversion and the encoded results confirm this. No idea why. It seems strange to have all these options if they don't work so maybe I'm just missing some important detail.

This is implementation detail. Convert format in the gui is fake filter. The actual work is done by implicit conversion which is indicated by "[C]" icon.
(btw I dont like this behavior)

I just tested it:
opened color_plate above (YUV420-601-FR)
added convert to YUV420-709
added crop to test colors: colors stay about the same, but format is now YUV420-709, so everything is ok.

Where exactly is "output override"? With so many different dialogs it's difficult to find for me.

https://sourceforge.net/p/vdfiltermod/wiki/compression/

see first picture

[shekh]

I d/led VirtualDub FilterMod(1.10.5), thex264vfw encoder crashes when saving. I tried your settings, I ran uncompressed output, viewed it, the range is still off.

I also got a "encoder failed" message at medium preset with your sample. However x64 version worked fine. Maybe insufficient memory issue?

And in the end I have very identical colors.

[RandomVHUser]

I d/led VirtualDub FilterMod(1.10.5), thex264vfw encoder crashes when saving. I tried your settings, I ran uncompressed output, viewed it, the range is still off.

I also got a "encoder failed" message at medium preset with your sample. However x64 version worked fine. Maybe insufficient memory issue?

And in the end I have very identical colors.

I downloaded the x64 version, the encoder now works, but the dynamic range is still not decoding right in VLC on the output file.

When playing your encode in VLC, does it state Decoded format as "Planar 4:2:0 YUV" or "Planar 4:2:0 YUV full scale"?

Could you post your encode?