Color keep changing and I cant get it to stop

[killerteengohan]

[ Problem I am having ]

I am trying to encode a Blu-Ray of mine and I'm having a color problem. This is obviously a rec601/rec709 issue but I cant get it fixed.


When I open the M2TS in any media player on any computer I have, the video and colors look like this. This is also how the DVD's colors look when I play it back in any media player on any computer of mine.





The problem I'm having is every time I run it through AVISynth the colors end up looking like this. (Notice the reds are really dark looking)





[ Things I have tried to fix this ]

Usually when I have this problem, I just specify --colorprim bt709 in the h.264 settings and it comes out how it's supposed to. I have tried specifying --colorprim bt709, --transfer bt709, and --colormatrix bt709 in the settings and it is not outputting the correct colors in the encode.

I have also tried adding ConvertToYV12(matrix="rec709") to the script and it does not come out with the correct colors.

I tried adding ConvertToRGB(matrix="rec709") to the script and that seems to get the correct colors, but my encoder says that's an incorrect colorspace and I would rather not change the colorspace to RGB if it can be helped.


[ My Question ]

The Blu-Ray is rec709 according to mediainfo so I don't know what else to do.

The M2TS, the Blu-Ray and the DVD all play fine when it comes to colors in my media players and on my TV.

I can encode the DVD and the colors remain correct. Why is this Blu-Ray giving me such a hard time even after trying the usual settings that always fixed this for me in the past?


[ MediaInfo ]

General
ID                                       : 0 (0x0)
Complete name                            : C:\Users\Desktop\Encoding\LN\BDMV\STREAM\00000.m2ts
Format                                   : BDAV
Format/Info                              : Blu-ray Video
File size                                : 19.1 GiB
Duration                                 : 1h 34mn
Overall bit rate mode                    : Variable
Overall bit rate                         : 28.8 Mbps
Maximum Overall bit rate                 : 48.0 Mbps

Video
ID                                       : 4113 (0x1011)
Menu ID                                  : 1 (0x1)
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L4.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 3 frames
Codec ID                                 : 27
Duration                                 : 1h 34mn
Bit rate mode                            : Variable
Bit rate                                 : 23.5 Mbps
Maximum bit rate                         : 33.0 Mbps
Width                                    : 1 920 pixels
Height                                   : 1 080 pixels
Display aspect ratio                     : 16:9
Frame rate                               : 23.976 fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.473
Stream size                              : 16.0 GiB (84%)
Writing library                          : x264 core 120 r2120 0c7dab9
Encoding settings                        : cabac=1 / ref=4 / deblock=1:1:1 / analyse=0x3:0x133 / me=umh / subme=10 / psy=1 / psy_rd=0.40:0.00 / mixed_ref=1 / me_range=24 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=6 / sliced_threads=0 / slices=4 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=1 / constrained_intra=0 / bframes=3 / b_pyramid=1 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=1 / weightp=1 / keyint=24 / keyint_min=2 / scenecut=40 / intra_refresh=0 / rc_lookahead=24 / rc=abr / mbtree=1 / bitrate=23500 / ratetol=1.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=33000 / vbv_bufsize=30000 / nal_hrd=vbr / ip_ratio=1.40 / aq=1:0.60
Color primaries                          : BT.709-5, BT.1361, IEC 61966-2-4, SMPTE RP177
Transfer characteristics                 : BT.709-5, BT.1361
Matrix coefficients                      : BT.709-5, BT.1361, IEC 61966-2-4 709, SMPTE RP177

Audio #1
ID                                       : 4352 (0x1100)
Menu ID                                  : 1 (0x1)
Format                                   : PCM
Format settings, Endianness              : Big
Format settings, Sign                    : Signed
Muxing mode                              : Blu-ray
Codec ID                                 : 128
Duration                                 : 1h 34mn
Bit rate mode                            : Constant
Bit rate                                 : 1 536 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Stream size                              : 1.02 GiB (5%)

Audio #2
ID                                       : 4353 (0x1101)
Menu ID                                  : 1 (0x1)
Format                                   : AC-3
Format/Info                              : Audio Coding 3
Mode extension                           : CM (complete main)
Format settings, Endianness              : Big
Codec ID                                 : 129
Duration                                 : 1h 34mn
Bit rate mode                            : Constant
Bit rate                                 : 448 Kbps
Channel(s)                               : 6 channels
Channel positions                        : Front: L C R, Side: L R, LFE
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Compression mode                         : Lossy
Stream size                              : 304 MiB (2%)

Audio #3
ID                                       : 4354 (0x1102)
Menu ID                                  : 1 (0x1)
Format                                   : PCM
Format settings, Endianness              : Big
Format settings, Sign                    : Signed
Muxing mode                              : Blu-ray
Codec ID                                 : 128
Duration                                 : 1h 34mn
Bit rate mode                            : Constant
Bit rate                                 : 1 536 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Stream size                              : 1.02 GiB (5%)

Text
ID                                       : 4608 (0x1200)
Menu ID                                  : 1 (0x1)
Format                                   : PGS
Codec ID                                 : 144

[sneaker]

Show your complete AviSynth script and encoding command, i.e. how you pass AviSynth to x264. Upload m2ts and encode sample. Are you creating the screenshots of the m2ts and the x264 encode with the same program? Which one? Renderer?

[jagabo]

It's likely your player simply isn't responding to the specified colormatrix and is using rec.601. That's very common. For example, VirtualDub always displays YUV with a rec.601 conversion. And your DVD is rec.601.

[killerteengohan]

Show your complete AviSynth script and encoding command, i.e. how you pass AviSynth to x264. Upload m2ts and encode sample. Are you creating the screenshots of the m2ts and the x264 encode with the same program? Which one? Renderer?

I didn't do anything but resize it with spline64 in the script. I'm using MeGui like I always do. I am doing the same encoding process I have for several years. This is not a common problem so it's not my doing anything I haven't already done before.

LoadPlugin("C:\Program Files (x86)\MeGui\tools\dgavcindex\DGAVCDecode.dll")
AVCSource("C:\Users\Aiden\Desktop\Encoding\Little Nemo\BDMV\STREAM\00000.dga")
crop( 2, 18, -2, -16)
Spline64Resize(848,464)

the m2ts is 20GB in size, I cant upload that.

here is the finished encode

https://mega.nz/#!A8QQgLKJ!Te0thZjbU0kKwYv7AlHJ9CxFpAuT3s8njKsRH1VHC9o

Screenshot is from M2TS and the finished encode, top is M2TS bottom is encode. Yes their being played with the same program and screenshots are being taken the same way. I tried 3 different media players and their all doing the same thing and on other computers.

[killerteengohan]

It's likely your player simply isn't responding to the specified colormatrix and is using rec.601. That's very common. For example, VirtualDub always displays YUV with a rec.601 conversion. And your DVD is rec.601.

That doesn't make sense to me, the M2TS responds and works fine. I could swear I set the color primaries and flagged it for use so the players would know what to use. I have never had this problem before and its always fixed it when this has happened before, so why this time?? I even tried 4 different players and different computers. So one of them should have responded to it I would think. Usually doing that always fixes it for me in the past.

EDIT: this makes more sense after more information was given to me in a later post.

Thanks Jagabo!

[jagabo]

That doesn't make sense to me, the M2TS responds and works fine.

Different containers, different results.

I could swear I set the color primaries and flagged it for use so the players would know what to use.

As I said, players often don't respond to colormatrix flags.

I have never had this problem before

You probably did have the problem and just never noticed it until now.

[sneaker]

The scripts is indeed ok (well, AVCSource is deprecated and buggy, but it shouldn't introduce color problems). Sample of m2ts can be cut using mkvtoolnix.

But your encoded mp4 looks like your upper screenshot on my PC. So it could be a problem of your player like jagabo said - or the original screenshot is wrong. I can't double check without a sample of the m2ts but since your script looks ok I assume it is just a playback problem. Note that many players/renderers do not respond to color matrix flags at all, they only choose according to resolution. That's why many people use ColorMatrix(mode="Rec.709->Rec.601") when converting from HD to SD.

I can recommend MPC-HC/LAV + madvr. It can detect color flags properly.

[killerteengohan]

Note that many renderers do not respond to color matrix flags at all, they only choose according to resolution. That's why many people use ColorMatrix(clip, mode="Rec.709->Rec.601") when converting from HD to SD.

I find that weird because I have not had this problem before until just now. Specifying in the h.264 settings and flagging it as rec709 has always fixed this for me before. It's just this one source being like this for me for some reason.

I'm already using MPC-HC/LAV.


I added this to the script as you mentioned and the colors came out fine. Thanks for mentioning that.
What exactly does this do? convert it to rec601?

Colormatrix(mode="rec.709->rec.601", clamp=0)

[sneaker]

Yes, it will convert to Rec.601. So don't flag as Rec.709 when applying ColorMatrix() like this.

I'm already using MPC-HC/LAV.

But not with madvr? (Renderer is chosen in Options->Output)
With other renderers it may depend on the GPU drivers to some extend. Some even react differently when you have multiple player windows open.

[killerteengohan]

You probably did have the problem and just never noticed it until now.

Oh believe me, I would have noticed it with how OCD I can be about those things.

I think it's more along the lines of this is the first time I have downscaled HD to SD resolution. Usually I don't scale below 720p. Apparently it changes to rec601 for SD resolutions at times.

Thanks for the information and clearing things up a bit for me.

[jagabo]

Just noticed your encoded sample:

LWLibavVideoSource("00000 480p colorprim 709 ConvertToYV12.mp4") 
StackVertical(ConvertToRGB(matrix="rec601").Subtitle("ConvertToRGB(matrix=''rec.601'')"), ConvertToRGB(matrix="rec709").Subtitle("ConvertToRGB(matrix=''rec.709'')"))

[Attachment 40948 - Click to enlarge]

Your player is ignoring the rec.709 flag and using rec.601. Maybe because you've downscaled to SD. Sometimes players use rec.601 for SD and rec.709 for HD.

[killerteengohan]

Yes, it will convert to Rec.601. So don't flag as Rec.709 when applying ColorMatrix() like this.

I'm already using MPC-HC/LAV.

But not with madvr? (Renderer is chosen in Options->Output)
With other renderers it may depend on the GPU drivers to some extend. Some even react differently when you have multiple player windows open.

Will converting to rec601 like that ruin anything like chroma or luma or colors?

would it be better if I did it like this?

ConvertToYV12(matrix="rec601")
Colormatrix(mode="rec.709->rec.601", clamp=0)

[sneaker]

Will converting to rec601 like that ruin anything like chroma or luma or colors?

No.

would it be better if I did it like this?

ConvertToYV12(matrix="rec601")
Colormatrix(mode="rec.709->rec.601", clamp=0)

No. The Blu-ray source is already YV12 so ConvertToYV12(matrix="rec601") should do nothing at all.

[killerteengohan]

Your player is ignoring the rec.709 flag and using rec.601. Maybe because you've downscaled to SD.

Thanks, I kind of figured that after reading what you and sneaker said.

All 4 of the players and on multiple computers are ignoring the flag? Kind of makes me wonder why you can even flag things if the players don't care about them and usually go by resolution.

I don't see any option about colorspace, rec601, or rec709 settings in MPC-HC. Is there anything I can do to force it use the flag, or am I stuck converting to rec601 with something like "Colormatrix(mode="rec.709->rec.601", clamp=0)"

Is converting like that a bad idea or going to hurt anything?

[jagabo]

All 4 of the players and on multiple computers are ignoring the flag? Kind of makes me wonder why you can even flag things if the players don't care about them and usually go by resolution.

In my experience players are slowly catching on. MCPHC in my computer follows the flagged matrix.

[sneaker]

If it works in MPC-HC depends on the chosen renderer and maybe even GPU driver plus splitter/decoder. With LAV+madVR it should always work (and madvr has options to force different flags).

[jagabo]

If it works in MPC-HC depends on the chosen renderer and maybe even GPU driver plus splitter/decoder.

I agree.

[killerteengohan]

Will my media player ignore flags and auto pick rec709 if I upscale SD material into an HD resolution? I wish to know what you think so I don't have to encode it twice to find out and I can just tell it what to be in the script.

How can I keep it at rec601?

This is for making rec709 into rec601. Is it also for making it stay in rec601 if it was never rec709 to begin with?

Colormatrix(mode="rec.709->rec.601", clamp=0)

[sneaker]

You can keep it at Rec.601 but most players will assume it's Rec.709. No AviSynth filter can change that. If your player assumes Rec.709 you have no choice but to either convert to Rec.709 or view with wrong colors.

If you want to keep at Rec.601 and want to play it correctly you need a player that can read colorimetry flags. Like MPC-HC/LAV+madvr. Then if you add --colormatrix smpte170m --colorprim smpte170m --transfer smpte170m (bt470bg for PAL)playback should be correct. But I would convert to Rec.709 so it plays correctly in all players.

This is for making rec709 into rec601. Is it also for making it stay in rec601 if it was never rec709 to begin with?

Colormatrix(mode="rec.709->rec.601", clamp=0)

No.

[killerteengohan]

So how do you convert to rec709?

like this?

Colormatrix(mode="rec.601->rec.709", clamp=0)

[sneaker]

ColorMatrix(mode="Rec.601->Rec.709")

[killerteengohan]

Am I supposed to allow clamping? I thought the clamping=0 kept colors from clipping or closer to how they should be

[jagabo]

Clamping eliminates some obviously illegal colors, those with Y<16, Y>235, UV<16, or UV>240. If your source has none of those clamping will make no difference. If you're working in full range YUV clamping will crush darks and brights.

Clamping won't eliminate illegal combinations of YUV (in rec.601 only about 1/6 of all YUV colors lead to valid RGB colors, even fewer in rec.709). See the "RGB Colors Cube in the YCbCr Space" diagram at https://software.intel.com/en-us/node/503873. The outer cube is the full YUV color space. The inner rectangle is the valid RGB color space.

[killerteengohan]

Is there a quick tool in avisynth that will easily tell me if its full YUV or not?

If I ever do not know whether or not its full or not, am I better off with not clamping or clamping and risk the crushing you mentioned?

[jagabo]

Histogram(), ColorYUV(analyze=true), VideoScope()

Almost all YUV acquisition and distribution formats use limited range YUV. One common exception is MJPEG which usually uses full range YUV. Some dSLR cameras use full range too.

[killerteengohan]

Thanks, those tools are helpful. I couldn't get the videoscope to work though.


coloryuv(analyze=true) gave this result

Frame: 68891 ( LumaY / ChromaU / ChromaV )
Average: ( 98.80 / 122.06 / 135.30 )
Minimum: ( 5 / 78 / 104 )
Maximum: ( 214 / 151 / 170 )
Loose Minimum: ( 16 / 92 / 109 )
Loose Maximum: ( 194 / 148 / 167 )


Does the fact that a number 5 is there mean that its full range YUV because it goes under 16 in several frames for the minimum section of Luma Y?? I have also seen the maximum go to 245 which is over 235/240. Does that also indicate its full range? I dont see 0 or 255 on any frames but I see it go close to it. Ive seen it go as low as 1.

Is it considered full or not?


When I add in colormatrix(hints=true, threads=0) nothing goes below 16 anymore. Thats clipping it by doing that as MeGui suggested with colour correction checked in is it not? I never knew this was going on until just now trying this tool. Do I not want to use that anymore?

[jagabo]

Does the fact that a number 5 is there mean that its full range YUV because it goes under 16 in several frames for the minimum section of Luma Y?? I have also seen the maximum go to 245 which is over 235/240. Does that also indicate its full range?

No. It's very common for oversharpening halos (especially with video tape based sources) to overshoot below 16 or over 235. The "loose" values are more indicative of the real range. But even that is not definitive. Viewing the waveform with Histogram or VideoScope (the latter only works with YUY2) is a much better way of checking levels. Like I said earlier, very few sources will be full range. And if it's your processing that producing large excursions outside the limited range -- you should fix your processing. And even if your source is full range you should change it to limited range because pretty much all distribution formats use limited range (DVD, Blu-ray, digital broadcast TV, youtube, etc.).

When I add in colormatrix(hints=true, threads=0) nothing goes below 16 anymore. Thats clipping it by doing that as MeGui suggested with colour correction checked in is it not?

Yes.

I never knew this was going on until just now trying this tool. Do I not want to use that anymore?

If you were really working with full range YUV all you have to do is set clamping=0 to prevent the clipping by ColorMatrix().

[jagabo]

Even innocuous operations like resizing can create excursions outside the limited range. For example, here's LanczosResize(width/2, height/2). The original had no excursions below Y=16 or above Y=235. But the resizing halos resulted in overshoots down to 0 and up to 255:


[Attachment 41182 - Click to enlarge]

In this shot it doesn't really matter because Y<16 will be drawn the same shade of black as Y=16, and Y>235 should be the same shade of white as Y=235. This waveform monitor display was created with TurnRight().Histogram().TurnLeft().

[killerteengohan]

As I understand it from reading, illegal colors are considered colors that cannot be displayed on a TV or certain devices, right? and supposedly anything under 16 is shown as black and over 240 is shown as white while watching it on a TV.

If the TV does the clipping/crushing for you and the computer is capable of viewing full range, why would anyone want to limit the colorspace with clamping? On their computer they could see it as full and the TV will just display them as black or white, right?

It sounds like not limiting it when it goes out of the limited range would be a good idea if you mostly watch on the computer and occasionally care to watch on your TV.

[killerteengohan]

Also I have another question. I noticed when I use

ColorMatrix(mode="Rec.601->Rec.709", clamp=0)

and then check it with

Limiter(16, 235, 16, 240, "luma")
ColorYUV(analyze=true)


That a lot of red spots are appearing all over the blacks. If I take mctemporaldenoise out of my script most of it goes away but it is still there.

When I move ColorMatrix(mode="Rec.601->Rec.709", clamp=0) to the end of the script, just above the checking limiter and coloryuv commands all the red dots go away.

Am I supposed to be placing ColorMatrix(mode="Rec.601->Rec.709") at the beginning of the script like megui was doing all this time, or at the end of the script after all of the filtering is done? Does it even really matter??