Avisynth nnedi3_rpow2 Color Shift Problem

[SameSelf]

I have searched the web but haven't found an answer. When I load some NTSC color bars in Avisynth, there is no color shift:

LWLibavVideoSource(bars, fpsnum=30000, fpsden=1001, format="YUV411P8", dominance=2)
ConvertToYUY2(matrix="Rec601", interlaced=true, chromaresample="Point")

However, when I add the following line after Convert, there is a color shift:

nnedi3_rpow2(2, cshift="PointResize", fwidth=960, fheight=720)

Looking at the Avisynth wiki, there doesn't seem to be any arguments to specify the colorspace. My guess is that the filter is doing some sort of transformation to BT.709 since the upscale is to HD dimensions. But when I try changing the matrix to Rec709 in Convert, the color shift still remains. Is there a solution to this problem or is the filter buggy? TIA

EDIT: The wiki says that the above line is equivalent to:

nnedi3_rpow2(rfactor=2)
PointResize(960, 720, src_left=-0.5, src_top=-0.5)

So I tried running just nnedi3_rpow2(rfactor=2) to see if the problem resided in the Resizer() filter. And the color shift still occurs. So the problem seems to be in the nnedi3_rpow2 filter.

[poisondeathray]

There should be no color shift on solid colors (ie. in the middle of a solid bar). It doesn't do anything extra.

How are you determining a shift? How are you converting back to RGB if you're viewing it ? or what program are you using to determine the shift ?

HD usually gets Rec709 treatment when converted back to RGB for display by convention, so the "proper" way to upscale to HD would be to use colormatrix filter (ie. change the color in YUV as if it had used 709 in the first place) . Similarly, the "proper" way to downscale from HD source would be to change 709 to 601 in YUV. So I'm guessing the application you're using is using the assumed 709 matrix because the upscaled content is now "HD" , but you haven't adjusted for it, thus the color shifting

[SameSelf]

Thanks, pdr. That makes sense. I think.

I am using the scopes in PP to determine if there is a color shift because I never trust my eyes. According to Adobe, PP uses the YUV data for the scopes, not the RGB data generated for display.

Stupid question here. Is there a way to tell PP which matrix to use for the RGB conversion on footage?

Also, what filter do you suggest using for converting to 709? TAA

[poisondeathray]

I am using the scopes in PP to determine if there is a color shift because I never trust my eyes. According to Adobe, PP uses the YUV data for the scopes, not the RGB data generated for display.

Stupid question here. Is there a way to tell PP which matrix to use for the RGB conversion on footage?

Also, what filter do you suggest using for converting to 709? TAA

But what format are you importing into Adobe ? Recall that not all YUV formats are "treated" as "YUV" in Adobe. The scopes only work in YUV if the format is treated as YUV.

There is no way that I know of to influence how it does the RGB conversion within Adobe, you have to use various workarounds. It has changed in the past too. In CS4 and prior, everything used to get 601 treatment, except v210. So the workaround for HD workflows was to convert everything to v210. Things are more accurate as to what they "should be" with the newer releases

You can prove it to yourself that avisynth isn't the culprit by converting it to RGB and specifying the matrix. Then compare RGB values in a central location of each bar. If nnedi3_rpow2 is doing something funky, it this would show a difference in solid bar colors. And it works as expected. The 2nd once uses 601, which of course is non standard for HD. But it demonstrates that nnedi3_rpow2 isn't doing anything unexpected in terms of color shifting on solid colors. (Edges are a different discussion.) . If you change the RGB conversion to 709 and compare the colors to the ones you're getting in Adobe, I'm guessing it's similar

colorbars(720,480,pixel_type="yuy2")
converttorgb24(matrix="rec601")

colorbars(720,480,pixel_type="yuy2")
nnedi3_rpow2(2, cshift="pointresize", fwidth=960,fheight=720)
converttorgb24(matrix="rec601")

A typical 8bit colormatrix conversion for upscaling would use
colormatrix(mode="rec.601->rec.709", clamp=0)

You can do it at higher bitdepths using dither tools or vapoursynth fmtconv, +/- dithering . There are pros/cons for doing it that way depending on the content but that's another discussion too. For example, in most scenarios you would want to dither, but typically not on solid patterns,color bars, clean graphics and titles

[SameSelf]

pdr, I agree with everything you said. I must be slowly catching on because in the past all that would have been gobblygook to me. Progress

Anyway, I should have mentioned that I am saving the Avisynth script as a UYVY AVI in vdub since we both know from our prior discussions that is only trustworthy codec for YUV sources in PP.

So I inserted ConvertToRGB24(matrix="Rec601"), and sure enough there is no color shift. Indeed, the problem lies somewhere in PP. So I replaced ConvertToRGB24 with the ColorMatrix command you suggested and now there is no color shift! Hurrah! It never occurred to me before that converting to BT.709 when uprezzing to HD was a necessary step. Good thing I test all my workflows now with color bars.

LWLibavVideoSource(bars, fpsnum=30000, fpsden=1001, format="YUV411P8", dominance=2)
ConvertToYUY2(matrix="Rec601",interlaced=true,chromaresample="Point")
nnedi3_rpow2(2, cshift="PointResize", fwidth=960, fheight=720)
colormatrix(mode="rec.601->rec.709", clamp=0)

[poisondeathray]

colormatrix takes interlaced=true/false argument as well, so it should be interlaced=true if you used it earlier for ConvertToYUY2. (It won't matter on static bars, but it's something you should do anyways or you'll forget to do it on real footage)

Those observations suggest the scopes do not work before the RGB conversion for the preview in PP; and that there is another conversion going on even with UYVY

[SameSelf]

Yeah, PP does a lot of things behind the scenes that aren't well documented. But then again, even Resolve has its quirks.

Thank you for the heads up on the interlaced flag. The documentation for colormatrix is pretty sparse.

[pandy]

Have no time to run your scripts and analyze outcome but remeber that chroma sample position vs luma sample position is different in various codecs (MPEG-1 chroma is located between 2 luma sample - average where in MPEG-2 is only on first luma sample and repeated trough zero order hold to next luma sample).
In interlace and vertical direction this is even more complicated.

[SameSelf]

Actually that is a much abbreviated version of my full script to focus on the line that was "causing" the problem, nnedi3_rpow2. But as pdr and I determined, it is not the script that is the problem, but how PP interprets the higher resolution file. Anyway I am not posting from my PC currently, so I don't have access to the full script.