Which format is better as intermediate? YUV420 full range or YUV422 10-bit?

[rgr]

I do not have the option to write in YUV444 or RGB, only I can in YUV420 (MP4 with full color range) or YUV422 (10-bit ProRes with limited color range).
Which of these formats gives a better picture quality for further editing - YUV420:FR or YUV422-10?
The size of the files generated by these formats is very close, suggesting the same quality.

Final video will be in YUV420:FR.

[poisondeathray]

Why do you need an intermediate ? What is preventing you from using the "original ?" What is the original source ?

The answer regarding quality would also depend on the source pixel format, how the editor/program handles the intermediate conversions if any (e.g. how it upsamples and downsamples if required) .

What editor ?

What compression format is your YUV420 MP4 choice ? What codec, what settings , lossy vs. lossless ?

[rgr]

Source: MTS file, YUV420 (range 16-255!), 25M bitrate, FHD

In Avisynth, I extend the color range to 0-255, stabilize in VirtualDub+Deshaker3.1 and want to run Topaz Video Enhance. And this program offers only such saving options. (Accepts almost everything).

Final: YUV420, H264, CRF 17-19.

[poisondeathray]

VEIA can export image sequences, that's the highest quality export option (eg. TIFF or PNG at 16bit) . Prores is a good compromise for the intermediate, but it's lossy . You can see the quality loss if you look carefully, not with normal viewing

But it probably doesn't matter what you choose, as long as the export quality is decent from VEIA - because your final format step will cause the most quality loss

[smartel]

deshaker only works in RGB i think ..

[poisondeathray]

deshaker only works in RGB i think ..

Yes, and same with VEIA

[lollo]

deshaker only works in RGB i think

and then expanding the levels from 16-235 to 0-255 prior to feeding a RGB filter is not appropriate

[poisondeathray]

deshaker only works in RGB i think

and then expanding the levels from 16-235 to 0-255 prior to feeding a RGB filter is not appropriate

His camera shoots 16-255 .

Source: MTS file, YUV420 (range 16-255!), 25M bitrate, FHD

The other option is to "legalize" Y 255 to 235 before RGB conversion, but you compress (squish) the data in the highlight range so there are overlapping values - this causes loss of data and banding. Full range YUV to full range RGB is technically the better option for intermediate processing (for him this would result in RGB 16-255), as long as you address the elevated black level at RGB 16 somewhere in the workflow. His final goal is full range YUV, so he's probably adjusting the black level to RGB 0 in the editor before the final YUV full range conversion. Elevated black levels in the intermediate steps often work better with many filters, especially denoising and upscaling. Often there is disproportionate noise in the shadow areas.

[lollo]

What I understand is that he's going from YUV 16-255 in AviSynth to a RGB VirtualDub filter. No editor adjustement.
If he expands to 0-255 the Y<16 will be lost, as well the Y>235 that is already there.
I think he needs to compress to 16-235 rather than expanding to 0-255, otherwise VirtualDub will crush/clip the levels.

[poisondeathray]

He's "extending the range in avisynth" beforehand. He can clarify, but I'm assuming that means he's using a PC / full range matrix in avisynth for the RGB conversion before deshaker, That would give you YUV 16-255 to RGB 16-255 (not really 0-255, unless you had 0-255 in YUV to begin with) , and less squishing or data compression than if you had compressed Y to 16-235 . He mentioned the final output is YUV Full range, so that would be a better higher quality option than squishing the highlights

Source: MTS file, YUV420 (range 16-255!), 25M bitrate, FHD

In Avisynth, I extend the color range to 0-255, stabilize in VirtualDub+Deshaker3.1

[lollo]

Maybe. I understand that the output of the AviSynth script is YUV 0-255, and this is loaded in VirtualDub, causing a potential problem. If the output of the AviSynth script is RGB 0-255 it is ok. Let's see

[rgr]

What I understand is that he's going from YUV 16-255 in AviSynth to a RGB VirtualDub filter. No editor adjustement.
If he expands to 0-255 the Y<16 will be lost, as well the Y>235 that is already there.
I think he needs to compress to 16-235 rather than expanding to 0-255, otherwise VirtualDub will crush/clip the levels.

May I explain:

Step 1 (avisynth to vdub)

v=ffms2("2022_05_14 15_31_15.mts",atrack=1)
v=convertbits(v,10) <--for better precision
v=converttoyuv444(v,matrix="PC.709",chromaresample="spline64")
v=coloryuv(v,off_y=-16,gain_y=128/7) <-- 16-254 to 0-255
v=converttorgb24(v,matrix="PC.709")

(a bit complicated, I planned output in RGB48, but the Deshaker filter in VDub expects RGB24/32, so I end up with RGB24)

Step 2:
I write in Vdub with x264 10-bit codec as RGB+lossless
(I know, unnecessary but before that I planned to use filters)

Step 3:
Use VEAI*

Step 4:
Final editing and saving as YUV420:FR

* - BTW I found a bug in VEAI. If we convert a YUV:FR file and save it as MP4, the output file will be marked as Full Range, but the color range will be 16-235. You have to "feed" the VEAI with RGB files, it seems that then this error does not occur. Edit: I just noticed that only limited range MP4 files can be converted in VEAI to MP4. If the source file is MP4/RGB, then VEAI generates an MP4 RGB file, but with the range 16-235... Funny

[rgr]

To get the final answer I did a short test (300 frames, Artemis Medium Quality):

Source: MP4, RGB48 (10-bit), full range

Comparison file: clip YUV444 10-bit limited range made from 16-bit TIFF images (CRF 0) - 190MiB

Output format (all limited range):
a) ProRes 4:2:2 HQ (Fast) - 293MiB, SSIM 98.1%
b) ProRes 4:2:2 Slow - 263MiB, SSIM 98.1%
c)* MP4, CRF 0 - 341MiB, SSIM 98.9%

* - due to a bug in VEAI, in order for the output file to be usable, I had to convert it (lossless - copy) to a limited range format (ffmpeg with bsf:metadata and color_range)

Well, I'm a bit disappointed with the results.