how to properly convert rgb signpost to yv12

[codemaster]

I use Premiere Pro CS3 to import the MPEG2 clips from the camcorder, edit them, and correct their tonality and colors. Then I export the Premiere sequence to uncompressed AVI signpost, with Debugmode FrameServer and it's RGB24 option. Then I use AviSynth to deinterlace, resize, denoise, etc, and frameserve to x264 encoder.

From what I understand, I have to convert to YV12, because those AviSynth filters cannot work with RGB input. But I don't understand how to properly convert from RGB to YV12 in AviSynth. For example, how can I make sure there will not be tonality shifts or color shifts, after the RGB to YV12 conversion ? Or how can I get exactly what I see in Premiere, in terms of tonality and colors, after the conversion to YV12 ?

There are 3 values than can be attributed to the "matrix" parameter from the ConvertToYV12() function:

ConvertToYV12(interlaced=true, matrix="pc.601") # keeps full range [0,255]
# or
ConvertToYV12(interlaced=true, matrix="rec709") # scales to TV range [16,235]
# or
ConvertToYV12(interlaced=true, matrix="pc.709") # keeps full range [0,255]

But what is confusing to me is that, after I analyse the original MPEG2 clips with DGIndex, in the Information window it says:

Colorimetry: BT.470-2 B,G*

So, several MPEG2 clips with colormatrix BT.470-2 are imported into Premiere Pro CS3 timeline, some cuts are made, some dip to black transitions are applied, then the Fast Color Corrector is applied on many of the clips, and the Three-Way Color Corrector is added on some clips. In this process:

1. Clips are converted from YCbCr to RGB ? What happens with the original colormatrix ? It converts that too ? From BT.470-2 to something else ?

2. When exporting the sequence to avi signpost, those exported frames have the exact same colormatrix as the frames inside the timeline, wich are displayed in Premiere UI ?

3. How do I determine the proper value for the matrix parameter for "ConvertToYV12()", so that what I see in Premiere UI is what I get after RGB to YV12 conversion ?

4. How do I determine the proper value for the --colormatrix parameter for x264.exe, so that what I see in VirtualDub (from the avs with the RGB to YV12 conversion) is what I get in the H.264 stream ?

[poisondeathray]

1. Clips are converted from YCbCr to RGB ? What happens with the original colormatrix ? It converts that too ? From BT.470-2 to something else ?

In CS3, most filters cause conversion to RGB. There are some exceptions, like brightness used with fast color corrector (allow you to stay in YUV). Premiere CS3 uses ITU Rec.601 for the conversion to RGB, regardless of flags in the original stream. Even HD streams get converted using 601 (incorrectly, they should be using Rec.709). The exception is v210 input, which gets 709 treatment (and this is the workaround you had to do in the past for proper color for HD streams)

2. When exporting the sequence to avi signpost, those exported frames have the exact same colormatrix as the frames inside the timeline, wich are displayed in Premiere UI ?

If you've used those RGB filters (you will see red render bar above the sequence) then you will be working in RGB. The term "colormatrix" no longer applies here , because you're going RGB timeline => RGB export. "Colormatrix" only applies to YUV<=>RGB conversions . (unless you mean what premiere originally did to convert to RGB, but that is a different step compared to the export signpost stage).

3. How do I determine the proper value for the matrix parameter for "ConvertToYV12()", so that what I see in Premiere UI is what I get after RGB to YV12 conversion ?

You use ConvertToYV12() for this specific conversion in CS3, the default is rec601. (other programs might be different, e.g. vegas uses studio RGB, so you need a different matrix)

4. How do I determine the proper value for the --colormatrix parameter for x264.exe, so that what I see in VirtualDub (from the avs with the RGB to YV12 conversion) is what I get in the H.264 stream ?

Vdub and most decoders ignore the flag. The flag is just a "label" or "sign". It does nothing to the actual YUV data. You would flag it bt470bg , which use the same coefficients as bt.601 . But even if you flagged it something different , it would look the same in most applications . It's a way of communicating 601 was used for the entire workflow