Color trouble during transcoding and denoising workflow

[effes]

Hello. I'm editing a short horror film and I'm pretty basic with AviSynth and stuff, I'm not a programmer. Can someone help me out? I'm on Win7 64bits.

This is what I want to do:

1.) Import original QT HD Canon T1i files to Premiere CS6.
2.) FrameServe project, open signpost.avi in VirtualDub via AviSynth script and apply MCTemporalDenoise (which asks to convert footage to YV12).
3.) Save denoised result as a lossless file (Uncompressed AVI or LAG or whatever).
4.) Import back again denoised lossless files to Premiere for editing (and in some cases also to After Effects for compositing, output again and then back to Premiere).

So far I managed almost everything. The thing that's really bothering me is ColorSpace - I would like to keep the color of the transcoded files as close to the original as possible. Canon T1i files are supposedly YUV 4:2:0 8bits. I started out making intermediates AVI files instead of frame serving but I got pretty bad results, from very dark clips to entire black screen, depending on how I transcoded from QT to uncompressed AVI (I tried rawvideo with ffmpeg, exporting with QT and with different output options in Premiere). Now I'm frameserving and it all seems ok, but I want to be completely sure I am doing it the right way.

I would like to know:

- What format should I tickle on Debug FrameServer Video Output window? (RGB24? RGB32? YUY2?)
- What mode should I tickle on VirtualDub LAG compression window? (RGBA? RGB (Default)? YUY2? YV12?)
- If I want denoised result as a Uncompresed AVI instead of LAG, is there anything color-related I should take care of, considering my source?
- In VirtualDub there's Video -> Color Depth option, should I tickle something there?

And finally, this is my avs script for MCTemporalDenoise, just in case. It's giving me really GOOD results, except in scenes where's there's too much concrete or textured walls. But I'm reading you guys and trying to find a few variations and tweaks.

AviSource("signpost.avi")
AssumeTFF()
ConvertToYV12
MCTemporalDenoise(settings="high", sigma=20, enhance=true, chroma=true, edgeclean=true, ECrad=3, stabilize=true, maxr=2, interlaced=false)

If someone can help me out, please do. I'll highly appreciate it. Thanks!

[poisondeathray]

- What format should I tickle on Debug FrameServer Video Output window? (RGB24? RGB32? YUY2?)

It depends what you did in PP . If you didn't use any filters or color correction (or only YUV labelled filters) , the timeline will still be in YCbCr so you would use YUY2. If you used and RGB filters then it will be converted to RGB , then you would use RGB24. RGB32 is only for alpha channel

- What mode should I tickle on VirtualDub LAG compression window? (RGBA? RGB (Default)? YUY2? YV12?)

Lagarith will automatically select same as original unless you force it otherwise . By default, always suggest RGB is checkmarked and RGB(default) is selected. .e.g If you feed it YV12 it will come out YV12. Make sure vdub is set to video=>fast recompress to avoid other colorspace conversions

- If I want denoised result as a Uncompresed AVI instead of LAG, is there anything color-related I should take care of, considering my source?

same as above, If you want uncompressed 8-bit YCbCr 4:2:0 after the script processing, just leave video set to "fast recompress" . This will disable all other colorspace conversions and subsampling options

- In VirtualDub there's Video -> Color Depth option, should I tickle something there?

in "fast recompress" mode this is disabled

And finally, this is my avs script for MCTemporalDenoise, just in case. It's giving me really GOOD results, except in scenes where's there's too much concrete or textured walls. But I'm reading you guys and trying to find a few variations and tweaks.

AviSource("signpost.avi")
AssumeTFF()
ConvertToYV12
MCTemporalDenoise(settings="high", sigma=20, enhance=true, chroma=true, edgeclean=true, ECrad=3, stabilize=true, maxr=2, interlaced=false)

Hard to suggest anything concrete from this; either provide a better description or sample

If it's too strong in sections, consider filtering selectively (applying different filters to different sections), or using masks

If you had frameserved RGB, then ConvertToYV12 would normally use ConvertToYV12(matrix="Rec709", clamp=0) for HD footage , but Canon DSLR footage (at least for 7D, T2i, not sure about yours) actually uses Rec601 for the conversion matrix (it's in the metadata), so you can leave it out (it defaults to 601) . The clamp=0 is important to include in order to not cut off superbrights/ superdarks (i.e. Y' <16, Y'>235 are clipped) for RGB<=> Y'CbCr convesions

[effes]

Thank you very much for your reply. You have helped me immensely to better understand this new Pandora box I just opened called color!

It depends what you did in PP . If you didn't use any filters or color correction (or only YUV labelled filters) , the timeline will still be in YCbCr so you would use YUY2. If you used and RGB filters then it will be converted to RGB , then you would use RGB24. RGB32 is only for alpha channel

They're just the raw t1i clips dragged onto timeline. No effects, nothing. I did this to FrameServe in order to avoid transcoding. But perhaps I am unaware and there is a direct way of opening avc1 files wrapped in a .MOV container with VirtualDub via an avs script? It's not a big deal to do it with Premiere, but if I can avoid the hassle...
Anyway thank for your tips, they will also be helpful when rendering the final edited output.

Hard to suggest anything concrete from this; either provide a better description or sample
If it's too strong in sections, consider filtering selectively (applying different filters to different sections), or using masks

Yes, I know. I just posted it because I wasn't sure of the "ConvertToYV12" line. This happening mostly during night shots filmed on a old train station - the floor and the walls get a bit too plastic. Also, I had to put sigma and enhance values in order to avoid banding. Either way I'll experiment and post sample in case I want better. Still, I'm getting way way better than the original. MCTemporalDenoise is really quite impressive. Thanks again!

[poisondeathray]

They're just the raw t1i clips dragged onto timeline. No effects, nothing. I did this to FrameServe in order to avoid transcoding. But perhaps I am unaware and there is a direct way of opening avc1 files wrapped in a .MOV container with VirtualDub via an avs script? It's not a big deal to do it with Premiere, but if I can avoid the hassle...
Anyway thank for your tips, they will also be helpful when rendering the final edited output.

Yes you can open mov files directly in avisynth either with ffms2.dll or qtsource.dll , or as a last resort directshowsource
http://code.google.com/p/ffmpegsource/
http://www.tateu.net/


Note many canon dslr h264 are opened with full range if you use ffms2, (because of a flag in the mov file and the way libav interprets the flag). You may have to convert the levels to "legal" levels. QTSource through quicktime will open them always clamped 16-235 (not clipped, but clamped or squished)

But why are you using premiere (are you editing in premiere) ? If not, don't even use premiere (huge waste of memory and system resources)

When you frameserve you have to transcode (or more precisely re-encode) , because frameserving serves uncompressed video

If you are doing this only to use MCTD, then use it directly in the script

[poisondeathray]

Another issue with your workflow might be needlessly denoising stuff that you're going to cut out anyway (as you know MCTD isn't exactly "fast" at those settings you're using, seems it could amount to a huge waste of time) .

A faster /better workflow might be to do rough cuts before MCTD and re-importing for final edits . Or do all your editing and MCTD later on . It depends on the specifics of the type of project your are doing

[effes]

It's all edited in Premiere already. I know mastering is likely part of the final stage but I wanted to denoise raw files (always keeping a copy of the original just in case of course) because I want two versions of the same short film: one all clean, all polished, the other one grindhouse-like (I might even put more noise, grain and scratches here and there for the second). Then when opening PP project I can just switch between denoised/noisy input folders and tada! a new version of the film is created which I can continue to make changes if necessary. And most of the shots ended up in the timeline anyway. But I get what you mean.
Now back on code track. I'm a noob. I downloaded ffms-2.17-cplugin but I couldn't get it to work. I believe my 2 line script is so basic is wrong. Plus I wouldn't know how to "legalize" levels.

LoadPlugin("E:\Denoise\ffms2.dll")
FFVideoSource("E:\Denoise\MVI_3375.mov")

With QTSource.dll I could open the MOV file but it is indeed clamped. Is there any way to fix that? Just for curiosity. I already can achieve what I originally intended but since I shot with DSRL cameras a lot I would like to know for future projects. Thanks again.

[poisondeathray]

If you are using the cplugin version of ffms2, you need to use LoadCPlugin(), not the regular LoadPlugin()

It's impossible to fix the clamping for QT decoder (that's how QT behaves) . You can expand the levels using levels() or color correcting software. If you don't like the clamping , use ffms2. There are other "negatives" to use QT for decoding , poor chroma upsampling when used in NLE's, added noise. Have you heard of 5DtoRGB? A lot of DSLR users use it for those reasons.

"Legalizing" the values of a full range (0-255) clip to Y' 16-235 in avisynth can be done by using levels or smoothlevels , but it sort of defeats the purpose of having full range if you do it "blindly" instead of carefully in grading software . If you wanted to do it, it would be :

levels(0,1,255,16,235, coring=false)

But this is essentially the same thing as the "clamping" that QT does. Many people prefer the data spread out 0-255, - When it's clamped it's more difficult to tease out and fine tune your grading . So they leave it "illegal" until you bring it into your color correcting or grading suite . Other people prefer it clamped because it's 1 less thing to do

You can use avisynth's histogram() which is really a waveform monitor to show Y' levels (its a waveform monitor turned on the side, areas in "brown" are "illegal" , that is 0-15 and 236-255)

[effes]

Thanks. Now I can open mov files in avisynth with both, ffms2.dll and qtsource.dll. (though with ffms2.dll the frames get messed up, but I can stick to qtsource.dll anyway). And 5DtoRGB sounds like the ultimate tool I was looking for the kind of work I do and for a no-compromise approach to quality. I'm investigating and testing already. Thank you!

[poisondeathray]

though with ffms2.dll the frames get messed up

What do you mean by this? Is it the levels issue ?

[effes]

What do you mean by this? Is it the levels issue ?

A mess
I don't know what it is. But mostly, it appears where there's fast movement.
Here's a sample:

[poisondeathray]

Looks like a decoding error. Might be an issue with the complied Cplugin that you're using - or did you try the vanilla build as well?

[effes]

I tried with ffms-2.17, ffms-2.17-cplugin, ffms-2.17-sdk and ffms-2.17-x64. Problem occurs while randomly seeking with FFVideoSource.

[poisondeathray]

I haven't seen this issue with ffms2

Is it reproducible? or does the error change each time and randomly? And linear playing doesn't reproduce the error ?

Do you have a small sample you can upload ?

[effes]

Here you go. I can open it, but if I scroll through the seeking bar or reproduce it, I get squares every here and there.
I put the basic script I used just in case in the rar too. I just tried with the one below too, same result.

LoadPlugin("ffms-2.17\ffms2.dll")
VID = FFVideoSource("MVI_3367.mov")
AUD = FFAudioSource("MVI_3367.mov")
AudioDub(VID,AUD)

[poisondeathray]

I had been using a different ffms2 version (than the 2.17 official one you're using ), but when I swapped out to that one I could finally reproduce the issue.

It appears to be a threading issue with that revision. If you use threads=1, the "seek pixellation" appears to go away

It doesn't seem to appear regardless of threads on r680 (I think r683 is on the beta download list , that should fix the threading issue)

EDIT: confirmed - threading appears to be ok on r683

[effes]

Yes, it is fine now. Adding threads=1 solves the issue.