Avisynth's "levels" function - 32-bit? Dithering?

[Asterra]

Once again I have a video project on the plate, and once again most of the footage (or maybe all of it) uses a luma range of 16-255, which means I need to find a way to preserve that detail before editing any of it.

As far as I can tell, the easiest method may be to use the following in Avisynth:

levels(16,1.0,255,16,235,coring=false)

But I am wary. Since the actual luma detail (before and after) is low, and since the adjustment is minor, I have a feeling that there's a pretty good chance of there being obvious contouring in the result. But I wouldn't expect there to be a problem if levels is doing something like this:

Converting to a higher-detail format, with dither.
Adjusting levels.
Converting back to the original level of detail, with dither.

Levels may or may not do this. The documentation doesn't say. But perhaps there is a way to force such a process?

[poisondeathray]

Y' in 16-255? How are you determining this?

Are you converting to RGB for editing? If so use PC matrix. This will preserve all the range 0-255

e.g.

ConvertToRGB(matrix="PC.709")

This will "map" Y' 0-255 => RGB 0-255 instead of Y' 16-235 => RGB 0-255 for "rec" matrices, where Y' <16 and >235 are clipped

Also, using smoothlevels (instead of levels), will dither and you will get less banding

Final Y'CbCr format should be dithered as well, and you need to set up encoder properly to keep the noise

[Asterra]

Y' in 16-255? How are you determining this?

I did a big video project a few years ago and this was one of the primary concerns I had. DV and HDV camcorders are known to have this feature-slash-problem. (My solution at the time involved a lot of juggling Avisynth, Vegas and Premiere Pro, in addition to handling everything as raw renders, and doing all the editing in After Effects. It wasn't terribly smooth.)

Are you converting to RGB for editing? If so use PC matrix. This will preserve all the range 0-255

Not planning on it this time. I may take the final edit and paste some AE stuff over it, eventually, but I'd really like to do the editing in Premiere Pro this time around.

Also, using smoothlevels (instead of levels), will dither and you will get less banding

That sounds more like it. Ideally, of course, it would convert to 24-bit or whatever, do the levels, and then convert back. It seems to me this would produce the best result. If smoothlevels doesn't do this, then I wonder if there is a series of Avisynth commands which will accomplish the same.

Final Y'CbCr format should be dithered as well, and you need to set up encoder properly to keep the noise

It's mostly DV and HDV. Inherently noisy. I'm going to do what I can with noise reduction because it frankly needs it. Final watchable format will be a 1080i60 Bluray, though I'll keep a UT raw because some of the footage was captured at true 1080p60 and frankly the only reason I'm settling for Bluray is because as far as standardized media players go, Bluray remains as good as it gets.

[poisondeathray]

Well if you use any color correction filters in premiere it will convert it to RGB . If you just to cuts type editing, native DV and HDV can still function in Y'CbCr timeline .

If you use avisynth or another intermediate format, 99% of the time premiere will treat it as RGB (and clip highlights). The only one it doesn't do that to is v210 (uncompressed 10-bit 422 Y'CbCr)

PP CS5 has "YUV" labelled filters that are applied before the RGB conversion, so you can salvage Y' >235 instead of clipping that data .

[Asterra]

If you use avisynth or another intermediate format, 99% of the time premiere will treat it as RGB (and clip highlights). The only one it doesn't do that to is v210 (uncompressed 10-bit 422 Y'CbCr)

Ah, this is probably the reason I had to drag out Vegas. Fortunately there are scopes which can be scrutinized to see if the whites have been destroyed, so I'm bound to find a workflow that suffices.

PP CS5 has "YUV" labelled filters that are applied before the RGB conversion, so you can salvage Y' >235 instead of clipping that data .

Don't I wish I had that. Although it seems like every new PPro also comes with new problems that have to be unearthed one by one. It's uncanny. Makes me feel unenthusiastic about upgrading.

[poisondeathray]

If you use avisynth or another intermediate format, 99% of the time premiere will treat it as RGB (and clip highlights). The only one it doesn't do that to is v210 (uncompressed 10-bit 422 Y'CbCr)

Ah, this is probably the reason I had to drag out Vegas. Fortunately there are scopes which can be scrutinized to see if the whites have been destroyed, so I'm bound to find a workflow that suffices.

Vegas clips a bunch of Y'CbCr formats too, like lagarith, ut video , huffyuv, etc... and treats them as RGB, even though they were in Y'CbCr mode

The one it allows full range pass through for is fourcc IYUV (other 8-bit uncompressed get clipped e.g. from vdub, which is YV12)

Native camera formats like HDV, DV are treated with "studio RGB" when you use 8-bit mode in vegas (ie. Y' 0-255 gets "mapped" to RGB 0-255 when an RGB conversion occurs, but the timeline still functions in Y'CbCr until you use a filter or cause an RGB conversion)

That's why you should work in RGB (and use avisynth to control the conversions), unless you are aware of the workarounds and what the behaviour is . Only if you do cuts type editing is the native format more beneficial (you can smart render DV and HDV in vegas for example) . The chroma upsampling of NLE's and AE are much lower quality as well, so when there is an RGB conversion upsample from YV12 (e.g. you're doing some color corrections or grading)., the color edges look really blocky and crappy

(Actually, Vegas 8 is slightly different than 9 and 10 - buth they all have bugs and quirks like Premiere , and slight differences between the versions)

[Asterra]

That's why you should work in RGB (and use avisynth to control the conversions), unless you are aware of the workarounds and what the behaviour is .

Alright. Let's say I decide to do the editing in Premiere Pro after importing as Avisynth RGB. How then do I export as YCbCr without allowing Premiere Pro (or Adobe anything) to do the conversion (Either as raw or UT, because DebugMode Frameserver works with every Premiere Pro except CS4), and also without predictable shenanigans like converting the full RGB range (0-255) to 16-235 which would squash the video's actual luma levels from 16-235 to something like 30-218? Simultaneously, how would I get Premiere Pro's preview window to show RGB's 235 as 255 and 16 as 0?

These are the items which made me want to stick with YCbCr but it's mainly just an issue of lack of knowledge.

[poisondeathray]

That's why you should work in RGB (and use avisynth to control the conversions), unless you are aware of the workarounds and what the behaviour is .

Alright. Let's say I decide to do the editing in Premiere Pro after importing as Avisynth RGB. How then do I export as YCbCr without allowing Premiere Pro (or Adobe anything) to do the conversion (Either as raw or UT, because DebugMode Frameserver works with every Premiere Pro except CS4), and also without predictable shenanigans like converting the full RGB range (0-255) to 16-235 which would squash the video's actual luma levels from 16-235 to something like 30-218? Simultaneously, how would I get Premiere Pro's preview window to show RGB's 235 as 255 and 16 as 0?

These are the items which made me want to stick with YCbCr but it's mainly just an issue of lack of knowledge.

RGB is RGB . The preview window, your monitor, TV's everything you see is RGB . If you import an RGB format, you get ...RGB and you see RGB displayed.

Even Y'CbCr video, is converted to RGB for display. (you're not viewing Y'CbCr directly)

That is where ALL the problems occur - because of Y'CbCr <=> RGB conversions and the differences in how they are converted. If you import a Y'CbCr video, you might get different treatment because of how the program handles those conversions. If everything was in RGB there would be zero problems like this

For example, Vegas does it differently with different formats (uses different conversions for different input formats. e.g. it might use studio RGB for some formats but computer RGB for others - so it's very confusing and inconsistent unless you know the little details)

But the problems with working in RGB are higher data rates (can be more sluggish to edit) .



If you're not doing any color work, then use the native format (HDV, DV) - it will edit like butter and you can smart render DV (but not HDV in premiere) . This is why I would use vegas for a mixed DV/HDV project. Smart rendering means only your transitions, effects , and cuts withing a GOP are re-encoded. Everything else is passed through with original quality untouched

If you were working in RGB, then DON'T let the program do the RGB=>Y'CbCr conversion. Always export RGB, then specify the conversion in avisynth. Better quality, more control.

Another option is to bring "illegal" ranges back to legal ranges in Y'CbCr before importing into NLE (e.g. using smoothlevels) and then use a Y'CbCr format; if everything is clearly within Y'16-235 and CbCr16-240, no clipping will occur in the NLE