DVD Luminance Value - How to do it right?

[SCDVD]

I don't want grave-dig a three and a half year old thread https://forum.videohelp.com/topic206484.html so I will just provide the link to it because it is related to my question. NTSC TVs in the US display a luminance range of Y=16 to 235. If values outside this range are input, they are hard clipped by the TV which results in washed out blacks and whites and other problems. It is my understanding that "most" DV Codecs convert the video to the 16 to 235 luma range when capturing from a DV camera. But "most" is a disturbing word. How can this be confirmed with a specific DV codec?

The second question is; is it possible to over correct this in the editor or encoder? It certainly wouldn't be good to compress the luma range twice - first when it is captured and then again in the editor or encoder. How can this be checked / avoided?

[jagabo]

It is my understanding that "most" DV Codecs convert the video to the 16 to 235 luma range when capturing from a DV camera. But "most" is a disturbing word. How can this be confirmed with a specific DV codec?

If you're capturing from a DV camcorder then the DV codec on the computer isn't doing anything but decoding the video so you can see it. What goes into the AVI file is an exact copy of the compressed DV stream from the camcorder. If your camcorder has bad black and white levels (and many do) you may need to correct them later though. There is at least one tool that can perform the correction without fully decompressing and recompressing the DV stream while capturing: Enosoft DV Processor.

If you're capturing from an analog capture card and compressing with a DV codec the capture card is probably capturing in the YUV colorspace so it should already have the correct luminance range. If your capture card is capturing with the wrong luminance you should adjust it's proc amp settings (brightness and contrast).

The second question is; is it possible to over correct this in the editor or encoder? It certainly wouldn't be good to compress the luma range twice - first when it is captured and then again in the editor or encoder. How can this be checked / avoided?

Yes, you can fix this in most editors with the brightness and contrast controls, though it's best to capture with the right levels in the first place. I know Vegas has tools that can display the luminance levels:

https://forum.videohelp.com/topic332734.html#1726325

I use VirtualDubMod with an AVISynth template to open Video files and convert to RGB without performing the usual luminance stretch.

Put the following script into a file called DirectShowSourceRetainLuma.avst and place it in VirtualDubMod's template folder.

#ASYNTHER DirectShowSource Retain Luma for Standard Definintion
DirectShowSource("%f")
ConvertToRGB(matrix="PC.601")

After opening with that template you can use VirtualDubMod's Levels filter to display a historgram of the luminance levels. You can also use AVISource() instead of DirectShowSource() in the template. You can see some post I made using this method in the above link.

Another thing to watch out for is who is doing the conversion from YUV to RGB. I prefer to use DV decoders that decode to YUY2 so I have control over the YUV to RGB conversion. Some DV decoders automatically convert to RGB and perform the standard luminance stretch so you lose blacker-than-black and whiter-than-white right off the bat.

[SCDVD]

jagabo, Thank you for your response. This seems to be a topic that is very conflicted. Until a week ago, I would have said pretty much what you said if asked the same question. But I have been doing some research on this and several very "authoritative" sources indicate that "most" DV codecs adjust the luma range to Y=16 to 235 when doing a firewire capture from a DV camera. I'm just trying to sort out this conflictive information.

[edDV]

jagabo, Thank you for your response. This seems to be a topic that is very conflicted. Until a week ago, I would have said pretty much what you said if asked the same question. But I have been doing some research on this and several very "authoritative" sources indicate that "most" DV codecs adjust the luma range to Y=16 to 235 when doing a firewire capture from a DV camera. I'm just trying to sort out this conflictive information.

All ITU Rec-601 based YCbCr video (e.g. DV,DVD,DVB,ATSC DTV, etc.) are supposed to pass 0-255 8bit video with black defined as level 16 and white at 235 but the overshoot areas 0-16 and 236-255 are supposed to be passed as well to handle transient overshoots and levels errors without hard clipping. A properly adjusted home TV will not display the 0-16 levels but should display 236-255.

All this is tightly regulated in the broadcast TV world where hundreds of clips from various sources must be run back to back without obvious levels shifts.

Commercial DVDs also follow these 16-235 levels with high quality control and do pass the overshoot regions. Pro and Prosumer camcorders also tighly control black to 16 and average white to 235. Consumer camcorders however often stretch whites up into the 236-255 region in an effort to "cheat" low light performance and signal to noise.

Computer video has traditionally operated in 0-255 RGB space with no overshoot provision. Conversion from 0-255 computer color space to ITU-Rec-601 YCbCr is a tricky business and is often mishandled. The quick and dirty method maps 0,0,0, RGB black to luminance level 16 and and 255,255,255 RGB white to luminance level 235. This generally works for converting graphics from computers to video space but incorrect handling of the overshoot regions causes problems for computer video captures.

Consumer camcorder material in particular gets mishandled when converted to RGB computer space. See this thread.
https://forum.videohelp.com/topic329440.html

The Prosumer computer editing packages (e.g. Premiere, Vegas, FCP, Media Studio Pro, AVID) generally follow Rec-601 levels properly. Their consumer versions generally pass the overshoot on a DV timeline but often have errors in graphics overlay modes.

The big issue is the ongoing practice of consumer camcorder manufacturers to put real video up in the 236-255 overshoot region. I recently bought a Canon HV20 HDV camcorder and was surprised to see the video extending from 16-255 consumer style and with hard clipping at 255.

Canon HV20 in full auto exposure mode.


Here's another HV20 example in auto mode.

[jagabo]

Computer video has traditionally operated in 0-255 RGB space with no overshoot provision. Conversion from 0-255 computer color space to ITU-Rec-601 YCbCr is a tricky business and is often mishandled. The quick and dirty method maps 0,0,0, RGB black to luminance level 16 and and 255,255,255 RGB white to luminance level 235.

I wouldn't call it tricky or quick and dirty. It's simply a matter of what matrix you use for the conversion:

http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html#RTFToC30
http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html#RTFToC31

The equations for the conversion are the same except for the values in the matrices. Every Windows program I know expands YUV luminance from 16-235 to RGB 0-255. I don't use Vegas but from posts you've made I suspect it might be an exception.

The real issue is to avoid YUV to RGB conversion if it all possible.

[SCDVD]

edDV, What software are you using in the screen shots?

[jagabo]

By the way, the Cedocida DV codec lets you specify the output colorspace (YUY2, YV12, RGB) and whether or not the contrast expansion takes place during the YUV to RGB conversion.

[edDV]

edDV, What software are you using in the screen shots?

The video software is Sony Vegas 6.

The screencap software is Gadwin Printscreen (free version)
http://www.gadwin.com/

[edDV]

Computer video has traditionally operated in 0-255 RGB space with no overshoot provision. Conversion from 0-255 computer color space to ITU-Rec-601 YCbCr is a tricky business and is often mishandled. The quick and dirty method maps 0,0,0, RGB black to luminance level 16 and and 255,255,255 RGB white to luminance level 235.

I wouldn't call it tricky or quick and dirty. It's simply a matter of what matrix you use for the conversion:

http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html#RTFToC30
http://www.poynton.com/notes/colour_and_gamma/ColorFAQ.html#RTFToC31

The equations for the conversion are the same except for the values in the matrices. Every Windows program I know expands YUV luminance from 16-235 to RGB 0-255. I don't use Vegas but from posts you've made I suspect it might be an exception.

The real issue is to avoid YUV to RGB conversion if it all possible.

I meant tricky from a user perspective. My attitude now is all hardware and software are guilty of levels errors until I can verify the process myself with tests. The waveform monitor in Vegas is my favorite measurement tool.

[SCDVD]

I understand another problem is inserting RGB images can cause a squashing effect on the video luma value because the RGB image has the 0 to 255 luma value.

[David K]

Jagabo,

I have followed your postings with interest and I successfully installed your template in VirtualDubMod.

My interest in this arises out of the well-known IRE problem in using a digital video camera to capture analog input like a VHS tape. US/Canada NTSC source video has a 7.5 IRE as the black level standard (a very dark gray). When doing a digital pass-through the DV capture has the brightness shifted up.

I’ve been trying to correct this with the VirtualDub levels filters, by setting the “input” slider to about 18. That seems to give me true blacks rather than dark grays.

My questions

1. Should I also move the “output” slider down, on the assumption that the IRE shift means that the highest whites need adjusting too? (Or will that be making the image too dark?)

2. Is there any advantage to using your DirectShowSourceRetainLuma.avst template in my situation?

Thanks

[jagabo]

When doing a digital pass-through the DV capture has the brightness shifted up.

I’ve been trying to correct this with the VirtualDub levels filters, by setting the “input” slider to about 18. That seems to give me true blacks rather than dark grays.

My questions

1. Should I also move the “output” slider down, on the assumption that the IRE shift means that the highest whites need adjusting too? (Or will that be making the image too dark?)

How you treat this will depend on what you send VirtualDubMod's output to. Most encoders will convert VirtualDubMod's RGB output back to YUV with the usual contrast compression. If you're using a codec that does that you want your blacks to be RGB 0 and whites to be RGB 255. Some encoders (Cedocida, TMPGEnc Plus, and CCE, for example) allow you to overide the contrast compression. When using one of those you want the blacks to be RGB 16 and the whites to be RGB 235 (a little undershoot and overshoot is OK).

2. Is there any advantage to using your DirectShowSourceRetainLuma.avst template in my situation?

Have a look at the DV image I posted here:

https://forum.videohelp.com/topic332734.html#1726415

You can see that the AVST script will allow you to recover some of the hilites that would otherwise be lost in the conversion to RGB. Or you could use Cedocida and speficy that Cedocida do the YUV to RGB conversion and that the contrast expansion should not occur. If your DV footage doesn't have the overbrite white problem, and only has the IRE 7.5 problem, you don't need to use the AVST script. You can just lower the black level.

[David K]

Jagabo,

So if I understand correctly, when I use my Digicam to make a digital transfer from a VHS tape, the following is happening:

1. NTSC signal coming from the VHS deck is converted by the digicam to a digital DV stream which is captured on the computer. The resulting DV avi file is in the YUV colorspace.

2. Because of the IRE problem, the digicam has mapped the darkest black in the YUV digital colorspace to IRE 7.5 not IRE 0. Consequently all the darkest blacks in the DV file are really dark greys and the brightness of the whole image is shifted up so it has a washed-out look. Question: Are the brightest whites also shifted up so that detail is lost because of the IRE problem? Or to put it another way, are the whites shifted up so that they are now above level 235.

3. Normally when opening a digital file in Virtual Dub, it converts it to RGB but only shows the 16-235 range (the RGB “stretch”) of the file. Anything below 16 and above 235 is cut off. Your template does the RGB conversion but includes the whole range from 0-255, so that one can have access to the detail in the levels above 235. To repeat question 2 in a different form: will the IRE capture problem ever result in above 235 levels, or is that a problem that normally only arises when shooting footage with consumer digicams. If I’m converting a commercial VHS tape will the analogue brightness levels be within such a norm that the IRE capture problem will not lift the white levels above 235 during the digital conversion despite the IRE problem?

[jagabo]

1. NTSC signal coming from the VHS deck is converted by the digicam to a digital DV stream which is captured on the computer. The resulting DV avi file is in the YUV colorspace.

Yes.

2. Because of the IRE problem, the digicam has mapped the darkest black in the YUV digital colorspace to IRE 7.5 not IRE 0. Consequently all the darkest blacks in the DV file are really dark greys and the brightness of the whole image is shifted up so it has a washed-out look.

Yes.

Question: Are the brightest whites also shifted up so that detail is lost because of the IRE problem? Or to put it another way, are the whites shifted up so that they are now above level 235.

I believe it can vary from camcorder to camcorder. Mine do have whites over IRE 100. I know many others do, including some Prosumer level camcorders.

3. Normally when opening a digital file in Virtual Dub, it converts it to RGB but only shows the 16-235 range (the RGB “stretch”) of the file. Anything below 16 and above 235 is cut off.

Yes.

Your template does the RGB conversion but includes the whole range from 0-255, so that one can have access to the detail in the levels above 235.

Yes.

To repeat question 2 in a different form: will the IRE capture problem ever result in above 235 levels, or is that a problem that normally only arises when shooting footage with consumer digicams. If I’m converting a commercial VHS tape will the analogue brightness levels be within such a norm that the IRE capture problem will not lift the white levels above 235 during the digital conversion despite the IRE problem?

Video is allowed to have a little overshoot above IRE 100 but it should not be large areas and not be routine. Analog TVs may still show details above IRE 100. That is, a spot of IRE 105 may be visible within an area of IRE 100.

[Wile_E]

The big issue is the ongoing practice of consumer camcorder manufacturers to put real video up in the 236-255 overshoot region. I recently bought a Canon HV20 HDV camcorder and was surprised to see the video extending from 16-255 consumer style and with hard clipping at 255.

So what is the consensus? Do you leave the range at 16-255 and encode to DVD?

I have some HI8 tapes that have obvious overshoot and hard clipping. Capturing using Canopus ADVC-300, I have tried manually limiting the Auto Gain, to limit the range up to 235 (IRE 100). But this makes the video much darker than the original analog video. The ADVC software has options for White Step, and White Peak Adjust. I'm not sure if I should be using that instead of just turning down Auto Gain.

Why should I compress down the luminance to 235, if the output will look much darker than the original video? Will Digital TV's clip above 235?

[Wile_E]

So values above digital 100 are "okay" for consumer video? The tapes I have appear to have used the full 0-255 range, instead of 16-235. Placing the Broadcast Colors on the timeline clamps down everything, washing out blacks. Apparently because of the black border that was captured from the analog video?

[SCDVD]

Apparently because of the black border that was captured from the analog video?

A zero black will have this effect. Inserting images can also cause the color range to get squeezed unless they have been precompensated.

[edDV]

I'm rebuilding my machines and don't have access to my sample files but in short, consumer digital camcorders work over the full 16-255 range and many DV to RGB conversions chop luminance at 235 (i.e. turn sky, clouds or highlights to clipped white).

I've found the higher end prosumer camcorders manange the majority of video luminance to 235 whites with only highlight excursions above. This is also typical of most broadcast video. The Sony PD-150/170, VX2100 and FX-1 are particularly good with levels.

Consumer camcorders including the Canon HV-20 use the 235-255 levels and can clip at 255 as shown in the pictures above.

When I export from Vegas or Premiere to external filters I usually do it as uncompressed YCbCr/RGB to avoid the DV to RGB conversion issues. The export can be scaled 16-235, 16-255 or 0-255 depending on the needs of the filter or external encoder.

[Wile_E]

The Hi8 Camcorder I'm using is purely analog output (s-video). So I suppose in my case, I should leave the ADVC auto-gain on, and let it use the full 16-255 range, and encode this to DVD. Because if I manually adjust the Gain on the ADVC to limit the capture between 16-235, the video appears noticeably darker.

The Mainconcept encoder in Sony Vegas appears to keep the full range and not clip to 235. In the end, I just want the DVD output to look close to the original analog tapes.

I hope I'm doing these Hi8 transfers correctly. Will most DVD players and newer TV's display the super-black and super-white values?

[edDV]

The Hi8 Camcorder I'm using is purely analog output (s-video). So I suppose in my case, I should leave the ADVC auto-gain on, and let it use the full 16-255 range, and encode this to DVD. Because if I manually adjust the Gain on the ADVC to limit the capture between 16-235, the video appears noticeably darker.

The Mainconcept encoder in Sony Vegas appears to keep the full range and not clip to 235. In the end, I just want the DVD output to look close to the original analog tapes.

I hope I'm doing these Hi8 transfers correctly. Will most DVD players and newer TV's display the super-black and super-white values?

The ADVC-100/110/300 captures Hi8 perfectly 16-235 (0-100 IRE) with 236-255 overshoots when the 7.5IRE switch is set. The DV file will be fine used in Vegas, Premiere, ULead, Windows Movie maker, etc.

A DV codec like the Panasonic DV to RGB codec in Virtualdub will saw off the peak whites mapping 7.5 IRE to zero and 100 IRE to 255. That results in sawing off all the bright video above analog 100 IRE (DV 235).

[Wile_E]

If I use the built-in Mainconcept encoder in Vegas to encode to MPEG2 for DVD, will the overshoot still be okay for DVD? Will DVD players and TV's clip the overshoot?

Some articles I've seen, said to keep the levels between 16-235 when capturing DV and wanting to output to DVD. Others say overshoot is fine. So is there any technical guide for DV to DVD transfer? What do the professionals do about overshoot?

[jagabo]

I just want the DVD output to look close to the original analog tapes.

That doesn't necessarily mean you want proper 16-235 levels. You're source may be off.

I think you already know this but how the captured video looks on the computer may be very different than how it looks on TV. Burn to DVD then compare the original Hi8 to the DVD on your TV.

[Wile_E]

Okay I compared again, and it looks like using auto-gain on the ADVC gets it pretty close to the source. Manually setting the white peak to 235 looks different.

I just figured that even with consumer video, most people adjust their incoming signal to capture at 16-235.

[edDV]

If I use the built-in Mainconcept encoder in Vegas to encode to MPEG2 for DVD, will the overshoot still be okay for DVD? Will DVD players and TV's clip the overshoot?

Some articles I've seen, said to keep the levels between 16-235 when capturing DV and wanting to output to DVD. Others say overshoot is fine. So is there any technical guide for DV to DVD transfer? What do the professionals do about overshoot?

DV format and the DVD standard (also ATSC and DVB) have identical levels. They are YCbCr format and obey similar standards. The problems arise from conversion to computer RGB or tight telecom transmission requirements that limit gray scale to 16-235 (219 levels )

If you stay within ITU-Rec 601 levels, overshoots are preserved in the 0-15 and 236-255 levels. If you must stay within 16-235, processing is required.