Help with video Black Levels

[vhelp]

hay, that would be great, fulci.. I would appreciate that. A few seconds
would be fine. Although I've never tried, what about one of those free
sites, like "yousendit.com" or something, if its not too much trouble.
(I leave that up to you to decide)

.
.

Anyway. I went ahead, and proceeded to process whatever you left from
your last post, the PAL original pic. And, I encoded that one with the
folowing setup, where I only resized the PAL pic to ntsc 720x480 pixels:

** PAL -> pal->ntsc.720x480 -> TMPGenc

Then..

** MPEG-2 -> vdubMOD[copy/paste] -> iStitch[bitmap]

warning.. Please excuse the clumsyness of this tools text.
Also, feel free to measure/analize this pic as you all see it.

ORIGINAL pal vs. TMPGenc processed


-vhelp 3869

[FulciLives]

How come your TMPGEnc Plus side looks like mine i.e., the masking? Did you do that on purpose to match mine?

As I understand it the original PAL is on the LEFT and your TMPGEnc Plus output is on the right. Well I have to say it looks good to me ... perfect match!

Did you checkmark that TMPGEnc option called, "Outout YUV data as Basic YCbCr not CCIR601" as I would imagine that the output is affected one way or the other.

Anyways I made a clip by loading the original PAL VOB file directly into Womble MPEG-VCR ... cut out a short clip ... then saved it. This should be 100% the same as working from the original VOB files otherthan MPEG-VCR might have done some re-encoding of the first few frames at the start and the last few frames at the end since I can't tell with that program if I am cutting on "I" frames or not ... it doesn't seem to tell you as far as I can tell. Anyways the clip is long enough that you can easily cut a couple of seconds from the start and end if you want. I tried to use VobEdit but never used it and couldn't make head or tails of it.

Anyways I am trying to upload the file now to one of those free file hosting websites but I tried one I never used before and it doesn't seem to be working ... no ftp ... only upload through a browser ... so I think the file is too large and it times out. So I might have to RAR the file into a few smaller files.

As soon as I get something uploaded I will let you know.

Try not to pass out but the file is about 29MB in size LOL

- John "FulciLives" Coleman

[edDV]

PAL image (correction this is the NTSC converted image)


NTSC converted Image (correction this is the PAL original image)

[FulciLives]

@edDV

Are you sure you didn't mix up the PAL and NTSC images?

Also some actual comment would be appreciated as I'm not sure what to make of what you posted on my own LOL

- John "FulciLives" Coleman

P.S.
I am uploading the PAL MPEG clip now and will report a working link once done.

[edDV]

@edDV

Are you sure you didn't mix up the PAL and NTSC images?

Also some actual comment would be appreciated as I'm not sure what to make of what you posted on my own LOL

- John "FulciLives" Coleman

P.S.
I am uploading the PAL MPEG clip now and will report a working link once done.

OOPS yes, they are reversed but the issues were the same in both.

The black levels are good per DVD spec (i.e. ~level 16) but the whites are too hot and are clipping at 255. The yellow area shows where the clipping is occuring.

I'll amend more comment in a minute.

Added:

The DVD authoring and player levels issue is poorly covered on the net.

Since I was around back when the CCIR-601 was being standardized, I can add a bit to the 16-235 issue. It was an analog world then and CCIR-601 devices were designed to plug directly into the analog stream without degrading the video. The D1 recorder advantage was wide bandwidth (detail) with low noise.

The 0-16 area was necessary to pass noisy blacks without clipping 236-255 is there to give headroom for transients and hot local gains (e.g. spotlights or sky).

Clipping at black or white is a bad thing for analog. Clipping creates analog harmonics which fall outside the video bandwidth. This causes analog circuits to ring or otherwise go unstable. For those here, VHS blacks are extremely noisy.

The white end of the scale has other issues. Analog TV sets have a limited range of dark to light (but still much more than plasma or LCD). When a CRT is new, it has a wider range of black to white. At the limit, max white is defined as the point where the picture begins to bloom. You adjust the TV white (aka "contrast" or "picture") until you see the picture bloom, then you back off a bit. As the tube ages, the white capability deminishes so the contrast adjustment will bloom at a lower brightness point.

Analog broadcasters are used to a -20 to 120 IRE levels scale. The idea is to adjust the cameras to put the bulk of white at or below 100IRE. Transients or small bright spots will be aborbed into the 100-120IRE overshoot range. When the user TV set is properly adjusted, any sustained white above 100 IRE will cause the picture to bloom. It is a no no to put too much energy up there. TV set high voltage circuits can only absorb and display small amounts of hot video. Broadcasters also need to worry about their transmitters exploding .

So the 236-255 range is built into the DVD standard and TV sets want to see white at level 235 which gets converted to 100 IRE analog. The 236-255 overshoot range for digital video corresponds to 101-108 analog IRE which is a much narrower range than the traditional 101-120 IRE.

[vhelp]

Did you checkmark that TMPGEnc option called, "Outout YUV data as Basic YCbCr not CCIR601" as I would imagine that the output is affected one way or the other.

This box should almost never be touched in TMPGenc, except in rare cases will
you feel it might help make your source closely match the original sources.
(I sometimes experiement with this, when I'm runing various testing scenarios)

How come your TMPGEnc Plus side looks like mine i.e., the masking? Did you do that on purpose to match mine?

Course not.. I'm surprised at you. Actually, here are a few processed
similarily:



I had since, deleted most of my many other examples of this (from various
topics requiring pics) I now regret deleting them, but I had to make room for
my small storage space.

Try not to pass out but the file is about 29MB in size L

You know.., that all right, during this testing phases, as there is something
to gain by all this.


@ edDV

The black levels are good per DVD spec (i.e. ~level 16) but the whites are too hot and are clipping at 255. The yellow area shows where the clipping is occuring.

This may be so.. true. But, the cause could be on the original source.
(we are only re-duplicating this issues, yes, even in proper processing)
NOT the encoding that fulci or I did. I think that this is something you
and others might need to consider all this. Because it seems that in most
of the pics that everyone here posts, (from time to time) and you measure
them, they all seem to be bad. I find this piculair. Seems like every
MPEG encoder is wrong. And, I don't believe this. (the users that operate
these tools may be doing so, incorrectly or the source is non-standard or
your method of measure is bad or Vegas is bad, or I don' know what) But, I
think that it is the fault of the original source. And, in your measuring/
analizing (yes, thank you very much) you are bringing all this to us to see.

On the other hand, could Vegas be at fault ? That, I wonder from time
to time.

You see, when I process video (encode to MPEG) I do so, in a manor that will
match perfectly, the original source. I dont' tamper with filtering, nor
any coloring, etc. (fwiw, I am working on a project to assist color correction)
when inside TMPGenc, I make sure that everything equates to a proper duplication
of the original source, whereever it may have originated, ie.. Cable; VHS;
Antenna; DVD Rip; etc. And, when I analize and compare them together, they
are always the same in color space or transparency.. as in the pics I just
demonstrated above (execpt for the one right above this one)

- As an excercise, why don't we all get together, and agree on a source that
... we can all grab hold of, process it to MPEG and then compare them all to
... some degree of measurement/analizing.

Then, we can make basic receipe on certain video processing steps to apply
on the video for testing.

--> "The Matrix" is a very common one that everyone seems to have..

And, if you find they all each have something wrong with them, then
either:

A - the source originate that way, and nothing is wrong after all, or
B - either, your method of analizing, or Vegas, or something else is flawed.

I'm gonna stick with my belief that A as the correct reason, because a lot
of what you post here seem correct to me

-vhelp 3870

[gshelley61]

@edDV

Are you sure you didn't mix up the PAL and NTSC images?

Also some actual comment would be appreciated as I'm not sure what to make of what you posted on my own LOL

- John "FulciLives" Coleman

P.S.
I am uploading the PAL MPEG clip now and will report a working link once done.

OOPS yes, they are reversed but the issues were the same in both.

The black levels are good per DVD spec (i.e. ~level 16) but the whites are too hot and are clipping at 255. The yellow area shows where the clipping is occuring.

I'll amend more comment in a minute.

edDV -

I just noticed something interesting that may explain why Fulci's posted still frames from the PAL DVD (and the NTSC conversion) have their whites too hot.

Check this out... here is the vectorscope, waveform monitor, and histogram for the THX multipurpose test pattern when the actual DVD MPEG2 video (VOB) is loaded into Vegas:


It charts totally normal, as you would expect.

But watch this - after saving a still frame from the VOB file to JPG using VirtualDubMod and IrfanView, the JPG picture file has re-mapped video levels from 0-255 RGB. Here's how the JPG still frame grab charts out in Vegas:



Here's the THX pattern JPG for reference:



This is something I did not realize was happening when saving MPEG2 video frames to JPG. So, either VirtualDubMod or IrfanView is re-mapping the MPEG2 file digital video levels to optimize viewing on the computer?

[FulciLives]

I got the clip uploaded to one of those free file hosting websites.

Now I had to split the file into 7 parts using WinRAR so that I could upload it. The free website I am using would not accept a single 29.2MB file.

Please note that the website RENAMED the files by adding a bunch of "crap" at the start of each file. So once you download all 7 parts you need to rename them by deleting the stuff that was added.

For instance when I downloaded the first file it was renamed like this:

[savefile]060401153711_PHANTASM3.part1.rar

So you need to edit the file name so it is simply ---> PHANTASM3.part1.rar

Do that to each file so that you have the following:

PHANTASM3.part1.rar
PHANTASM3.part2.rar
PHANTASM3.part3.rar
PHANTASM3.part4.rar
PHANTASM3.part5.rar
PHANTASM3.part6.rar
PHANTASM3.part7.rar

You can now go ahead and unrare the files. You will get a FOLDER called PHANTASM3 and inside that folder is the file P3_PAL_TEST_CLIP.mpg

I downloaded all the files and tried it myself so I know it works as long as you change the file names as I indicated.

Here is the website link:

http://www.savefile.com/projects.php?pid=156814

Again this is a MPEG file created by loading the original PAL DVD VOB file directly into MPEG-VCR where I cut out a small clip and then saved it.

- John "FulciLives" Coleman

*** EDIT ***
I just wanted to add that frame 312 in this MPEG clip is the frame that I captured before in my other screen grabs. This way we can all capture the exact same frame if need be.

[FulciLives]

This is something I did not realize was happening when saving MPEG2 video frames to JPG. So, either VirtualDubMod or IrfanView is re-mapping the MPEG2 file digital video levels to optimize viewing on the computer?

Great.

Just what we need

How the hell are we supposed to do screen shots then if we can't use the method we have always used i.e., VirtualDubMod.

- John "FulciLives" Coleman

[edDV]

OK I'm back. See the additions above.

[vhelp]

more Confusion added, yes.

Well, he was refering to what I believe was *his* method of producing the
.jpg for posting here for analizing. So it's probably his .jpg he's blaiming.

(gshelley69, we've been telling over and over, "stop using .jpg's, else
they risk taint )

I usually post .PNG or .BMP files because both these formats are lossless.
But, if we are really serious in our studying and researching, then maybe
we should re-consider .BMP files for ultimate accuracy.

@ gshelley69, could you redo that demo above, but this time, store it in
.BMP format. Don't worry about the size. Just get it right, the first time.

Then, post the results again - thanks

-vhelp 3871

[vhelp]

Thanks fulci, for your latest upload. (making somall headway, via dial-up, hehe)

-vhelp 3872

[vhelp]

(sorry to waist bandwidth)

@ edDV, what are the changes. Looks the same to me

-vhelp 3873

[edDV]

I just noticed something interesting that may explain why Fulci's posted still frames from the PAL DVD (and the NTSC conversion) have their whites too hot.

Check this out... here is the vectorscope, waveform monitor, and histogram for the THX multipurpose test pattern when the actual DVD MPEG2 video (VOB) is loaded into Vegas:
(see image above)


It charts totally normal, as you would expect.

But watch this - after saving a still frame from the VOB file to JPG using VirtualDubMod and IrfanView, the JPG picture file has re-mapped video levels from 0-255 RGB. Here's how the JPG still frame grab charts out in Vegas:
(see image above)

Here's the THX pattern JPG for reference:
(see image above)

This is something I did not realize was happening when saving MPEG2 video frames to JPG. So, either VirtualDubMod or IrfanView is re-mapping the MPEG2 file digital video levels to optimize viewing on the computer?[/quote]

I'm conviced that Vegas is a calibrated environment. I always save JPGs out of Vegas. Vitualdubmod does work in 0-255 RGB space. The corrections need to be made while MPeg2 encoding to get from 0-255 to 16-235 standard levels. There is often and encoder setting called something like "Input video is RGB".

To complicate things, there are two kinds of YUV to RGB conversion. Products like old Premiere and Vegas are native RGB, but use 16-235 scaling to maintain the over and undershoots. Programs that place black at 0 and white at 255 will clip the overshoot areas of captured video.

I hope to test this further.

[edDV]

@ edDV

The black levels are good per DVD spec (i.e. ~level 16) but the whites are too hot and are clipping at 255. The yellow area shows where the clipping is occuring.

This may be so.. true. But, the cause could be on the original source.
(we are only re-duplicating this issues, yes, even in proper processing)
NOT the encoding that fulci or I did. I think that this is something you
and others might need to consider all this. Because it seems that in most
of the pics that everyone here posts, (from time to time) and you measure
them, they all seem to be bad. I find this piculair. Seems like every
MPEG encoder is wrong. And, I don't believe this. (the users that operate
these tools may be doing so, incorrectly or the source is non-standard or
your method of measure is bad or Vegas is bad, or I don' know what) But, I
think that it is the fault of the original source. And, in your measuring/
analizing (yes, thank you very much) you are bringing all this to us to see.

On the other hand, could Vegas be at fault ? That, I wonder from time
to time.

I agree we need a standard set of images from known sources. For starters there are the following candidates.

THX test patten
SMPTE DV color bar from Vegas (I can post this)
Bell-Nuit Test pattern http://www.belle-nuit.com/testchart.html **

Then we can select representative well balanced images (DVD sourced).

** I just evaluated the Belle-Nuit NTSC and NTSC-DV test patterns to see what was different. All the levels are the same 16-235 reference. The difference is number of scan lines: 720x486 (true NTSC) vs. 720x480 (ITU Rec-601)

[vhelp]

So, in all this latest discoveries, if we can deteremine without a doubt
that vdubMOD should not be sued (under certain analizing conditions)* and
should we consider looking for an alterntative method of obtaining the
video frame from a given MPEG VOB commercial dvd source ??

* note, this does not mean that we have to give up vdubMOD all-together.
.. Just under certain analizing scenarios, we might not want to use it.
.. (but we probably benefit under which of those scenarios)

However, I'm still not convinced that we need to go this way. After all,
my eyes do not deceive me when I compare two clips processed. Thus, there
must be a missing explaination somewheres

But, something else to consider here, (for testing purposes) and that is,
determining exactly what exact format in stored on a given commerical
DVD. I mean..

** We have MPEG-2 in YUV color space
** The YUV is in the form YCbCr
** Where the Y: 16-235, Cb: 16-240, Cr: 16-240
** and, the YUV (YCbCr) is sub-sampled to 4:2:0 and in YV12 (packed) format.

If the frames are in this format, (compressed of course, w/ DCT etc) we
would then need to obtain (decode) the compressed frame, but stop there.
Keep it in the YUV (YV12) format. Then, run it through some analizing
software, such as Vegas

This to me, would be the ultimate method of obtaining a non-tainted frame.
I think that AVIsynth could do this part, but I am not clear on the exact
steps to use with it at this time. I could research it though.

-vhelp 3874

[edDV]

No need to jump to conclusions. Let's test the assumptions.

I need to go do things but will be back later this evening.

[vhelp]

Lucky you, I also have to break way from this discussion to do a
few errands and things.. hair-cut, dinner (spagetti w/ Prego meat sauce)

See you all later..

-vhelp 3875

[BrainStorm69]

OK, I'm going to need some help here to interpret this correctly. I downloaded the trial of Vegas, so I have the scopes. I have a Datavideo TBC3000 that will generate colorbars. I don't know how accurate the colorbars are, but I decided to try a test with them and my Avermedia DVD EZMaker PCI capture card. I used Virtual VCR to capture three short AVI clips using HuffyUV set to capture YUY2.

In Virtual VCR, with my DVD EZMaker, there is a digital proc amp function I can use to alter the capture. I did one capture with everything at default. I did one capture with the brightness turned way up. And one capture with the brightness turned way down.

Then I did the scope on each capture in Vegas with the setting for 7.5 IRE checked, and the setting for Studio RGB unchecked. Screen caps of the Vegas scopes for each follow. What I don't understand is that I don't get over 100 IRE in the overbright cap, nor do I get under 7.5 IRE in the underbright cap. How can this be???

I was reading some posts over at Doom9 a few days ago about YUY2 needing to be converted to RGB for some applications, and that depending on how Windows is configured and what .dll is doing the converting, the whites and blacks may get clipped. Is that what is happening here?

Help understanding this will be appreciated!


TBC3000 Colorbars at default proc amp settings in Virtual VCR


TBC3000 Colorbars at overbright proc amp settings in Virtual VCR


TBC3000 Colorbars at underbright proc amp settings in Virtual VCR

[jagabo]

I've verified that VirtualDubMod stretches the luminance range from 16-235 to 0-255 when it converts to RGB.

I started with a short MPEG sequence ripped off a commercial DVD. The frame I chose had white text on a black background.

Saving a BMP image from VirtualDubMod gave an image with the black background at RGB 0,0,0, and white text at ~255,255,255.

I then opened the MPEG file with Graphedit. The output from the MPEG decoder pin was YV12. I piped that to the dump filter to get a raw byte stream. Examining the file I could see that the black background had a luma of 16, and the white text around 235.

VirtualDubMod export:


Graphedit export (YV12 viewed as 8 bit greyscale):

[FulciLives]

OK I got Sony Vegas 6.0a

I got the PAL video frame from the MPEG file (loaded the MPEG directly into Vegas).

I got the Videoscope up but what mode do I want?

You know the checkmark for the 7.5 IRE Setup and for the Studio RGB (16 to 235)

Which get checkmarked and/or not checkmarked?

I want to do the same frame from the original PAL and compare to the NTSC I made from it.

- John "FulciLives" Coleman

[gshelley61]

Leave 7.5 IRE Setup unchecked, but do check Studio RGB

[FulciLives]

Well I took the MPEG file I cut from the PAL DVD (download link above) and loaded it into Sony Vegas 6.0a and here is the Vectorscope/Waveform but I didn't know what settings to use so I used all the combinations that there are:

7.5 IRE Setup = CHECKMARKED
Studio RGB (16-235) = CHECKMARKED


7.5 IRE Setup = NOT CHECKMARKED
Studio RGB (16-235) = NOT CHECKMARKED


7.5 IRE Setup = CHECKMARKED
Studio RGB (16-235) = NOT CHECKMARKED


7.5 IRE Setup = NOT CHECKMARKED
Studio RGB (16-235) = CHECKMARKED


Again this is from the PAL MPEG. I might try and post a MPEG clip from the NTSC DVD but I'll need to hunt down more free website space. I guess I could just do with the NTSC DVD what I did here for the PAL DVD but not until I am sure which of the four ways is the "correct" way.

- John "FulciLives" Coleman

P.S.
I don't know if this makes a difference but when I imported the PAL MPEG into Vegas I used the PAL DV widescreen template setting.

*** EDIT ***
I just had to make another change but I am done editing this post now LOL

[FulciLives]

Leave 7.5 IRE Setup unchecked, but do check Studio RGB

Figures I saw this only after I made my post above where I did all the combinations LOL

Picture 4 above (the last picture) is the one that matches this description.

- John "FulciLives" Coleman

[edDV]

Agree: 7.5 unchecked, Studio RGB 16-235 checked.
PAL should be identical levels but different project setting (PAL DV 720x576, 25.0fps)

Here is a SMPTE color bar (DV scaled 16-235) from Vegas for those who need one.




Here is my reference path:

NTSC_7.5IRE->ADVC(sw2 on)->DV-AVI->Vegas(DV-NTSC)->Mainconcept-SONY MPeg2->DVD

If I drag the MPeg2 or VOB back to the timeline, all levels match. Same if I record to a DV camcorder and transfer back.

Next, I'm working on VirtualDubMod.

[FulciLives]

OK I created a NTSC MPEG from the 16x9 WS NTSC DVD that I made from the 16x9 WS PAL DVD. So the PAL MPEG can be downloaded (see above) and now also can the NTSC MPEG.

Here is the link for the NTSC MPEG clip:

http://savefile.com/projects.php?pid=184994

As before it is in RAR format although there are only 6 files this time as the NTSC is smaller than the PAL although I did the editing at the exact same points.

Again each file has a bunch of "crap" added on to it so you have to edit each file so you get something like P3NTSC.part1.rar so in the end you will have:

P3NTSC.part1.rar
P3NTSC.part2.rar
P3NTSC.part3.rar
P3NTSC.part4.rar
P3NTSC.part5.rar
P3NTSC.part6.rar

When you unrar this you get a FOLDER called P3NTSC and in that folder the file is called P3_NTSC_TEST_CLIP.mpg

So now anyone can download the PAL and NTSC clips to verify me.

Anyways I loaded the NTSC mpeg into Sony Vegas 6.0a as I did with the PAL mpeg only for the NTSC mpeg I used the NTSC DVD widescreen template.

NTSC MPEG
7.5 IRE Setup = Not checkmarked
Studio RGB (16-235) = Checkmarked



PAL MPEG
7.5 IRE Setup = Not checkmarked
Studio RGB (16-235) = Checkmarked


The PAL MPEG image here is the same as the one in the post earlier but I thought I would put the NTSC and PAL in one place.

So finally no VirtualDubMod in the way.

So edDV how does it look now?

- John "FulciLives" Coleman

[edDV]

THX Monitor Test Slide



THX White Test


THX Black Test

[edDV]

So edDV how does it look now?

- John "FulciLives" Coleman

White and black levels look fine here.

[FulciLives]

So edDV how does it look now?

- John "FulciLives" Coleman

White and black levels look fine here.

So the problem before had to be either VirtualDubMod and/or Photoshop. Right?

- John "FulciLives" Coleman

[BrainStorm69]

Agree: 7.5 unchecked, Studio RGB 16-235 checked.
PAL should be identical levels but different project setting (PAL DV 720x576, 25.0fps)

OK, I guess I may have screwed up my post above by unchecking Studio RGB and checking 7.5 IRE.

I got somewhat confused by the help section of vegas which says:

RGB values on your computer can range from 0 to 255. Studio RGB values range from 16 to 235. If you want to limit the display of the Video Scopes window to studio RGB standards, perform the following steps:

Click the Settings button in the Video Scopes window.

Select the Studio RGB (16 to 235) check box in the Video Scopes Settings dialog.

Whether you need to use the Studio RGB (16 to 235) setting depends on the DV codec you will use to render your video before printing to tape. Suggested settings follow. Refer to the codec's documentation to determine whether the Studio RGB (16 to 235) check box should be selected.

Codec
Studio RGB Setting

Sony Media Software DV
On

Microsoft DV
Off

Matrox DV
Off

MainConcept DV
Off

Canopus
On

Apple QuickTime DV
Off


If the Use Microsoft DV codec check box is selected on the General tab of the Preferences dialog, the Microsoft DV codec will be used.

If the Ignore third-party DV codecs check box is selected and the Use Microsoft DV codec check box is cleared on the General tab of the Preferences dialog, the Sony Media Software DV codec will be used.

You can render using a third-party DV codec by choosing a specific codec from the Video format drop-down list on the Video tab of the Custom Template dialog.

If you aren't using a DV codec, but have used huffyuv to capture, do the scopes still work right?