Colour band on conversion

[Tafflad]

Have just had an NTSC VHS video captured as both
huffyuv and MPEG2

Video and audio is pretty good, plenty of colour audio pretty clear.

There is one issue I would like advice on ...
Across the top of the clips, whenever there is bright blue sky there is magenta discolouration ... not tearing ... just top 10% of screen affected.

On some occasions the whole blue sky takes on magenta tinge .... any specific approach to help clean this up ?

uploaded 20 sec sample .... hopefully one of you may be able to advise

[Tafflad]

20 sec sample

[sanlyn]

Your black levels and gamma are too high, the video looks washed out and brights are blown away.

Some players can cause the stripes along the top border. There are a couple of Avisynth plugins that might solve the problem. I'll have to look again later, or other members might pitch in.

[Tafflad]

welcome any suggestions on how to improve.

[DB83]

@sanlyn

To assist the OP can you clarify how you arrived at your analysis. To my un-trained eye, the video played out fine and with plenty of clarity.

Bear in mind that this is a NTSC tape captured with PAL equipment.

[sanlyn]

Load the video into Avisynth and pull up a histogram on it. Or try with Avisynth's histogram in YUV. There are no dark blacks. The bright end is crashing against the right-hand side of the 'scopes. You don't need a 'gram to see it. Most of the shots have no color below RGB 32 or so, and many don[t go below RGB 60. About half the shots exceed 255 by a wide margin. A few look OK.

Will post more a little later. The wife is nagging me to empty the trash and come to dinner. Wives just don't understand that stuff has certain priorities!

[Tafflad]

I have never used avisynth, although I would try if someone could walk me through it.
I have used VirtualDub a little.

[DB83]

I would regard myself as an avisynth 'virgin' as well. Although I certainly do not have 20/20 vision, I rely on my eyes to tell me what looks right.

I have been assisting the OP with this project so I do have a slight vested interest in getting it right.

Does the 'fault' lie with the capture method, the capture device, the software used or a bit of everything.

Capture Device : Hauppauge WinTv-Express PCI (dates from pre 2007). No internal corrections apart from sound due to severe clipping.
Capture Software : DScaler - there are some colour adjustments but I thought those affected playback and not capture.
Capture Method: YUY2 (native Format). Would changing that to RGB improve ?.
Codec Huffyuv v2.1.1 - CCESP Patch with 'Predict Median' compression. Field threshold 480.

[jagabo]

I'd start with somwhere around: ColorYUV(off_y=-20, cont_u=-50, cont_v=-50). Before and after (no other filtering here):



Note how a lot of detail has appeared in the bright splash. That's an average adjustment for the entire clip. You could filter each shot separately for better results. Also, I don't know what the rest of the video looks like so the above values may need to change if this clip isn't representative.

Actually, I'd start by recapturing with the proc amp adjusted to get levels as close to legal as possible. Because the severely blown out parts of the current cap can't be improved with software filtering.

The flickering color at the top of the frame can be greatly reduced with McTemporalDenoise(settings="very high"). But that's too much filtering for the luma channel. Try something like (very slow!):

ffVideoSource("GD_sample.avi", seekmode=0) # real HuffYUV didn't like this file
AssumeTFF()
ColorYUV(off_y=-20, cont_u=-50, cont_v=-50)
ConvertToYV12(interlaced=true)
QTGMC()
MergeChroma(McTemporalDenoise(settings="medium"), McTemporalDenoise(settings="very high"))

There's probably some faster methods to reduce the flickering colors at the top. And, of course, MCTD is pretty slow for general noise reduction. So some other filter might be used for that. Someone else might be able to help out with the noise reduction issues but I don't want to spend the time. I'm just giving you a rough idea what can be done.

I'd try a different VHS deck to see if that gives less flicker.

There's something odd going on with the pulldown pattern, maybe a mix of frame rates. I just used QTGMC() rather than analyze it.

[sanlyn]

Does the 'fault' lie with the capture method, the capture device, the software used or a bit of everything.

Capture Device : Hauppauge WinTv-Express PCI (dates from pre 2007). No internal corrections apart from sound due to severe clipping.

Well....kinda screwed up the frame structure. I see progressive frames, interlaced frames (hard-coded pulldown), and duplicate frames. I didn't know a Hauppauge card captured to lossless AVI. MAybe it does. Now all we need is someo0ne who can figure out how to restore the original frame rate. The source seems to be film, converted to PAL, then to NTSC.

Capture Software : DScaler - there are some colour adjustments but I thought those affected playback and not capture.

Why wouldn't they affect capture? It seems that it did, unless the source is that bright to begin with.

Capture Method: YUY2 (native Format). Would changing that to RGB improve ?.

No. Not until later. The top stripes can't be fixed in RGB.

Now all we need is someone who knows about film24FPS->PAL25FPS->NTSC30FPS and screwy frame timing/pulldown structure than I do.

[jagabo]

A lot of the shots are slow motion -- by duplicating frames. So one can't reduce the entire video down to a single frame rate.

[sanlyn]

I noticed the slow-mo business after cruising for a while. It's the hard-coded pulldown I'm worried about (that's what it looks like, ayway, as if interlaced). Far as I can tell, it's very irregular: 1 frame of 5, then 2 of 5. Still looking at those. The color+levels in most shots won't be so difficult, even if a few quick segments are so blown out the brights won't be recovered, but most of it will look better witjh out much of a hassle. It's the hard-coded stuff I can't find the pattern for.

[jagabo]

Yes, TFM().TDecimate() gives you 24 fps. But the frames are screwed up. The time base problem that's causing the discoloration at the top of the frame is also causing the frame to bounce up and down. That's confusing TFM().

[sanlyn]

Hmm. Plays smoother in MPC-BE than in VLC and others. But there's still something not-smooth going on. I'll just have to keep working with this 24FPS and see if encoding with pulldown makes a difference, or set it back to exactly 23.976. As for the top border disturbance, we had a similar case within the past few weeks (and a lot worse than this!). Gentlemen, give me overnite to fetch the routines that jagabo and I used on that project...

To elaborate on jagabo's example, You'll likely be surprised at what fairly simple level and gamma adjustments can accomplish. Not a good idea to try bit with eyeballs alone -- eyes aren't as accurate as histograms and other tools. The image below is an unadjusted frame (and still has pulldown artifacts) on the left, a YUV histogram in the middle, and an RGB colorspace histogram on the right. In the middle YUV chart, note how the top white graph is shoved to the right of the histogram, and white luma levels actually start climbing up the right-hand wall in the "stay-outta-here" blow-out zone for brights. In the RGB chart on nthe left, you can see that luma (white) and all three colors are doing the same thing in an RGB display. Smashing against the sides of histograms indicate bright or dark detail that is being clipped, or cut off from the resulting image. Clipped = no detail. Once gone, stays gone. Note the loss of detail in the brightest parts of the left-hand image.

[Attachment 20003 - Click to enlarge]

Below, the image has been adjusted with the ColorYUV function in Avisynth, to the point where the YUV histogram shows the white level markers pulled back from the no-fly zone at the right and into the inner "valid" value areas of the histogram. Note the right-hand side of the 'gram, where the white markers have a tall right-hand "spike". That indicates clipped data in the capture that can't be recovered. Still, some of the bright and upper midtones were rescued; the adjusted image looks more detailed and three dimensional.

[Attachment 20004 - Click to enlarge]

Below, these before (left) and after (right) frames used the same settings as the above demo -- basically it's a corrective negative offset of luma and color about 35 RGB points lower than the original, with an added gamma adjustment. This is different from the "Brightness and Contrast" controls in many apps. Both reveal "missing" detail and color; the bottom image has recovered some facial highlights that were nearly blown away in the "before" image.

[Attachment 20005 - Click to enlarge]

[Attachment 20006 - Click to enlarge]

Will try out some scripting ideas and return tomorrow with some (hopefully) simplified ideas for those top borders bad guys.

[DB83]

Thanks for the input guys. I will await the scripts with baited breath.

One thing that puzzles me. The source video, which is still in my possession, c1992, was produced by a specialised production company in Florida. What gives you to think that this started its life out as PAL and then converted to NTSC ?

The Proc-amp settings on DScaler are as follows (I changed nothing prior to capturing and it is many moons since I last used this so I can not remember where these came from):

Brightness = 0
Contrast = 219
Hue = 0
Color = 236
Color U = 254
Color V = 219

There are also settings on the card itself but they are all in the middle of the park.

The capture card has the BT878 chip which enables DScaler to exploit its PAL60 capabilities.

The other thing is that the frame rate for the capture is 30 not the 29.97 as I would have expected. I also did a DV capture which also produced 30 fps but mediainfo told me that the original rate for that one was 29.97

edit: Could the sound clipping (corrected but still evident when looking at the vid in a nle) affect the video quality ?

[sanlyn]

Thanks for the input guys. I will await the scripts with baited breath.

Hopefully it's just "baited" and not not "held".

If you haven't used Avisynth, things might be little puzzling at first. But only at first. I'm looking up some old threads and scripts now.

One thing that puzzles me. The source video, which is still in my possession, c1992, was produced by a specialised production company in Florida. What gives you to think that this started its life out as PAL and then converted to NTSC ?

Probabl I read an earlier post too quickly. I re-read the post last night and cleared up my own confusion.

The Proc-amp settings on DScaler are as follows (I changed nothing prior to capturing and it is many moons since I last used this so I can not remember where these came from):

Brightness = 0
Contrast = 219
Hue = 0
Color = 236
Color U = 254
Color V = 219

There are also settings on the card itself but they are all in the middle of the park.

I don't use any of that gear or software, but I've seen its use previously. I don't know what the "middle" or null positions would be. Raising a "contrast" setting is not usually the best place to start with video source that is already exposed fairly correctly -- as you will eventually learn. Such modifications made during capture are difficult to undo afterward, which is why many users avoid them. But some sources might call for it.

The other thing is that the frame rate for the capture is 30 not the 29.97 as I would have expected. I also did a DV capture which also produced 30 fps but mediainfo told me that the original rate for that one was 29.97

There are ways to attempt a correction in Avisynth and VirtualDub. I'll have to play with that idea.

edit: Could the sound clipping (corrected but still evident when looking at the vid in a nle) affect the video quality ?

I shouldn't think so, although video rate might have an effect. Having never had that problem unless I did something really silly that I could correct later, I'll have to leave that for the sync experts. But sound level itself shouldn't be a problem unless your playback gear or software has trouble with it. Sound volume does affect the amount of data used, as I understand it.

[DB83]

Just checked the DScaler settings and those numbers are the 'default' ones.

As for avisynth, even this 'old dog' is prepared to learn some new tricks.

My problem with the before and and after pics is that I find the sky now to be a little un-natural. But I see that when using a dslr. Maybe its because I live in part of the world where we rarely see blue sky and people with perma-tans

[jagabo]

My problem with the before and and after pics is that I find the sky now to be a little un-natural.

Yes, too purple. But I didn't address the color except to crank the saturation down to match the lowered luma levels. I didn't think it was worth fiddling with until the OP gets a decent cap.

[DB83]

@jagabo

In reply #9 you said "Actually, I'd start by recapturing with the proc amp adjusted to get levels as close to legal as possible."

Could you define legal in a way that this ejit can follow.

On the haupaugge, the 'middle of the road' settings are all at 128

[jagabo]

Could you define legal in a way that this ejit can follow.

Y: 16 to 235
U and V: 16 to 240

Use AviSynth's Histogram(). Keep out of the yellow areas.

For example using TurnRight().Histogram().TurnLeft()

bad levels:


better levels:


Of course, not every shot should have the full excursion from 16-235, and a little below 16 or above 235 is ok. But over the entire video, most if it should be in the 16-235 range.

[sanlyn]

@DB83: there are example histograms in post $14, above.

The attached GD_Sample_Test01.mkv is for basic levels fix and top border stripes. Not perfect (What is?). The original audio was non-compliant for DVD/DB/AVCHD, so I adjusted that to 48KHz in my encoder. Color needs tweaking but can be tweaked later; and there's some mild chroma bleed, dot crawl, more rainbows, and a constant stream of horizontal tape noise. Those can be addressed later as well. The mkv has restored 29.976 frame rate. Pulldown can be added later for 29.972.

I attached a .txt file of the script I used. Actually, I ran it in 2 steps to speed things along. You'll need about 20 Avisynth plugins and support files to run it. Better to wait for your questions on that.


You probably don't realize it, but there's a faint yellow stain on the right border. If you didn't notice, I doubt others would.

[DB83]

@sanlyn

It appears to be a massive improvement. One to get the OP salivitating Even Friday the 13th can have its moments

I did notice a slight tear at the top of the screen. I suspect that was when the Martians were kicked off.

I was aware about the audio but again that appears to be a limitation of the capture software.

As for the script, I know about temporal noise being present in VHS recordings and I usually filter for that. This was a 'Raw' cap since we more or less agreed that any filtering would be done once the OP had the full capture - the sample came from my copy.

Is there any advantage to save as Lagarith since the source is Huffyuv ?

I will try to run that script on the full video just to see if there are any side-effects.

[sanlyn]

Lagarith makes a somewhat smaller file than huffyuv. Also, lagarith doesn't have the compatibility problems that huffyuv has with some system setups, because there are so many versions of it. Some versions of huffyuv (yours, I think) won't compress to YV12 (YUY2 and RGB only). Depends on which version you're using.

If you have problems finding all the support files -- likely you will if you haven't been digging too deeply into Avisynth -- let us know. Note that chubbyrain2 mentioned in the script must be used as early in processing as possible.

If by a "tear" you mean something in the upper left background around frame 411 of the MKV (17.1 seconds in), that's an onshore concrete wall in the background.

[Tafflad]

I'm reading every post ... but admit having trouble following some of it ... improvement is impressive.

Having no knowledge of AVIsynth .. is there a primer you can point me to ?

[DB83]

Digging too deeply ? Nah, the spade has not left the shed

No. Look at the sky at approx 8 secs - top left corner

[sanlyn]

I posted a corrected script in post #24. Just some typos in the comments, the rest of it is unchanged.

Digging too deeply ? Nah, the spade has not left the shed

No. Look at the sky at approx 8 secs - top left corner

The very top left corner looks like the edge of a lens shade. Frame 191 has pink noise remnant, upper right -- still a few of those remnants now and then. The lens shade shows up on frame 211, 230, and later. It's on the original AVI. Maybe some of those remnants can be replaced with frame interpolation, but the problem here is fast motion and, in the case of 191, there is no "before" frame that can be used for that.

One might also notice that first 2 frames in the mkv are unfiltered or only partially filtered -- there were no leading frames for the filters to work with.

I'm reading every post ... but admit having trouble following some of it ... improvement is impressive.

Having no knowledge of AVIsynth .. is there a primer you can point me to ?

Uh-oh. Well, it's not impossible. Hell, I did it (while jagabo and a few others dragged me kicking and screaming into it).

There's a rudimentary guide here: http://avisynth.org/mediawiki/Getting_started . It's actually from the same documentation that comes with the Avisynth download. When you run the Avisynth installer, it creates a program folder that shows up in your program listing. Open the Avisynth program listing and you'll see its documentation subfolder listed. The same stuff is also on the web.

There's also the Avisynth Wiki (http://avisynth.nl/index.php/Main_Page) with a directory of various articles for newcomers. Again, the same thing is in the installed documentation. Once you know what a script looks like, the best way to get the hang of it is to look over more scripts. The actual settings in some of the code require a little study, but it shouldn't take long to figure out what's generally happening.

An Avisynth script is really a text file with ".avs" on the end of its name. When installing Avisynth, you can specify that .avs files should be associated with Notepad -- when you click on it, it will open in that text editor. VirtualDub is programmed to recognize .avs as a video file; upon opening it, VirtualDub treats the running script like any other video. When you "play" that file in VirtualDub or move it frame by frame, the script is executed line by line, from top to bottom. And also in reverse.

Avisynth has tons of built-in functions. One of them is AviSource(). It opens an avi video. A caution here: "AVI" is a container, not a "format", and it can accept many codecs. e.g.,If you have huffyuv or Lagarith in your system, using that function will open the avi and send it to VirtualDub. AviSource() as you can see is followed by "()", and inside that () you type the path and filename you want, within quotes:

AviSource("E:\forum\tafflad\GD_sample.avi")

That statement shows where your GD_sample.avi was located in my 'puter. The statement after that performs an inverse telecine -- removes telecined effects and makes those frames progressive....that's a simplified description, but basically it's deinterlacing telecined frames and discarding (TDecimate()) duplicates. The 3rd line uses Avisynth's ColorYUV() function to adjust contrast (mostly brights), to shift all luma and colors toward the left side of the histogram (off_y, which means "offset of luma"), and slightly moves all blue values away from the the brightest end of the spectrum (off_u, which means "offset of blue"). And so on.

Which occasions the question: do you have Avisynth installed?

[DB83]

My apologies about the corner.

Now here is a puzzling thing (remember that I am a avisynth virgin as well). Now I did have it installed(2.5.8) as it came with another program. 'Getting Started' writes that you can create a one line .avs with the content as "Version" (no quotes). It then states you can open that with Windows Media Player. Except that WMP would not recognise the file. Vdub had no problem and loaded the video. But the kinda thing would put newbies off straight away.

[sanlyn]

WMP has changed since then. Not for the better.

[jagabo]

It then states you can open that with Windows Media Player. Except that WMP would not recognise the file.

32 bit AviSynth and 64 bit WMP? Both must be 32 bit, or both must be 64 bit. I don't know if there's some other issue though.

[sanlyn]

I've never been able to make WMP read an avs script for years. Probably works now. Maybe. I'm just too busy to try it.

ED: Hm. WEMp just shows one frame every few seconds, skips about 60 frames, shows another one. Not impressed. It did open an avs script. After I waited about 20 seconds. Why bother?