Scripts and filters for an Opera (sample videos included)

[sanlyn]

Example using badframes.dll in AviSynth. Thanx to manono's suggestion, I retrieved this long-lost plugin from a stash of old Windows folders.

Bet you thought I was sick of this video (I'm gettin' there). Still tweaking away, though. Here are links to 4 versions of the same short clip on 4share.com.

Clip "A" is from the original VOB. Shows a "flash" from a damaged frame (it's frame 148 in the "A" clip).
About 4 MB. http://dc181.4shared.com/download/7SlcvoJF/A_A01a9_original.mpg

Clip "B" is an early cleanup run after some (much?) cleanup. Frame #135 is the culprit in Clips B, C, and D.
(5.5 MB) http://dc181.4shared.com/download/9uzVVHow/B_A01a9_badframe.mpg

Clip "C" has frame 135 deleted. It works, but it's jarring when the cut occurs during a sudden, emphatic motion like the one shown here.
(5.4 MB) http://dc181.4shared.com/download/IHeaISYc/C_A01a9_cutframe.mpg

Clip "D" shows frame 135 replaced by interpolation using Fizik's Badframes plugin, which blends previous and following frames, according to Fizik. (5.4 MB) http://dc181.4shared.com/download/bnW8D7HX/D_A01a9_newframe.mpg

The scene is from a continuous 4-minute segment in which Mignon crosses back and forth across the stage, going thru several stage lighting areas -- lights with red gels, then cyan gels, then amber, then neutral. In this scene Mignon is apparently under a hot amber spot, not a floodlight. All recordings of stage plays have artsy lighting schemes. I've been taking overall color balance from more naturally lighted scenes. Much of the remaining background noise seen here doesn't show up much on a TV (thank goodness). Could be cleaner, but this damaged video just won't tolerate much processing.

Cap of bad frame #135:


cap of bad frame 135 replaced:

[2Bdecided]

mvtools motion interpolation might be nicer here, but I don't know of a simple neat script to feed it a list of frame numbers to replace. It's surprising if no-one had made one though. Probably won't work directly on interlaced video.

The mvtools page just includes this...

To recreate bad frames by interpolation with MFlowInter:

AVISource("c:\test.avi") # or MPEG2Source, DirectShowSource, some previous filter, etc
super = MSuper()
backward_vectors = MAnalyse(super, isb = true, delta=2)
forward_vectors = MAnalyse(super, isb = false, delta=2)
inter = MFlowInter(super, backward_vectors, forward_vectors, time=50, ml=70)
# Assume bad frames are 50 and 60
trim(0,49) ++ inter.trim(49,-1) \
 ++ trim(51,59) ++ inter.trim(59,-1) ++ trim(61,0)

...which is clunky!

Cheers,
David.

[sanlyn]

Thanks,2B. Already tried motion interp, it made a mess of it, very blurry. Note that the motion in this piece is really quick anyway, the original and the "fixed" look pretty much alike but without the "flash". There are other scenes that are far, far worse, but nothing to be done about them; they are 100% noise, with nothing to "interpolate" for a dozen frames olr more. Will just have to let those go as-is. The only flahes worth working with are the 1-fgrame or 2-frame problems. When they hit 140 frames (like the one climactic boo-boo near the very end), it's just no use. The only way to recover those frames would have been to record them again after trying a few tricks with the original tape, like rewinding a few times, etc.. Alas, nobody can locate the source.

[sanlyn]

This crummy video has become a long-term project, maybe a couple of afternoons a week. I agree with many who've viewed this thread: to most hobbyists the original transfer from VHS is so awful it would hardly seem worth the effort. But it's such a good operatic performance, I see it as a great learning tool to find ways to repair some of the dumbest recording errors I've ever seen, anywhere, anytime.

The worst is the frequency of brightness/contrast changes made by the original user. In one instance I counted 7 up/down brightness changes in a 15-second scene, in another a long and beautiful aria starts out just find but the user suddenly decides to brighten the video so the singer's face is burned into a white blob of glare for the rest of the scene! Amazingly stupid. I tried many auto-level filters, all of them made it worse and none could handle 5 or 6 dumb adjustments in a row. It's bad enough to work on damaged video without having a maniac at the controls. I worked today on one scene where for 8 seconds the user made the scene so dark that 1/4 of the RGB curve disappeared off the left end of the histogram (*OMG!). But I'm picking up many tricks with AviSynth and VirtualDub that help make this thing viewable (sorta). Will share later. Maybe these tricks will be helpful to others. Thanks to all who've contributed.

[jairovital]

Hi sanlyn

It's wonderful to see that you are still working on this project. As you said, yes, it's a great learning tool to find ways to repair wrong handling with movies, mainly when transfering from tape. Superb!

I worked today on one scene where for 8 seconds the user made the scene so dark that 1/4 of the RGB curve disappeared off the left end of the histogram (*OMG!).

LOL

I suggest, if you accept the deal and the challenger, to write a list of steps and advices to fix and repair the most of the problems of that opera, when you finally reach the end of the work. Deal? Will you remember that?

It is because it would be valious for many people who come here to read our comments in the future. I think it would become a big lesson for those who want to learn more about restoration and ripping from tape.

[sanlyn]

...write a list of steps and advices to fix and repair the most of the problems of that opera, when you finally reach the end of the work. Deal?

I'm now near the end of Act 2. Some of the physical damage is truly horrible. I'm saving quite a few "steps and advices", enough to fill another entire forum, just from this tape. And in every scene, The Maniac is still fiddling with the controls. I was hoping to finish this by Christmas but, alas, this unavoidable series of events called Life keeps getting in the way, along with o0ther projects. But the challenge continues.

[sanlyn]

Damage Control 101:

Having taken a one week hiatus from this Opera From Hell , I foolishly took this project on again.
Alas, if readers thought the earlier samples looked bad, they'd weep at what's on cassette #2. Really bad damage. I chanced upon a few tricks that might help others.

I posted "A_Damage_original.mpg" video (27 secs, 11-MB). The Audio is Layer-II.
http://dc281.4shared.com/download/yoLcBVpW/A_Damage_original.mpg

Folks, this ain't just noise, it's chaos. Likely the tape doesn't even contact the video heads half the time. At 3.5 seconds into this clip (frame 98 - 109) a huge chunk of damage occurs. The audio tells me the recorder dropped a handful of frames by frame 108. The tape jerks again at frames 539-541 (18 seconds in). Besides all the flashing, border stains, sparkles, etc., note the geometric swirls in the background, dancing like sugar plum fairies. Here are some caps of bad frames (left=frame 106, right=frame 541):


The shake, shimmy and wiggles didn't respond well to my usual filters. So I developed these steps:

1. Make an avi from the worst segment of this original, using AviSynth and the aforementioned chubbyrain script. As the frames load into VirtualDub, run the script with VDub's temporal smoother set to 3. Smoothes out a lot of wiggles, almost as good as a TBC -- well, not quite, of course). Doesn't work nearly as well if I just run chubbyrain and temp smoother in 2 separate steps. Works only when reading output from AviSynth. Don't know why (any takers on this?).

2. Run the resulting avi in a separate step with a previously configured NeatVideo filter. Work on mid- and low-level temporal noise only; high-frequency filter turned down to a mere pittance.

3. Run the 2nd avi thru AviSynth using FreezeFrame() and BadFrames().

The last step smoothed out a few top-border flashes and wild wiggles, and helped mask/replace bad frames. TMPGenc made final color and level tweaks (such as were possible, anyway). The damage at frame 99-109 has dark frames that gradually lighten for another 80 frames. The jerky movement you see in this "first try" clip is due only partially to FreezeFrame for a few frames, but mainly due to the recorder dropping frames. It's hard to tell from the original, but this scene had zero contrast -- not to mention really soft "detail" (I use the term loosely). So whadya want from 352x480 video recorded at a 4-hour bitrate? The result is posted in "B_Damage_first_try.mpg" (27-secs, 16-MB).
http://www.4shared.com/video/X-m0xfmp/B_Damage_first_try.html

But the dark frames were an eyesore. The caps below show, from left to right, frames 99, 104, 185.


Not being quite satisfied (i.e, not yet pummelled bloody enough by this disaster), I spent countless hours trying every autolevel device I could find, from odd scripts on Doom9 of which I had no understanding whatever, to stuff I never heard of, etc. Finally worked the dark frames with AviSynth's autogain. Listen, people, this thing really screws up the colors, which were pretty grim to begin with. The levels worked ok, but OMG! the noise and colorations! After several hours I took the "B" version thru autogain in AviSynth and then umpteen filter settings in TMPGenc, and ended up here: C_Damage_autolevels.mpg (27-secs, 16-MB).
http://www.4shared.com/video/cZaeugqE/C_Damage_autolevels.html

Shucks. I just don't get that old my-cup-runneth-over sensation. Oh, well. On to Act 3. I'll re-visit this clip later...after I see Santa.

[sanlyn]

Flashes and wiggles and burps, Oh my!
New uses for FreezeFrame and BadFrames.dll

Herewith, a sample 16-second video (another?) of severe tape damage. Looks like the tape at this point was crinkled and probably the texture of much-used aluminum foil Christmas wrap. After major denoising and some ColorYUV work, the tape has red and green blotches running across the top border, "flashes" of bubbling and boiling what-not and all manner of visual schmutz.

E1_92_E3_98.jpg (below) shows 2 frames from the sample. In the left frame find one of many small green (or red) fireballs in the upper right. Note the vertical lines in the background wood panels; during the video they wiggle and shrink and bow out(!). In the right-hand frame note the background at the right side behind the gal (woo-hoo!) in the pale pink dress. The walls will appear to bubble and boil, and the stage lighting will flash and flicker.


I used FreezeFrame and BadFrames.dll to fix 267 of the 5324 frames in this section of video. The sample video shows 16 seconds of the problems and 16 seconds of results. Sorry, no way to make it cleaner.

E_MoreDamage.mpg (16 secs, 9.5-MB) <- NO AUDIO
http://dc281.4shared.com/download/iB6rT9Hu/E_MoreDamage.mpg

E_MoreDamageFix.mpg (16 secs, 9-MB) <- NO AUDIO
http://dc281.4shared.com/download/kJHoFlbX/E_MoreDamageFix.mpg

Major advice, for those who try recording VHS to DVD on uncalibrated monitors. For the entire length of this opera the user repeatedly fiddles with the copy adapter's image controls. It's enfuriating to see brightness and contrast zooming up and down a dozen times during a 2-minute sequence. Autolevels might even things out, but to no avail. This user always overcorrected. For much of Act 2 the user apparently set dark levels too low and brights too high. Result? Brights above RGB 190 or so are blown to hell, dark detail below RGB 40 is crushed to blobby oblivion, and stretched contrast in both directions thinned out the midtones. You can use filters to correct levels, but the darkest and brightest details are destroyed. Period. Trying to equalize images on an on an outtta spec monitor is useless. Aside from that, once you find decent white and black levels for recording (and you better have a calibrated monitor or a luminance meter), set the controls and then LEAVE 'EM ALONE!

Yes. I see the red and green rainbows. If there was more detail to work with, I'd tackle those a bit. But this crummy video can't take much more processing.

[sanlyn]

Even more uses for FreezeFrame and BadFrames():

Act2 of this g.d. opera is the worst damage I've ever seen. If you consider that after 5 or 6 major damage spots the tape is so crinckled it's hardly rising over the video heads, you wonder why there's any image at all up. Herewith, a screen cap of a piece of damage, which was followed by 52 frames of pure garbage.



31 frames look like the one shown, except for 2 frames right in the middle that somehow survived the holocaust. The next 20 minutes of tape have beautiful singing and really ugly noise. Here's a 9-second clip of a piece of the original source:
2A_Damage_E01C_original.mpg from the original disc (3.9 MB, NO AUDIO)
http://dc304.4shared.com/download/OwB4ZeTf/2A_Damage_E01C_original.mpg

Just spent a week going so far as to try delinterlace/reinterlace and dozens of plugi9ns. But results just went from bad to worse to more worse. So I returned to the method mentioned earlier: AviSynth/chubbyrain with VDub's temp smoother in the final output. Then NeatVideo. That's the major noise reduction routine that has worked (I use the term loosely, considering the source). With the worst of the noise removed, what's left are flashes, dancing backgrounds, distorted objects, twists, shimmering, etc., etc., ad nauseam:
2B_Damage_E01C_Intermediate.mpg (5.1MB, NO AUDIO)
http://dc301.4shared.com/download/IBrmT5bu/2B_Damage_E01C_Intermediate.mpg

With the liberal use of FrezeFrame and the BadFrames AviSYnth plugin, the results are actually better than I could hope for...which means, of course, I've learned not to hope for much from this horrible source. A few stalwarts out there might find these techniques useful. Let's Hope you never have to use them to this extent.
2C_Damage_E01C_framefix.mpg (5.1MB, NO AUDIO)
http://dc304.4shared.com/download/VoE1Hvmr/2C_Damage_E01C_framefix.mpg

[sanlyn]

Thought I'd reply to my own post, especially since I got on the 'net and found a few people had this same problem.

Even more uses for FreezeFrame and BadFrames():

On top of this, my VirtualDub suddenly decided to stop working. Click "save as avi...", give the file a name, and...nothing. Nada. No output. VDub just stared at me. Period. Checked stuff, uninstalled, reinstalled, uninstalled again, rebooted, installed again, etc. No soap. VirtualDub just plain refused to output (Was it trying to tell me something?). The old VDubMPEG worked ok, but it's too old for some of my plugins. I found a new VDub v1.9 or whatnot, downloaded, installed. Works. Just something else to warn me about working on this crazy project.

Actually, after a couple of days it didn't work any more. I shoulda figured it out a few weeks earlier when my new copy of AVG AntiVirus told me VirtualDub was a virus (a trojan, yet!) and I clicked AVG's "ignore" button. Didn't do any good. I removed my $30 AVG and replaced it with ESET. Good fix. No problema. Onward.

I just spent 3 weeks on the last 12 minutes of this horrible video. What a mess. Learned much. I didn't think I'd ever get this close to the end. Yes, I know -- you gotta be nuts for this. Beautiful opera, though. Too bad it isn't in print.

[jairovital]

Sanlyn

I'm still here. Never miss a post from you. And glad to know that you're close to finish this "neverending" story.

About Virtualdub, very weird to hear that kind of behaviour. I know how you feel: have to think about million other things about color space and other video stuffs and to have to stop all, to give attention to "why this damn thingie doesn't want to run".

Good because you could find the guilty one. AVG, huh?

Stay alive. I'm waiting the grand finale and you could post, "enough. it's over!"

[2Bdecided]

Nah, he'll just finish it and some TV station will decide to rebroadcast it in pristine quality.



Amazing work though - reminds me of the painstaking work the Doctor Who Restoration Team go through. I think they're getting paid though!

Cheers,
David.

[sanlyn]

Nah, he'll just finish it and some TV station will decide to rebroadcast it in pristine quality.

Sure wish they would, but likely not in the USA. Both of my DVD recorders are waiting.

reminds me of the painstaking work the Doctor Who Restoration Team go through. I think they're getting paid though!

Wish I had the $$multi-million worth of equipment they use.

Sorry to say this effort leaves me with much to be desired IMHO, but at least it's getting, uh, sorta viewable. Sort of. Much of the time.

[sanlyn]

Color Timing 101

In movie making and video archiving, there's a necessary technique called "color timing". The term hails from the earliest days of movies. Today, "timing" pertains to luma and color matching. When a director calls for different camera shots in a scene, each shot should look as if it came from the same scene. Inappropriate luma and color changes between long shots, medium shots, etc., makes viewers become (to quote a movie title) dazed and confused. Besides the awful tape damage, the scene matching in this video is horrible. That's a problem with VHS playback anyway, but here it really gets on your nerves.

To those who wonder why I'm spending over 3 weeks on the last 4.5 minutes of this video (insanity aside), look at this frame sequence. The caps are 1/2 the original 352x480. These are from AviSynth/chubbyrain before additional color work. Main problem is that some scenes are too light, others too dark, so a middle ground had to be chosen with AviSynth/tweak/ColorYUV. These frames are from the 5th AviSynth version. Often I mix versions to try to match severe contrast differences. Anyway, the stage lighting here is apparently supposed to be grim and high-contrast - it is, after all, a death and angst scene. As for trying to restore dark and bright extremes, forget it. The kamikaze who recorded this tape apparently set dark/bright levels for his uncalibrated monitor, but the settings often exceed the capabilities of PAL or NTSC.





The contrast and colors change with each shot of the same scene. The rear curtain, the clothing, hair color, saturation, etc., all differ. Most shots are joined with straight cuts, but all shots beyond frame 645 are joined with a sloooow dissolve, often up to 5 seconds. If you correct the contrast/color for one shot, the dissolve into the next weird shot displays a "snap" of luma/color change where the shots are joined. I tried joining long dissolves with AviSynth, but the results of dissolve-on-dissolve were really strange. So in some dissolves the join is sometimes obvious, sometimes not.

Another glitch: the sequence involves a change in stage lighting. As the sequence approaches the opera's climactic kick-the-bucket finale, the peripheral lights dim and the central spots get very bright. The lighting also gradually changes from bluish 15 minutes earlier to amber here. The uneven stage lighting is OK for human eyes, but for video it sucks.

Frames 116 and 645 were corrected and joined OK because there's another shot between them (not shown), joined with straight cuts. But scene 645 dissolves into scene 2581, and 2581 into 3366. Note, all 3 shots are of the same scene but no shot looks the same. 645 is too low-contrast and red, 2581 is too red and dark but with hints of green in skin tones, 3366 has oversaturated green. It's possible to match any two scenes fairly well in a dissolve/join, but not all three.

Frame 3366 to the end is a single shot, to a straight cut at frame 5438. From there, the heroine falls into the arms of the Death Figure midget waiting to catch her (and he does, too. But he doesn't get a spot in the curtain calls. Talk about thankless work!).

Then there's tape damage: frames 4988 to 5450 (462 frames). The worst of it abates at 5225. I managed to salvage 5 decent (?) frames that I can use for a solution that covers those frames. Meanwhile for the whole ending the orchestra is blasting a crescendo and everyone on stage is yelling high-c's like mad, kettle drums pounding away.

The heroine collapses just after frame 5438. From there the scene has a very-slow-dissolve to white. Frame 5438 actually looks close to the "real" color, because by the time the next 185 frames go white, the final color really is white (RGB 224). So any major change in the color balance of 5438 wouldn't end looking white. After that, the white dissolves to a shot of the audience. That shot, too, looks about right. So from 5438 on, mild corrections will do.

The object is to go from the start near Frame 116 and end up with the color balance (or near it) of 5438. Look at the continuous sequence starting near frame 3366. Green is too strong, the darks too dense. Some of that density is a lighting effect, but too dense here. And, yes, highlights and darks are just gone, period.

The just-before-end at Frame 5274 is too green (look at the "blacks"). Look at the midget out front in those frames, then look at the color of the his hooded robe in 5438. From previous scenes I know the hood to be brownish, not green, and it looks about "right" in Frame 5438. From all appearances, it looks as if the final bright light is likely an amber and cyan mix. The preceding frames aren't even close.

Another problem: In frame 4796 Mignon's gown hits a bright RGB 224. Then the damage occurs, and the following frames are darker at max RGB 188. I fixed that with VirtualDub's gradation filter, but I had to make and join three different AVI clips to do it.

For the knit-pickers (this video is knit-pick heaven): Yes, there is apparently a blue-gelled flood off to the left. No, it's not the blue edge stain seen earlier in this tape, it's a different blue that's also used to light an earlier sequence of the same scene.

These frames were previously subjected to noise filters and BadFrames. The samples above look dark, but a little darkening helped control the fried highlights. The finals look more natural -- these are earlier Photoshop layers used to compare scenes. Note that the heroine's gown isn't white. It's slightly pinkish off-white, at this point lighted with amber gels. Her hair is neither green nor black as it sometimes appears: it's very dark red, nearly black. The hero's shirt is white, his vest purple, and the other big guy wears black. Those are the colors that develop when I correct for whites, flesh tones, black, and lighting gels. It also looks as if the green rocks are variously lighted with red and/or blue or cyan overheads. From the original disc, you can only guess:



Onward. The end is, well...near.

[sanlyn]

Dissolving Dissolves

These caps show one of the problematic dissolves whose scenes, after they've had some color and contrast work, just don't match. One way to join the scenes is to find a spot somewhere during the dissolve where the two scenes blend seamlessly. Often this works, but sometimes it's a visual "snap" where the two scenes meet.

The caps below are scenes from the godawful source video. They connect with a sloooww 10-second dissolve. The originals are so discolored that any problems connecting the two aren't apparent. The short scene starts with the frame on the left and dissolves into the shot on the right.


After the greenie original has been through AviSynth/chubbyrain (twice), Tweak, ColorYUV, denoisers and BadFrames, it looks this way before further correction. The scene on the left is too cyan and has no oomph. The stage is supposedly dimly lighted but with a brighter amber gel on Mignon and the "rocks" at left, and some magenta floods farther off-camera. The right frame looks almost as washed out and is deficient in blue. Both are under-saturated.


After some VirtualDub work, the scenes look more real, even if the low bitrate left no fine detail. I fiddled with contrast, color, gamma, etc., for a long time. The two just don't match after being corrected.


Below are two frames from about the middle of the dissolve. The frame on the left is the dissolve as seen in frame 479 from clip #1. The frame on the left is frame 480 clip #2. You can see that the two shots look quite different in contrast and color. In the original video it works OK over a 10-second dissolve if you want one (or both) scenes to remain uncorrected (in the original video, the right-hand scene looks weird, it's fuzzy and has even lower contrast. It almost looks like something from another video).


One way to join these shots is to cut clip #1 at frame 479 and join it in a straight cut to frame 480 of clip #2. The test video below shows at frame 480 that you get a real jolt from two dissimilar shots. Direct link:
http://dc206.4shared.com/download/j1Forrgm/Clip_Join.mpg
(MPGE2, Layer-2 audio. Bitrate somewhat lower than the final version. 14MG, 28secs).

Enter the Dissolve function from AviSynth. The two AVI's are cut in the middle of the dissolve and joined for a 2-second dissolve-on-dissolve starting at frame 440. The join is almost undetectable. Direct link:
http://dc206.4shared.com/download/A6HD0f6u/Dissolve_On_dissolve.mpg
(MPGE2, Layer-2 audio. Bitrate somewhat lower than the final version. 13.5MG, 28secs).

At first try I cut the clips to the desired length. Then I realized that one clip lost audio for a few frames, so I remade them with some leeway on each side of the join to avoid audio problems. Too bad I couldn't go through every dissolve in the 3-hour opera. It just takes too long. The sample videos are from my first test run.

Here's one version of the avs script:

clip1=AviSource("[insert path and file name]")
clip2=AviSource("[insert path and file name]")
c1=clip1.Trim(0,-440)
c2=clip2.Trim(440,0)
Dissolve(c1,c2,60,fps=AudioRate(c1))

[Gavino]

Dissolve(c1,c2,60,fps=AudioRate(c1))

The 'fps' parameter is for joining audio-only clips and is ignored if a video stream is present.
http://avisynth.org/mediawiki/Dissolve

[sanlyn]

Dissolve(c1,c2,60,fps=AudioRate(c1))

The 'fps' parameter is for joining audio-only clips and is ignored if a video stream is present.
http://avisynth.org/mediawiki/Dissolve

oops. You're correct. I revised the posted script, and thanks.

Is this the sort of absent-mindedness that causes so many of us to have to re-do stuff?

[sanlyn]

I'm still at it.

Reached the end about 2 weeks ago. Since then I've acquired a new IPS-display monitor, recalibrated everything over and over, and got TMPGenc MPEG Editor v3. Now going back thru 2hrs-56min of opera, tweaking areas that might respond better to tricks I've learned during the past months.

Yes, I know: it still looks like crap. Somebody took a damaged, low resolution, low-detail, off-color VHS source and recorded it to an even lower-resolution bitrate with lossy MPEG compression onto a second-rate DVD recorder at an eensty-weentsy frame size (!). One can make the images look passable when played back at the original 352x480, and even at 640x480. But upscale this awful recording to 720 or 1080 lines on a TV, and -- as they say down South -- you got nothin' but 10 miles of bad road.

This has been one hell of an experience.

[2Bdecided]

Postage stamp it:

Encode it to a 1920x1080 frame, with just a 640x480 image in the middle, surrounded by black. EDIT: beware of screen burn if watching this on a plasma.

Or sit further away from your TV! (across the street should do the trick )

Cheers,
David.

[sanlyn]

Going bananas with bad VHS (Another Chapter): As if tape damage were not enough (are we to be spared nothing?). . .

In the three images below, note the comic relief character in the pale blue costume. BTW, he's a great actor, a great tenor, and a very funny guy. These images are about 15 seconds apart, all from the same sequence of shots. In the left image his costume is the same color as in the rest of the opera. In the middle shot, his suit has turned green. In the right image, it's blue again.



This VHS tape has many problems, but scene and color matching are the originating producer's fault. Note in the middle image that the cape worn by the dressy countess is supposed to be pretty much a pure blue -- at least, it's blue in the entire Act 2-Scene2. In the middle image, the blue is under-saturated toward blue-gray. This is the kind of junk production work for which VHS is justly infamous. The middle shot is only 2.5 seconds, and there's no way to balance the color with the other shots from the same scene. So to heck with it. The background colors also change frequently.

For a filmed opera, the lighting is terrible. The use of colored floods is overdone. Nice for stage work, terrible for video. This scene appears to be lighted by blue floods at the left, ungelled floods at center, and amber or yellow floods at the right. It's supposed to be a late nite festival scene, lit by torches and moonlight.

But . . . I notice something about this opera, seen here and just plain obvious throughout the play. Below are two images. They are taken 1 frame apart. In the left-hand image, note the guy on the right with the letter "A" over his head (I put the A there). In the left-hand shot, Mr. A has just begun to turn slightly toward someone to his right. The camera angle changes, and in the next frame Mr. A has a different stance, bent farther to his right and partly backward, closer to the guy next to him. Mr. A's costumne has magically gone from red to purplish in a single frame. Also, the hero on the left-hand side has a different posture. Just as I suspected: this tape was assembled from videos of more than one performance. And each performance has a different color balance and light levels.



Anyway, this whole project is one big series of guesses about what the colors are supposed to be. Here's the scene in the original (godawful) DVD recording, at 352x480:



These scenes are 2 hours in, near the end of Act 2 and the fire-and-smoke extravaganza -- a pi9ece of business with problems all its own. As nearly as I can tell from other shots in this long scene, this shot has a strong blue flood off-image at the left.

Here are direct download links to the full-sized 352x480 image originals for this po0st. Your cup runneth over.

http://dc394.4shared.com/download/cjS4Ri4b/Match1.jpg
http://dc394.4shared.com/download/X11fgFRO/Match2.jpg
http://dc394.4shared.com/download/8znj_q8g/Match3.jpg
http://dc394.4shared.com/download/m2QowOT8/Match4.jpg
http://dc394.4shared.com/download/mjF_B15n/Match5.jpg
http://dc394.4shared.com/download/4woBoXGk/Match6.jpg

[sanlyn]

Struggling with faded video:

I've never worked with a video source whose image is as badly faded as this. Add the usual VHS aging (greatly accelerated by apparently storing this tape in a virtual oven for a long period!), and you have scenes where the image signal is so far below the noise level, there's no image to work with. Then add sloppy production work on the tape itself, where a scene is inserted that has little contrast or color to begin with, and you have a hopeless situation.



From the original video, no corrections. This 10-second scene occurs a minute into Act2 Scene2, which is on the 2nd damaged cassette. I thought casette #1 had damage, but #2 is a pip. Besides being faded, the scene has no contrast and low saturation to begin. It should look a high-contrast image, an interior night scene with low foreground lighting, some concealed floods behind the little "window" (which also has a thin scrim or transparent cotton stretched across it, just to make for very difficult video). One of the lights behind that window has a blue gel and is off to the left, which you can detect from other scenes. What you see behind the figure of the countess is lots of noise, but some of that "noise" is actually a backdrop used throughout the play -- in this scene, that backdrop with painted trees has a very dim blue flood on it. Any blue in this scene has been crushed off to the side of the histogram, and there's no bright blue at all even tho it's seen in the shots before and after this one.



After loading the original .d2v file with AviSynth running chubbyrain/Tweak/ColorYUV, and in tandem with that is VirtualDub's temp smoother set to 3. I have no diea why this combo of AvySynth/tempsmoother in tandem has far more effective results with motion noise than they do when running those processes separately. The original has lots of motion noise and flicker, but much of it it's gone by using this method. Anyone who knows why it works better than separate processing is welcome to offer an explanation. Here, even with measures to deepen color and add some contrast, there's still not much image to play with.



Above, after two runs. The first runs used AviSynth and more chubbyrain along with an added dose of Cnr2, in tandem with a NeatVideo filter set to work on high-frequency noise and some temporal filtering. Note, NeatVideo has been set to a very low power; there's precious little "detail" to waste here. Again, this is running AviSynth/NeatVideo in tandem. Run #2 was without AviSYnth, but NeatVideo and tempsmoother were run together; NeatVideo was set for only medium-and-low level cleaning and a mere 10% sharpening, and temp smoother has been set to a lowly 1 to help with more of the simmering fine-grain stuff and some remianing motion blobs. There's still a lot of noise, but it's reduced to a viewable (?) level especially inside that little window. IMHO, this crummy video just can't tolerate any more filter runs.



The only color filters used were with the TMPGenc 2.5 encoder. Extending high blue (what there was of it) and darkening the darkest blues to help reduce the blue stain in dark areas, was abouyt all I could do without ruining everything. The histogram for this scene is a total mess; there's only so much you can with a badly deteriorated magnetic layer in VHS. At first I increased saturation by as much as 80%, which is pretty stiff, but the fluttering noise and deep blue stain were painful to watch. So this saturation level was cut to 40%.

When I mention a blue satin, I refer to a 20-pixel wide blue discoloration at the lower left (not too visible here because of the foreground colors) and a blue stain in the darkest areas of the background behind that "window". The stain dissipates somewhat later in the btape. But by "satin" I mean a highgly visible blue discoloration about RGB-20 to 30 on the left-hand side. There's also a green stain across the stop and a big red rainbow in the upper 1/3. These have resisted every attempt at cleaning. Here, the top green is reduced by all those filter runs, but blue is persistent.

When I say "stain", I can't think of any other word except "discoloration" or "layer distortion". Usually you see a rainbow that has some added color overlaying the others. In this case, the blue or green don't increase, except maybe a little. What really happens in the stained areas is that the two other colors suffer a sudden and precipitous drop. The area beside the stain might say Red-45 Green-45 Blue 30, but just inisde th stain you get Red-28 Green-25 Blue 31. Note that Blue hasn't changed much, but Red and Green have melted away. I'm thinking that this why AviSynth and VirtualDub and other filters just don't see these stains as "noise".

Also, I still don't see why running AviSynth\YUV filters and then adding VirtualDub filters in tandem on output works better than running the same filters separately. The results are sharper and cleaner. I have no explanation for it.

[sanlyn]

AviSynth, MVTools, and Simmering Blobs. Can MVTools fix this?

Believe it or not, I made it to the end of this 2-hr 54-min opera. I stopped calling it "the opera". Now it's TCFH, The Capture From Hell.

Time to get deep into specialized AviSynth tools. I'd use AviSYnth more often if its infrequent documentation and examples weren't so cryptic. In assembling all the processed parts of TCFH into a final 2-disc DVD I quick-reviewed from the beginning, tweaking here and there. Then I reach a scene like this, and I realize I've gone as far as I can without turning to AviSynth.

Sample: There must be some AviSynth plugin or script that can handle the problem of "simmering blobs" without destroying video. I just spent a whole week, all day long, reading up on motion vectors, etc., and searching for any use at all of MVTools to fix this kind of disturbance. I know it's damaged tape rattling over the video heads of a budget Aiwa VCR, a miserly 2500 bitrate (!), etc. It's not just on TCFH. I see some form of simmering blobs in many VHS captures (except mine. Maybe I'm more careful?). The image below has an arrow pointing to the crap I'm talking about.


Here's a zoom of what this gunk looks like when not moving. Some kinda ugly amoeba design that theatrical set designers like to use.


The blobs are in this clip I posted earlier as an example of BadFrames.dll. Yes, I know, it's over sharpened. Fixed that later, but this version shows blobs in motion more clearly. NTSC MPEG, 352x480, 12 seconds, 5.2-MB:
http://dc181.4shared.com/download/bnW8D7HX/D_A01a9_newframe.mpg

Here's the original unprocessed clip from the TCFH DVD. NTSC MPEG, 352x480, 12 seconds, 4-MB:
http://dc181.4shared.com/download/7SlcvoJF/A_A01a9_original.mpg

I know TCFH is beyond hope (it's even beyond despair). I see blobs all the time but no one seems concerned. Temporal filters can't touch it. It needs more than "a filter" anyway. Just fiddling with example scripts aimed at other problems can't work and doesn't teach us anything. Or maybe I'm on the wrong track. Maybe simmering blobs can't be fixed. Maybe MVTools is the wrong approach; the blobs stay fixed for multiuple frames, then move or change shape -- not a good omen for this plugin. But how the hell does MVTools work?

[poisondeathray]

Nice work , this is a marathon . I especially like what you've done with the screwed up colors so far



RE: "simmering blobs"
Not sure mvtools is suitable for this sort of thing. It calculates motion vectors, but how would that be useful here , even when used for with other functions?

The artifact is definitely more perceptible on more brightly calibrated setups, and it doesn't look that bad if you consider the original source quality LOL

Already tried motion interp, it made a mess of it, very blurry. Note that the motion in this piece is really quick anyway, the original and the "fixed" look pretty much alike but without the "flash". There are other scenes that are far, far worse, but nothing to be done about them; they are 100% noise, with nothing to "interpolate" for a dozen frames or more. Will just have to let those go as-is. The only flashes worth working with are the 1-frame or 2-frame problems. When they hit 140 frames (like the one climactic boo-boo near the very end), it's just no use.

IMO, it does a better job (less blurry) than a blend, if you apply it on fields. In the grand scheme of things, this is a very low motion scene, so mvtools should have no problems with it. (e.g Compare to a race car video, now that's what I would call high motion, where there is a large difference between frames or fields). In the original video, only the top field is damaged, the lower field isn't; but in the "fixed" video, since you replaced it with a blend, both top and bottom fields have to be replaced.

You're right in that it's useless for long stretches. Only single bad frames or fields that have surrounding "good" frames or fields might be improved effectively

[sanlyn]

Nice work , this is a marathon . I especially like what you've done with the screwed up colors so far

Thank you. Color took more time-n-labor than the noise.

RE: "simmering blobs"
Not sure mvtools is suitable for this sort of thing. It calculates motion vectors, but how would that be useful here , even when used for with other functions?

I keep asking the same question. But I don't fully understand all of MVTools' functionss. I get what you mean, especially since most examples seem aimed at noise reduction rather than motion "reconstruction" -- the latter is what I think I'm getting at, but MVTools might not be the answer. I keep looking at other methods. I see examples of MVTools used to fix several seconds of bad edge streaking; the streaked objects are somehow being rebuilt, but I can't tell from the sample scripts how it's being done. And I'm still playing with motion interp filters, as you suggested earlier.

The artifact is definitely more perceptible on more brightly calibrated setups, and it doesn't look that bad if you consider the original source quality LOL

Quality? I second that LOL. The blobs are easier to spot on a PC. On the plasma or CRT, not so obvious. Kinda frustrating that I could get this far, then hit speed bump blobs every few minutes.

In the original video, only the top field is damaged, the lower field isn't; but in the "fixed" video, since you replaced it with a blend, both top and bottom fields have to be replaced.

I saw that. NeatVideo's interface shows interlaced video like SeparateFields(). It's obvious that one of the video heads was ready to fall off the drum. But I know what you mean. BadFrames and even FreezeFrame were expedients. The opening scene of Act2 Scene1 has 1049 frames, 310 of them busted in some way. A day's work for 33 seconds of video. If I had a spare month just for that, I think I could have learned more about motion interp/MVTools.

This scene is from diskette #1. Diskette #2 is worse. I fixed maybe 75% of the motion noise, judder, and breaks. Oh, well...This will be one of those projects whose pieces one keeps going back to again and again. I'll package the TCFH pretty much as-is. But I'll keep looking. I'm bound to run into something like this again.

[poisondeathray]

Here is an idea that might work to help treat your "bouncy titles":

You can use strong temporal filter to average out the titles , and to limit the damage to the rest of the "good parts", composite the "fixed" titles back onto the "good" version. It's the same idea as the decolored overlay for the colored borders - you want to limit the damage by composting. You could do this with stackhorizontal, stackverticals, or if you have AE or even NLE's you can use more refined non rectangular masks... I'm sure you get the gist of it...

This is just a quick proof of concept example that has been bob deinterlaced, where the titles have treated with QTGMC, TTempSmooth with strong settings, then overlayed on top of the original titles. (In vdub, "temporal smoother" would be similar to TTempSmooth). There are a couple blips that you can smooth out with tweaked settings or even with mflowinter, and for the real project you can reinterlace and place back on the "fixed interlaced" version.

[sanlyn]

Gotcha, poison. Great Idea, and nice demo. (See? Toldja I'd be coming back to this godawful project again and again). The opera opens with 2 minutes of slow dissolves overlaid with horrible opening titles. Ends with 5 minutes of curtain calls + credits. Last couple of minutes of credits roll vertically (Hmm, now there's a challenge). Wish I had AE and a PC to run it, but I'm about $1500 short. Thanks. Will give it Try. It's already giving me ideas.

[poisondeathray]

Gotcha, poison. Great Idea, and nice demo. (See? Toldja I'd be coming back to this godawful project again and again). The opera opens with 2 minutes of slow dissolves overlaid with horrible opening titles. Ends with 5 minutes of curtain calls + credits. Last couple of minutes of credits roll vertically (Hmm, now there's a challenge). Wish I had AE and a PC to run it, but I'm about $1500 short. Thanks. Will give it Try. It's already giving me ideas.

You don't need AE for this approach, any decent NLE can do basic compositing with keyframed masks (vegas, premiere). You can do it in Blender for free as well, but there is a bit of a learning curve.

Scrolling credits on black "should be" easy to do - you shouldn't even have to worry about masks (there's nothing to "damage" with the surrounding black areas) . Of course "should be" = famous last words

Scrolling credits on real stuff like actors & the stage requires animated masks (a real b!tch to do in avisynth IMO, but it's possible with animate() ) .

Static credits that fade in/out will have a side effect of longer duration fade ins/outs depending on the settings used for the temporal smoother. The higher the "maxr" setting, the more frames it samples/averages from, the longer the fade in/out compared to the original - but I don't see this as a problem - there is enough gap between the credits timing-wise so they shouldn't "bump" into one another.

Glitches , bad frames , other defects , should be "fixed" before applying TTempSmooth or equivalent smoother, otherwise the glitches & errors get propogated into other frames . So you would probably be better off fixing the titles last (after applying the other fixes, color correction etc...)

TTempSmooth seems to work much better on the titles than vdub's temporal smoother

[poisondeathray]

Debugmode wax is another free compositor, there are tutorials for masking, using mattes on youtube and other sites

[jagabo]

There must be some AviSynth plugin or script that can handle the problem of "simmering blobs" without destroying video.

You might try a stronger temporal filter on the darker areas. Something like:

SeparateFields()
light=TTempSmooth(maxr=3, lthresh=10, cthresh=15, strength=3) # light filter version
heavy=TTempSmooth(maxr=7, lthresh=20, cthresh=30, strength=7) # heavy filter version
mask=GrayScale(light).Invert() # make an alpha mask from the luma channel inverted
# you may want to further adjust the brightness, contrast, and/or gamma of the mask
# overlay the heavy filtered clip onto the lightly filtered clip using the alpha channel
Overlay(light, heavy, 0, 0, mask)
# the dark areas will be more heavily filtered than the light areas
Weave()

[poisondeathray]

Also for the "simmering blob" , I noticed the black level was a tad high - there is a bit of room to bring it down and that would disguise the "blob" a bit better as well