It's a shame the movie "Lili" (1953) with such beautiful color work has to get short changed with inferior VHS editions and an even worse laser disc. I'm making a 2nd effort at restoring the movie's color from my 2 old VHS tapes. I've learned a little in this forum since I first tried this 4 years back. The problem here is the opening titles. Both tapes had the titles devoured by a faulty VCR (since dumped, and good riddance).
Frankly, the VHS titles aren't just noisy; they're really, really ugly. I have 4 cable broadcast recordings (I think all cable stations show the same crummy laser disc version) with intact titles and so-so audio. I borrowed the titles from my 7500-bitrate DVD recording off a PBS station. They're the cleanest I've seen (that still doesn't say much for the LD). What the LD titles have is some noise in the margins of the MGM lion's roar. But worst is the spots and tiny explosions during 1-min 13-sec of the movie's start.
After struggling for a week with every de-spot filter I found, I devised an AviSynth script that de-spots the hard way: frame by frame -- field by field, in fact. The caps below show some of the debris that the laser disc people call "digital remastering". IMHO they just copied some bad film stock without using so much as a dust cloth.
The MGM lion's name isn't Leo, BTW. It's "Jackie" (Yes, Mulder, the truth is finally out there).
The opening credits are just as bad:
Here is a sample NTSC MPEG2 of cuts from the titles - not the whole thing, just some of the worst frames. 11.7 MB (No audio, from the unmodified DVD recording).
http://dc426.4shared.com/download/PqmKlPjf/Sample_Spots.mpg
If you care to see all 13 minutes of the original, unchanged titles that I just spent 3 days with, here 'tis (NTSC DVD, no audio, 46-MB):
http://dc255.4shared.com/download/hHacWlrU/TitleA_AllSpots.mpg
Here are samples of the script I came up with. The DVD was extracted to AVI with DGIndex. All work was in YUY2. Before bobbing the clip, a few lines of FreezeFrame and BadFrames were used for frames having no motion and big spots covering only one frame. I couldn't use those commands with Jackie the Lion or during any dissolve or fade.
What the script does with bob'd fields is to use the variable "a2" (the original input clip) to move bobbed fields around to cover bad frames. Then the "b0" variables are used to contain small square patches of "clean" area that are overlayed onto the current, progressively-built output clip (the "a4" and "a6" variables). Yeah, I know: not kosher. But the overlays are very small, barely 4 to 8 pixels square. Later denoising makes them invisible.
The result of de-spotting Jackie the MGM Lion:Code:MPEG2Source("E:\Videos\LIL2\LIL13_56k\TitleE\LIL56k_TitleApcm.d2v") ColorYUV(off_y=+1,cont_y=-15.0) # --- skip the easy few easy lines using Bad Frames() AssumeTFF().Bob() a1=last a2=a1 # -- example of operation on one spot that appears # -- on 2 frames (3 consecutive bob'd fields) b0=a1 b1=a2.FreezeFrame(99,99,97).FreezeFrame(100,100,98).BadFrames(101,blend=true).Crop(520,90,-186,-376) b2=Overlay(b0,b1,x=520,y=90) # -- 99-101 a4=ReplaceFramesSimple(a1,b2,mappings="99 100 101") # -- example of operation on 5 bad areas that appear # -- on 2 frames (4 consecutive bob'd fields). # -- "aJ61" below is clip of accumulated "clean" frames. b0=a6J1 b01=a2.FreezeFrame(1136,1136,1135).FreezeFrame(1138,1138,1140).BadFrames(1137,1139) b02=b01.Crop(162,246,-522,-212) b03=b01.Crop(198,254,-508,-206) b04=b01.Crop(212,268,-474,-206) b05=b01.Crop(246,258,-418,-152) b06=b01.Crop(312,338,-360,-128) b07=Overlay(b0,b02,x=162,y=246) b08=Overlay(b07,b03,x=198,y=254) b09=Overlay(b08,b04,x=212,y=268) b10=Overlay(b09,b05,x=246,y=258) b11=Overlay(b10,b06,x=312,y=338) a6J=ReplaceFramesSimple(a6J1,b11,mappings="[1136 1139]") # -- after more than 50 of these routines, go here # -- and then feed the output to a very low-octane # -- NeatVideo for mild smoothing and some color filters. New1=A7i.AssumeTFF().SeparateFields().SelectEvery(4,0,3).Weave()\ .ConvertToRGB32(matrix="Rec601",interlaced=true) return New1
NTSC MPEG2, 6 seconds, no audio, 3.5-MB
http://dc234.4shared.com/download/_feM7ikb/Jackie_fix.mpg
After three days of this unconventional script, here's 1-minute 13-seconds of the opening titles -- with audio and everything!
NTSC DVD, 13-min, Dolby audio, 55.4-MB. It's anyone's guess what the background colors look like (they keep changing). I made the letters look like those in the final "The End" titles.
http://dc223.4shared.com/download/i61jCet5/TE3.mpg
Probably broke every rule in the AviSYnth Bible with that script. Anyone wishing to haul me onto the carpet for this method, have at it. While you're at it, consider what might be done with all the noise in the margins of the MGM logo shot. I have many classic color film recordings, and that noise appears on quite a few shots of this 1950's logo. I just might have to buckle down and learn some masking techniques in After Effects to clean this:
You might also note that the opening titles change color every 10 seconds or so. This happens on the VHS and the LD broadcasts. It gradually clears after about 10 minutes of the movie. I don't know what causes it, but autolevel filters made it look awful.
Try StreamFab Downloader and download from Netflix, Amazon, Youtube! Or Try DVDFab and copy Blu-rays! or rip iTunes movies!
+ Reply to Thread
Results 1 to 30 of 212
Thread
-
-
Shouldn't this be IVTC'ed instead of bobbed ? unless they screwed up or processed the titles and credits differently than the main movie ? but usually only animated features do that
-
Haven't done that in years (and had great problems with it). If I'm just covering spots, what advantage would that be? (Serious question, actually). After the titles fade, I won't use any more of the LD video.
I did play around with separating fields, etc., and the LD broadcasts all do the same things: Fields seemed to have been reversed, progressive, interlaced, back and forth. They must have put the LD together from so many different sources. . .it would be a lifetime project just figuring out how many ways it was patched together. And just too many pops and sparkles, all the way through. -
Film vs. video. Film sources will be progressive 23.976. Deinterlacing will degrade quality, you will lose resolution and generate deinterlacing artifacts. IVTC will reverse the telecine process (removes the 3:2 pulldown) and give you the original progressive frames
Sometimes credits/titles and intro sequences have different cadence ( maybe interlaced) than the main movie, often anime is like that; but for standard film / hollywood productions, usually the intro/outro/credits are film rate as well
Most "automatic" filters work better on progressive content. If this is hard telecined, then temporal filters, dirt removal, temporal cleaners etc.. won't work as well because of the repeats. They look at differences between frames. IVTC will field match and decimate the dupes, so you have the original progressive sequence to work with.
But if you are doing things manually field by field, not using "semi automatic" filters, and keeping hard telecine (Treating it as interlaced), then there would be no advantage to the restoration, but even when you re-encode interlace is not as efficient as progressive (25% more content, and motion vectors are truncated). It's better to encode progressive as progressive and use soft pulldown flags for DVD
I'm busy right now but I'll have a closer look later, I'm just guessing it's a film source, not interlaced -
It's certainly telecine, there are duplicate fields and smears everywhere (some with spots or raspberries, too). I might give that a try this weekend. Problem would be interlacing and keeping the same pace; the music is continuous for the intro and mixed with dialog just a few frames later. But, then, there are title screens that just sit there long periods. . .
BTW, two short camera shots later, the fields look reversed for one scene, then continue TFF in the next. Brother! The VHS doesn't look that way.
That MGM lion isn't the same size or colors as the one usually seen in the early 1950's. Wonder where they got it? I've seen it before, with the edge noise. -
Yes, IVTC first (TFM().TDecimate()) then use RemoveDirt(). That will take care of most of them.
http://www.removedirt.de.tf/
http://avisynth.org.ru/docs/english/externalfilters/removedirt.htm
RemoveDirt() also degrains. I have a slight mod that only removes spots:
Code:function RemoveSpots(clip input, bool "_grey", int "repmode") { _grey=default(_grey, false) repmode=default(repmode, 16) clmode=17 clensed=Clense(input, grey=_grey, cache=4) sbegin = ForwardClense(input, grey=_grey, cache=-1) send = BackwardClense(input, grey=_grey, cache=-1) alt=Repair(SCSelect(input, sbegin, send, clensed, debug=true), input, mode=repmode, modeU = _grey ? -1 : repmode ) restore=Repair(clensed, input, mode=repmode, modeU = _grey ? -1 : repmode) corrected=RestoreMotionBlocks(clensed, restore, neighbour=input, alternative=alt, gmthreshold=70, dist=1, dmode=2, debug=false, noise=10, noisy=12, grey=_grey) return corrected }
Last edited by jagabo; 11th Nov 2011 at 19:03.
-
I tried RemoveDirt (and a dozen others), and several variations of same. Plus a handful of bobbers. They all did something a little different, but I still had plenty of work to do. I don't mind the little pixel patches, but the last time I used IVTC/Decimate w2 of every 5ork 'way-back- when was to get progressed video for PC. This project is going into DVD, so I'm reading up on how to cut out duped fields but still end up with 29.97 fps interlaced DVD that lasts for exactly 4470 interlaced frames to synch with music and dialog. That's the part I never touched on. Reading up on that now.
Last edited by sanlyn; 22nd Dec 2011 at 09:15.
-
TFM().TDecimate() will leave you with 23.976 fps progressive film frames. You encode that as 23.976 fps progressive MPEG 2 with 3:2 pulldown flags. That will retain the original running time.
RemoveDirt() and RemoveSpots() don't work well during fade ins and fade outs, or when motions get too large.
Some samples where it worked well (original left, RemoveSpots() right):
Import("C:\Program Files (x86)\AviSynth 2.5\plugins\RemoveSpots.AVS")
Mpeg2Source("Sample_Spots.d2v", CPU=2)
TFM(d2v="Sample_Spots.d2v")
TDecimate()
StackHorizontal(last,RemoveSpots(last))Last edited by jagabo; 11th Nov 2011 at 20:12.
-
An IVTC will cut the frame count but the length stays exactly the same (fewer frames playing at a lower framerate). Retaining the audio synch is a non-issue. You do want to IVTC the thing.
-
Some deflicker plugins work on more than luminance only, so you could try smooth out the color changes that way. Donald Graft's deflicker plugin in vdub does this IIRC
You could remove the logo "TVPG" using various logo removal techniques . The background plate is supposed to be static, so one way you could do it is stabilize the frame and overlay that section with a static cropped frame (or alternatively motion track a replacement area if you want to keep that old film look where the frame bounces around) -
I'm still giving IVTC a try this weekend, and thanks for your guidance. Encoding might be tricky; the 1-min 13-sec segment is actually 2 segments, each color corrected separately and joined with a dissolve in AviSynth during the blackout starting in frame 208 (there's only 24 frames to work with). The lion shot is too dark (with crushed blacks), the titles too bright; one is too green, the other needs blue. The fade to black at the end of the titles fades up quickly to the first real-world shot of the movie -- which is on the much-better-looking VHS capture thru the end of the flick.
RemoveDirt did remove some of the junk, but RemoveSpots() never showed on any Avisynth filter search -- which is why I never found it, so I didn't try it. I'll give that a run. Versions of RemoveDirt that I used left many biggies around, but the overall image looked more crispy-clean (those versions called a chroma cleaner).
About 1/2 of the bad guys occur during dissolves and fades.
MCRemoveDirt doesn't show up anywhere, except in a brief doom9 post and this line from RemoveDirt's html:
The "neighbour2" is for using RemoveDirt in combination with motion compensation filters like MVtools (see MCRemoveDirt below)
The html makes no further mention of MCRemoveDirt or mvtools....Whooops! Hold on. It's name is "RemoveDirtMC", not MCRemoveDirt. I've had that script on my 'puter for quite a while. Just found it (that'll teach me not to doubt jagabo). I knew I'd seen it somewhere! You'd think the author would remember the title. It appears to call plugins that work only in YV12 -- I was hoping to avoid that, but I'll give the script another look.
... And to think I was halfway thru this movie when I just got tired of looking at those sparks and pops. Serves me right for being so picky. -
Quoting my own silly remarks:
What was I thinking? That's the way I did it the first time -- color corrected in AVI, joined in Avisynth, then encoded! I'll just do it that way again.
Must be time for my cookies 'n milk and 'jammies.Last edited by sanlyn; 11th Nov 2011 at 21:57.
-
Thanks for the tip. I was considering that. Autogains/auto levels just a made a mess, especially at the fades.
The logo never bothered me. Maybe because I have hundreds of recordings with "TCM", "PBS", etc., popping up now and then. I have a feeling I won't remove the grainy noise around Jackie the Lion without learning about masks in After Effects. That would be a good time to go deeper into AE. I've had AE a whole month arready, and just now cracking the book! And I have yet to get past Page 1 of Mark Christiansen's AE book!
If I do get after the TV/PG, I'll have to work around the fade-in and fade-out. I'm sure there's a way. -
You could also try using UtoY() and VtoY() and run regular Y deflicker filters. Then merge the chroma channels back in with the luma with YtoUV().
<edit>
After thinking about this a bit more, it probably won't work. U and V are based around 128 (ie Y=U=128 --> gray). Using deflicker filters for Y on the U and V channels will result in color shifts if the base gets moved up or down.
</edit>Last edited by jagabo; 11th Nov 2011 at 22:46.
-
I'll take your word on that, poisondeathray. I see scripts moving U or V to Y, vice versa, and I have no idea what the purpose is. But whenever I see it mentioned, I keep looking for sources somewhere that tell me what's going on with this method. If you or somone in the know gets a chance, I'd appreciate a reference to some site, book, article, whatever, explaining it. I see it mentioned too often to let it pass. Thanx in advance.
-
I think I can remove that noise in script. Both the stuff around the lion and the spots. Not at this moment though.
As for utoY, I use it all the time. For one reason, to do calculations between color channels, for example to convert to YSL, in one of my scripts. It's a workaround because we have a great plugin to do fast arbitrary calculations, but it only works on the respective channels between clips, not within clips.
Another reason is to view the chroma easier. For example today there was a problem with chroma 'lagging' luma, which looked like color ghosting. I had to step through a chroma only image to see which luma frame it corresponded with.
I've used it to generate test charts for color frequency response. I've used it to create a pseudo component format video for recording on a normal VHS. I've used it to decode the image of a special technique I invented to let any video card record component video (even with no compoonent inputs). And so on. -
Basically, UtoY() and VtoY() create grayscale images from the chroma channels. YtoUV() is the complementary function to convert the grayscale images back to chroma channels.
-
OK, I have the general idea idea about moving YUV channel info. My knowledge gap lies in the details of channel data, but that's something I'll have to look up on my own. I recall having downloaded a PDF onto another hard drive somewhere, on my "other" video PC in storage. Shucks. Looks like I still have a lot to catch up with (sorry for that double preposition, folks, but I'm running outta time on this task).
Last edited by sanlyn; 12th Nov 2011 at 08:43.
-
To visualize the three channels of YUY2 video:
StackHorizontal(Greyscale(), UtoY(), VtoY())
StackHorizontal(Greyscale(), StackVertical(UtoY(), VtoY())) -
Ok, that's easy to explain.
Think of a rainbow. Now twist that into a color wheel. Now U is the x axis of a particular color and V is the Y axis. -
Yes, I familiar with color wheels and x,y coordinates (and grids, CIE charts, cubes,etc. Was seeing them in my sleep for a while). The StackHorizontal routine could be instructive. Need to find a way to get actual numbers on what that would show me and how they'd be manipulated. Going over some old masktools scripts, I might understand them a little better now.
Been running TIVTC all morning. Softens the image quite a bit, so I couldn't use it for other scenes from the cable/LD version (I'm only using the titles anyway. The LD version is soft to begin with). RemoveSpots got 'em all except 7 booboo's, 6 of which are in fades or dissolves. Jackie's yellow spot is still there (?!). So at least I'll just use 7 of the overlay routines from my script instead of 50-plus of them. The pixel positions will be the same, I'll just change the frame numbers. Also, this eliminates using BadFrames() on 35 non-moving images.
Looking over the low-octane NeatVideo I used before: it's mainly used as a smoother here rather than major denoiser, because its temporal filter does clean work on fades without making them grainy or choppy. Some low-power NV sharpening (which includes a "smooth edges" setting) cleaned up the file-tooth edges that showed up on the titles after IVTC, and tightened up the lines without increasing grain. I had to make a new noise sample patch for NeatVideo; the noise on the test frame looked different after IVTC. The test frame isn't in the posted video, it's from dark gray frames that precede the movie's start.
TMPGenc matter: I made a 29.97 fps MPEG in TMPGenc Plus, then added the audio in TMPGenc Editor. Sync was OK. But TMPGenc Plus 2.5 wouldn't let me set the following:
Frame rate: 29.97 fps
Video format: NTSC
Encode mode: 3:2 pulldown when playback
The error message I get is "For encode mode, 3:2 pulldown when playback can not be used if the flame [sic] rate is not 23.976fps or 24fps"
Confusing message, because the input AVI's frame rate is already 23.967 fps. I think they refer to the output "frame rate" above, which is set for 29.97. So I used these settings:
Frame rate: 29.97 fps
Video format: NTSC
Encode mode: interlaced
The resulting MPEG is 4469 frames interlaced, exactly like the pre-IVTC clip, and audio sync added later was perfect. Using VDub's bob filter, I see that TMPGenc performed the usual 3:2 in every 5 frames. This is different and more consistent than the duplicate-frame pattern I saw when bobbing the original clip. The original clip had a bobbed duplicate image pattern wherein every second image appeared as 3 duplicate bob's. I don't think I've seen that before.
The IVTC is softer, but has less sandy noise and less "gunk" overall. Didn't get a chance to run RemoveDirtMC yet.
Now the wife's after me about Saturday chores. Later. -
I've never seen TFM().TDecimate() soften the picture. All it does is weave fields together then pick out the frames that don't have comb artifacts.
You don't set TMPGEnc to 29.97 fps. You set it to "23.976 fps (internally 29.97 fps)" and set Encode Mode to "3:2 pulldown when playback". What you did by setting the frame rate to 29.97 fps is duplicate every 4th frame. -
It's the other way around. For progressive content, Bob() will soften and degrade the image much more than IVTC. This is more noticable on higher quality material, but you can still see the difference here. Click and Save the images to the desktop at full resolution and flip back & forth
IVTC done properly will just return the identical progressive frames. You shouldn't deinterlace progressive content. Bob will reduce the resolution, and introduce aliasing artifacts . In fact dumb bob() is alot worse than other smart deinterlacers, it reduces resolution on everything, every frame, instead of using motion compensation or trying to interpolate. Theoretically, interlaced content with no motion should be weaved, so you get same resolution as progressive.
Not sure about TMPGEnc, but it sounds like you're encoding interlaced, defeats the purpose and wastes bitrate, lower quality -
OK, IVTC looks somewhat cleaner.
Now we have a problem. The remaining 1-hr 23-min of the movie is 29.97 fps interlaced NTSC DVD. I'm going to patch a non-interlaced 24fps IVTC clip on the front of that DVD? -
It's not a "cleaner" vs. "more dirty" issue . It's primarily a vertical resolution loss vs. original image resolution issue, and secondarily an aliasing issue. Bob() will cut vertical resolution in half. You might not see a big difference in this example because the original resolution (VHS sourced) is so low to begin with
This is from another thread (questioning whether or not youtube deinterlaced progressive material). Compare screenshot 1 vs. 3, especially section 1 for the vertical resolution
https://forum.videohelp.com/threads/340249-program-to-optimize-for-youtube?p=2119597&vi...=1#post2119597
Now we have a problem. The remaining 1-hr 23-min of the movie is 29.97 fps interlaced NTSC DVD. I'm going to patch a non-interlaced 24fps IVTC clip on the front of that DVD?
I 'm not entirely sure, but I don't think it's possible to join soft telecine section with hard telecine section -
I get your point about Bob(). I've since cranked up yadif into the plugins folder.
OK, fans, it's recap time. The movie LILI is on NTSC VHS. I had 2 copies. One's titles got chewed up by an aging JVC player, so they're useless. The other tape is otherwise OK but its titles had damage as well. The VHS movie was captured via s-video to YUY2 AVI thru a Panasonic DVD recorder as pass-thru TBC, a PA-1 proc amp, and ATI AIW 9600XT in VirtualDub. They look OK, after the titles.
The 1-min 13-sec title I'm attaching to the almost-done VHS movie version is from a PBS cable broadcast. The recording I'm using came off analog cable (not thru the box) directly to a Toshiba RD-XS34 DVD recorder at 90-minute (7500) bitrate. It appears to be a broadcast of the old laserdisc version, with a greenish undersaturated image that often looks worse than VHS. I have other recordings, but they are at crummy low-bitrate transmissions that "digital" cable companies jokingly call an improvement.
Can't get back to my main PC now, the wife has me vacuuming a damn closet (wives just don't understand that this project should have priority!). I'll try step 1 again with yadif and RemoveSpots a bit later. -
Yadif will cause the same problem - it's still deinterlacing it. It might be marginally better than vanilla bob(), but you will still lose resolution and it will generate artifacts. In some cases bob() is actually better than yadif() - IF you are deinterlacing an interlaced source. (By "interlaced" I mean content, not method of encoding).
Again, there might not be a huge visible difference on this source, but it's an indisputable mathematical resolution loss when you deinterlace progressive material. By the time you denoise it , the difference will be even less. But for best practices treat progressive as progressive and use IVTC, it's up to end users if they want to make any trade offs. But I don't see any reason NOT to IVTC progressive content, telecined material. Conceptually , you usually don't want to throw away 50% right off the bat
Moreover, deinterlacing without decimation will yield frame repeats (every 5th frame), so your dirt or spot removal filters, or temporal noise removers, etc. won't work as well
You could IVTC edit/filter as progressive then re-introduce the hard telecine to join the mpeg2 segments. If you have a choice, progressive encoding is much better, higher quality, more efficient than interlaced
PS how does "LILI" compare to "GIGLI" -
Which DVD? You mean, the VHS-> to -> DVD transfer? 24fps progressive will play on all NTSC DVD players and CRT TV's?
-
If it's noticeably softer after the IVTC, then perhaps the deinterlacer is kicking in too much. You might try putting on 'Display=True' to find out if it's being deinterlaced and when. You can also have it, when necessary, use a different and better deinterlacer. It's explained in the TFM doc.
Yes, it's quite easy. Just make sure the one encoded as progressive has had pulldown applied so both sections output interlaced 29.97fps. You mux/author using Muxman and open the earlier section, hit the 'Add' button, and then select the next section. Depending on the kind of scene where the join is, it's often better to have encoded the last frame of the first part as an I-frame or sometimes you'll get a couple of frames of corruption. I join sections encoded differently quite often (like, for example, an IVTC'd video joined to interlaced credits, or an IVTC'd movie joined with IVTC'd end credits encoded in lower quality).
But none of that should be necessary because, by definition, movies aren't interlaced 29.97. And as the others have asked, why can't you IVTC the rest? Is it time for a sample of this interlaced 29.97fps section? Does it not display the telltale 3 progressive/2 interlaced in every 5-frame cycle?
Yes, once pulldown has been applied to make it DVD compliant.
Similar Threads
-
Set "Output filename" As Default Global "File/segment title" In MkvMerge?
By LouieChuckyMerry in forum Video ConversionReplies: 0Last Post: 9th Jul 2011, 02:52 -
Mini DV + S-VHS Combo Decks: "SR" Pro Series vs "HR" Consumer Series
By jbd5010 in forum CapturingReplies: 0Last Post: 12th Apr 2011, 23:34 -
WMV files: Changing "Recorded Date", "Media Created" fields in metadata
By axhack in forum EditingReplies: 5Last Post: 18th Sep 2010, 02:27 -
"stretching" or "cramming" aspect ratio when I convert .avi to MP4
By Agent Bauer in forum ffmpegX general discussionReplies: 3Last Post: 21st Mar 2010, 11:30 -
JVC S-VHS machine w/ stabilizer "on" plus inline TBC = disaster?
By justin81 in forum RestorationReplies: 8Last Post: 7th Oct 2009, 01:23