Help to get rid of duplicate frames on PAL source

[rhaz]

Hi. I got this old DVD of a movie I was looking for some really long time. And I like to digitize my video collection, but this one is very bad source. I tried using Decimate and Srestore various settings, but didn't help, I either get missing or duplicate frames or both at the same time.

I attached a 15 seconds long source file. Please someone who has time, maybe you can try encode it with correct settings and get best of it and then paste me the settings you used on your .avs script?

Thanks,
Cheers.

[jagabo]

Use whatever bobber you want then Tdecimate(mode=2, rate=23.3) comes very close. I saw only one duplicate in the sequence, no drops. So maybe 23.25 or 23.2.

[rhaz]

Hi, I am new to this. What's a bobber? Google gives bunch of motorcycle websites.

[jagabo]

What's a bobber?

In AviSynth: Bob(), Yadif(mode=1), QTGMC(), Interleave(TFM(field=1), TFM(field=0)), etc.

Of course, you wouldn't want to use Bob() because it's very low quality.

[rhaz]

So you mean a deinterlacer? I recognize Yadif from this list.

[jagabo]

A "bob" is a double frame rate deinterlace. Ie, it converts your 25i source to 50p.

Mpeg2Source("test-001_track1_und.d2v", info=3)
AssumeTFF()
Trim(2,0) # open GOP has two junk frames at the start, skip them
QTGMC(preset="very fast") # 25i to 50p
Tdecimate(mode=2, rate=23.27) # remove duplicates

or

Mpeg2Source("test-001_track1_und.d2v", info=3)
AssumeTFF()
Trim(2,0) # open GOP has two junk frames at the start, skip them
Yadif(mode=1, order=1) # 25i to 50p
Tdecimate(mode=2, rate=23.27) # remove duplicates

[rhaz]

40th frame is duplicate in your attached file. I tried 23.2 it gave every second frame as duplicate. I also tried 23.35 it gave like every 220th duplicate frame.

[jagabo]

I'll leave the fine tuning up to you.

[sanlyn]

yadif is better than Bob(). QTGMC is miles ahead of both, but slower: http://forum.doom9.org/showthread.php?t=156028

40th frame is duplicate in your attached file. I tried 23.2 it gave every second frame as duplicate. I also tried 23.35 it gave like every 220th duplicate frame.

Then it qwould appear that the original source is film # 23.876 or 24fps, and some form of pulldown was applied at one point to bring it up to 25 fps.

[rhaz]

I want to try that QTGMC, but I cannot get SSE2Tools.dll from Google. The homepage link is dead. Could you share it please. It comes with RemoveGrain.zip

[jagabo]

I want to try that QTGMC, but I cannot get SSE2Tools.dll from Google. The homepage link is dead. Could you share it please. It comes with RemoveGrain.zip

SSE2Tools.dll in ZIP attached.

40th frame is duplicate in your attached file. I tried 23.2 it gave every second frame as duplicate. I also tried 23.35 it gave like every 220th duplicate frame.

Then it qwould appear that the original source is film # 23.876 or 24fps, and some form of pulldown was applied at one point to bring it up to 25 fps.

I don't now where you come up with 23.876. If 23.35 results in one duplicate every 220 frames then 23.35 * 219 / 220 = 23.24386 would be about right. Of course, TDecimate() isn't always perfect.

[sanlyn]

All of the plugins required by QTGMC are on the doom9 page I linked to earlier. Look for the text "Plugins package" in RED on the linked post. The package has instructions as well as plugins for Avisynth 2.5 and 2.6 (use the 2.5 plugins. 2.6 is hassle on many systems). http://forum.doom9.org/showthread.php?t=156028

The mpg has some pretty ugly red chroma noise and low-bitrate artifacts -- another reason I'd never record analog tape directly to lossy MPEG2, especially at a low 3900 bitrate. But if that's the only source available, one has to work with one one gets. I cleaned up many of the compression artifacts (and QTGMC itself will get some of itas well ), but the red flutter won't go away (look at the man's dark coat).

[rhaz]

I am sorry I run on Windows 8 and most of the vob cutting tools I used do not work properly or don't work at all. So I tried some tool after cutting .vob sample gives it in .mpg. So if you know any working .vob cutting tool for making short samples I would like to know the titles.

Also I tried this QTGMC tool and I still get duplicate frames. Using rate 23.27 the ~220th frame gets fixed, but the 41th frame still duplicates. And I tried all values from 23.20 to 23.35

There's my full .avs script:

LoadPlugin("E:\DGDecode.dll")
LoadPlugin("E:\TIVTC\TIVTC.dll")
LoadPlugin("E:\mt_masktools-25.dll")
LoadPlugin("E:\mvtools2.dll")
LoadPlugin("E:\nnedi3.dll")
LoadPlugin("E:\RemoveGrainSSE2.dll")
LoadPlugin("E:\RepairSSE2.dll")
LoadPlugin("E:\RSharpenSSE2.dll")
LoadPlugin("E:\SSE2Tools.dll")
Import("E:\QTGMC-3.32.avsi")
DGDecode_mpeg2source("E:\test.d2v")
AssumeTFF()
Trim(2,0) # open GOP has two junk frames at the start, skip them
QTGMC(preset="very fast") # 25i to 50p
Tdecimate(mode=2, rate=23.27) # remove duplicates
crop(6, 0, -10, 0)
Lanczos4Resize(704,540) # Lanczos4 (Sharp)

[sanlyn]

Well, a couple of notes:

LoadPluginstatements: not needed. dll's load automatically.
Import statement: not needed. avsi files load automatically.
resize:
(a) Don't resize noisy video until it's as clean as you can get it.
(b) 704x540 is neither a 4:3 image nor PAL DVD compliant
(c) Don't use Lanczos. Use Spline16Resize, Spline36Resize, or Spline64Resize.

You'll probably never get rid of all that low-bitrate flutter and noise. The original transfer is just too poorly done.
You'll probably never get rid of all the bad frames. Not without a lot of experimentation.
The final output will have to be reset for 25FPS and the audio adjusted to match it.

[rhaz]

Good news. The correct rate is 23.178. I use that formula jagabo posted above. With 23.27 every 254th frame was duplicate, so I did 23.27 * 253 / 254 = 23.178 Now I didn't find any duplicates (at least in that sample).

So to finish this great thread off I got couple questions. Do you use VirtualDub to open the .avs and see how it looks or do you have to encode it and then open the container in VirtualDub and see the result? Because now opening .mkv it gives some crazy stuff, with rate less than 23.25 encoded video in VirtualDub shows me that every second frame is a dupe which is invalid.

What's wrong with Lanczos? And what's so good with Spline? And what those numbers mean 16 36 64?

Why to reset for 25FPS? How's that even possible after applying such .avs?

And how DLL's load automatically and etc? I use MeGUI, then save the .avs with it and then I edit it with notepad and add the correct plugins I need. With MeGUI I get size resolutions and etc to get a starting .avs to work on.

[jagabo]

You might try adding the m2PA=true arguement to TDecimate().

Mpeg2Source("D:\Downloads\test-001_track1_und.d2v")
AssumeTFF()
Trim(2,0)
QTGMC(preset="very fast")
Tdecimate(mode=2, rate=23.24386, m2PA=true)

[sanlyn]

I can't advise on MeGUI. I had nothing but hassles with it, and one-button GUI's never seem to do what I want. I use separate apps for every processing step, including edit, encoding, etc. Like most advanced users I don't use VirtualDub as a link to encoders. The encoders I use are HCenc, TMPGEnc Plus 2.5, TX264, X264 encoder, and TMPGenc Mastering Works 5. My editors are VirtualDub, TMPGenc MPEG Editor v3, and TMPGEnc Smart Renderer 4, and a lot of cutting, joining, frame rate adjust, pulldown fixes, etc., are with Avisynth and something like DGPulldown.

I didn't play with the frame rate yet (I'm still struggling with that low bitrate noise and the ugly red stuff!). But basically you could take the 23.178 frame rate and use AssumeFPS() to set it at PAL_Film, which is 24 fps. I also extracted the audio by opening the mpeg in VirtualDub and saving only the audio as uncompressed .wav PCM (which would be saved at the original frame rate of 25 FPS). I haven't tried it yet (I don't have audio in your sample), but one could open the 12.178 fps AVI in Avisynth, join the .wav audio to it with AudioDub(), then use AssumeFPS("PAL_film",sync_audio=-true), which would sync audio and video at 24FPS. Then you would encode that to PAL DVD and lie to the encoder, telling it to encode at 25FPS. Finally, you could use DGPulldown and lie to that app, too: tell it to add pulldown to adjust 24FPS to 25FPS; the audio sync would adjust automatically.

Yes, that is tricky. But it works.

Getting extracts from VOB's: If DGDecode worked in your script in Win7/8, then DGIndex should work to extract video samples from VOB's. The audio doesn't always come thru as desired (OK, so DGindex ain't perfect). But here's a recent guide to how it's done: https://forum.videohelp.com/threads/359295-Record-without-interlacing?p=2272359&viewful...=1#post2272359

For mpg samples I use something else that is more convenient and let's me get audio with it: TMPGEnc Smart Renderer 4 or TMPGenc MPEG Editor v3. I'm sure users of free tools could offer other suggestions.

[jagabo]

Because now opening .mkv (in VirtualDub) it gives some crazy stuff, with rate less than 23.25 encoded video in VirtualDub shows me that every second frame is a dupe which is invalid.

That's a problem with the ffmpeg source plugin for VirtualDub.

[rhaz]

Thanks for all the help, but I will probably stay with the result I get with that final .avs I am not so picky as you on picture quality ^_^

[sanlyn]

Then why bother? I played that sample in VLC and MPC-BE, and it looked OK to me with deinterlacing turned on.

[sanlyn]

[QUOTE=rhaz;2279648I am not so picky as you on picture quality ^_^[/QUOTE]Neither was the person who made the original transfer. Anyway, since I put some time on it.....but still didn't get exactly what I was looking for. Can't have everything, but always looking for another learning experience.

[rhaz]

Wow sanlyn that picture quality is sick. Would be nice to see .avs and the tools list. And it has no duplicate frames?

Anyway I came back here because I just finished encoding the entire movie which took 8 hours with QTGMC... Why not just Yadif? So I finished it and now found out that every switching scene has a duplicate frame! In this sample we did there was no switching scenes, sadly... So in overall the movie has no duplicate or skipping frames while watching the scenes, but when scenes switches it has an extra frame in which you can see the new scene coming in and the leftovers of the previous scene.

What do I use for removing those?

[manono]

So I finished it and now found out that every switching scene has a duplicate frame!

You mean a blended frame, right? At the scene change there's a blend of frames from the last and next scenes? That's the price you pay for crappy sources such as yours. All of us that work with field-blended garbage have it. The unblender didn't cause it. It's in the source. There's a fix, but you might not like it:

prev = restore.selectevery(1,-1)
next = restore.selectevery(1,1)
SCclean = restore.SCSelect(next,prev,restore,dfactor=2.0) # 2.0 ~ 5.0

return(SCclean)

It needs v0.9 of RemoveDirt. There will be no blended scene changes but it will create duplicate frames both before and after the scene change.

[sanlyn]

Wow sanlyn that picture quality is sick. Would be nice to see .avs and the tools list. And it has no duplicate frames?

Sick? I hope you mean to say "slick"?

Whatever. It does look over filtered. The source is so horrible, it's an all or nothing situation. A copy of the two-step script is attached, with some notes. Step 1 restores film speed. Step 2 does the rest. It's probably not worth this much trouble, but I saw it as a challenge. I don't know if there are duplicate or lost frames. The results played well, so I saw nothing I couldn't live with. The source is ruined anyway. Anything would be an improvement.

As for using yadif over QTGMC: there is no way you will get the same clarity, sharpness, or denoising with yadif. I use yadif for testing and for tweaking other filters. But now and then yadif proves more useful with some types of video, especially anime. It would not work so well here. I used QTGMC at its slow default setting to repair many problems. You can make QTGMC run faster by using faster presets such as "medium" or "very fast".

[jman98]

Wow sanlyn that picture quality is sick. Would be nice to see .avs and the tools list. And it has no duplicate frames?

Sick? I hope you mean to say "slick"?

"sick" can mean "very good". http://www.urbandictionary.com is your friend when the youngsters start to use expressions you don't understand. I have to use it myself at times as I'm not as young as I'd like to be.

[sanlyn]

Thanks for the tip, jman98. Guess I can't keep up as well as I used to. I'd better check the 'net on the latest jive and get with it.