Pal retail dvd with blends, looks bad

[spiritgumm]

I'm not sure how to analyze this Pal retail dvd which appears to have blended frames and looks bad on my Ntsc dvd player. I believe it's progressive.
I tried the usual simple scripts to improve it - TFM, Yadif, Yadif+SRestore, but none were right.
The film is listed on imdb as 100 minutes, the dvd is 97. I havent compared the audio to a Ntsc TV copy yet so I'm not sure if it was sped-up, or a different edited version (which does exist).
If it's improvable, I would like to reencode it for Ntsc, slowing for speed-up if necessary.
Any help will be greatly appreciated.
VTS_01_1.demuxed.m2v

[jagabo]

Yes, it has too many blends for Yadif(mode=1).SRestore() to work well. Still, it's better than nothing. Maybe manono knows something better.

[spiritgumm]

Thanks for checking it out. Given the distracting blends and odd playback on the retail, my TV copy is preferable.
I dont recall reading any negative feedback on the retail dvd quality. I wonder if it would somehow play better on European machines. Also wonder why the video would be made this way (distributed by eurocinefilms.com)?

[jagabo]

It was probably made from an NTSC video tape using a cheap NTSC to PAL converter. This doesn't look too unbearable:

Mpeg2Source("VTS_01_1.demuxed.d2v", CPU=2)
QTGMC(preset="fast")
SRestore()
MCTEmporalDenoise(settings="medium")
BilinearResize(720,480)
ChromaShift(c=-6, l=-4)

[jman98]

Thanks for checking it out. Given the distracting blends and odd playback on the retail, my TV copy is preferable.
I dont recall reading any negative feedback on the retail dvd quality. I wonder if it would somehow play better on European machines. Also wonder why the video would be made this way (distributed by eurocinefilms.com)?

Why would they do that? Stupidity I guess. I know of several foreign studios (I'm looking at YOU Ruscico and IVL) who paid big bucks for professional film restoration to be done on their catalog. I know for a fact that in in the case of IVL that they were given 24 fps masters by the remastering companies and I suspect strongly that Ruscico was too but I cannot prove it. Both companies took those 24 fps masters, which IVL still has and will license out on demand, and immediately converted them to 25 fps interlaced PAL video which they keep on hand as their day to day "master copy". They take those interlaced PAL masters and convert those, not the 24 fps originals, to interlaced NTSC. In the case of Ruscico I think that they really do know better but are just cheap. In the case of IVL, I am not sure that they know any better. I've seen a lot of Asian studios make really weird decisions about how they handle their catalogs and I just think that they really do not fully grasp the technology involved.

To be honest with you, many consumers may not care. I've seen a lot bitching here about ghosting and while I am sure that it is a super big deal to some people, it doesn't bother me at all. I'm not sensitive to it in the slightest and there are other issues that I consider to be completely unacceptable that might not bother other people at all.

[spiritgumm]

It was probably made from an NTSC video tape using a cheap NTSC to PAL converter.

This doesn't look too unbearable:

Dont be so modest, looks great! I'm just not sure i can implement all of the script on my machine. Will test.

jman98, thanks for the input. While I know the mind's eye can adapt to audio/video defects, this looks blatantly bad on too many levels to ignore.

[intracube]

Nice repair work jagabo.

spiritgumm's video looks like an already telecined NTSC transfer has been converted straight to PAL (24p->59.97i NTSC->50i PAL) instead of IVTC the NTSC version to get back to 24p, then doing a simple 4% speed up.

At least some seasons of Six Feet Under on DVD (Warner Video) suffered from a similar half a*sed conversion to PAL.

[spiritgumm]

jagabo, I started to run the script on a new 64bit PC (Sandy Bridge i5) using Avisynth 2.58. To encode to dvd video with HCenc, the first pass estimate is 30 hours. Is there a way to speed this up? Should I be using Avisynth 64bit?

[jagabo]

You can use QTGMC(preset="faster") for faster processing. Also "very fast", "super fast", and "ultra fast". Each is faster but does less processing. Or use Yadif(mode=1) instead.

McTemporalDenoise has similar settings. McTemporalDenoise(settings="low") and "very low" perform less processing and are faster.

You can try enabling multithreading by using SetMtMode(2,8) at the very beginning of the script.

If you are doing a 2-pass encode render first to a losslessly compressed (lagarith) intermediate file. That way you only perform the filtering once. Then run your 2 pass encode on the intermediate file.

[Gavino]

You can try enabling multithreading by using SetMtMode(2,8) at the very beginning of the script.

Mpeg2Source doesn't like mode 2, so better to put SetMTMode(5, 8) at the start of the script and SetMTMode(2) after Mpeg2Source().
And of course you will need a multithreading version of Avisynth.

[jagabo]

Mpeg2Source doesn't like mode 2, so better to put SetMTMode(5, 8) at the start of the script and SetMTMode(2) after Mpeg2Source().

What's the problem/symptom? I haven't noticed anything.

[Gavino]

I'm sure I've seen reports of random crashes, but in truth I can't actually find any right now.
But certainly, 'received wisdom' is to use mode 5 (or mode 3) for all source filters, see for example
http://forum.doom9.org/showpost.php?p=1312666&postcount=1.

[jagabo]

I'm sure I've seen reports of random crashes, but in truth I can't actually find any right now.
But certainly, 'received wisdom' is to use mode 5 (or mode 3) for all source filters, see for example
http://forum.doom9.org/showpost.php?p=1312666&postcount=1.

I have had an occasional crash while running AviSynthMT. I don't remember if it was when using Mpeg2Source(). I'll keep the issue in mind though. Thanks.

[spiritgumm]

I see various MT versions for Avisynth -is there one in particular I should use, like revision 0.7?
http://avisynth.org/mediawiki/MT_support_page#SetMTMode.28.29_Mode
Also near the bottom of above wiki, says mode 5 is slower than not using MT at all. If so, why use it?

Do I need to "convert" my current Avisynth to 64 bit? I only saw unofficial builds:
http://forum.doom9.org/showthread.php?t=152800

I dont expect the audio to change since the video will be the same length, but I did get an error when I loaded the nicac3source(ac3). As this is a new PC, it seems like I'll have to start from scratch, finding multiple script errors which I will have to research, etc. (ie like putting FFtw3.dll in your System folder).

I started using AvsPmod instead of the original, and it indicates I need to rename dlls which had "undesirable" symbols. Is that advisable?

[poisondeathray]

You don't "need" to use 64bit avisynth, in fact many people have problems with it, but it works ok for others

Why are you loading audio in avisynth with nicac3source? Why not just copy the original audio - this way you keep same as source quality. Unless you are making edits ?

It's safe to rename .dll's

I would advise against using MT with srestore or temporal filters, sometimes you get frames mixed up or crashes (but again , some people seem to do ok with it)

[Gavino]

I see various MT versions for Avisynth -is there one in particular I should use, like revision 0.7?

SET's build is generally regarded as the most reliable/stable version.

Also near the bottom of above wiki, says mode 5 is slower than not using MT at all. If so, why use it?

It's used to suppress multi-threading for part of a script, where there are filters that don't work multithreaded. You wouldn't use it on its own in a script.

I started using AvsPmod instead of the original, and it indicates I need to rename dlls which had "undesirable" symbols. Is that advisable?

It's just a warning and can be suppressed if you prefer (as I do) to keep the original names.
But renaming will do no harm either.

[spiritgumm]

Why are you loading audio in avisynth with nicac3source

I would advise against using MT with srestore or temporal filters, sometimes you get frames mixed up or crashes (but again , some people seem to do ok with it)

well, that's a big caveat about MT. I dont want to spend 20 hours processing to find out I get mixed frames.

As I said, I dont expect to have to process the audio, but sometimes the new video doesnt sync with the old audio. My point was I couldnt even load the audio. Also, I usually use MPC as the external player in AvsP but it wont run correctly, either. Just a pain switching to a new PC.

[poisondeathray]

Why are you loading audio in avisynth with nicac3source

I would advise against using MT with srestore or temporal filters, sometimes you get frames mixed up or crashes (but again , some people seem to do ok with it)

well, that's a big caveat about MT. I dont want to spend 20 hours processing to find out I get mixed frames.

As I said, I dont expect to have to process the audio, but sometimes the new video doesnt sync with the old audio. My point was I couldnt even load the audio. Also, I usually use MPC as the external player in AvsP but it wont run correctly, either. Just a pain switching to a new PC.

IMO, it 's not worth the headache. I'd rather have it work properly and be 10% slower (sometimes you don't even gain 5% speed boost with MT, it depends where the bottlenecks are , the particular script and specific situation)

Audio sync shouldn't be affected with that script, but I see you're just making a "plan B" just in case.

Can you run MPC / MPCHC separately to preview (ie. open the avs directly in MPC / MPCHC , not through avspmod)

[spiritgumm]

MPC itself wont play a script, and I pared the script down to just loading the d2v file. It also wont play a couple of video formats so apparently my PC is missing some codecs.

Hopefully having 800 GB free will allow me to do the first pass in lagarith as jagabo suggested, which should speed things up. It's a 97 minute movie - I'm not sure how big that will be uncompressed.

[poisondeathray]

Yes, the lossless intermediate will speed things up. Uncompressed 8bit 4:2:0 PAL is ~ 70GB/hr , expect lagarith to compress to ~1/2 that

You won't get realtime playback with that slow script in MPC (QTGMC + MCTD are slow, expecially stacked together) , so previewing avs script in realtime player is a futile exercise)

[jagabo]

sometimes you don't even gain 5% speed boost with MT

On a script similar to the one I gave earlier, SetMtMode(5,8)/SetMtMode(2) gave nearly a 3x performance increase. But there was an error message imprinted in the video -- caused by SRestore(). Changing the script to:

SetMtMode(5,8)
import("C:\Program Files (x86)\AviSynth 2.5\plugins\Deblock_QED.avs")

Mpeg2Source("VTS_01_1.demuxed.d2v", CPU=2) 
SetMtMode(2)
QTGMC(preset="fast")
SetMtMode(5)
SRestore()
SetMtMode(2)
UnDot()
MCTEmporalDenoise(settings="medium")
BilinearResize(720,480)
ChromaShift(c=-4, l=-2)

delivered a clean encode and was still 2.8x faster than the non-MT encode. Of course, it's only a short sample so I can't rule out problems with longer videos. And I didn't scrutinize the videos frame by frame so I can't rule out minor differences.

[spiritgumm]

are the script changes necessary for MT, or another reason? I know it's mainly just MT lines added, but I dont have alot of time to hunt down more plugins and debug issues when I cant even run the new PC with the old plugins, etc.

In the old script, I was curious what effect Chromashift had. To speed things up, I omitted MCTEmporalDenoise when processing the sample. I didnt see any difference with and without Chromashift. Is it required after using MCTEmporaldenoise?

[jagabo]

I was playing around with different settings. The import of Deblock_QED was for the deblocking option in MCTD(...deblock=true), not used in the script here. I was comparing it to the deblocking option in Mpeg2Source(...CPU=2). You can remove the import. I added UnDot to remove some DCT ringing artifacts. You can do that instead with Mpeg2Source(...CPU=6), or just leave it out. ChromaShift() was used to shift the colors left and up (relative to the greyscale image). You should look through other parts of the video to see how much the colors should be moved, if at all. Or you could use awarpsharp2() to sharpen and align the chroma channels.

[Gavino]

On a script similar to the one I gave earlier, SetMtMode(5,8)/SetMtMode(2) gave nearly a 3x performance increase. But there was an error message imprinted in the video -- caused by SRestore().

Can you remember what the message was? Was it something like "I don't know what d32 means"?

This is a mysterious error that has been reported from time to time but never explained - according to the function logic it should 'never' happen.
It would be interesting to know if multithreading can (sometimes) cause it to appear.

[jagabo]

Actually, there were a few different errors:

Evaluate System exception - Access Violation
(????, line 109)
([ScriptClip], line 1)

Evaluate System exception - Access Violation
(????, line 96)
([ScriptClip], line 1)

Evaluate System exception - Guard Page Violation
(????, line 110)
([ScriptClip], line 1)

Evaluate System exception - Access Violation
(BReC@ReC, line 110)
([ScriptClip], line 1)

LoadPlugin: unable to load "r \Program Files (x86)\AviSynth 2.5\plugins\GRunT.dll"
(????, line 109)
([ScriptClip], line 1)

I don't know what "lfr" means
(,line 96)
([ScriptClip], line1)

???? is some random assortment of characters, usually six, often <' ' or >'z':



The errors only seem to occur when the number of threads exceeds the number of cores (I'm running on a quad core, non hyperthreaded i5 2500K). And the higher the number of threads the more often they occur.

And the behavior is different with different programs. VirtualDub might hang or crash on opening the script. x264 runs to completion with the error messages embedded into the frames.

Well crap, it looks like I'm running version 2.57, not 2.5.8. Odd, I reinstalled AviSynth just a few days ago.



I'll uninstall and reinstall and see if there's any improvement.

[Gavino]

Wow, that's a bizarre set of errors - looks like there is some random memory corruption going on.
2.57 MT is known to have some multithreading errors that were fixed in later MT versions, so it's worth trying another, but it seems the most reliable way to avoid problems is to run SRestore single-threaded (using mode 5 or 3).

[jagabo]

Well this sucks. Given that I was using the outdated version of AviSynthMT I updated to 2.5.8 and then 2.6 MT build. The latter is more stable but there's a big drop in performance.

Using the 2.6 MT build I got about 2.7 fps running the earlier script with no SetMTMode(), about the same as I was getting with the regular 2.6 build and the 2.57 build. Running the full script with only SetMtMode(2,0) at the start increases the speed to about 3.7 fps and the resulting video looks ok. Using only SetMtMode(2,8) drops the speed down to about 2.0 fps but the video was still fine. With AviSynth 2.57 that increased the encoding speed to about 7.5 fps but resulted in the error messages mentioned earlier. Using the full "safe" script with multiple SetMtMode() commands (in post 21) drops the encoding speed to a mere 0.9 fps! Version 2.57 ran that script at about 7.5 fps and generated no errors. I may go back to 2.57!

Am I doing something wrong?

[Gavino]

MT performance behaviour is difficult to predict (and hence to optimise) with complex scripts, especially when temporal filtering is concerned. Try using different values of SetMemoryMax and adjusting the number of threads as suggested in this Doom9 post by the author of QTGMC.

[spiritgumm]

so Poisons's caveat was well-founded, for the moment. I started jagabo's original script with cpu=6, compressing to lagarith. I tried the sample first, seemed fine. For the whole movie, Vdub estimated 17 hours. Hopefully the power wont go out, and I dont do something negligent.

[spiritgumm]

I encoded and muxed the video, and took a few samples to watch on TV before burning a disc, and noticed some problems - if I had started my submitted sample a second earlier, someone would have noticed the problem after processing and addressed it. There's some clunky motion in some people walking. Here's a clip of two scenes demonstrating it (xvid of the finished lagarith).
clip problem2.avi
The guy walking in top left at the start (very brief), and as the woman enters room at 13 seconds, and possibly as she walks forward at 22 seconds. Since I noticed this randomly, I'm thinking it probably happens frequently. I'm wondering what part of the script can be tweaked or perhaps removed. I can upload the approximate same m2v clips for experimenting.