DirectShowSource and FFVideoSource Problems

[ndjamena]

I've been using an AVISynth scripts with X264 to encode some doctor who DVDs for my iPhone. Everything had been going fine but a few days ago the encodes began stalling partway through. If I keep trying eventually some will finish and the final encodes are perfect. I tried googling for an answer but all I found was that it was unlikely to be X264 and was probably the codec. I have Shark007 installed, I had fiddled with it a bit but have since returned it to it's default settings and it still happens. LAV splitter is added to my taskbar every time I begin an encode. I tried switching from DirectShowSource to FFVideoSource and the stalling stopped but upon watching the results the movement was off. I found a section in the opening credits of one episode that showed the text 'Planet of the Daleks' being replaced with 'Terry Nation'. Viewing the original frame by frame in VLC shows a full switch between one frame and the next but in the FFVideoSource encode it goes like this f1:'Planet of the Daleks', f2'Terry Nation'(with a ghost of POTD), f3'POTD', f4'Terry Nation'. It continues like that throughout the entire encode and makes the video unwatchable. I thought maybe it had something to do with the framerate and the video was being telecined by FFVS in the opposite order to the deinterlacer so I tried DSS with a fixed fraterate but it still crashed.

DirectShowSource("Doctor Who - 10x18 - Planet of the Daleks (4).mkv", 25, convertfps=true)
QTGMC(Preset="Very Slow")
SelectEven()
fft3dfilter(sigma=2.0, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.6, plane=4)
Crop(12, 2, -12, -6)
Spline144Resize(752, 576)
ChangeFPS(25)

I leave 4 instances running at once to use the majority of my CPU power. This was working perfectly a few days ago, I know nothing about codecs, all I can think to do is use this opportunity to upgrade the firmware on my Revodrive, reinstall windows and all my programs from scratch and hope that the problem goes away.

[manono]

You said the source was Dr Who DVDs. Why the MKV?

I leave 4 instances running at once to use the majority of my CPU power.

Why aren't you using a multi-threaded version of AviSynth? Then you wouldn't have the problem you're having and would encode a single episode at a time.

[ndjamena]

I'm ripping with MakeMKV, it's was either that or DVDFab. I used to use DVDFab but it takes up more room and is harder to work with. With MakeMVK and MKVToolNix I can easily process the video, audio and subtitles separately then mux them all together later with MP4 box.

I only started using Avisynth a fortnight ago and the only mention of multithread that I've seen said that it was unreliable. I'll look into it further.

[ndjamena]

I've altered my batch file to add this to the AVS:

MT(filter=QTGMC(Preset="Very Slow"), threads=8)
MT(filter=fft3dfilter(sigma=2.0, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.6, plane=4), threads=8)

Is this the correct syntax?

The next encode will start in less than two hours so I'll see how it goes. That doesn't explain what the problem was, hopefully DSS won't stall if left to a single instance.

[manono]

I'm ripping with MakeMKV,

If I'm not mistaken, because it's still MPEG-2 video inside you can still create a D2V project file using DGIndex and then open it using MPEG2Source, the way normal people do. Then you wouldn't have to use the unreliable DirectShowSource.

Is this the correct syntax?

It's not like anything I've ever seen. Read about multi-threaded usage of QTGMC (and other filters) here:

http://forum.doom9.org/showthread.php?p=1423459#post1423459

I'm not quite sure why you're using QTGMC in the first place, but I'm sure you have your reasons. Is it because of the mix of progressive and interlaced video found in the PAL DVDs?

Use an MT version of AviSynth as found here:

http://forum.doom9.org/showthread.php?p=1312666#post1312666

That first post also explains (again) how to use it correctly.

[ndjamena]

Thanks.

I looked up 'Best deinterlacing filter' and it was universally QTGMC. I realise QTGMC works best as double rate but the iPhone screen only works at 30fps. I used to use handbrake with yadif and mcdeint which I've realised is better for what I'm trying to do but handbrake isn't really suited to the kind of messing around I do and I haven't found an AVISynth equivalent of mcdeint. The iPhone screen is only a few inches in area so I figured the lost temporal data won't have that much effect and they even look fine when played on my 40 inch bravia. I'm open to suggestions.

OK, so I downloaded DGIndex, removed all my plugins then downloaded the QTGMC multithread package and QTGMC, then downloaded fft3dfilter and Splineresize again and put them back. I also downloaded AviSynth MT and put it in my SysWOW64 folder, after figuring out how to add DGIndex to my Batch files through the command line I ran it and got this as a script

SetMTMode(3, 8)
mpeg2source("C:\Users\N'Djamena\Desktop\Videos\Wor k\Doctor Who - 10x16 - Planet of the Daleks (2).d2v")
SetMTMode(2)
QTGMC(Preset="Very Slow", EdiThreads=2)
SelectEven()
fft3dfilter(sigma=2.0, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.6, plane=4)
Crop(14, 4, -10, -4)
Spline144Resize(752, 576)
ChangeFPS(25)

It's running, but it's no faster. I'm guessing I missed something. I have to go out now, when I get back I'll try to figure out what's wrong, I'll just run three more instances in the meantime.

Thanks for the help, appreciated!

[manono]

It's running, but it's no faster.

If you have 8 cores you might try 10, 12, 14 , or even 16. For me QTGMC runs fastest using 14. You want to get the higest percentage of CPU usage without going to a number too high and without crashing the encode.

In the QTGMC thread to which I linked last time it says this about setting it up for the fastest encoding speed:

SetMTMode(3, X) # See below for value X, could try 5 instead of 3 for non-standard source-filter/avisynth combinations .
.
.
.
Setting X
- Start at the number of logical cores in your machine (note: with HyperThreading enabled, logical cores = 2x physical cores)
- If it crashes, decrease 1 at a time
- Otherwise increase 1 at a time until CPU usage just reaches 100%, don't go too far or it will slow down.

And get all the filters required for multi-threaded QTGMC from the Modded Plugin Package linked in the same post.

[ndjamena]

AviSynth_130114.exe
avisynth_20130220.7z
QTGMC 32-bit Plugins [Vit-2.6].zip
QTGMC-3.32.zip
SplineResize_v02.zip
fft3dfilter211.zip
dgmpgdec158.zip

SetMemoryMax(512)
SetMTMode(3,2)
mpeg2source("C:\Users\N'Djamena\Desktop\Videos\Wor k\Doctor Who - 17x02 - Destiny of the Daleks (2).d2v")
SetMTMode(2)
QTGMC(Preset="Very Slow", EdiThreads=1)
SelectEven()
fft3dfilter(sigma=2.0, bt=5, bw=32, bh=32, ow=16, oh=16, sharpen=0.6, plane=4)
Crop(16, 2, -18, -2)
Spline144Resize(736, 576)
ChangeFPS(25)

Avisynth: access violation at 0x0001F0CC in C:\Program Files (x86)\AviSynth 2.5\plugins\mt_masktools-26.dll, attempting to read from 0x565B695D
(QTGMC-3.32.avsi, line 776)
(QTGMC-3.32.avsi, line 386)

i7 3770 4.1ghz(OC) 16g 1600m DDR3RAM Win8pro 64bit

I uninstalled AVISynth, deleted the plugins directory then re-downloaded everything, re-installed it all, then restarted the computer. I feel like a real noob at the moment... Should I have the 64-bit version of AVISynth for this?

[manono]

From what I understand the 64-bit version of AviSynth isn't all that stable and a lot of filters only work in 32-bit anyway, but others know more about that than I. I believe you should have 32-bit everything.

Avisynth: access violation at 0x0001F0CC in C:\Program Files (x86)\AviSynth 2.5\plugins\mt_masktools-26.dll, attempting to read from 0x565B695D

Your AviSynth is a 2.6 version and so is the MT version you're using?

[ndjamena]

I've got it working, wow, AVISynth is a mess! It's soooo cool.

[ndjamena]

First I switched to Mask 2.5 but that didn't work, then I installed avisynth 20120516 and it's working. ~33% CPU usage in VirtualDUB, I'll see how it works with X264.

SetMemoryMax(2048)
SetMTMode(3,14)

I'm not sure how high I should go...

[ndjamena]

Grrr, fft3dfilter won't work with multithreading enabled, what do I do about that?

[ndjamena]

SetMemoryMax(2048)
SetMTMode(3,14)
mpeg2source("C:\Users\N'Djamena\Desktop\Videos\Wor k\Doctor Who - 17x02 - Destiny of the Daleks (2).d2v")
SetMTMode(2)
QTGMC(Preset="Very Slow", EdiThreads=2, FPSDivisor = 2, NoiseRestore=0, GrainRestore=0, sigma=3.0, Sharpness=1.8, SLRad=3)
SetMTMode(6)
Crop(16, 2, -18, -2)
Spline144Resize(736, 576)
ChangeFPS(25)

Around 50% CPU, not exactly impressive but better than it was.

[manono]

Grrr, fft3dfilter won't work with multithreading enabled, what do I do about that?

I believe it does work with MT, but maybe not with SetMTMode(2). Anyway, FFT3DFilter is included within QTGMC and is the default denoiser used. There's no point in using a separate call to it outside of QTGMC. Or use DFTTest within QTGMC instead. From the QTGMC doc:

Removing Noise
Denoiser .... (string) .... Select denoiser to use for noise bypass / denoising. Select from "dfttest" or "fft3dfilter". dfttest is better, but slower. Unknown value selects "fft3dfilter"

Try this in a test and see if you get better CPU usage:

SetMTMode(2,14)
mpeg2source("C:\Users\N'Djamena\Desktop\Videos\Wor k\Doctor Who - 17x02 - Destiny of the Daleks (2).d2v")
QTGMC(Preset="Very Slow", EdiThreads=2, FPSDivisor = 2, NoiseRestore=0, GrainRestore=0, sigma=3.0, Sharpness=1.8, SLRad=3)
Crop(16, 2, -18, -2)
Spline144Resize(736, 576)
ChangeFPS(25)

And why the 'ChangeFPS' if your source is already 25fps?

[ndjamena]

I think it's likely installing the new Office killed DSS. I used Excel to create a list of Optimum Croppings to remove anamorphic while maintaining modulus 16 without distorting the videos through stretching. Everything started going wrong once I'd rebooted.

Removing ChangeFPS got VirtualDUB working with almost 100% CPU, but the display pane wasn't updating so I couldn't see the result. I'm trying it with x264 now and CPU has only increased to about 60%. I re-encoded a video using the new settings and compared it to the old one and the new settings produced a significantly larger file. 287mb vs 332mb at CRF22. I want the file smaller because the iPhone is only 64gig and I thought denoising would do a smarter job than an increase in CRF, although looking at the source it would probably need denoising anyway. Somehow I've got to tweak the new settings to match the old ones or go back to what I was doing before. ChangeFPS was there to make sure I really got 25fps since I don't completely trust the file output. For something that's not doing anything it has quite a large impact on speed.

[ndjamena]

I set sigma to 2.0 and sharpening to 0.6 and the new encode is a tiny bit bigger than the first. I compared a frames from each using vlc and they're pretty such the same, thus proving that adding together the settings of two sequential denoises and feeding the numbers into a single denoise gets you nowhere. Since I couldn't get full CPU utilization I decided to try for a little less than half so that I could run two instances. I set MTMode to 2, 6 thinking less threads would slow it down. The LCD screen on my G510 is now happily showing that my CPU utilization is hovering around the 100% mark. End.