Multi-Threaded Avisynth optimization help needed

[VideoFanatic]

I've got Avisynth 2.60 MT installed as well as the QTGMC MT version. I use it like this as I found this gives the fastest FPS speed at around 13-14 fps when encoding standard definition h264 files and I made sure the CPU usage is between 98-100%:

setmtmode(5,9)
Mpeg2Source("E:\Video.d2v")
setmtmode(2,0)

I have an 8 core CPU. Some videos only use 80% CPU usage. The FPS on all videos varies from 8 to 14 Fps. I can't understand why it would be slow at 11 fps or lower?

I use the below script on all my videos:

setmtmode(5,9)
Mpeg2Source("E:\Video.d2v", CPU=6)
setmtmode(2,0)

McTemporalDenoise(settings="medium", interlaced=true) 

AssumeTFF() # or AssumeBFF() depending on the source
QTGMC(Preset="Super Fast") 
Vinverse() RemoveSpots() AddGrainC(var=1.0,uvar=1.0) SeparateFields() SelectEvery(4,0,3) Weave()

Also I encode about 8 videos per day and 1 of them always fails yet I've installed everything correctly and I believe the plugins I use are supposed to be stable in multi-threaded.

[Selur]

I can't understand why it would be slow at 11 fps or lower?

What did you expect?
I mean the speed of the encoding and some of the filters depends on the content, so it's not a surprise the speed is fluctuating,...
also you are
a. using a bunch of slow avisynth filters
b. probably using relatively slow encoding settings

-> if you need more speed adjust you filters e.g. using some gpu based filters might help speed things up a bit,..

[jagabo]

General observations about MT in AviSynth:

Mpeg2Source(...,CPU=6) can be a bottleneck. Deblocking and deringing can be slow. And running it in mode 5 doesn't help. From the MT docs: "Mode 5 is slowest(Slower than not using SetMTMode)". (This probably isn't an issue since you are using other very slow filters.)

The codec you're using to render can be a bottleneck too. Most MPEG2 encoders are not well multithreaded. Xvid and Divx aren't either.

Try different numbers of threads. Up to 3x (sometimes even more) the number of cores. See what works best. The sweet spot is usually around 1.5x to 2x. But where that sweet spot is varies depending on the filters used. MCTD().QTGMC() on my quad core i5 2500K requires about 12 threads to maximize throughput.

[VideoFanatic]

I spent several hours coming up with setmtmode(5,9) a few months ago. 9 threads gave the fastest FPS. I thought I was already using the fastest filters - I'm using the QTGMC multi-threaded version. McTemporalDenoise doesn't have any faster dependency plugins that I could use for it to my knowledge. I need to use both QTGMC and McTemporalDenoise as per my script so I don't know what other plugins I could use.

That's suprising that you use 12 threads. What's your script for that? I have a core i7 2700K (8 core) and found that 9 threads was the fastest! Do you think using more threads will be more stable or faster on average than what I get at the moment which is sometimes a fast FPS and sometimes a slow one?

I thought mode 5 was the most stable which is why I used it? What about the green text, should I change that?

setmtmode(5,9)
setmtmode(2,0)

[Selur]

it's relatively easy to find out where your bottleneck is, simply use AVSMeter and build up your script filter by filter,...

I thought mode 5 was the most stable which is why I used it? What about the green text, should I change that?

setmtmode(5,9)
setmtmode(2,0)

I would recommend to stick with mode 5 for source filters
also 'setmtmode(2,0)' should be setmtmode(2), only the first setMTMode call gets two parameters.

[jagabo]

I spent several hours coming up with setmtmode(5,9) a few months ago. 9 threads gave the fastest FPS.

With exactly that filter sequence? I don't think so. And given you're only getting 80 percent CPU usage -- obviously not. The number of threads required depends on the filters used. I already pointed that out.

I thought mode 5 was the most stable which is why I used it?

Yes, you have to use mode 5 with Mpeg2Source().

[manono]

setmtmode(5,9)
Mpeg2Source("E:\Video.d2v", CPU=6)
setmtmode(2,0)
McTemporalDenoise(settings="medium", interlaced=true)
AssumeTFF() # or AssumeBFF() depending on the source
QTGMC(Preset="Super Fast")
Vinverse()
RemoveSpots()
AddGrainC(var=1.0,uvar=1.0)
SeparateFields()
SelectEvery(4,0,3)
Weave()

You can skip MCTemporalDenoise and use DFTTest within QTGMC if you want to save
some time and you like the results DFTTest gives you. And why use Vinverse when
there's no interlacing at all at that point? And I never quite understood the rationale
behind adding grain after running a video through a denoiser to remove it. Especially
since adding grain makes it less compressible and one of the reasons (for me, anyway)
for removing grain is to make a video easier to compress.

For me with a quad-core i860, the max encoding speed when using QTGMC is with 14 threads.

[VideoFanatic]

QTGMC needs to go after McTemporalDenoise otherwise the encodings will slow to a crawl and RemoveSpots needs to go after QTGMC otherwise it doesn't work properly. I found out that AddGrainC actually covers up more spots that weren't removed by RemoveSpots. So that's why I can't have AddGrainC before the denoiser. I can't even see any grain that it adds and it barely adds to the encoding size so it doesn't bother me.

If you have a media player set top box then you may see interlacing artifacts for some reason (AC Ryan Playon HD & WD TV Live, yet you don't get those if you author to a DVD or Bluray. So for those media players, Vinverse removes any remaining interlacing artifacts that QTGMC didn't remove.

Do you have a link to a shop page for the i860 as I can't even find out anything about it?

So with the filters I use should only use this, is that correct - there's no settings which are more stable? And it looks like I need Mode 5 for MPEG2Source.

setmtmode(5,9)
Mpeg2Source("E:\Video.d2v", CPU=6)
setmtmode(2)

So the only thing I need to do is change the red text based on how many threads I want?

[manono]

QTGMC needs to go after McTemporalDenoise

My suggestion was to skip MCTemporalDenoise entirely and denoise with DFTTest which is included in QTGMC. That's up to you, of course.

So for those media players, Vinverse removes any remaining interlacing artifacts that QTGMC didn't remove.

Now I remember, we've been through all this before. And it still makes no sense. If by 'interlacing artifacts' you mean QTGMC leaves behind some actual interlacing (mice teeth, feathering, etc.), then I don't believe it. But maybe you're referring to something else.

Do you have a link to a shop page for the i860 as I can't even find out anything about it?

I apologize. It's called the i7-860 and it's old now:

http://ark.intel.com/products/41316/Intel-Core-i7-860-Processor-%288M-Cache-2_80-GHz%29

So with the filters I use should only use this, is that correct - there's no settings which are more stable? And it looks like I need Mode 5 for MPEG2Source.

setmtmode(5,9)
Mpeg2Source("E:\Video.d2v", CPU=6)
setmtmode(2)

I'll leave that to my betters. I just use:

SetMTMode(2,14)

once at the top. This is for AVISource. And not using filters that don't multi-thread well. With QTGMC, RemoveSpots, Stab, LimitedSharpenFaster, etc. in the script. I'm doing VHS tape to DVD conversions.