TemporalDegrain problem with SetMTMode

[RegiOween]

Hello, folks!

First I would like to say that this post is a bit longer than the usual, but the reason is to provide as much information as possible in order to help you help me .

I would like to fix some of the BluRay titles I acquired recently, due to a seeming tendency to have a lot of graining in this kind of media.
My goal is to backup all of my BluRay discs at my local storage to MKV container encoding to AVC format using x264 with CRF value 20, all other settings default.

I have been testing many methods of degraining, including high quality ones like MCTemporalDenoiser and MC_Spuds, but in my personal opinion, and for my specific needs, TemporalDegrain is the hands down winner.

As well stated in the docs, this script function is very slow, so I'm trying to make it work a little bit faster, but the problem is that I couldn't find a way to make it work using SetMTMode instruction, cause every attempt I made led to crash the process at the start of my script, and the error is always related to FFTW3.dll.
There's absolutely no errors using this amazing function without SetMTMode, but the speed is barely above 1 fps.

Follow below all details of my issue:

Script sample:
SetMTMode(3, 0) # Also tried many different values for (mode, threads) here
DSS2("BluRay Title.m2ts") # Also tried with FFVideoSource (FFmpeg) and AVCSource (DGDecode)
SetMTMode(2)
TemporalDegrain()

VirtualDub error when trying to open the script:
An out-of-bounds memory access (access violation) occurred in module 'fftw3' writing address 00000020.

Media Player Classic error when trying to open the script:
Problem Event Name: APPCRASH / Fault Module Name: fftw3.dll

AviSynth versions tried:
2.5.8 MT and 2.6.0 MT with the most recent avisynth.dll, both 32-bit.

FFTW3.dll versions tried at SysWOW64 folder:
old 30.01.2004 (available at avisynth.org site) and newest 25.11.2012 (available at fftw.org site)

Relevant machine specs:
Intel i7 3770 3.4Ghz (quad-core hyper-threading), 16 Gb RAM DDR3 1600 MHz

Operating System:
Windows 7 SP1 x64

Considering my machine is fairly good, I would like to get at least 3 fps, but when I tried using TemporalDegrain_MT wrapper script with AviSynth 2.5.8 (that uses MT instruction instead of SetMTMode), I just got around 1.7 fps, which is far from my expectations.

I know I can gain speed increasing the blksize value or lowering the degrain parameters, but in my tests, this shows considerable quality loss, so I want to avoid it.

If you can help me to solve this issue, and need some more details about it, feel free to ask, and I would be very grateful.

Thanks in advance,

RegiOween

[jagabo]

I don't normally use DSS2 but I'm getting close to 3 fps (near 100% CPU usage) on my quad core 2500K with this:

SetMtMode(5,6)
import("C:\Program Files (x86)\AviSynth 2.5\plugins\TemporalDegrain.avs") 
DSS2("filename.mkv") # Blu-ray rip muxed into MKV
SetMTMode(2)
Crop(0,132,1920,816) # 2.35:1 movie
TemporalDegrain()

[jagabo]

As I feared, DSS2 isn't frame accurate and delivered a lot of glitches. Try using the ffmpeg source plugin instead, ffVideoSource("filename.ext").

[RegiOween]

Jagabo, I don't think this problem is related to DSS2 or any other source plugin, since I already tested with FFVideoSource too, and also with the most frame accurate free tool around, DGAVCIndex/DGAVCDecode combo.
Thanks for your input.

[jagabo]

If you're using 32 bit AviSynth I think you're running out of memory. With 32 bit AviSynth and the cropping above, 4 threads eats up about 3 GB of memory (SetMemoryMax) with CPU usage around 50 percent. I'm not sure exactly what the limit is for 32 bit AviSynth on 64 bit Windows but you certainly can't use more than 4 GB. If you're lucky you'll get a malloc error (indicating no more memory could be allocated) but usually you just get a crash.

[RegiOween]

Hello, Jagabo!

All things considered, I think I will give up about multi-threading, and instead, focus on multi-tasking. I have to make some tests to check if my little baby can handle maybe four or five x264 sessions at the same time, without becoming unstable. This wouldn't get the job done faster, as I would like to, but at least will take advantage of the resources in a more productive way.

Many thanks for your attention and valuable information.

RegiOween

[jagabo]

I have to make some tests to check if my little baby can handle maybe four or five x264 sessions at the same time, without becoming unstable.

Each 32 bit process runs in its own virtual space. With 16 GB of memory you shouldn't have any problems running several x264 sessions at once.

[RegiOween]

Hello, folks!

Just to update the topic with more concerning info, I would like to inform that after many hours of reading, testing, comparing, etc., I finally managed to get a satisfying result, combining good performance with high quality output.
As I said before, until this multi-threading thing got more mature in Avisynth, from now on I prefer to focus on multi-tasking to get the best out of my machine resources.

The method I choose is the following:

# Script_Part_1.avs

DSS2("BluRay Title.mts")  # DSS2 is very fast and reliable for me, unlike native DirectShowSource
TemporalDegrain()  # Defaults are pretty much ok for me, but if I want to tweak, I would
                   # play with degrain, sad1 and sad2 parameters.
LSFmod(defaults="slow")  # A limited sharpening after degraining looks good for me, and
                         # the "slow" preset gives much better quality
Trim(0, FrameCount/2) # Get the first half of the video

# Script_Part_2.avs

# The same script above, except for the last line:
Trim((FrameCount/2)+1, 0) # Get the second half of the video

Then I enqueue both scripts to my favourite front-end (MeGUI), thas is capable of run both serial and parallel jobs, and start a parallel encode using x264 with CRF 20 (very good balance between quality and file-size), preset "very slow" (very slower indeed, but much better compression), and all other settings default.

The two jobs are running at about 4.8 fps (2.4 fps each), so in the end I get much more than what I wanted to achieve in the first place (3 fps). The 4 cores of my CPU are running at full load, so I see no reason to start more than 2 jobs. When the jobs finish, I just merge both parts together with the audio-track in a MKV container, and it's done.

Note 1: To get my machine running at full load for many hours uninterrupted without become unstable, I had to disable hyper-threading in my bios setup, otherwise, x264 was crashing after some time, and in a few occasions I even got BSOD.

Note 2: I often use the term "for me" in my sentences on purpose, so opinions may vary.

I hope this little experiment can help other users, and if you have some suggestions or advices, or just comment about something, you'll be very welcome.

RegiOween

[jagabo]

Nice to hear about your success. You need better cooling if you can't run indefinitely at 100 percent CPU load with hyperthreading.