Avisynth MT

[logicom]

Hi,

is there any better performance with avisynth+ then with Avisynth 2.6MT + QTGMC [vit mod]

is there 64bit combo for any of those two?

on I5 650 CPU based system, I am getting 40fps for SD video at QTGMC fast setting . (no other filters involved, only deinterlacing)
For ex. should it be possible to get 20 more fps with the same settings, on the same computer or my expectations should be set lower?
I try to tweak MT parameters, but maybe someone could help with that a bit ?

Regards.

[jagabo]

64 bit won't be significantly faster.

Without encoding, On my i5 2500K I get about 110 fps and nearly 100 percent CPU usage with QTGMC(preset="fast") when encoding 720x480 material.

SetMtMode(5,4)
Mpeg2Source("filename.d2v")
SetMtMode(2)
QTGMC(preset="fast")

When encoding with x264 with the veryfast preset it drops down to about 90 fps. With the slow preset about 50 fps.

x264.exe --preset=slow --crf=18 --keyint=50 --sar=8:9 --output %1.mkv %1

Watch CPU usage in Task Manager while filtering/encoding. If you see less than 100 percent CPU usage you may be able to squeeze out more performance. With some filters you need more threads than cores to max out the CPU. Maybe 1.5x to 2x the number of cores. And with some sources decoding of the source is the bottleneck.

[LigH.de]

There is no 64 bit version of AviSynth 2.60 MT (because the development of the original AviSynth 2.50+ project is dead).

And there is a 64 bit version, but no MT version of AviSynth+ ... yet?

[jagabo]

By the way, my results were with 2.6 MT, 32 bit.

[logicom]

Hi,

QTGMC is well threaded, I see that cpu is squeezed at 100%, I am using 2.6 MT, 32 bit and I noticed that delay introduced by the chain FFmpegSource2, DV decoder and ffmpeg is actually the reason for having maybe a bit lower FPS than expected. Source is 4:2:0 DV25>MOV
Just for reference, there is a SSD raid 0 and 8 GB of ram. Any suggestion for ffmpeg version/flavor ?

Regards.

[poisondeathray]

Since you're already at 100% usage, not much is going to help

The only thing that will make it faster is to use faster settings : either in QTGMC settings, or encoding settings, or both

[logicom]

Hi,

I do not encode, but just for testing I will run it now on dual X5470 and e5 2697 v2

Regards.

[poisondeathray]

I do not encode.

What are you using ffmpeg for ? uncompressed mov ?

[poisondeathray]

If you're not encoding , how are you measuring the script speed? avsmeter ?

[jagabo]

If you're not encoding , how are you measuring the script speed? avsmeter ?

I used VirtualDub's File -> Run video analysis pass.

[logicom]

I do not encode.

What are you using ffmpeg for ? uncompressed mov ?

There is no call to x264 , I am just piping trough ffmpeg to null. And there is no "uncompressed mov" in this case.

ex:

ffmpeg -i deintMOV.avs -vcodec copy -an -f null out.null

Once I run video analysis pass in virtualdub, with show in and out video disabled, I am getting lover FPS, for about 15%, instead of 42 it is 35 and CPU is in both cases at 100%.

Tautologically speaking, CPU mark is one most important parameter for filtering or encoding, or processing in general , but I was looking into possibilities for further optimization with the goal of getting better result in FPS units.

Regards.

[LigH.de]

A possible solution; but avsmeter has some features specifically implemented for benchmarking, and is probably easier to use, and a lot smaller without libav codecs.

[logicom]

A possible solution; but avsmeter has some features specifically implemented for benchmarking, and is probably easier to use, and a lot smaller without libav codecs.

I partially agree on that, but taking ffmpeg as a reference would be mandatory as it just reflects one real scenario so that it is wise to have a comparison testing,taking into account possible delay produced by its I/O routines , while not kilning the cpu with encoding task for the moment, nor rewrapping nor performing any other additional processing. I would rather look into the chain of processes and individual actions, trying to minimeze impact, and then go for real drive.

All benchmarking methods are giving slightly higher fps initially, but I wait a while to get the stable average resault.
Ironically enough, benchmarking tool you have mentioned gives higher result than vdub but lower than ffmpeg

c:\AVSMETER>avsmeter C:\ffmpeg\deintMOV.avs -l

AVSMeter 1.9.0 (x86) by Groucho2004
AviSynth 2.60, build:Sep 28 2013 [15:09:12] (2.6.0.4) (MT)

Active MT Mode: 1
Number of frames: 85710
Length (hh:mms.ms): 00:57:08.400
Frame width: 720
Frame height: 576
Framerate: 25.000 (25/1)
Colorspace: YV12

Frames processed: 1232 (0 - 1231)
FPS (min | max | average): 2.599 | 346396 | 38.98
CPU usage (average): 98%
GPU usage: GPU monitor disabled
Thread count: 37
Memory usage (phys | virt): 1145 | 1169 MB
Time (elapsed): 00:00:31.602

c:\ffmpeg\ffmpeg -i deintMOV.avs -vcodec copy -f null out.null
ffmpeg version N-65991-g8c63a0d-Sherpya Copyright (c) 2000-2014 the FFmpeg developers
  built on Aug 31 2014 05:10:22 with gcc 4.9-win32 (GCC)
Input #0, avisynth, from 'deintMOV.avs':
  Duration: 00:57:08.40, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 720x576, 25 fps, 25 tbr, 25 tbn, 25 tbc
Output #0, null, to 'out.null':
  Metadata:
    encoder         : Lavf56.3.100
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 720x576, q=2-31, 25 fps, 25 tbn, 25 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame= 1759 fps= 42 q=-1.0 Lsize=N/A time=00:01:10.36 bitrate=N/A
video:110kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: unknown
Received signal 2: terminating.

Regards.

[LigH.de]

Thread count: 37

Sounds exhausting? Despite MT mode = 1?

[logicom]

Thread count: 37

Sounds exhausting? Despite MT mode = 1?

SetMTMode(5,8) is the cause, reminiscence from tweaking but 3,4-8 is just good enough.

Regards.

[LigH.de]

May be too extreme, except you really have 2 Octocore with HT. Remember, QTGMC also spawns EDI threads on its own.

[logicom]

May be too extreme, except you really have 2 Octocore with HT. Remember, QTGMC also spawns EDI threads on its own.

I assigned 2 thread for EDI , at this moment I dont see extra threads by it, however MT have to be higher, and not extreme, otherwise, CPU load variations are noticeable or suboptimal result in FPS is present. I am looking forward to put it on double dodeca core with HT - e5 2697 v2.

[poisondeathray]

You should be testing real life scenario. While script testing is important for bottlenecks, the effect on actual encoding scenario might be different. You have other threads that required for encoding. In your setup you will get better results with parallel encoding multiple instances (faster overall, higher quality, fewer problems)

[jagabo]

ex:

ffmpeg -i deintMOV.avs -vcodec copy -an -f null out.null

Once I run video analysis pass in virtualdub, with show in and out video disabled, I am getting lover FPS, for about 15%, instead of 42 it is 35 and CPU is in both cases at 100%.

When I run that ffmpeg command line I see only 25% CPU usage and about 38 fps (compared to 100 percent CPU usage and 115 fps in VirtualDub).

[logicom]

ex:

ffmpeg -i deintMOV.avs -vcodec copy -an -f null out.null

Once I run video analysis pass in virtualdub, with show in and out video disabled, I am getting lover FPS, for about 15%, instead of 42 it is 35 and CPU is in both cases at 100%.

When I run that ffmpeg command line I see only 25% CPU usage and about 38 fps (compared to 100 percent CPU usage and 115 fps in VirtualDub).

Now we are talking I must admit that is unusual and not expected for me. May you provide output sample, with ver included ?

[Selur]

try adding:

SetMemorMax(768)

to the start of your avisynth script and

distributor()
return last

to it's end when decoding the script with ffmpeg

[jagabo]

Here's the ffmpeg log:

C:\Users\John\Desktop>"G:\Program Files\ffmpeg\bin\ffmpeg.exe" -i C:\Users\John\
Desktop\VTS_01_1.avs -vcodec copy -an -f null out.null
ffmpeg version N-67331-g547fce9 Copyright (c) 2000-2014 the FFmpeg developers
  built on Nov  1 2014 22:01:52 with gcc 4.9.1 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libbs2b --enable-libcaca --
enable-libfreetype --enable-libgme --enable-libgsm --enable-libilbc --enable-lib
modplug --enable-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrw
b --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinge
r --enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --en
able-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-libvorbis
 --enable-libvpx --enable-libwavpack --enable-libwebp --enable-libx264 --enable-
libx265 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      54. 11.100 / 54. 11.100
  libavcodec     56. 10.101 / 56. 10.101
  libavformat    56. 12.100 / 56. 12.100
  libavdevice    56.  2.100 / 56.  2.100
  libavfilter     5.  2.101 /  5.  2.101
  libswscale      3.  1.101 /  3.  1.101
  libswresample   1.  1.100 /  1.  1.100
  libpostproc    53.  3.100 / 53.  3.100
Input #0, avisynth, from 'C:\Users\John\Desktop\VTS_01_1.avs':
  Duration: 00:00:33.37, start: 0.000000, bitrate: 0 kb/s
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 720x480, 59.94 fp
s, 59.94 tbr, 59.94 tbn, 59.94 tbc
Output #0, null, to 'out.null':
  Metadata:
    encoder         : Lavf56.12.100
    Stream #0:0: Video: rawvideo (I420 / 0x30323449), yuv420p, 720x480, q=2-31,
59.94 fps, 59.94 tbn, 59.94 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (copy)
Press [q] to stop, [?] for help
frame= 2000 fps= 38 q=-1.0 Lsize=N/A time=00:00:33.36 bitrate=N/A
video:125kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing o
verhead: unknown

Here's VirtualDub running the video analysis pass:

[logicom]

SetMemoryMax(1024)
SetMTMode(3,4)
FFMpegSource2("C:\ffmpeg\intput.mov")
SetMTMode(2)
QTGMC(preset="very fast", EdiThreads=2)
SelectEven()
SetMTMode(1)
GetMTMode(false) > 0 ? distributor() : last

try this script.

[jagabo]

With that script (opening an NTSC VOB file, 720x480) I get about 115 fps with ffmpeg and VirtualDub. distributor() is making the difference with ffmpeg.

Interestingly, with this script:

SetMtMode(5,4) 
Mpeg2Source("VTS_01_1.d2v")
SetMtMode(2)
QTGMC(preset="very fast")
Distributor()

I get about 180 fps with ffmpeg, 150 fps with VirtualDub. But if I remove Distributor() I get about 70 fps with ffmpeg, 180 fps with VirtualDub.

[logicom]

QUOTE=poisondeathray;2359604]You should be testing real life scenario. While script testing is important for bottlenecks, the effect on actual encoding scenario might be different. You have other threads that required for encoding. In your setup you will get better results with parallel encoding multiple instances (faster overall, higher quality, fewer problems)[/QUOTE]

What are you talking about ?

[logicom]

With that script (opening an NTSC VOB file, 720x480) I get about 115 fps with ffmpeg and VirtualDub. distributor() is making the difference with ffmpeg.

Nice!

[jagabo]

With that script (opening an NTSC VOB file, 720x480) I get about 115 fps with ffmpeg and VirtualDub. distributor() is making the difference with ffmpeg.

Nice!

I added a bit more to my last post.

[poisondeathray]

You should be testing real life scenario. While script testing is important for bottlenecks, the effect on actual encoding scenario might be different. You have other threads that required for encoding. In your setup you will get better results with parallel encoding multiple instances (faster overall, higher quality, fewer problems)

What are you talking about ?

Are you asking about the quality hit ? or fewer problems

I'm referring to the encoder side of things, there is an inverse relationship: the more threads , the lower the quality (eg. SSIM, PSNR) . But it's negligible at low thread counts. Once you get into higher thread counts (eg. workstation / server encoding like your setup) it becomes more relevant

[logicom]

With that script (opening an NTSC VOB file, 720x480) I get about 115 fps with ffmpeg and VirtualDub. distributor() is making the difference with ffmpeg.

Nice!

I added a bit more to my last post.

Can you please try input from mov ? Just ot check if it is the same as with d2v files.

Regards.

[logicom]

You should be testing real life scenario. While script testing is important for bottlenecks, the effect on actual encoding scenario might be different. You have other threads that required for encoding. In your setup you will get better results with parallel encoding multiple instances (faster overall, higher quality, fewer problems)

What are you talking about ?

Are you asking about the quality hit ? or fewer problems

I'm referring to the encoder side of things, there is an inverse relationship: the more threads , the lower the quality (eg. SSIM, PSNR) . But it's negligible at low thread counts. Once you get into higher thread counts (eg. workstation / server encoding like your setup) it becomes more relevant

We just discussed about filtering performance and possible optimizations, but thanks for the input, again nothing to do with encoding
But anyway, could you please provide some reference paper which further discuss this phenomena that you mentioned? Like impact on video quality and implicitly video quality measurement parameters due to the high number of threads ? I would love to dig into that inverse equation.