Video conversion speed - How to improve

[tsk1979]

Hello
My machine details are in my profile, just hover link.
4GB ram, win 7 core i5 2.53GHz, and Nvidia 460M

I have seen, that my video encoding is around 0.8FPS when there are no "operations" or complex things happening in avisynth. If I am doing stuff like ScriptClip then it drops to 0.43FPS

This I have observed with
1. Xvid default settings
2. H264 default settings

I even enabled 3D accr in virtualdub settings.

Now, when I am encoding my final copy, its fine. However, sometime I need to quickly see what actually is happening in my video. First I tried uncompressed, but filesize is tooooo big.
Is there a codec which is not good quality wise but can give me 5FPS or greater with a small file size so I can quickly test out my scripts.

[Selur]

First use AVSMeter to figure out how fast simply working through your Avisynth script is.
Assuming your Avisynth script, is the one thing slowing the whole process down, tweaking the encoding setting won't help.

[jagabo]

Bring up Task Manager and look at CPU usage while encoding. If you see close to 100 percent you can just use faster settings in x264. x264 with the VeryFast preset is several times faster than the default settings. If you aren't seeing close to 100 percent CPU usage the bottleneck is somewhere else. You can try using a multithreaded build of AviSynth but those are somewhat buggy.

[poisondeathray]

If this is for your timelapse and you're using 20+ Megapixel photos for the input source, it will be very taxing memory wise for avisynth / scriptclip .

It might help to batch resize the photos ahead of time , and it might help to use SetMemoryMax(1024) or some other value

If you're doing this only for preview renders, you can use a lower resolution (resize to some smaller dimensions) to increase the speed , use full resolution for final renders

[tsk1979]

Thanks jagabo and poisondeathray

Yes, this is for my time lapses. However I observe < 1FPS even for my action cam clips which are .mov
In my scripts I use Dissolve + Trim to get segments and create composite video.

CPU usage went low when I switched on 3D acceration, and FPS stayed same for reduced CPU usage too!

What is the fastest codec? Intel IUV etc.,?

If I create a 640x480 type of SD video I think it should encode faster, and later I can go with full rez once I am satisfied.

[jagabo]

Fast codecs include lossless codecs like HuffYUV and UT Video Codec (but they don't give much compression). Most MJPEG encoders are fast and give you a choice of quality vs. compression. Divx (ASP) at it's fastest settings is as fast as MJPEG. x264 at the Ultrafast preset is pretty close.

[tsk1979]

Fast codecs include lossless codecs like HuffYUV and UT Video Codec (but they don't give much compression). Most MJPEG encoders are fast and give you a choice of quality vs. compression. Divx (ASP) at it's fastest settings is as fast as MJPEG. x264 at the Ultrafast preset is pretty close.

I downloaded the free H.264.
Where do I go and set the ultrafast. Or is the x264 different?
Is it in the video->compression->codec select->configure in virtualdub

[jagabo]

Fast codecs include lossless codecs like HuffYUV and UT Video Codec (but they don't give much compression). Most MJPEG encoders are fast and give you a choice of quality vs. compression. Divx (ASP) at it's fastest settings is as fast as MJPEG. x264 at the Ultrafast preset is pretty close.

I downloaded the free H.264.
Where do I go and set the ultrafast. Or is the x264 different?
Is it in the video->compression->codec select->configure in virtualdub

You're using x264vfw in VirtualDub? There are different builds. One of them has a pulldown for presets. If you don't have that one you can try configuring the options to match the presets. I once posted a list of parameters specified by the presets:

https://forum.videohelp.com/threads/347067-VirtualDub-H-264-Encoder-Speed?p=2169830&vie...=1#post2169830

Just as a quick reference, I encoded a 1280x720 video using the x264 CLI encoder with some of the different presets. Here are the resulting frame rates:

ultrafast: 203 fps
veryfast: 112 fps
medium: 43 fps
slow: 25 fps
placebo: 2 fps

So there's about a 100 fold difference between the slowest and fastest presets.

[tsk1979]

Yes, I am using the x264vfw. I downloaded from videohelp only. Can you point me to a build with "presets".
I saw that when encoding a normal video(some trip + 180degree rotate) full hd it was 8FPS with veryfast in avidemux 64bit.
Virtualdub should give me similar speed too I think.
Is there a way to use the power of GPU. The 460M is a powerful card, as compared to my 2.53 GHz i5(2010 generation)

[poisondeathray]

The x264vfw version with presets/drop down menus is the one by MasterNobody hosted on sourceforge

http://sourceforge.net/projects/x264vfw/

[jagabo]

Is there a way to use the power of GPU. The 460M is a powerful card, as compared to my 2.53 GHz i5(2010 generation)

If you have a quad core or better CPU, the GPU based h.264 encoders are both slower and deliver lower quality than x264 at the veryfast preset. They also don't scale much with GPU power so the difference between $100 and $500 graphics cards isn't great. The exception is Intel's Quick Sync encoder which can encode a little faster on Sandy Bridge, maybe twice as fast on Ivy Bridge or Haswell based CPUs. Quality is still lower though. Most programs that support GPU encoding don't support AviSynth scripts as input.

[tsk1979]

Is there a way to use the power of GPU. The 460M is a powerful card, as compared to my 2.53 GHz i5(2010 generation)

If you have a quad core or better CPU, the GPU based h.264 encoders are both slower and deliver lower quality than x264 at the veryfast preset. They also don't scale much with GPU power so the difference between $100 and $500 graphics cards isn't great. The exception is Intel's Quick Sync encoder which can encode a little faster on Sandy Bridge, maybe twice as fast on Ivy Bridge or Haswell based CPUs. Quality is still lower though. Most programs that support GPU encoding don't support AviSynth scripts as input.

Thanks for this
I will switch off 3D acceration and use only CPU.
I will also download the x264 as specified by you.

For speed I have seen a low rez divX video is the way to go. Quickly encode, see the content, and if satisfied, go for the final veryFast preset run.

[jagabo]

Keep in mind that faster settings result in slightly lower quality. When using CRF mode the general trend is faster settings deliver larger files. But the major differences there are between ultrafast, superfast and veryfast (for example ultrafast files will sometimes be 3x larger than veryfast, depending on the content) . From veryfast on up the differences are pretty small. (typically about 10 percent between veryfast and placebo). I think x264 at ultrafast is about as fast as Divx at it's fastest settings. So there's no reason to use Divx.

I recommend you try different presets and decide for yourself where the best compromise is.

[lordsmurf]

Quality videos takes time.
No way around it.

If you think this is slow, you should see what we had to deal with to encode MPEG-2 in 2001.

[jagabo]

Also, don't forget that encoding speed can be a small part of the overall process if you are running very slow filters.

[tsk1979]

Thanks, I will keep that in mind
jagabo, I think its scriptclip that is killing my speed(along with Lancozresize) rather than the encoding.

What is the preferred video editing methodolgy? Virtualdub+avisynth, or go with Avidemux(64bit advantage)+avisynth

[jagabo]

I use VirtualDub to preview AviSynth scripts. I do my encoding with the x264 CLI encoder. All 32 bit. There's not much difference in speed between 32 bit and 64 bit software when encoding. But with HD material and some filters you may run out of memory with 32 bit filtering.

[poisondeathray]

If these are full res photos, that's not HD, that 7-8x more pixels than 1920x1080 . That's going to be very slow, and even slower with script clip . Very difficult to edit smoothly in any software, especially when you throw various effects/filters on top of that

What I usually do with image sequences (not necessarily timelapses) , is I have 2 folders, one for full resolution, one for "proxy" or lower resolution photos that have been batch resized. Many programs will allow you to switch between them with a button . In avisynth it's simply changing the path . You might use 3 folders: full res, 1920x1080, preview resolution folder

or go with Avidemux(64bit advantage)+avisynth

Theoretical 64bit avidemux advantage is pretty much negated when you use 32bit avisynth (and 64bit avisynth is unstable / full of problems)

[poisondeathray]

Maybe I should have noted this in your other thread, but for your timelapse, most photos have an AR of ~1.5 . 1920x1080 has an AR of ~1.78 (16:9) . If you're not pillarboxing it (black borders on the L+R sides) , or other manipulations like reframing or cropping - you will distort the image

[tsk1979]

Maybe I should have noted this in your other thread, but for your timelapse, most photos have an AR of ~1.5 . 1920x1080 has an AR of ~1.78 (16:9) . If you're not pillarboxing it (black borders on the L+R sides) , or other manipulations like reframing or cropping - you will distort the image

While processing my RAWs I chose a 16:9 crop for output file
Thanks!
For perfomance, I am using Lancos4resize before doing anything else. Here is my script now

s1=ImageSource("...converted\A-%d.jpg", 1, 129, 12).Lanczos4Resize(1920,1080)
.
.
.

This way, all the filters do not get big files.

[jagabo]

Is it just me or does the ScriptClip() segment cause a crash after a while? I can see memory usage increasing until the process (x264 CLI encoder) has eaten up 3.5 GB of memory (about 550 frames into a 1280x720 source video, even with SetMemoryMax(1000) or less) then the process crashes. Maybe this problem is limited to the multithreaded AviSynth? ScriptClip() doesn't work at all if I use SetMtMode(2,n).

[tsk1979]

Happening to me also. I switched to xvid and now the crash happens near the end.


Here is my script

#Luster Stars segment

s1=ImageSource("....converted\A-%d.jpg", 1, 129, 12).Lanczos4Resize(1920,1080)

s1a=Trim(s1,1,50)
s1b=Trim(s1,51,0)
last=s1b
b=blankclip(last)


s1trails=ScriptClip("""
  b = b.Loop(2,0,0).Overlay(last, mode="lighten")
  return b
""")

s1final=s1a+s1trails
s1finalreverse=Reverse(s1final)

#Skibotn clouds segment
sclouds=ImageSource("....converted\A-%d.jpg", 1, 301, 12).Lanczos4Resize(1920,1080)


aurora1=ImageSource("....converted\A-%d.jpg", 1, 92, 12).Lanczos4Resize(1920,1080)
aurora2=ImageSource("....converted\A-%d.jpg", 1, 56, 12).Lanczos4Resize(1920,1080)
aurora3=ImageSource("....converted\A-%d.jpg", 1, 143, 12).Lanczos4Resize(1920,1080)
aurora4=ImageSource("....converted\A-%d.jpg", 1, 48, 12).Lanczos4Resize(1920,1080)
aurora5=ImageSource("....converted\A-%d.jpg", 1, 150, 12).Lanczos4Resize(1920,1080)
aurora6=ImageSource("....converted\A-%d.jpg", 1, 88, 12).Lanczos4Resize(1920,1080)
aurora7=ImageSource("....converted\A-%d.jpg", 1, 269, 12).Lanczos4Resize(1920,1080)
aurora8=ImageSource("....converted\A-%d.jpg", 1, 200, 12).Lanczos4Resize(1920,1080)
aurora9=ImageSource("....converted\A-%d.jpg", 1, 64, 12).Lanczos4Resize(1920,1080)
aurora10=ImageSource("....converted\A-%d.jpg", 1, 133, 12).Lanczos4Resize(1920,1080)

Dissolve(s1,sclouds,aurora1,aurora2,aurora3,s1final,aurora4,aurora5,aurora2,aurora6,aurora1,aurora7,aurora8,aurora9,aurora5,aurora10,s1finalreverse,30).ConvertToRGB32

SmoothDeinterlace(doublerate=true)

[poisondeathray]

I would PM gavino

1) I think MT can be quite unstable with scriptclip

2) I think there are instances where scriptclip a certain way exacerbates a known memory leak - I don' t know the details , I remember reading about it somewhere (PM gavino he probably knows a workaround) .

3) Reverse() is very difficult , and even more difficult with scriptclip . I would consider processing it normally to a lossless intermediate then calling that to reverse()

4) Why are you using smoothdeinterlace?

[tsk1979]

I want to double fps from 12 to 24
Smoothdeinterlace gives good results.

[tsk1979]

@poisondeathray, I think I want to start a new thread on my access error issue. What would be the best forum section for such threads?

[jagabo]

I want to double fps from 12 to 24
Smoothdeinterlace gives good results.

Try ChangeFPS(24) to duplicate frames or ConvertFPS(24) to duplicate and blend frames.

[poisondeathray]

or MVTools2 to interpolate frames . You don't want to deinterlace progressive content to increase the FPS

I would break up the script into sections to reduce the complexity . Perhaps the access error is from reverse() . If you render out the "normal" section to a lossless intermediate you will reduce your memory issues and it will be easier to process (scriptclip with overlay is intensive enough without reverse() thrown in to the mix)

You can ask at Doom9 if you don't get a good answer here

[jagabo]

No, the ScriptClip() memory leak problem isn't caused by Reverse(). Though that may exacerbate it.

[tsk1979]

I removed Reverse as well as Smoothdeinterlace and it still crashes. So its a memory leak in scriptclip

[poisondeathray]

Yes, as mentioned above, there is a known memory leak with scriptclip . Gavino found this with srestore a few years back and re-wrote the function
http://forum.doom9.org/showthread.php?p=1325921#post1325921

Apparently there is a way to write the functions differently to reduce the leak, but it' s WAY above my head . That's why I suggested to PM him . You can read from this post and the next few ones
http://forum.doom9.org/showthread.php?p=1652758#post1652758

Breaking out the script into smaller sections will reduce memory related issues. (Using more memory , or more filters, more functions won't help and will certainly make it worse)

For example, can you render only that 129 frame section with scriptclip separately into a lossless intermediate without crashing ? Use SetMemoryMax(512), use "vanilla" avisynth , not MT . Batch resize the clips beforehand instead of using Lanczos in the script . IMO - having it successfully render at slowly at 0.5 fps is better than not at all ; or having some crash

If not, then you might have to use another method or use an AE trial version like smrpix suggested