Converting h264 to avi

[ziggy1971]

I need some help speeding up my h264 to avi conversion.

I'm going from either mkv or mp4 to avi (CineForm).

This is a script I'm using:

LoadPlugin("D:\MeGUI 2525 x86\tools\ffms\ffms2.dll")
SetMTMode(5, 8)
FFVideoSource("I:\Temp\ROVERS\Beginning.mkv", fpsnum=24000, fpsden=1001, threads=1)

As you can see, there is little CPU usage.
Any suggestions on what I can do to speed things up. ie. different decoder, Avisynth versions, scripts, etc.

If I extract the .264 elementary stream and remux to mp4 GoPro CineForm Studio can transcode it in nearly half the time with about 50% CPU usage.

Thanx

[LigH.de]

H.264 (MPEG-4 AVC) is content. AVI is a container. You cannot "convert" content to a container. Your target content format is GoPro CineForm.
_

Apart from that, you should not try to have one input file read by multiple instances of a source plugin, that would possibly result in competitive threads accessing different parts of the source at the same time, stressing the harddisk in the worst case so that it will slow down the whole process. Setting an MT mode in a script with only a source function, but no filters afterwards, is completely superfluous (and mode 5 would load only one instance, anyway). Parallelizing in AviSynth MT should only apply to video processing filters, not to a source filter.

On the other hand, you limited the decoder threads of FFVideoSource to only 1. Is that necessary to have the source video decoded without errors? Try to keep the decoder threads set up automatically by omitting ', threads=1' and see if the decoded video is still correct. Alternatively, try the same with LwLibavVideoSource from L-SMASH Works.

And finally: What is the reason to resample the fps while reading the video?
_

P.S.: In general, little CPU usage means that either the CPU is waiting for other hardware, or only one core of many is used. Maybe both.

[ziggy1971]

I know that H.264 (MPEG-4 AVC) is content/codec and AVI is a container. However, x/h264 can be extracted/demuxed into an elementary/raw stream. While the elementary/raw stream using the CineForm codec cannot be extracted thereby remaining in the avi container. I know it makes for unnecessary confusion, but it is what it is I guess.

Now, moving forward.

I have no idea what you mean by this:

you should not try to have one input file read by multiple instances of a source plugin

Is it the Active MT mode?

I used MeGUI to open and index the file and left the defaults in the script. Therefore, the ", fpsnum=24000, fpsden=1001, threads=1" is included in the script. Previous versions of MeGUI only included the ", threads=1", but being a newer version I simply assumed the ", fpsnum=24000, fpsden=1001" was needed now.

Is that necessary to have the source video decoded without errors?

I certainly hope that ALL my videos are decoded without errors, doesn't make for good viewing/processing when having unnecessary errors.

[LigH.de]

I did not see you mention that this script was generated by MeGUI ... it may have inserted "threads=1" for reliability and stability. There are some cases where video is decoded wrong when "threads=1" is omitted. But they should be rare. Therefore you should try to remove this part, allowing FFMS2 to use more decoder threads, that may speed up things a bit.

The second line with SetMTMode(5, 8) is superfluous when you have only a source filter, but no more processing filters. It is quite useless to set an MT mode when there is no activity which would profit from parallelization. When access to a file or other hardware is involved, multithreading may even cause issues. So better remove this second line completely, if there are no more following filters.

Specifying fps resampling is usually not only "not necessary" when the source video has a constant frame rate which is detected correctly; it may even cause problems when the declared values don't match the source frame rate (like lost or doubled frames, thus jerky motion). A MediaInfo analysis of your source may tell us if your source is simply compatible, so you can omit the fps parameters.

[ziggy1971]

I think I saw a script somewhere using the SetMTMode(5, 8), so I was giving that a try. Obviously I'm no expert with the scripts or plugins and how they interact with each other.

I'm not familiar with LwLibavVideoSource. Is it faster, better quality, etc.?

[pandy]

Instead AVISynth use ffmpeg directly - i see nothing in your script requiring AVS, perhaps x264 will support your source directly - try it.

[LigH.de]

Multi-threading in AviSynth may speed up the filtering. But it cannot speed up reading and decoding the source. Not limiting the decoder may do so.

L-SMASH Works is an alternative to FFMS2. They are very similar, yet different in a few details (e.g. L-SMASH Works always relied on L-SMASH or libavformats as demultiplexers for MP4/MOV/3GPP or any other container, while FFMS2 used other software which turned out to be not reliable, except for MKV, so it changed to libavformats recently as well). How reliable FFMS2 works, depends a lot on the version you are using. Speed should be almost the same under the same conditions. Quality is certainly the same, because both use libavcodec as decoders.

__

@ pandy:

ziggy1971 wants to create CineForm in AVI; x264 won't help here.

But ffmpeg may if it contains a CineForm encoder. Which I doubt. Not even my latest "license-free" ffmpeg build contains even a decoder. He will need the VfW codec shipped with GoPro software.

[hello_hello]

As you can see, there is little CPU usage.
Any suggestions on what I can do to speed things up. ie. different decoder, Avisynth versions, scripts, etc.

If I extract the .264 elementary stream and remux to mp4 GoPro CineForm Studio can transcode it in nearly half the time with about 50% CPU usage.

Thanx

Just some thoughts..... higher CPU usage when decoding mightn't necessarily be ideal (ASVMeter is just measuring the Avisynth output isn't it?) because the encoding process would require a fair percentage of the CPU's time.

I've never used AVSMeter but if it's showing an average frame rate of about 80fps I'd imagine the encoding process would be lucky to keep up.

Does GoPro CineForm Studio use the same encoder and settings as MeGUI?

What's the CPU usage like when MeGUI is encoding and what's the speed?

ffmsindex is generally best left in single threaded mode, in my opinion. Or change the thread count in MeGUI's settings, and if it crashes or doesn't decode properly, enable threads=1 again.

There'd be no harm in indexing with L-Smash and trying again.

The main reason I tend to use ffmsindex is MeGUI can tell it only to index the video. L-Smash indexes everything, which usually takes longer. No big deal for small files, but for large ones.....

Without any filtering in a script you should be able to decode/encode while keeping the CPU pretty busy, especially at 1080p. If it's not busy for some reason (which would generally be slow Avisynth filtering), you can run more than one encode simultaneously. MeGUI lets you run jobs in a temporary "worker" via the right click menu in the job queue, or you can use the "Worker" menu at the top to create additional workers. In the real world you'd probably mostly create just a second worker, add a bunch of encoding jobs to the queue, click the start button, and they'll run two at a time.

For single encoding jobs you can always copy a script, use each copy to encode half the video, run them simultaneously, and append the encoded video with MKVMergeGUI. The "stitchable" option in the x264 encoder configuration under the Misc tab should be enabled for that method.

MeGUI adds the frame rate conversion stuff to the script for ffmsindex.... well because I thought it might be a good idea and nagged Zathor.
It already adds conversion when decoding via DirectShow (using DirectShow's convertfps=true), it adds AssumeFPS() instead for AVISource (which I put in the "must have seemed like a good idea at the time" category), and it seemed like it'd be potentially clever for MeGUI to add it for the other indexers too, because then when variable frame rate goes in, at least something with audio sync comes out, even if it's not ideal.
It was also because ffmsindex and L-Smash treat constant frame rate video with null frames (I think that's the description) as variable frame rate, even for AVI which is theoretically never VFR, so they output the average frame rate and it ends unpleasantly. The frame rate conversion was supposed to prevent that.
The idea was if the frame rate is added to the script matching the source frame rate it'd effectively do nothing, and for VFR video it'd at least convert to a CFR. It was implemented for ffmsindex but not for L-Smash (I don't think it had frame rate conversion back then) and it's often a good thing, but sometimes not. It sometimes causes ffmsindex to drop or duplicate frames when the source is AVI for no apparent reason (when the frame rate shouldn't be changing) and very occasionally MeGUI gets the frame rate wrong which obviously isn't ideal. For good quality sources, it realy doesn't matter if it's there or not.

If you open video in MeGUI's script creator and MeGUI adds the conversion stuff, you can check the total frame count in the preview, remove the frame rate stuff from the script and refresh the preview. If the frame count doesn't change (or maybe only changes by one) the frame rate conversion isn't necessary. If the total frame count changes by more than a couple of frames, it probably needs to be there, or something's up. For standard Bluray sources, it should never be needed.

You can also info() to the end of a script in order to preview the frame rate that way, with and without the frame rate conversion. If you remove it and the frame rate doesn't change you don't need it.

[LigH.de]

I've never used AVSMeter but if it's showing an average frame rate of about 80fps I'd imagine the encoding process would be lucky to keep up.

AVSMeter is a tool specifically designed to benchmark AviSynth scripts by requesting but ignoring its output (similar to "> NUL"). Groucho2004 took some efforts to make the results as reliable as possible.

If you feel like your conversion is slow, but AVSMeter confirms a rather quick delivery of the video source by AviSynth, then you can be quite certain that most of the time is spent in the encoder, and trying to optimize the AviSynth script won't widen the bottleneck.

[ziggy1971]

OK, just so we're clear.
I DON'T use MeGUI for encoding TO CineForm.
I DON'T use GoPro CineForm Studio TO encode h/x264.
I usually use MeGUI to open the source file (because it opens virtually anything) and make the script, if it can.

The saved script is opened in VirtualDub, I load my preset settings (the CineForm codec is available in VirtualDub), then I save to a new avi file on a different hard drive than the source so that the source HDD is only reading and the target HDD is only writing.

When encoding x264 with MeGUI my CPU uses 99-100% of all 12 threads and around 80-120 fps for 1080p, temperature is around 75-85 deg C and vCore is around 1.456v depending on content.

Here is a new test with the same file, only I removed the ", fpsnum=24000, fpsden=1001, threads=1" and "SetMTMode(5, 8)"


Obviously, the script and decoder are working much better.

However, just for the heck of it, I opened the script in GraphStudio and much to my surprise "Nero Resize"?


If that's the chain the video is using for decoding, it might explain some of the slowdown. If so, how do I get rid of the "Nero Resize"?

Edit:
Encoding is now down to 0:09:31 or up to 112.25 fps with FFVideoSource
Encoding is now down to 0:09:32 or up to 112.05 fps with L-SMASH Works
Encoding is now down to 0:07:40 or up to 139.33 fps with GoPro Studio

[hello_hello]

When encoding x264 with MeGUI my CPU uses 99-100% of all 12 threads and around 80-120 fps for 1080p, temperature is around 75-85 deg C and vCore is around 1.456v depending on content.

At least that's what you'd expect, or hope for. Was that before you removed the frame rate conversion and threads=1?

Here is a new test with the same file, only I removed the ", fpsnum=24000, fpsden=1001, threads=1" and "SetMTMode(5, 8)"
Obviously, the script and decoder are working much better.

Most of that's probably due to removing threads=1 rather than the frame rate stuff, but if the CPU was running flat out for x264 encoding before that stuff was removed, it'll probably only make a small difference to encoding speed when it is.

However, just for the heck of it, I opened the script in GraphStudio and much to my surprise "Nero Resize"?
If that's the chain the video is using for decoding, it might explain some of the slowdown. If so, how do I get rid of the "Nero Resize"?

I don't know a lot about the graph edit thing, but I don't think Nero resize would be involved, assuming VirtualDub is encoding with a video for windows codec which I assume the CineForm encoder is? Maybe it's a limitation of VFW. Someone else might know. Are you using "fast recompress" as the video compression method for VirtualDub, just to make sure it's not doing anything like converting to RGB. I think it still does by default for the other compression methods.

PS You could probably use the ffmpeg input driver plugin to open the source directly rather than via Avisynth, given you're only using AVIsynth for decoding. It probably won't speed up encoding, but it'd save having to create a script. It's listed on VideoHelp's VirtualDub page.

[ziggy1971]

Yeah, I'm using "fast recompress" in VirtualDub.

Encoding x264 isn't a problem and I probably know more about that than decoding it.
I was just looking for a way to decode the x264 to a CineForm avi without having to demux the video from mkv then remux it to mp4 so the GoPro Studio would recognize the video.

[LigH.de]

VirtualDub will certainly not use a DirectShow filter chain to serve VfW codecs with video input; GraphEdit will not show you what VirtualDub does. As long as you don't use DirectShowSource in AviSynth, this Nero filter should not be any concern during the conversion.

How to remove it ... you would have to discover which *.ax or *.dll file contains this filter, unregister it (regsvr32), then delete it. GSpot 2.70a may be outdated as media analysis tool, but can help you identifying DirectShow filters. RadLight Filter Manager would be another tool suitable for managing your installed filters.