Fastest possible mp4 video encoding solution (quality is not important)

[rezpower]

Hello guys,

I edit 4K and HD mp4 videos with adobe premiere. The problem is when editing these videos, the seeking and playback becomes very laggy
due to high video quality and it makes the editing painful. What I do is that I re-encode my video to much lower resolution and then do my
work on the low quality edited video and all goes nice and smooth. Once editing is done, I replace the file with the original and do my export.

The only problem is that I loose a lot of time doing the conversion. It would be really great if someone would help me find a way to do this the
fastest possible. What software can do this fast (or what settings/profiles) should I use to make sure that the software is not loosing time trying
to get best possible quality.

[poisondeathray]

What kind of GPU do you have ?

Fastest is going to be NVEnc based solution if you have a pascal GPU , by a large margin. But even Maxwell v2 or v1 will be significantly faster than "CPU" based encoder

I-frame (intra) only encoding will edit the fastest (seek latency the lowest), but you need larger filesizes for the same level of "quality". If you're resizing to smaller resolutions, you might consider a compromise to a short GOP interval like 1 second

[rezpower]

Hi and thanks for your response. My GPU is a NVIDIA K3000M. For the conversion what I have done till now is to resize the videos to 640x360 which is largely enough for the small adobe preview window. Once this is done seeking and editing is real smooth and I can do it easily. Only thing I'm struggling with is the time it takes to convert as I have hundreds of 20 to 40 minute clips that I have to edit and each time I loose a lot of time converting.

[jagabo]

Why don't you just encode the proxy files for you next project at low priority in the background while working on the current project?

QSVEncC on my i5 2500K encodes a 4K AVC source downscaled to 640x360 to h.264 all intra-frame at about 70 frames per second (decoding the source is the limiting factor). Without any noticeable drag on the system. Using x264 with the ultrafast preset and keyint=1 is about the same.

[poisondeathray]

Why don't you just encode the proxy files for you next project at low priority in the background while working on the current project?

QSVEncC on my i5 2500K encodes a 4K AVC source downscaled to 640x360 to h.264 all intra-frame at about 70 frames per second (decoding the source is the limiting factor). Without any noticeable drag on the system. Using x264 with the ultrafast preset and keyint=1 is about the same.

But what are your CPU/ GPU usages ?

Yes, the bottleneck will be SW/CPU decoding UHD/4K , and that will in turn reduce the amount of CPU for CPU encoding for other tasks

You can use combinations of GPU/CPU decode / encode / scale with HW or software , using a mixed pipeline with intel GPU / NV GPU / software to full utilize resources

**BUT, in my experience, GPU decoding isn't very reliable unless it's indexed. You can get problems with mixed frames, dropped frames, corrupt frames. The only reliable way that I know of is through avisynth , DGSource. It's not free and avisynth adds some overhead and extra steps...

For NVEnc, you can only run 2 simultaneous instances on a single Nvidia (GTX) consumer card, not sure about Quicksync I think it might be limited to 1 . But Maxwell v2 Quadros and above can run unlimited sessions, you are basically limited by GPU memory .


Unfortunately the OP has an older kepler generation card, it's significantly slower than the newer generation cards. But I still suspect some combination of GPU / CPU with multiple instances will be faster than CPU only.

[jagabo]

Why don't you just encode the proxy files for you next project at low priority in the background while working on the current project?

QSVEncC on my i5 2500K encodes a 4K AVC source downscaled to 640x360 to h.264 all intra-frame at about 70 frames per second (decoding the source is the limiting factor). Without any noticeable drag on the system. Using x264 with the ultrafast preset and keyint=1 is about the same.

But what are your CPU/ GPU usages ?

Yes, the bottleneck will be SW/CPU decoding UHD/4K , and that will in turn reduce the amount of CPU for CPU encoding for other tasks

The background encodings are consuming 100 percent of the CPU . But they are running at idle priority so they have essentially no impact on foreground processes -- they only get CPU time that higher priority process don't need. Task priority doesn't directly control how much CPU time a program gets, it controls which programs get more CPU time when multiple programs are requesting it -- higher priority process get more CPU time. When only one process is requesting CPU time it gets as much as it wants, regardless of its priority. An editor running at normal priority in the foreground will run just as if the background encodings weren't taking place because they are only getting CPU time that would otherwise go to System Idle Process (disk I/O may have some small impact here and there, especially on a laptop with a single hard drive).

You can use Task Manager to set the priority of the background encodings after they're started. Or explicitly start them at "low" priority using a batch file with:

start /b /low "window name" x264.exe --preset=slow etc.

I run almost all my encodings this way so the computer is still usable while the encodings are going on. Of course, if the foreground processes are using lots of CPU time the background processes take longer to complete. But most other things I do with the computer are only using a few percent of the CPU. I usually don't care if my encoding takes 5 percent longer because I'm browsing the internet, checking my email, or writing some document.


Or the OP could just set up a batch process to encode all his proxy files overnight. There's a big thread around here with many examples of how to batch process with ffmpeg.

[poisondeathray]

Yes for sure you can set lower priority, you can even set affinity for specific cores reserved for PP or whatever other tasks .

My point is you can offload some of that CPU work or do some work with GPU. If you offload some tasks like encoding, resizing to GPU, that leaves more CPU for other things. For example 2 SW decode/ NVEnc processes will encode faster (total fps) than 1 GPU or CPU only process . If you're getting 70FPS total for 1 GPU or CPU, you might get 110FPS for 2 (it won't scale perfectly)

QS on an old HD4600 using the fastest /low quality preset actually encodes at 170-180 FPS with full HW decode/resize/encode from a UHD source => 640x360 , but some corrupt green frames . The only usable ways are CPU decoding or GPU indexed

[rezpower]

Thanks a lot for all the precious info and help. I could go the batch convert way over night as some suggested but I almost never know what videos I will be editing first in advance. So best option for me is re-encode a video in the background while editing another or re-encode first and edit after. The laptop I use to work with is now an old one but still performant.
Its an HP Elitebook8770W Work Station with Nvidia Quadro K3000M gpu and Intel i7-3630QM quad core cpu@2.40GHz. I did a little testing on my side and re-encoded few of my files using a free software called XMedia Recode. The result output files were totally acceptable @lowest settings (down scaled to 640X360 mp4). I was getting an around 270 fps during the encode process.

Now only remaining question for me is that to achieve similar result is there a software that can do the job faster? Also knowing that adobe premiere supports other formats like: AVI, DV, M1V, M2TS, M4V, MOV, MPEG, MTS, VOB, WMV, is there a format that could lead into a faster result over the original mp4?

[poisondeathray]

Thanks a lot for all the precious info and help. I could go the batch convert way over night as some suggested but I almost never know what videos I will be editing first in advance. So best option for me is re-encode a video in the background while editing another or re-encode first and edit after. The laptop I use to work with is now an old one but still performant.
Its an HP Elitebook8770W Work Station with Nvidia Quadro K3000M gpu and Intel i7-3630QM quad core cpu@2.40GHz. I did a little testing on my side and re-encoded few of my files using a free software called XMedia Recode. The result output files were totally acceptable @lowest settings (down scaled to 640X360 mp4). I was getting an around 270 fps during the encode process.

Now only remaining question for me is that to achieve similar result is there a software that can do the job faster? Also knowing that adobe premiere supports other formats like: AVI, DV, M1V, M2TS, M4V, MOV, MPEG, MTS, VOB, WMV, is there a format that could lead into a faster result over the original mp4?

But that 270fps was for 1920x1080 => 640x360 test , right ?

The format doesn't really matter, because it's not the bottleneck. Your "job" is to identify where is the bottleneck and how to remove it, and/or how to allocate other resources (if you have 2 idle GPU's , you should be able to use them in some fashion , either parallel or in the same process). Take a look at CPU and GPU usage, and if they are not 100%, that means you have "idle" resources .

So it depends on what you used for decoding, scaling, and encode in xmediarecode. That 270fps is probably a decoding or 1080p downscaling bottleneck because 640x360 should encode much faster. x264 can encode a 2000-4000fps without bottleneck at that resolution, but in real usage scenario you are limited by decoding speed or something like filters (scaling)

[rezpower]

But that 270fps was for 1920x1080 => 640x360 test , right ?

The format doesn't really matter, because it's not the bottleneck. Your "job" is to identify where is the bottleneck and how to remove it, and/or how to allocate other resources (if you have 2 idle GPU's , you should be able to use them in some fashion , either parallel or in the same process). Take a look at CPU and GPU usage, and if they are not 100%, that means you have "idle" resources .

So it depends on what you used for decoding, scaling, and encode in xmediarecode. That 270fps is probably a decoding or 1080p downscaling bottleneck because 640x360 should encode much faster. x264 can encode a 2000-4000fps without bottleneck at that resolution, but in real usage scenario you are limited by decoding speed or something like filters (scaling)

Oh now I feel really stupid as I never thought about it this way! at no moment I was thinking that the decoding was the process that was slowing things down! Yes the 270fps was for 1920x1080 => 640x360. I did a new test and while encoding a video I was editing another encoded one in premiere and having the encode run in background did not have any disturbing impact on my editing work! So the problem is solved for me. While I edit I encode the next video and editing always take me more time then the encode.

[poisondeathray]

You can probably change the scaling algorithm and get it maybe 10-25% faster, assuming you're using default scaling . Something like a bilinear resizer will be faster than a bicubic or other sharper resizer like lanczos 3tap or spline 16/36 . Most default scalers use bicubic . But test to see if it's too "blurry" for your proxy