HCenc multi-thread capability

[SameSelf]

I just finished a 2-pass encode of a 90 minute video using HCenc and it took nearly 7 hours. My main complaint is it only averaged about 13% CPU usage on my machine. Is there some sort of multi-thread option that I am missing? Thanks.

[LigH.de]

From the documentation of v0.28 (misc\HC028.pdf):

HCenc 028 is a multi-threaded application, it's also possible to run multiple instances of HCenc.

But HCenc can't encode faster than it receives the video stream from the AviSynth 2.6 frameserver. May it be waiting for a very complex script with slow filters, executed in a single-threaded AviSynth? If you have a CPU with 8 logical cores, a maximum utilization of 12.5% (1/8) would be expectable in this case.

[jagabo]

There is an SMP option, but like most MPEG 2 encoders, it's not well multithreaded. Its speed doesn't increase much beyond two threads.

[SameSelf]

Thanks. So my problem is Avisynth? I am loading some HD Lagarith lossless intermediates and doing some downrezzing. Maybe the resizer, BlackmanResize, is the problem? Single threaded?

I have used Avisynth in MT before for QTGMC.

[KarMa]

I deleted my previous comment about QTGMC as you are not using it here.

This topic finally got me to test out HCenc, and got to say it's pretty great. I have a 6 core AMD FX-6300 and I seem to be getting ~65% CPU usage across all the cores during Pass 1, and ~65% CPU usage during Pass 2. Using the best profile.

I used a x264 SD video for this test and so only had avisynth resize horizontally.

[jagabo]

HD Lagarith lossless intermediates

Not a problem.

Maybe the resizer, BlackmanResize, is the problem?

No.

I have used Avisynth in MT before for QTGMC.

QTGMC() definitely needs AviSynth MT. But it sounds like you're not using it here.

HCEnc is the big problem. It can barely use two threads to their fullest. Don't expect it to use more than about 25 percent of an eight thread CPU under the best of circumstances.

[SameSelf]

Sorry for the confusion. No, I am not using QTGMC for this script. I just mentioned it because I have successfully sped up Avisynth in the past using QTGMC, and I currently run the Avisynth MT dll because of QTGMC. So I was wondering if maybe I needed to insert some MT lines into my script.

Thanks jagabo for the clarification. That is too bad to hear though.

I deleted my previous comment about QTGMC as you are not using it here.

This topic finally got me to test out HCenc, and got to say it's pretty great. I have a 6 core AMD FX-6300 and I seem to be getting ~65% CPU usage across all the cores during Pass 1, and ~65% CPU usage during Pass 2. Using the best profile.

I used a x264 SD video for this test and so only had avisynth resize horizontally.

I am going to start calling you Special Sauce. I am sooper jealous. What is your secret other than that you have an Athlon? Also, are you using the HCenc GUI of CLI?

[jagabo]

Lagarith might be a problem if you're reading HD video from a slow drive.

What's your exact script? There are other slow filters, like TemporalDegrain, and pretty much everything that uses mvtools.

[SameSelf]

My script:

video1 = AVISource("E:\1.avi", audio=false).AssumeFPS(30000,1001).ConvertToYV12(interlaced=false, matrix="rec601")
video2 = AVISource("E:\2.avi", audio=false).AssumeFPS(30000,1001).ConvertToYV12(interlaced=false, matrix="rec601")
...
video12 = AVISource("E:\12.avi", audio=false).AssumeFPS(30000,1001).ConvertToYV12(interlaced=false, matrix="rec601")

video1 ++ video2 ++ ... ++ video12

Sharpen(0.5, 0)
BlackmanResize(704, 480, taps=4)
Blur(0, 0.7, false).Sharpen(0, 0.4, false)

Yes, sadly, I have to stitch together 12 different AVIs for this encode. Also, yes, these files are on a slow drive—rust that spins at 7200 rpm through a SATAIII interface. My life sucks.

EDIT: I am not doing any NR, I did all all my post-production in AE before writing out the lossless intermediates. The only thing I am trying to do is get them into a DVD compliant resolution before encoding.

[jagabo]

What's the resolution of the source AVI files? The 1920x1080p30 videos you mentioned in another thread?

[SameSelf]

Yes, 1920x1080p29.97, written out from ProRes HQ 422 masters.

[KarMa]

I am going to start calling you Special Sauce. I am sooper jealous. What is your secret other than that you have an Athlon? Also, are you using the HCenc GUI of CLI?

Yeah just used the standard GUI that came in the .zip file. But my source was 800MB, unlike your 1080p 4:2:2 lossless Lagarith.

[jagabo]

Yes, 1920x1080p29.97

I just tried a 1920x1080p30 lagarith video on a hard drive with your script. The file was about 8 minutes long, 60 GB (big enough that Windows couldn't cache it all in memory). Hcgui encoded it at about 22 frames per second, consuming about 18 percent of the CPU (as reported by TaskManager) on my i5 2500K. It didn't matter much whether I used SMP in hcenc or AviSynth. Reading the lagarith avi was the bottleneck.

Doing the same with a 1920x1080 MPEG 2 source of the same length (a 1 GB file), with SMP, encoded about 3x faster.