x264 limited by CPU cores/threads?

[TE16]

I am thinking about getting a new CPU for x264 encoding but I have seen online that x264 performance is limited to 24 threads per instance, is this true?

I have also seen that too many threads being utilised by one instance of x264 can actually have worst final results, is this also true?

[poisondeathray]

I am thinking about getting a new CPU for x264 encoding but I have seen online that x264 performance is limited to 24 threads per instance, is this true?

The hard limit is set to 128, but I think you can change that if you compile it yourself

I have also seen that too many threads being utilised by one instance of x264 can actually have worst final results, is this also true?

Yes . More threads => lower quality at a given bitrate (worse compression)




There are other factors that come into play, other bottlenecks that affect speed. For example resolution - if you encode a 64x64 video, that's going to be different than UHD . Some encoding settings have poor multithreading and may become bottleneck like --me tesa

[sophisticles]

The hard limit is set to 128, but I think you can change that if you compile it yourself

As you said it's a "hard limit" meaning the code is built around that limit, you can't change that during the compilation process, you would need to go into the code and change it in their but it's not as simple as changing a "128" to some other number, you would need to make significant changes.

Regardless, it's been shown that past a certain point x264 fails to scale linearly and as you already pointed out the quality goes down as the number of threads goes up, there used to be a formula based on resolution and number of b frames for determining the point maximum number of threads you could use before you start to notice the degraded quality.

It's interesting to note that when MSU conducts their yearly codec shootouts and asks the x264 and x265 devs for what command line parameters they should use they are always told "-threads 1", so according to both the x264 and x264 devs when push comes to shove and you want maximum quality for a comparison you only use 1 thread.

[KarMa]

It's interesting to note that when MSU conducts their yearly codec shootouts and asks the x264 and x265 devs for what command line parameters they should use they are always told "-threads 1", so according to both the x264 and x264 devs when push comes to shove and you want maximum quality for a comparison you only use 1 thread.

I checked the H.264/AVC 2012 comparison, the HEVC 2015 comparison (incl. x264), the HEVC 2016 comparison (incl. x264), and the HEVC 2017 comparison (incl. x264). x265 and x264 were included in all the comparisons I listed except for x265 in 2012 (naturally). I don't see any mention of "-threads 1" in any of the settings used which are provided near the end of the comparison PDFs. Nor do I see any settings which would indirectly force 1 thread in either of the x264 and x265 settings used.

My understanding of the thread/quality is that it tends to become noticeable in the double digit range (just off the top of my head) and just keeps getting worse and worse as you go higher. But it would kind of be suicide for x264/x265 developers to ever try and force 1 thread considering the latest tests were performed on a 4 Core/8 Thread Intel CPU which is nothing crazy in need of restricting. If speed were not a factor in these tests and if speeds were not disclosed, then it would probably be wise to use 1 thread just to get that tiny amount of extra potential compression. But speed is certainly a factor in these tests, and they even added minimum fps encoding speeds for each of the parameters (Fast/Universal/Ripping). Cause I guess MSU is tired of some of the proved codecs taking forever.


http://www.compression.ru/video/codec_comparison/h264_2012/mpeg4_avc_h264_video_codecs...comparison.pdf
http://www.compression.ru/video/codec_comparison/hevc_2015/MSU_HEVC_comparison_2015_free.pdf
http://www.compression.ru/video/codec_comparison/hevc_2016/MSU_HEVC_comparison_2016_free.pdf
http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_free.pdf

[poisondeathray]

It' s easier to see the effect of increasing threads on quality "degradation" at lower bitrate ranges. I posted some examples before, but it's quite easy to do a comparison for anybody, just encode something at a lower bitrate range with threads=1 and some high number, then some higher number .

EDIT: I posted a b-frame example of 1 vs 36 here
https://forum.videohelp.com/threads/378203-Captured-a-60gb-2hr-avi-with-Vdub-Huff-enco...e2#post2442659

[jagabo]

To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.

[hello_hello]

The info here says the quality loss is "mostly negligible" until you exceed 16 threads, whatever "mostly negligible" means.
http://www.chaneru.com/Roku/HLS/X264_Settings.htm#threads

[hello_hello]

It's interesting to note that when MSU conducts their yearly codec shootouts and asks the x264 and x265 devs for what command line parameters they should use they are always told "-threads 1", so according to both the x264 and x264 devs when push comes to shove and you want maximum quality for a comparison you only use 1 thread.

I checked the H.264/AVC 2012 comparison, the HEVC 2015 comparison (incl. x264), the HEVC 2016 comparison (incl. x264), and the HEVC 2017 comparison (incl. x264). x265 and x264 were included in all the comparisons I listed except for x265 in 2012 (naturally). I don't see any mention of "-threads 1" in any of the settings used which are provided near the end of the comparison PDFs. Nor do I see any settings which would indirectly force 1 thread in either of the x264 and x265 settings used.

Plus the comparisons include encoding speed, so limiting an encoder to a single thread would put it out the running.
This year the three categories were "Fast", "Universal" and "Ripping" encoding. To qualify for Fast and Universal the encoders had to use settings enabling them to encode at 60fps and 25fps respectively.

[poisondeathray]

Breaking up and stitching later also reduces quality, as more IDR frames are introduced, fewer redundancies can be re-used, motion vectors truncated . The more pieces, the lower the potential quality . Also there is the small chance of non-union or playback "issues"

The info here says the quality loss is "mostly negligible" until you exceed 16 threads, whatever "mostly negligible" means.
http://www.chaneru.com/Roku/HLS/X264_Settings.htm#threads

It depends on your point of view / reference , the specific situation and source, the encoding settings, and especially bitrate range applicable to that source. You can only "easily" see problems when using very low bitrate ranges. At "typical" bitrate ranges you usually cannot see much difference, even on still frames , until the threads get much higher

But it's a very consistent finding, more threads => lower quality. Always. It's measureable with metrics, visible with human eyes in some situations

[jagabo]

Breaking up and stitching later also reduces quality, as more IDR frames are introduced, fewer redundancies can be re-used, motion vectors truncated . The more pieces, the lower the potential quality .

Yes, encoding a three minute long video as 128 segments would be a problem. But encoding a two hour long movie in four segments will introduce 3 extra i-frames at most, negligible.

There are other possible problems with segmented encoding. If you're using bitrate based encoding you won't have optimal bitrate distribution across the entire video. At the joins you may end up with i-frames very close together, resulting in a bitrate spike higher than desired.

[hello_hello]

I always start each segment on the first frame of a scene, so unless the previous scene is between 250 and 275 frames in length (I use the default -keyint 250 and --min-keyint 25) that shouldn't add any extra keyframes, and I'm OCD enough to check the previous scene's length too.... at least some of the time. I even tend to split encodes somewhere where there's not much action happening, if possible.

More often than not I'd run complete encodes simultaneously rather then split them anyway.

[m00511]

To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.

i did this but with ffmpeg and when i concatenate the final file have glithes on the segment point that i breaked,

[jagabo]

To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.

i did this but with ffmpeg and when i concatenate the final file have glithes on the segment point that i breaked,

Did you specify the --stitchable option?

[m00511]

To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.

i did this but with ffmpeg and when i concatenate the final file have glithes on the segment point that i breaked,

Did you specify the --stitchable option?

i tested it now and it was not helpfull,

ffmpeg -i "input.mkv" -c copy -f segment -segment_time "time" -reset_timestamps 1 "output"

, is there any way to do that?maybe i am doing wrong with ffmpeg

ffmpeg -f concat -safe 0 -i "seg.txt" -c copy -fflags +genpts "output"

[Selur]

I think jagabo meant something like:
a. segment the source (at key frames)
b. encode while using x264s stitchable option
c. join the encoded parts

[m00511]

I think jagabo meant something like:
a. segment the source (at key frames)
b. encode while using x264s stitchable option
c. join the encoded parts

how to segment the video at Key Frames? Can i do that with ffmpeg or mkvmerge?

[Selur]

Yes, you can cut at key frame using ffmpeg, read http://trac.ffmpeg.org/wiki/Seeking

[m00511]

Yes, you can cut at key frame using ffmpeg, read http://trac.ffmpeg.org/wiki/Seeking

okay, so i wanna split in every ten minutes because i want to run several simultaneous encodings for each segment, i have cmd batch file for doing it automatically but http://trac.ffmpeg.org/wiki/Seeking didnt example for it, it there a way for doing it?

[Selur]

You have to create a batch file to split every X min. (the segment option might also work, simply it and check if each of the segments can be played on it's own)

[m00511]

You have to create a batch file to split every X min. (the segment option might also work, simply it and check if each of the segments can be played on it's own)

okay, i know it and i have one and after splitting it will encode each file but i am using this

ffmpeg -i "input.mkv" -c copy -f segment -segment_time "time" -reset_timestamps 1 "output"

and

mkvmerge --split size:(for example 700m) -o output.mkv input.mkv

, so both way dose not work means after concatenating the result have a little glitch on splits point, if you can please write me a code

[Selur]

Did you actually try to playback the splitted parts, since I'm not sure that your concatenate line will work on such splitted files when they haven't been reencoded,...
If you split with mkvmerge, I for example would also append the parts with it.

[m00511]

Did you actually try to playback the splitted parts, since I'm not sure that your concatenate line will work on such splitted files when they haven't been reencoded,...
If you split with mkvmerge, I for example would also append the parts with it.

for example i have a Full BD Disk so i want to have it on my hard drive in a smaller size , i am re-encode it but i have a high core cpu (36 cores) so the x264 does not use a whole power of cpu and it's take about 100 fps but i can build a remux of the Disc and then split that mkv file into a smaller file size (like a 10 GB to 1 GB Files) then re-encoding each file , i am getting about 200 to 250 fps here, so yes splitted parts will play properly and after re-encoding and concatenating the result will play properly but in the splits point i have a little glitch but another part o movie is good

[jagabo]

What I was suggesting was something along the lines of this:

start /b /low "part1" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:00:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.1.mkv"
start /b /low "part2" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:05:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.2.mkv"
start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:10:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.3.mkv"
start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:15:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.4.mkv"

pause

echo file '%~1.1.mkv' > list.txt
echo file '%~1.2.mkv' >> list.txt
echo file '%~1.3.mkv' >> list.txt
echo file '%~1.4.mkv' >> list.txt

"g:\program files\ffmpeg\bin\ffmpeg.exe" -y -safe 0 -f concat -i list.txt -c copy %1.new.mkv

del "%~1.1.mkv"
del "%~1.2.mkv"
del "%~1.3.mkv"
del "%~1.4.mkv"
del list.txt

Put that in a batch file (you'll have to change the path to ffmpeg.exe) and drag/drop a video on it. It starts four instances of ffmpeg to encode 5 minutes each of the source. Once the encodings are done press Enter to concatenate the new segments and delete them. I only have a quad core CPU so I set the encoding up to use only one thread. Adjust that to an appropriate value for your setup. And you'll want to adjust the length and number of segments for your source.

[Atak_Snajpera]

What I was suggesting was something along the lines of this:

start /b /low "part1" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:00:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.1.mkv"
start /b /low "part2" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:05:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.2.mkv"
start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:10:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.3.mkv"
start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:15:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.4.mkv"

pause

echo file '%~1.1.mkv' > list.txt
echo file '%~1.2.mkv' >> list.txt
echo file '%~1.3.mkv' >> list.txt
echo file '%~1.4.mkv' >> list.txt

"g:\program files\ffmpeg\bin\ffmpeg.exe" -y -safe 0 -f concat -i list.txt -c copy %1.new.mkv

del "%~1.1.mkv"
del "%~1.2.mkv"
del "%~1.3.mkv"
del "%~1.4.mkv"
del list.txt

Put that in a batch file (you'll have to change the path to ffmpeg.exe) and drag/drop a video on it. It starts four instances of ffmpeg to encode 5 minutes each of the source. Once the encodings are done press Enter to concatenate the new segments and delete them. I only have a quad core CPU so I set the encoding up to use only one thread. Adjust that to an appropriate value for your setup. And you'll want to adjust the length and number of segments for your source.

Are you 100% that ffmpeg is frame accurate?

[jagabo]

Are you 100% that ffmpeg is frame accurate?

No, but the batch file worked without duplicate or missing frames at the seams on the samples I tried. I even used AviSynth scripts to check the new vs. original video.

I thought I might have to change the segment lengths to 4.59.999 or something like that but it wasn't necessary. Of course, it might make a difference with some other videos.

[Atak_Snajpera]

Are you 100% that ffmpeg is frame accurate?

No, but the batch file worked without duplicate or missing frames at the seams on the samples I tried. I even used AviSynth scripts to check the new vs. original video.

I thought I might have to change the segment lengths to 4.59.999 or something like that but it wasn't necessary. Of course, it might make a difference with some other videos.

I would be more confident if FFMPEG allowed frame range instead of timecodes. For example 0-1999 , 2000-2999 and so on.

Btw. you can improve your script like this

start /b /low "part1" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:00:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.1.mkv"
start /b /low "part2" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:05:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.2.mkv"
start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:10:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.3.mkv"
start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:15:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.4.mkv"

:LOOP
tasklist | find /i "ffmpeg.exe" >nul 2>&1
IF ERRORLEVEL 1 (
  GOTO CONTINUE
) ELSE (
  Timeout /T 5 /Nobreak
  GOTO LOOP
)

:CONTINUE
echo file '%~1.1.mkv' > list.txt
echo file '%~1.2.mkv' >> list.txt
echo file '%~1.3.mkv' >> list.txt
echo file '%~1.4.mkv' >> list.txt

"g:\program files\ffmpeg\bin\ffmpeg.exe" -y -safe 0 -f concat -i list.txt -c copy %1.new.mkv

del "%~1.1.mkv"
del "%~1.2.mkv"
del "%~1.3.mkv"
del "%~1.4.mkv"
del list.txt