VideoHelp Forum
+ Reply to Thread
Results 1 to 26 of 26
Thread
  1. Member
    Join Date
    Jul 2016
    Location
    United Kingdom
    Search PM
    I am thinking about getting a new CPU for x264 encoding but I have seen online that x264 performance is limited to 24 threads per instance, is this true?

    I have also seen that too many threads being utilised by one instance of x264 can actually have worst final results, is this also true?
    Quote Quote  
  2. Originally Posted by TE16 View Post
    I am thinking about getting a new CPU for x264 encoding but I have seen online that x264 performance is limited to 24 threads per instance, is this true?


    The hard limit is set to 128, but I think you can change that if you compile it yourself


    I have also seen that too many threads being utilised by one instance of x264 can actually have worst final results, is this also true?
    Yes . More threads => lower quality at a given bitrate (worse compression)




    There are other factors that come into play, other bottlenecks that affect speed. For example resolution - if you encode a 64x64 video, that's going to be different than UHD . Some encoding settings have poor multithreading and may become bottleneck like --me tesa
    Quote Quote  
  3. Originally Posted by poisondeathray View Post
    The hard limit is set to 128, but I think you can change that if you compile it yourself
    As you said it's a "hard limit" meaning the code is built around that limit, you can't change that during the compilation process, you would need to go into the code and change it in their but it's not as simple as changing a "128" to some other number, you would need to make significant changes.

    Regardless, it's been shown that past a certain point x264 fails to scale linearly and as you already pointed out the quality goes down as the number of threads goes up, there used to be a formula based on resolution and number of b frames for determining the point maximum number of threads you could use before you start to notice the degraded quality.

    It's interesting to note that when MSU conducts their yearly codec shootouts and asks the x264 and x265 devs for what command line parameters they should use they are always told "-threads 1", so according to both the x264 and x264 devs when push comes to shove and you want maximum quality for a comparison you only use 1 thread.
    Quote Quote  
  4. Dinosaur Supervisor KarMa's Avatar
    Join Date
    Jul 2015
    Location
    US
    Search Comp PM
    Originally Posted by sophisticles View Post
    It's interesting to note that when MSU conducts their yearly codec shootouts and asks the x264 and x265 devs for what command line parameters they should use they are always told "-threads 1", so according to both the x264 and x264 devs when push comes to shove and you want maximum quality for a comparison you only use 1 thread.
    I checked the H.264/AVC 2012 comparison, the HEVC 2015 comparison (incl. x264), the HEVC 2016 comparison (incl. x264), and the HEVC 2017 comparison (incl. x264). x265 and x264 were included in all the comparisons I listed except for x265 in 2012 (naturally). I don't see any mention of "-threads 1" in any of the settings used which are provided near the end of the comparison PDFs. Nor do I see any settings which would indirectly force 1 thread in either of the x264 and x265 settings used.

    My understanding of the thread/quality is that it tends to become noticeable in the double digit range (just off the top of my head) and just keeps getting worse and worse as you go higher. But it would kind of be suicide for x264/x265 developers to ever try and force 1 thread considering the latest tests were performed on a 4 Core/8 Thread Intel CPU which is nothing crazy in need of restricting. If speed were not a factor in these tests and if speeds were not disclosed, then it would probably be wise to use 1 thread just to get that tiny amount of extra potential compression. But speed is certainly a factor in these tests, and they even added minimum fps encoding speeds for each of the parameters (Fast/Universal/Ripping). Cause I guess MSU is tired of some of the proved codecs taking forever.


    http://www.compression.ru/video/codec_comparison/h264_2012/mpeg4_avc_h264_video_codecs...comparison.pdf
    http://www.compression.ru/video/codec_comparison/hevc_2015/MSU_HEVC_comparison_2015_free.pdf
    http://www.compression.ru/video/codec_comparison/hevc_2016/MSU_HEVC_comparison_2016_free.pdf
    http://www.compression.ru/video/codec_comparison/hevc_2017/MSU_HEVC_comparison_2017_free.pdf
    Quote Quote  
  5. It' s easier to see the effect of increasing threads on quality "degradation" at lower bitrate ranges. I posted some examples before, but it's quite easy to do a comparison for anybody, just encode something at a lower bitrate range with threads=1 and some high number, then some higher number .

    EDIT: I posted a b-frame example of 1 vs 36 here
    https://forum.videohelp.com/threads/378203-Captured-a-60gb-2hr-avi-with-Vdub-Huff-enco...e2#post2442659
    Quote Quote  
  6. To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.
    Quote Quote  
  7. The info here says the quality loss is "mostly negligible" until you exceed 16 threads, whatever "mostly negligible" means.
    http://www.chaneru.com/Roku/HLS/X264_Settings.htm#threads
    Quote Quote  
  8. Originally Posted by KarMa View Post
    Originally Posted by sophisticles View Post
    It's interesting to note that when MSU conducts their yearly codec shootouts and asks the x264 and x265 devs for what command line parameters they should use they are always told "-threads 1", so according to both the x264 and x264 devs when push comes to shove and you want maximum quality for a comparison you only use 1 thread.
    I checked the H.264/AVC 2012 comparison, the HEVC 2015 comparison (incl. x264), the HEVC 2016 comparison (incl. x264), and the HEVC 2017 comparison (incl. x264). x265 and x264 were included in all the comparisons I listed except for x265 in 2012 (naturally). I don't see any mention of "-threads 1" in any of the settings used which are provided near the end of the comparison PDFs. Nor do I see any settings which would indirectly force 1 thread in either of the x264 and x265 settings used.
    Plus the comparisons include encoding speed, so limiting an encoder to a single thread would put it out the running.
    This year the three categories were "Fast", "Universal" and "Ripping" encoding. To qualify for Fast and Universal the encoders had to use settings enabling them to encode at 60fps and 25fps respectively.
    Quote Quote  
  9. Breaking up and stitching later also reduces quality, as more IDR frames are introduced, fewer redundancies can be re-used, motion vectors truncated . The more pieces, the lower the potential quality . Also there is the small chance of non-union or playback "issues"

    Originally Posted by hello_hello View Post
    The info here says the quality loss is "mostly negligible" until you exceed 16 threads, whatever "mostly negligible" means.
    http://www.chaneru.com/Roku/HLS/X264_Settings.htm#threads
    It depends on your point of view / reference , the specific situation and source, the encoding settings, and especially bitrate range applicable to that source. You can only "easily" see problems when using very low bitrate ranges. At "typical" bitrate ranges you usually cannot see much difference, even on still frames , until the threads get much higher

    But it's a very consistent finding, more threads => lower quality. Always. It's measureable with metrics, visible with human eyes in some situations
    Quote Quote  
  10. Originally Posted by poisondeathray View Post
    Breaking up and stitching later also reduces quality, as more IDR frames are introduced, fewer redundancies can be re-used, motion vectors truncated . The more pieces, the lower the potential quality .
    Yes, encoding a three minute long video as 128 segments would be a problem. But encoding a two hour long movie in four segments will introduce 3 extra i-frames at most, negligible.

    There are other possible problems with segmented encoding. If you're using bitrate based encoding you won't have optimal bitrate distribution across the entire video. At the joins you may end up with i-frames very close together, resulting in a bitrate spike higher than desired.
    Quote Quote  
  11. I always start each segment on the first frame of a scene, so unless the previous scene is between 250 and 275 frames in length (I use the default -keyint 250 and --min-keyint 25) that shouldn't add any extra keyframes, and I'm OCD enough to check the previous scene's length too.... at least some of the time. I even tend to split encodes somewhere where there's not much action happening, if possible.

    More often than not I'd run complete encodes simultaneously rather then split them anyway.
    Quote Quote  
  12. Originally Posted by jagabo View Post
    To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.
    i did this but with ffmpeg and when i concatenate the final file have glithes on the segment point that i breaked,
    Quote Quote  
  13. Originally Posted by m00511 View Post
    Originally Posted by jagabo View Post
    To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.
    i did this but with ffmpeg and when i concatenate the final file have glithes on the segment point that i breaked,
    Did you specify the --stitchable option?
    Quote Quote  
  14. Originally Posted by jagabo View Post
    Originally Posted by m00511 View Post
    Originally Posted by jagabo View Post
    To use a large number of threads you can run several simultaneous encodings with fewer threads each. For a single video, break it into segments encoded separately and simultaneously, then concatenate them when done. Be sure to use the --stitchable option.
    i did this but with ffmpeg and when i concatenate the final file have glithes on the segment point that i breaked,
    Did you specify the --stitchable option?
    i tested it now and it was not helpfull,
    Code:
    ffmpeg -i "input.mkv" -c copy -f segment -segment_time "time" -reset_timestamps 1 "output"
    , is there any way to do that?maybe i am doing wrong with ffmpeg
    Code:
    ffmpeg -f concat -safe 0 -i "seg.txt" -c copy -fflags +genpts "output"
    Quote Quote  
  15. I think jagabo meant something like:
    a. segment the source (at key frames)
    b. encode while using x264s stitchable option
    c. join the encoded parts
    users currently on my ignore list: deadrats, Stears555
    Quote Quote  
  16. Originally Posted by Selur View Post
    I think jagabo meant something like:
    a. segment the source (at key frames)
    b. encode while using x264s stitchable option
    c. join the encoded parts
    how to segment the video at Key Frames? Can i do that with ffmpeg or mkvmerge?
    Quote Quote  
  17. Yes, you can cut at key frame using ffmpeg, read http://trac.ffmpeg.org/wiki/Seeking
    users currently on my ignore list: deadrats, Stears555
    Quote Quote  
  18. Originally Posted by Selur View Post
    Yes, you can cut at key frame using ffmpeg, read http://trac.ffmpeg.org/wiki/Seeking

    okay, so i wanna split in every ten minutes because i want to run several simultaneous encodings for each segment, i have cmd batch file for doing it automatically but http://trac.ffmpeg.org/wiki/Seeking didnt example for it, it there a way for doing it?
    Quote Quote  
  19. You have to create a batch file to split every X min. (the segment option might also work, simply it and check if each of the segments can be played on it's own)
    users currently on my ignore list: deadrats, Stears555
    Quote Quote  
  20. Originally Posted by Selur View Post
    You have to create a batch file to split every X min. (the segment option might also work, simply it and check if each of the segments can be played on it's own)
    okay, i know it and i have one and after splitting it will encode each file but i am using this
    Code:
    ffmpeg -i "input.mkv" -c copy -f segment -segment_time "time" -reset_timestamps 1 "output"
    and
    Code:
    mkvmerge --split size:(for example 700m) -o output.mkv input.mkv
    , so both way dose not work means after concatenating the result have a little glitch on splits point, if you can please write me a code
    Quote Quote  
  21. Did you actually try to playback the splitted parts, since I'm not sure that your concatenate line will work on such splitted files when they haven't been reencoded,...
    If you split with mkvmerge, I for example would also append the parts with it.
    users currently on my ignore list: deadrats, Stears555
    Quote Quote  
  22. Originally Posted by Selur View Post
    Did you actually try to playback the splitted parts, since I'm not sure that your concatenate line will work on such splitted files when they haven't been reencoded,...
    If you split with mkvmerge, I for example would also append the parts with it.
    for example i have a Full BD Disk so i want to have it on my hard drive in a smaller size , i am re-encode it but i have a high core cpu (36 cores) so the x264 does not use a whole power of cpu and it's take about 100 fps but i can build a remux of the Disc and then split that mkv file into a smaller file size (like a 10 GB to 1 GB Files) then re-encoding each file , i am getting about 200 to 250 fps here, so yes splitted parts will play properly and after re-encoding and concatenating the result will play properly but in the splits point i have a little glitch but another part o movie is good
    Quote Quote  
  23. What I was suggesting was something along the lines of this:

    Code:
    start /b /low "part1" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:00:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.1.mkv"
    start /b /low "part2" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:05:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.2.mkv"
    start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:10:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.3.mkv"
    start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:15:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.4.mkv"
    
    pause
    
    echo file '%~1.1.mkv' > list.txt
    echo file '%~1.2.mkv' >> list.txt
    echo file '%~1.3.mkv' >> list.txt
    echo file '%~1.4.mkv' >> list.txt
    
    "g:\program files\ffmpeg\bin\ffmpeg.exe" -y -safe 0 -f concat -i list.txt -c copy %1.new.mkv
    
    del "%~1.1.mkv"
    del "%~1.2.mkv"
    del "%~1.3.mkv"
    del "%~1.4.mkv"
    del list.txt
    Put that in a batch file (you'll have to change the path to ffmpeg.exe) and drag/drop a video on it. It starts four instances of ffmpeg to encode 5 minutes each of the source. Once the encodings are done press Enter to concatenate the new segments and delete them. I only have a quad core CPU so I set the encoding up to use only one thread. Adjust that to an appropriate value for your setup. And you'll want to adjust the length and number of segments for your source.
    Quote Quote  
  24. Originally Posted by jagabo View Post
    What I was suggesting was something along the lines of this:

    Code:
    start /b /low "part1" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:00:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.1.mkv"
    start /b /low "part2" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:05:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.2.mkv"
    start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:10:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.3.mkv"
    start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:15:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.4.mkv"
    
    pause
    
    echo file '%~1.1.mkv' > list.txt
    echo file '%~1.2.mkv' >> list.txt
    echo file '%~1.3.mkv' >> list.txt
    echo file '%~1.4.mkv' >> list.txt
    
    "g:\program files\ffmpeg\bin\ffmpeg.exe" -y -safe 0 -f concat -i list.txt -c copy %1.new.mkv
    
    del "%~1.1.mkv"
    del "%~1.2.mkv"
    del "%~1.3.mkv"
    del "%~1.4.mkv"
    del list.txt
    Put that in a batch file (you'll have to change the path to ffmpeg.exe) and drag/drop a video on it. It starts four instances of ffmpeg to encode 5 minutes each of the source. Once the encodings are done press Enter to concatenate the new segments and delete them. I only have a quad core CPU so I set the encoding up to use only one thread. Adjust that to an appropriate value for your setup. And you'll want to adjust the length and number of segments for your source.
    Are you 100% that ffmpeg is frame accurate?
    Quote Quote  
  25. Originally Posted by Atak_Snajpera View Post
    Are you 100% that ffmpeg is frame accurate?
    No, but the batch file worked without duplicate or missing frames at the seams on the samples I tried. I even used AviSynth scripts to check the new vs. original video.

    I thought I might have to change the segment lengths to 4.59.999 or something like that but it wasn't necessary. Of course, it might make a difference with some other videos.
    Last edited by jagabo; 18th Dec 2018 at 17:54.
    Quote Quote  
  26. Originally Posted by jagabo View Post
    Originally Posted by Atak_Snajpera View Post
    Are you 100% that ffmpeg is frame accurate?
    No, but the batch file worked without duplicate or missing frames at the seams on the samples I tried. I even used AviSynth scripts to check the new vs. original video.

    I thought I might have to change the segment lengths to 4.59.999 or something like that but it wasn't necessary. Of course, it might make a difference with some other videos.
    I would be more confident if FFMPEG allowed frame range instead of timecodes. For example 0-1999 , 2000-2999 and so on.

    Btw. you can improve your script like this


    Code:
    start /b /low "part1" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:00:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.1.mkv"
    start /b /low "part2" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:05:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.2.mkv"
    start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:10:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.3.mkv"
    start /b /low "part3" "g:\program files\ffmpeg\bin\ffmpeg.exe" -i "%~1" -ss 00:15:00.00 -t 00:05:00.00 -c:v libx264 -preset veryfast -x264opts stitchable:threads=1 -c:a copy -y "%~1.4.mkv"
    
    :LOOP
    tasklist | find /i "ffmpeg.exe" >nul 2>&1
    IF ERRORLEVEL 1 (
      GOTO CONTINUE
    ) ELSE (
      Timeout /T 5 /Nobreak
      GOTO LOOP
    )
    
    :CONTINUE
    echo file '%~1.1.mkv' > list.txt
    echo file '%~1.2.mkv' >> list.txt
    echo file '%~1.3.mkv' >> list.txt
    echo file '%~1.4.mkv' >> list.txt
    
    "g:\program files\ffmpeg\bin\ffmpeg.exe" -y -safe 0 -f concat -i list.txt -c copy %1.new.mkv
    
    del "%~1.1.mkv"
    del "%~1.2.mkv"
    del "%~1.3.mkv"
    del "%~1.4.mkv"
    del list.txt
    Last edited by Atak_Snajpera; 19th Dec 2018 at 04:53.
    Quote Quote  



Similar Threads

Visit our sponsor! Try DVDFab and backup Blu-rays!