Thread message queue blocking - Image encoding

[SM1872]

Placed this in newbie as I'm sure I'm doing something incredibly stupid.

I've a folder full of images to piece together and I've attempted to make up a script

ffmpeg -i e\%06d.tif -vf scale=744:576 -level 4.1 -8x8dct 1 -fast-pskip 0 -pix_fmt yuv420p -bf 16 -b-pyramid normal -me_method umh -subq 11 -me_range 64 -mbtree 1 -rc-lookahead 250 -keyint_min 25 -direct-pred auto -partitions all -trellis 2 -mixed-refs 1 -aq-mode 1 -aq-strength 1.15 -vcodec libx264 -profile:v high444 -refs 16 -crf 20 -maxrate 6250k -bufsize 7812k -preset veryslow -r 25 81120.mkv

I get the error:
Thread message queue blocking; consider raising the thread_queue_size option (current value: 1)

I've no wish to encode in realtime. Going for quality over speed. At least that's the aim.

Could anyone help me tweak this?

[october262]

Placed this in newbie as I'm sure I'm doing something incredibly stupid.

I've a folder full of images to piece together and I've attempted to make up a script

ffmpeg -i e\%06d.tif -vf scale=744:576 -level 4.1 -8x8dct 1 -fast-pskip 0 -pix_fmt yuv420p -bf 16 -b-pyramid normal -me_method umh -subq 11 -me_range 64 -mbtree 1 -rc-lookahead 250 -keyint_min 25 -direct-pred auto -partitions all -trellis 2 -mixed-refs 1 -aq-mode 1 -aq-strength 1.15 -vcodec libx264 -profile:v high444 -refs 16 -crf 20 -maxrate 6250k -bufsize 7812k -preset veryslow -r 25 81120.mkv

I get the error:
Thread message queue blocking; consider raising the thread_queue_size option (current value: 1)

I've no wish to encode in realtime. Going for quality over speed. At least that's the aim.

Could anyone help me tweak this?

see this thread - https://video.stackexchange.com/questions/25643/ffmpeg-thread-queue-size-warning-what-does-this-mean

[SM1872]

Thanks, october262, however I'm still stumped.

I get what the error is. I'd just rather tweak the script so that it encodes very slowly but I don't risk the errors associated with that particular message, but not sure how to set it up.

[jagabo]

Did you try adding "-thread_queue_size 4096" to your command line? Add it before "-i".

[SM1872]

May I offer my thanks to you again, Jagabo. I'd tried using that switch but I'd obviously put it in the wrong place. The two videos I'd encoded with the error appear to be entirely fine. Perhaps I was just lucky due to having 16GB RAM and nothing else running, but I was afraid that it'd cause problems if I was even browsing YouTube or something whilst encoding.

Thanks once again.

ON SECOND THOUGHTS...

I used this command:
ffmpeg -thread_queue_size 4096 -i e\%06d.tif -vf "scale=744:576" -level 4.1 -8x8dct 1 -fast-pskip 0 -pix_fmt yuv420p -bf 16 -b-pyramid normal -me_method umh -subq 11 -me_range 64 -mbtree 1 -rc-lookahead 250 -keyint_min 25 -direct-pred auto -partitions all -trellis 2 -mixed-refs 1 -aq-mode 1 -aq-strength 1.15 -vcodec libx264 -profile:v high444 -refs 16 -crf 20 -maxrate 6250k -bufsize 7812k -preset veryslow -r 25 81120.mkv

Edit: It started encoding, then I got:
[image2 @ 000001637c636b40] Thread message queue blocking; consider raising the thread_queue_size option (current value: 4096)

There are 106149 frames in all. Wondering how high it can go without crippling the machine, or is there a better way?

I have files that I encoded using the script I originally had. Wondering if there's a way of pulling the FFMPEG command line from one of those using MediaInfo or something? Opened one in MediaInfo GUI but can't see the info. Would the command line version be able to pull my old script?

[jagabo]

I've seen some people indicate that encoding without audio eliminates the problem. You can always mux the video and audio in a second step.

[SM1872]

I use MKVToolNix to add the audio afterwards, Jagabo. As I said, I'm pretty hopeless, but I don't think there's any audio in my script, is there?

[Selur]

Does adding '-an -sn -dn -vsync drop' help`?

[SM1872]

Does adding '-an -sn -dn -vsync drop' help`?

I'm afraid not.

thread_queue_size 8192 seems to be the lowest that I'll get away with.

Machine has AMD Ryzen 7 4800H processor and 16GB RAM. Also a 6GB graphics card helping to keep the machine running.

I only just started the process with 8192. Just to check, will it damage the encoding output? Is it likely to cripple my machine and stop me doing other things? (Basically just browsing)

Strange that this has happened as I've never experienced it before. Could it be the new FFMPEG?

[Selur]

Should not influence the output.
To be able to browse etc. you need to lower the priority of ffmpeg.
As a general side note: make sure to add exceptions to your output and input folder in your virus scanner, otherwise the virus scanner checking each file on access might slow things down. (similar for search indexing)

[october262]

Does adding '-an -sn -dn -vsync drop' help`?

I'm afraid not.

thread_queue_size 8192 seems to be the lowest that I'll get away with.

Machine has AMD Ryzen 7 4800H processor and 16GB RAM. Also a 6GB graphics card helping to keep the machine running.

I only just started the process with 8192. Just to check, will it damage the encoding output? Is it likely to cripple my machine and stop me doing other things? (Basically just browsing)

Strange that this has happened as I've never experienced it before. Could it be the new FFMPEG?

try adding thread_queue_size parameter with max possible value - https://video.stackexchange.com/questions/32960/ffmpeg-showing-thread-message-queue-bl...ng-non-real-ti

[SM1872]

So I've disabled the virus scan in the video encoding thread.
The thread_queue_size is now at 106150.
And the priority of FFMPEG in task manager is now at the lowest it can go.

Tested running a YouTube video at the same time and we seem to be OK, but any higher with FFMPEG's priority and the machine grinds to a halt.

Honestly cannot recall coming across this before my machine went for repair and they reset the contents of the hard drive - when replacing a broken fan! Will have to find out how to automatically set the priority of FFMPEG as it does seem to jump back to normal each time I use it, and if I don't set it to minimum immediately it could result in a force closure/restart.

[SM1872]

Well, the encode looks nice... what there is of it anyway, but it errored just after half an hour of content:

Output #0, matroska, to '81120.mkv':
Metadata:
encoder : Lavf59.34.101
Stream #0:0: Video: h264 (H264 / 0x34363248), yuv420p(tv, progressive), 744x576, q=2-31, 25 fps, 1k tbn
Metadata:
encoder : Lavc59.50.101 libx264
Side data:
cpb: bitrate max/min/avg: 6250000/0/0 buffer size: 7812000 vbv_delay: N/A
C:\ffmpeg\bin\e\%06d.tif: Cannot allocate memory00:31:32.52 bitrate=1827.3kbits/s speed=0.641x
C:\ffmpeg\bin\e\%06d.tif: Cannot allocate memory00:31:32.68 bitrate=1830.5kbits/s speed=0.64x
Last message repeated 4 times
C:\ffmpeg\bin\e\%06d.tif: Cannot allocate memory00:31:32.88 bitrate=1830.3kbits/s speed=0.64x
Last message repeated 7 times
C:\ffmpeg\bin\e\%06d.tif: Cannot allocate memory00:31:33.20 bitrate=1830.0kbits/s speed=0.64x
Last message repeated 1 times
frame=47619 fps= 16 q=-1.0 Lsize= 425625kB time=00:31:44.64 bitrate=1830.6kbits/s speed=0.641x
video:425277kB audio:0kB subtitle:0kB other streams:0kB global headers:0kB muxing overhead: 0.081635%



And here's the Mediainfo :

General
Complete name : 81120.mkv
Format : Matroska
Format version : Version 4
File size : 416 MiB
Duration : 31 min 44 s
Overall bit rate : 1 831 kb/s
Writing application : Lavf59.34.101
Writing library : Lavf59.34.101
ErrorDetectionType : Per level 1

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings : CABAC / 16 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 16 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 31 min 44 s
Bit rate : 1 794 kb/s
Width : 744 pixels
Height : 576 pixels
Display aspect ratio : 1.292
Frame rate mode : Constant
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.167
Stream size : 407 MiB (98%)
Writing library : x264 core 164 r3101 b093bbe
Encoding settings : cabac=1 / ref=16 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=11 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=64 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=18 / lookahead_threads=4 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=250 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=6250 / vbv_bufsize=7812 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.15
Default : No
Forced : No
Color range : Limited




Whereas here's one I did prior to my machine going away, which processed without any errors at all. I'm just not sure where I'm going wrong, or how to fix this.


General
Complete name : 84.mkv
Format : Matroska
Format version : Version 4
File size : 1 014 MiB
Duration : 1 h 7 min
Overall bit rate : 2 113 kb/s
Encoded date : UTC 2022-08-17 20:24:30
Writing application : mkvmerge v70.0.0 ('Caught A Lite Sneeze') 64-bit
Writing library : libebml v1.4.2 + libmatroska v1.6.4

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : High@L4.1
Format settings : CABAC / 16 Ref Frames
Format settings, CABAC : Yes
Format settings, Reference frames : 16 frames
Codec ID : V_MPEG4/ISO/AVC
Duration : 1 h 7 min
Bit rate : 1 998 kb/s
Width : 752 pixels
Height : 576 pixels
Display aspect ratio : 4:3
Frame rate mode : Constant
Frame rate : 25.000 FPS
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.185
Stream size : 959 MiB (95%)
Writing library : x264 core 164 r3095 baee400
Encoding settings : cabac=1 / ref=16 / deblock=1:-1:-1 / analyse=0x3:0x133 / me=umh / subme=11 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=64 / chroma_me=1 / trellis=2 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=0 / chroma_qp_offset=-2 / threads=18 / lookahead_threads=4 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=16 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=250 / rc=crf / mbtree=1 / crf=19.5 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / vbv_maxrate=6250 / vbv_bufsize=7812 / crf_max=0.0 / nal_hrd=none / filler=0 / ip_ratio=1.40 / aq=1:1.15
Default : No
Forced : No
Color range : Limited

Audio
ID : 2
Format : AAC LC
Format/Info : Advanced Audio Codec Low Complexity
Codec ID : A_AAC-2
Duration : 1 h 7 min
Bit rate : 112 kb/s
Channel(s) : 1 channel
Channel layout : C
Sampling rate : 48.0 kHz
Frame rate : 46.875 FPS (1024 SPF)
Compression mode : Lossy
Stream size : 54.0 MiB (5%)
Title : Mono
Default : Yes
Forced : No

[jagabo]

To start a program at low priority use Start:

start /b /low "WindowName" "c:\path\to\ffmpeg.exe" [rest of command line]

Keep in mind that priority effects scheduling, not overall CPU usage. Ie, when multiple programs are requesting CPU time the one(s) with higher priority get more CPU time, those with lower priority get less. But when only one program is requesting CPU time it still gets as much as it can use. So you can't reduce CPU temperatures by running a program at low priority.

[SM1872]

GOT IT!



Well, I might be a funk-wit, but I'm intelligent to know that when something's not working it's because I'm a moron.

Here's what the problem was:

FFMPEG page re-directs you here:



I went with this - ffmpeg-git-full.7z

Talking with a friend tonight and he asked me to check if I had the right build. Went back to the same page, scrolled down, and whoops...



Installed ffmpeg-release-full.7z

And we can all get down and groove... we dance all day in this house.


Guys, I'm really sorry for wasting your time. I apologise for my moronic error. Yes, I am embarrassed.

[jagabo]

So a buggy build?

[SM1872]

Perhaps, but I've seen other threads about the error and suspect others have done the same as me - using the build at the top of the page rather than the latest official one. I'll certainly keep a mental note of it in case I see someone asking about it - Don't use the git master build, scroll down for the release build. Doesn't help that they have the git master build above the release one.

Sorry for being a daft-arse!

[jagabo]

It's not really your fault. From here: https://www.gyan.dev/ffmpeg/builds/#about-these-builds

The git master builds are built from the master branch of the repository, which is where all development occurs whereas the release builds are built from the latest release branch. These are essentially a snapshot of the master branch at a given point of time. They are updated with bug and security fixes for a while but no new features. It is highly recommended to choose a git master build; unlike many software, releases are primarily made for the convenience of OS distributors and package managers and don't signify greater stability or maturity.

So they actually recommend the master build over the release build. But in this case the master build may have introduced a bug.