ffmpeg prores: Packet too small

[Gwar]

Hey there,
not always but a lot of times when I encode something to ffmpeg prores_k I'm getting those weird Packet too small errors. If a file is causing them it doesnt matter if you encode the video directly or through avs lshmash or LWLibavVideoSource or also with DVD's through Mpeg2Source. It's always causing it.
I never saw any issues in the video but would feel better if there is a possibility to let this go. Maybe someone is famliar with the issue and know how to solve it.
I tried aprox 10 different versions and recently upgraded to the latest but it didn't change anything
thanks

  Metadata:
    encoder         : Lavf59.27.100
  Stream #0:0: Video: prores (apcn / 0x6E637061), yuv422p10le(tv, bt709, progressive), 768x576, q=2-31, 200 kb/s, 25 fps, 12800 tbn
    Metadata:
      encoder         : Lavc59.37.100 prores_ks
[prores_ks @ 0000021b3d6854c0] Packet too small: is 177272, needs 158627 (slice: 1547). Correct allocation is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[prores_ks @ 0000021b3d6854c0] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[prores_ks @ 0000021b3d664180] Packet too small: is 177272, needs 157542 (slice: 1542). Correct allocation is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[prores_ks @ 0000021b3d664180] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[prores_ks @ 0000021b41af0300] Packet too small: is 177272, needs 154070 (slice: 1526). Correct allocation is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[prores_ks @ 0000021b41af0300] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[prores_ks @ 0000021b3d62be80] Packet too small: is 177272, needs 147777 (slice: 1497). Correct allocation is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[prores_ks @ 0000021b3d62be80] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
[prores_ks @ 0000021b41abb480] Packet too small: is 177272, needs 153636 (slice: 1524). Correct allocation is not implemented. Update your FFmpeg version to the newest one from Git. If the problem still occurs, it means that your file has a feature which has not been implemented.
[prores_ks @ 0000021b41abb480] If you want to help, upload a sample of this file to https://streams.videolan.org/upload/ and contact the ffmpeg-devel mailing list. (ffmpeg-devel@ffmpeg.org)
frame=22412 fps=138 q=-0.0 size= 3783936kB time=00:14:55.96 bitrate=34597.5kbits/s speed=5.52x

[Selur]

Those are warnings, not errors, but warnings.
I can't reproduce this here, I tried with:

ffmpeg -y -noautorotate -nostdin -threads 8 -i "G:\TestClips&Co\files\test.avi" -map 0:0 -an -sn -vf scale=768:576,zscale=rangein=tv:range=tv -pix_fmt yuv422p10le -strict -1 -vsync 0  -sws_flags spline  -vcodec prores_ks -profile:v 2 -vtag apcn -aspect 768:576 -f mov "G:\Output\test.mov"

What does your command line look like? (do profile and content fit together?)

I tried aprox 10 different versions and recently upgraded to the latest but it didn't change anything

What ffmpeg version are you using? (is it up-to-date?) (My ffmpeg version reports encoder Lavf59.34.102 while yours is Lavf59.27.100, so yours is probably a few months older.)

Cu Selur

[Cornucopia]

I don't use this kind of workflow so won't comment on the toolset compliance, but I see a strangeness in the command line: 768x576 is basically SD rez (for most valid prores SD, 720x486 and 720x576 is required). Yet, bt709 colorspace is meant for HD use, not SD.

Don't know this is what is making it give a warning, but it's a good possibility.

Scott

[Gwar]

I can upgrade my ffmpeg but almost guarantee that this will not change much. It wasn't there in older 32 bit versions and the more updated the version the more it appears. A resolution of 720x486 doesnt make much sense to me and the resolution shouldnt really matter for the codec that would be super weird. 768x576 is pretty common for PAL 4:3 SD content.
I know it's not the right color space but I have no choice. I can set the matrix to 470bg. But there are no 470bg for primaries so 709 would need to be used for instead. For transfer I can use either 601 or 709 but they're the same anyway. It doesn't support the analogue PAL 470bg transfer function.
And in the end when I import my stuff to Davinci resolve it will be converted to BT709 anyways. There are no 601 or even 470BG in Davinci's color matrix options.
I could use sRGB or YUV tho' but would that really matter?

My command line for SD prores recordings is

"%FFMPEG%" -i %INPUTFILE%  -an -sn -dn -c:v prores_ks -profile:v 2 -f MOV %1%.mov

-pix_fmt yuv422p10le is usually set automatically. And I never set the aspect in the ffmpeg commandline as it is set automatically or I do a resize with an AVS script I pass through.

-vsync 0 sounds pretty good I will test that here for the next projects and see how it goes.


thanks guys

[Cornucopia]

re: the resolution, 768x576 is really only a valid/common PAL SD resolution *AFTER* an analog SD stream has been already captured and then converted to 1:1 PAR for computer.
ProRes, like many editing/intermediate formats (Cineform, DNxHD), assumes the source is NOT yet converted to 1:1 PAR, and for compliance, expects square 1:1 PAR for HD sources (except 4:3 PAR for 1440x1080 when coming from something like HDV), or non-square PAR for SD sources, both NTSC and PAL.
It is not unusual at all for certain pro editing/intermediate codecs to only consider certain resolutions as valid. Cannot say for certain here, but there is a good chance this is acting this way also. Hence, the "warning" when it isn't one of the expected legit resolutions.

re: the color space. when dealing with SD color spaces, you should be using 601. when dealing with HD color spaces you should be using 709. PAL 740bg is the PAL subset version of the color transfer function of 601, so picking 601 as an option in ffmpeg SHOULD work correctly for you when you are not able to explicitly state 470bg. IMHO, if ffmpeg were working perfectly, it WOULD understand NTSC=SMPTE170M color primaries, PAL=470BG color primaries both as integral functions of 601, and SHOULD choose them properly based on the contents' resolution, including doing the conversion automatically when uprezzing to HD so the outcome then is 709. But I'm sure I'm asking too much.
If your stuff is SD and 601 it should convert better to HD and 709 than if it isn't flagged/encoded properly.

I would suggest you DO NOT convert to sRGB, etc as those are lossy conversions, regardless of codec.

Unlike lots of stuff procured or generated from various means and methods, pro codecs do have an expectation of compliance with their formats, and so giving them non-compliant material may throw them for a loop.

Don't know if that is really happening here, but if it is I wouldn't be surprised why.


Scott

[poisondeathray]

The metadata is not related to the packet warnings

But-

I know it's not the right color space but I have no choice. I can set the matrix to 470bg. But there are no 470bg for primaries so 709 would need to be used for instead. For transfer I can use either 601 or 709 but they're the same anyway. It doesn't support the analogue PAL 470bg transfer function.
And in the end when I import my stuff to Davinci resolve it will be converted to BT709 anyways. There are no 601 or even 470BG in Davinci's color matrix options.

No - Resolve interprets SD prores correctly as 601 if you have it flagged correctly. You can use smpte170m for the matrix, and mediainfo will say "BT.601" and Resolve will interpret it correctly. If matrix field is not flagged or flagged incorrectly, yes Resolve will interpret it as 709 . You can verify this behaviour with SD colorbars

Yes, transfer is the same value for 601/709, so it doesn't matter

You can use -bsf:v prores_metadata to set the metadata afterwards if you need to
https://ffmpeg.org/ffmpeg-bitstream-filters.html#prores_005fmetadata

eg

ffmpeg -i input.mov -bsf:v prores_metadata=color_primaries=bt470bg:color_trc=bt709:colorspace=smpte170m -c copy output.mov

[Gwar]

ah ok. I think I used

-vf zscale=matrixin=470bg:matrix=470bg:transferin=601:transfer=601:primariesin=709:primaries=709

at one point but shure I can switch to

-bsf:v prores_metadata=color_primaries=bt470bg:color_trc=bt709:colorspace=smpte170m

However I'm curious as why to set a colorspace of smpte170m for PAL sources. As far as I remember smpte170m is used for NTSC and bt470bg PAL. But that probably only relates to the primaries and the general color space is smpte170m for both PAL and NTSC right?
I noticed that the colors are a bit off if I export SD content even if it is recognised correctly as BT601 to SD with 709. So would you suggest to switch davinci to its colorspace option which they call Y'UV? Thats the only one I saw which could be considered to exporting as 601

[Gwar]

still having the packet size too small error and the resolution really doesnt matter. it always happens and in 720x576 too

[poisondeathray]

However I'm curious as why to set a colorspace of smpte170m for PAL sources. As far as I remember smpte170m is used for NTSC and bt470bg PAL. But that probably only relates to the primaries and the general color space is smpte170m for both PAL and NTSC right?

Primaries have different actual values, but actual matrix values are the same for 170m and bt470bg : KR = 0.299; KB = 0.114

For the ffmpeg prores bitstream filter, "bt470bg" is not accepted for matrix_coefficients, that's why you're using 170m .
https://ffmpeg.org/ffmpeg-bitstream-filters.html#prores_005fmetadata

matrix_coefficients

    Set the matrix coefficient. Available values are:

    ‘auto’

        Keep the same colorspace property (default).
    ‘unknown’
    ‘bt709’
    ‘smpte170m’

        BT 601
    ‘bt2020nc’

https://github.com/FFmpeg/FFmpeg/blob/f4098bbc3b10926f618cf89e24780c9e6ae9b8b5/libavco...metadata_bsf.c

{"colorspace", "select colorspace", OFFSET(matrix_coefficients), AV_OPT_TYPE_INT, {.i64=-1}, -1, AVCOL_SPC_BT2020_NCL, FLAGS, "colorspace"},
{"auto", "keep the same colorspace", 0, AV_OPT_TYPE_CONST, {.i64=-1}, INT_MIN, INT_MAX, FLAGS, "colorspace"},
{"unknown", NULL, 0, AV_OPT_TYPE_CONST, {.i64=0}, INT_MIN, INT_MAX, FLAGS, "colorspace"},
{"bt709", NULL, 0, AV_OPT_TYPE_CONST, {.i64=AVCOL_SPC_BT709}, INT_MIN, INT_MAX, FLAGS, "colorspace"},
{"smpte170m", NULL, 0, AV_OPT_TYPE_CONST, {.i64=AVCOL_SPC_SMPTE170M}, INT_MIN, INT_MAX, FLAGS, "colorspace"},
{"bt2020nc", NULL, 0, AV_OPT_TYPE_CONST, {.i64=AVCOL_SPC_BT2020_NCL}, INT_MIN, INT_MAX, FLAGS, "colorspace"},

I noticed that the colors are a bit off if I export SD content even if it is recognised correctly as BT601 to SD with 709. So would you suggest to switch davinci to its colorspace option which they call Y'UV? Thats the only one I saw which could be considered to exporting as 601

Checking the export is another can of worms. Short answer is depends on how you are viewing it and how it's being converted to RGB for display.

You need to go read the 100's of pages in the resolve forum about gamma interpretation and flags between Windows and Mac. Use the variation that fits your situation and workflow, because it's different for other situations and workflows.

Besides that difference, some programs take account for the primaries difference , others do not (only matrix), when converting to RGB for display - that also accounts for some observed differences. Technically using primaries is more correct, but many programs, broadcast, web still do not in 2023. ITU recommendations even mentions this common practice of ignoring primaries

=> Basically you'll never get the same color everywhere. You choose which programs/situation you want to mostly satisfy, knowing full well that the others will for certain will be visibly different

still having the packet size too small error and the resolution really doesnt matter. it always happens and in 720x576 too

Not sure, maybe some issue with your build or source files. I can't reproduce the warning