ffmpeg mp4 to bitmap to Prores

[chris319]

This article will give you an idea of what I'm trying to do:

https://batchloaf.wordpress.com/2017/02/12/a-simple-way-to-read-and-write-audio-and-vi...-part-2-video/

I'm trying to read video frames into a bitmap and write the bitmap out to a Prores file. I've got it working with mp4 on a Linux vm but am having trouble writing the frames out to Prores.

Opening an input pipe works fine:

FILE *pipein = popen("ffmpeg -i KodakChart.MP4  -f image2pipe -vcodec rawvideo -pix_fmt rgb24 -", "r");

The problem seems to lie in the output pipe:

FILE *pipeout = popen("ffmpeg  -y  -i KodakChart.MP4 -c:v prores -profile:v 3 -c:a copy  output.mov", "w");

The above code give me a playable Prores file with audio, but merely re-encodes the video from the original file and bypasses the bitmap.

Here is the code that writes the bitmap named "frame":

fwrite(frame, 1, (H*W*3)/2, pipeout);

I have tried the following without success:

//FILE *pipeout = popen("ffmpeg  -y           -f rawvideo  -vcodec rawvideo                -c:v prores -profile:v 3 -c:a copy  output.mov", "w");

//FILE *pipeout = popen("ffmpeg  -y  -f rawvideo  -vcodec rawvideo -pix_fmt yuv422p  -c:v prores -profile:v 3 -c:a copy  output.mov", "w");

Any help is appreciated. Thank you.

[poisondeathray]

Why would you want to do this ? What are you trying to do exactly ?

There is potential for problems with pixel format conversions; ie. whatever the source is , 8bit RGB bitmap , 10bit422 YUV prores ... Matrix issues, rounding errors

Why is it necessary to read from the physical bmp for this? Are you doing something to the bmp sequence in another program? why not read from pipe before the bmp and specify multiple outputs (eg if you wanted to include the rgb conversion before the prores write)? You could use filter_complex with 2 splits for example

[chris319]

Why is it necessary to read from the physical bmp for this? Are you doing something to the bmp sequence in another program?

I'm modifying the pixels elsewhere in the same program.

Ouch! Forgot about the 8- to 10-bit aspect of it.

[poisondeathray]

Why is it necessary to read from the physical bmp for this? Are you doing something to the bmp sequence in another program?

I'm modifying the pixels elsewhere in the same program.

Ouch! Forgot about the 8- to 10-bit aspect of it.

If you're performing operations in other programs, then there is a timing issue here as well; You can't access a bmp in the other program until it's written out, and presumably you don't want to read the bmp sequence until it's been modified in the other program (and the modified version written out by the other program)

So you might as well just read a bmp sequence separately later (assuming you don't care about the pixel format , bit depth, matrix conversions) . ie. just separate the 2 stages

[chris319]

I'm modifying the pixels elsewhere in the same program.

As I said previously, the bitmap modifications are taking place in the same program, after the bitmap is read in and before it is written out, just like in the article I linked to in the O.P.

[poisondeathray]

I'm modifying the pixels elsewhere in the same program.

As I said previously, the bitmap modifications are taking place in the same program, after the bitmap is read in and before it is written out, just like in the article I linked to in the O.P.

What modifications ? If you're doing exactly the same modification, ie channel inversion in 8bit RGB, why not just use a ffmpeg filter ? (I know this doesn't address your original question but it seems much easier)

[chris319]

I know this doesn't address your original question but it seems much easier

I realize that, but there is a reason I'm doing it this way which I don't care to divulge.

[poisondeathray]

For the output to prores pipe, you have -i KodakChart.MP4 , which means take the original video as input

Usually stdin is just -

so

-i -

Raw video pipes need all their characteristics specified. So pixel format, dimensions, fps . A recieving rawvideo pipe would look something like

ffmpeg -f rawvideo -s 1920x1080 -pix_fmt rgb24 -r 24  -i - .....

Adjust your dimensions, fps to whatever yours is; also your pixel format might be bgr24

[chris319]

Here is the complete code. It is strictly experimental and may not even work. It disregards the 8-bit 10-bit difference.

The idea is to clip off all luminance pixels > 235 to 235.

if (lum[y*W+x] > 235) lum[y*W+x] = 235;

// Video processing example using FFmpeg
// Written by Ted Burke – last updated 12-2-2017
// Now works in YUV space
// Outputs to prores
// To compile: gcc Ted2.c -o Ted2
// Note: make sure .MP4 file extension matches case
 
#include <stdio.h>
 
// Video resolution
#define W 1280
#define H 720
 
// Allocate a buffer to store one frame
unsigned char frame[((H)*(W)*3)/2];
 
int main(void)
{
int x, y, count;

    // Create a pointer for each component's chunk within the frame
    // Note that the size of the Y chunk is W*H, but the size of both
    // the U and V chunks is (W/2)*(H/2). i.e. the resolution is halved
    // in the vertical and horizontal directions for U and V.

unsigned char *lum;//, *u, *v;
unsigned char *u, *v;

    lum = frame;
    u = frame + H*W;
    v = u + (H*W/4);
 
FILE *pipein = popen("ffmpeg -i KodakChart.MP4  -f image2pipe -vcodec rawvideo -pix_fmt rgb24 -", "r");

FILE *pipeout = popen("ffmpeg  -y     -f rawvideo   -s 1280x720   -pix_fmt rgb24       -vcodec rawvideo    -r 59.94     -i KodakChart.MP4 -c:v prores -profile:v 3 -c:a copy  output.mov", "w");

// Process video frames
while(1)
    {
        // Read a frame from the input pipe into the buffer
        // Note that the full frame size (in bytes) for yuv420p
        // is (W*H*3)/2. i.e. 1.5 bytes per pixel. This is due
        // to the U and V components being stored at lower resolution.
        count = fread(frame, 1, (H*W*3)/2, pipein);
         
        // If we didn’t get a frame of video, we’re probably at the end
        if (count != (H*W*3)/2) break;
 
// Process this frame
for (y=0 ; y<H ; y++)
        {
            for (x=0 ; x<W ; x++)
            {

if (lum[y*W+x] > 235) lum[y*W+x] = 235;

            }
        }
 
  // Write this frame to the output pipe
        fwrite(frame, 1, (H*W*3)/2, pipeout);
    }
 
    // Flush and close input and output pipes
    fflush(pipein);
    pclose(pipein);
    fflush(pipeout);
    pclose(pipeout);
}

[poisondeathray]

Not sure about linux, but for windows piping, you'd also have to map the rawvideo pipe, and map which stream the audio is coming from

Another way to clip would be to use -vf lutrgb or lutyuv . The filter also works at >8 bit depths now

[chris319]

Another way to clip would be to use -vf lutrgb or lutyuv . The filter also works at >8 bit depths now

Good idea.

I've tried that and itdoesn't work very well. It doesn't hard clip at the specified values. ffmpeg has issues with video levels. Also, when you re-encode to, let's say H.264 it puts the artifacts >235 back in.

[richardpl]

lutyuv and limiter filters do clip at specified values, perhaps you do not know how to use filters.

Non-lossless encoding can produce various artifacts you can not avoid that.

[chris319]

If you wish to limit pixels like that use -vf scale=in_range=pcut_range=tv
Otherwise, command will do nothing.

This compresses the video levels, which I do not want to do.

If you just want to clamp pixels, use lut or limiter filter.

I want to clip them without altering the levels, which lutyuv seems to have trouble doing.

[poisondeathray]

Perhaps you do not know how to correctly measure video levels.

I assure you, richardpl does know how to measure video levels. He is a ffmpeg developer

lutyuv does work, I've verified this in the past for both 8bit and 10bit



If there are problems , then document the issues or post a detailed bug report so they can be fixed. Rounding errors are expected for some operations, and lossy codecs can produce excursions as well

[chris319]

If there are problems , then document the issues or post a detailed bug report so they can be fixed.

Already have.

[poisondeathray]

If there are problems , then document the issues or post a detailed bug report so they can be fixed.

Already have.

Was it the invalidated ticket, or did you post another ?

[chris319]

Have a look at the scope shot below. Here is the code used to generate it:

ffmpeg -i stairstep.mp4 -y -vf lutyuv=y='clip(val,16,235)' outfile.mp4

I don't see any clipping. I see video (green traces) above and below 16 and 235. Is there something wrong with my code? Shouldn't it clip at 16 and 235?

http://www.chrisnology.info/videos/Cliptest.jpg

[chris319]

The TV levels filter works as expected. It compresses the video levels to the range 16 - 235, thus altering the levels.

I'm looking to clip the video without altering the levels.

bin\ffmpeg -i stairstep.mp4 -y -vf "scale=in_range=pc:out_range=tv" -c:a copy outfile.mp4

http://www.chrisnology.info/videos/TVLevels.jpg

The problem is that these ringing artifacts pop up when you encode the video to a deliverable format, be it h.264, h.265 or even prores. They can be tamed to an extent by applying some "unsharpening" and removing some of the high-frequency content.

[poisondeathray]

Have a look at the scope shot below. Here is the code used to generate it:

ffmpeg -i stairstep.mp4 -y -vf lutyuv=y='clip(val,16,235)' outfile.mp4

I don't see any clipping. I see video (green traces) above and below 16 and 235. Is there something wrong with my code? Shouldn't it clip at 16 and 235?

http://www.chrisnology.info/videos/Cliptest.jpg

Possible explanations: an issue with your source video, or testing method, or scopes ? It definitely works here

I can check your video if you want to upload it

I'm looking to clip the video without altering the levels.

Understood; then you want to use lutyuv or limiter

The problem is that these ringing artifacts pop up when you encode the video to a deliverable format, be it h.264, h.265 or even prores. They can be tamed to an extent by applying some "unsharpening" and removing some of the high-frequency content.

You can use lossless h264 or h265, or other lossless formats - but they are typically not used for mezzanine or intermediate deliverables. The final end delivery format will have some of those lossy issues anyways.

[poisondeathray]

8bit YUV full range test pattern before , and lutyuv clipping after

[richardpl]

If you didn't know ffmpeg have own waveform monitor: http://trac.ffmpeg.org/wiki/WaveformMonitor

[chris319]

an issue with your source video, or testing method, or scopes ?

Here it is with no filtering at all:

ffmpeg -i stairstep.mp4 -y outfile.mp4

http://www.chrisnology.info/videos/Nofilter.jpg

Nearly perfect reproduction of the video levels. No problem with source video, testing method or scope.

So the problem is either with my ffmpeg filter syntax or with ffmpeg itself.

[poisondeathray]

an issue with your source video, or testing method, or scopes ?

Here it is with no filtering at all:

ffmpeg -i stairstep.mp4 -y outfile.mp4

http://www.chrisnology.info/videos/Nofilter.jpg

Nearly perfect reproduction of the video levels. No problem with source video, testing method or scope.

So the problem is either with my ffmpeg filter syntax or with ffmpeg itself.

Please upload your video. Likely it's either a problem with your video, or you're not actually measuring YUV, but RGB in a standard range conversion

[chris319]

Here is the code used to create the video from a bmp.

bin\ffmpeg -y  -loop 1 -t 10  -i stairstep.bmp  -pix_fmt yuv420p  -crf 17  -c:v libx264  -vf scale=out_color_matrix=bt709  -color_primaries bt709  -color_trc bt709  -colorspace bt709  -r 59.94  -c:a copy  test_pattern.mp4

Please upload your video. Likely it's either a problem with your video, or you're not actually measuring YUV, but RGB in a standard range conversion

You miss the point. Near-perfect reproduction with no lutyuv filter. It just doesn't clip.

Please post the code used for your clipping and point out any differences between your code and mine.

http://www.chrisnology.info/videos/outfile.mp4

The scope is beyond reproach. It has been thoroughly and rigorously calibrated and tested with Matlab and it passes muster. Again, it perfectly reproduces the gray scale.

[poisondeathray]

Here is the code used to create the video from a bmp.

bin\ffmpeg -y  -loop 1 -t 10  -i stairstep.bmp  -pix_fmt yuv420p  -crf 17  -c:v libx264  -vf scale=out_color_matrix=bt709  -color_primaries bt709  -color_trc bt709  -colorspace bt709  -r 59.94  -c:a copy  test_pattern.mp4

Please upload your video. Likely it's either a problem with your video, or you're not actually measuring YUV, but RGB in a standard range conversion

You miss the point. Near-perfect reproduction with no lutyuv filter. It just doesn't clip.

Please post the code used for your clipping and point out any differences between your code and mine.

http://www.chrisnology.info/videos/outfile.mp4

The scope is beyond reproach. It has been thoroughly and rigorously calibrated and tested with Matlab and it passes muster. Again, it perfectly reproduces the gray scale.

The reason is there is nothing to clip . It' s not a full range video . Your RGB to YUV conversion for the test video uses limited range (RGB 0,0,0-255,255,255 gets "mapped" to Y 16-235, CbCr 16-240)

If nothing happens after lutyuv clipping, the most likely explanation is your video has nothing to clip - it is limited range Y (16-235) , and you are measuring standard range converted RGB values . That is exactly what is happening here

Technically , a Y' waveform is supposed to measure Y values, not their RGB converted values

If you look at a real Y' waveform, there are no values <16 or >235 in your video

[chris319]

First problem: the ffmpeg scope is confining the levels to 16 - 235. That's not what's in the file. The stairstep ranges from 0 - 255.

http://www.chrisnology.info/videos/ffmpegscope.jpg

Here is the PureBasic code used to generate the stairstep. Now you've got all the evidence.

Global screenWd, screenHt

Procedure DrawGrayScale()
      StartDrawing(ImageOutput(1))
           Box(0,0,screenWd,ScreenHt, 0)

        Steps = 16
        X = 0
        Lum.f = 0
        LumStep.f = 255 / (Steps-1)
        XStep = ScreenWd/Steps
        For ct = 1 To Steps
          Box(X,0,XStep,ScreenHt, RGB(Lum,Lum,Lum))
          Lum + LumStep
          X   + XStep
        Next
        StopDrawing()
EndProcedure  

CreateImage(1,1280,720)

ExamineDesktops()
      ScreenWd = DesktopWidth(0):ScreenHt = DesktopHeight(0)
      OpenWindow(0,0,0, ScreenWd,ScreenHt, #Null$,#PB_Window_BorderLess)
      AddKeyboardShortcut(n, #PB_Shortcut_Escape, #PB_Shortcut_Escape) ;"ESCAPE" KEY TO QUIT PROGRAM
      DrawGrayScale(): ShowCursor_(#False)

SaveImage(1,"stairstep.bmp")
StartDrawing(WindowOutput(0))
DrawImage(ImageID(1),0,0)
StopDrawing()

    Repeat
      event = WaitWindowEvent()
      If event = #PB_Event_Repaint
        DrawGrayScale()
        
      ElseIf event = #PB_Event_Menu ;KEYBOARD INPUT
        menuItem = EventMenu()
       
        Select menuItem
           
          Case #PB_Shortcut_Escape
            Break
           
        EndSelect
       
      EndIf
    ForEver
    End

[chris319]

Clearly you don't understand how the lum. component of a video signal is derived.

[poisondeathray]

Your source video is limited range , it's already standard range ,with nothing to clip. (Just look at the command line you used, it converts RGB to YUV with limited range, that's already proof enough)

Your scope actually measures RGB values if it shows the 0 and 255 values.

You can verify this with multiple methods , not necessarily ffmpeg (I like using other methods to verify). For example , you can verify this with broadcast NLE or hardware SDI out

I am 100% certain of these facts.

[chris319]

Steps = 16
        X = 0
        Lum.f = 0
        LumStep.f = 255 / (Steps-1)
        XStep = ScreenWd/Steps
        For ct = 1 To Steps
          Box(X,0,XStep,ScreenHt, RGB(Lum,Lum,Lum))
          Lum + LumStep
          X   + XStep
        Next

The stairstep is 0 to 255. Here's the source code.

ffmpeg -y  -loop 1 -t 10  -i stairstep.bmp  -pix_fmt yuv420p  -crf 17  -c:v libx264  -vf scale=out_color_matrix=bt709  -color_primaries bt709  -color_trc bt709  -colorspace bt709  -r 59.94  -c:a copy  test_pattern.mp4

 it converts RGB to YUV with limited range

where do you see a filter that would limit levels to 16 - 235? It's not there.

Your scope actually measures RGB values if it shows the 0 and 255 values.

No, it calculates lum. from RGB. I hope you know that YUV can range from 0 to 255.

Now view the video and count the steps (there are 16 steps, 16 decimal values apart. Now take an eyedropper program or use Colorzilla to measure the colors. They range all the way from 0 to 255.

Your ffmpeg scope is hiding everything outside the range of 16 - 235, but you're resistant to the notion that it's basically lying to you.

Where do you think the lum. component comes from? It's calculated from the R, G and B components. Kr, Kg and Kb are the lum. coefficients. I'm surprised you don't already know this.

Y = R * Kr + G * Kg + B * Kb

I will scour the docs and try to find out it it's possible to get the scope to show the 0 - 255 range.

[poisondeathray]

The stairstep is 0 to 255. Here's the source code.

Your RGB test pattern is probably OK. That's not what the concern is. The concern is the RGB=>YUV conversion, and your scopes

ffmpeg -y  -loop 1 -t 10  -i stairstep.bmp  -pix_fmt yuv420p  -crf 17  -c:v libx264  -vf scale=out_color_matrix=bt709  -color_primaries bt709  -color_trc bt709  -colorspace bt709  -r 59.94  -c:a copy  test_pattern.mp4

 it converts RGB to YUV with limited range

where do you see a filter that would limit levels to 16 - 235? It's not there.

I never said there was a filter. I said that commandline is for a limited (standard) range conversion, not a full range conversion

Read up on ITU Rec 709 conversion. If I recall, you coded some RGB<=>YUV functions. Might be a good idea to revisit
https://www.itu.int/rec/R-REC-BT.709/en
https://en.wikipedia.org/wiki/Rec._709

Yes you started with a nice RGB test pattern 0-255. But your RGB conversion to YUV gets "mapped" to Y 16-235, CbCr16-240 . So you have no values Y<16 or Y>235. You have to use a "full range" conversion to map RGB0-255 to YUV0-255. It's know as a "full range" conversion.

Your scope actually measures RGB values if it shows the 0 and 255 values.

No, it calculates lum. from RGB. I hope you know that YUV can range from 0 to 255.

This tells me it's measuring RGB values. Your source file only started with Y 16-235. If it's reading 0-255, it's wrong. Likely Rec conversion back to RGB and reading that. ie. the reverse transform of YUV back to RGB. , not the actual YUV values

Your ffmpeg scope is hiding everything outside the range of 16 - 235, but you're resistant to the notion that it's basically lying to you.

I'm not resistant, because I wasn't using a ffmpeg scope. I use multiple tools to confirm things. Look at the screenshot I posted. That is a broadcast NLE scope. You can confirm with other professional programs or a SDI hardware scope - they all say the same thing.

This is your outfile.mp4 source. Notice it's limited range, there are no values >100 or <0 IRE. Compare to the full range one I posted above before applying lutyuv. Values are preset <0 IRE, and >100IRE (which correspond to Y=16, Y=235 in 8bit)


But in this case, the ffmpeg waveform is correct too (it's not always, some formats have decoding and levels issues, or certain flags can confuse the issue). This is a true full range YUV video, and with lutyuv clipping applied . Your outfile.mp4 shows no values <0 or >100. It's limited range




Again, I'm 100% certain about this.