Lossless (10 Bit RGB 444) and (10 Bit YUV 422) Compression Codec's

[JasonCA]

I am looking for a Lossless (10 bit RGB 444) and (10 Bit YUV 422) compression codec in a standard 720 x 480 frame format for archival purposes...

I had some 8mm Film transfer work done and had them store the film uncompressed for me. They look fantastic! I am happy with the results. However, before I get more transferred, I want to find a way to store them a bit more efficiently.

I'll have to look again, but I believe the video is stored with a Black Magic codec in (10 Bit YUV 4:2:2 - 'v210') uncompressed and another format (10 Bit RGB 4:4:4) uncompressed also. These were transferred at different times.

Perhaps in the future and when I have more time to edit them, I'll convert them to a lossy format such as XVID or something...but that's in the future and when I can spend time to do it correctly. However, for now, I'd like to compact these videos down and keep them in a compressed lossless format instead of as I have them right now as FULLY uncompressed. So, please keep that in mind.

I heard that ffmpeg may be able to compress video in a lossless (10 bit RGB 444) and event (10 Bit YUV 422)? Can anyone confirm this to be true?

It seems like the runner up that I may investigate is one from SheerVideo.

http://www.bitjazz.com/en/products/sheervideo/

They seem to offer a compression codec for what I may want to do, although I'm not sure if anyone else has look at it either?

If ffmpeg does this, that would be great!

[poisondeathray]

FFV1 using ffmpeg supports 10bit YUV 444, 422, 420 for sure. I don't think FFV1 supports 10bit RGB

[JasonCA]

FFV1 using ffmpeg supports 10bit YUV 444, 422, 420 for sure. I don't think FFV1 supports 10bit RGB

Lossless?

Would you know the correct syntax for such a conversion? A good test would be:

10bit YUV 422 uncompressed -> 10 bit YUV 422 lossless compressed
10 bit YUV 422 lossless compressed-> 10bit YUV 422 uncompressed

It would be good if FFV1 did RGB too *crossing fingers*

[poisondeathray]

Yes, FFV1 is lossless

Actually it does support 10bit RGB, just tested it

for v210 (10bit YUV 422)
ffmpeg -i input.avi -pix_fmt yuv422p10le -vcodec ffv1 -an 10bit422output.avi

for r10k (10bit RGB)
ffmpeg -i input.avi pix_fmt gbrp10le -vcodec ffv1 -an 10bitRGBoutput.avi

(if you want to stream copy audio, use "-acodec copy" instead of "-an"

[JasonCA]

Yes, FFV1 is lossless

Actually it does support 10bit RGB, just tested it

for v210 (10bit YUV 422)
ffmpeg -i input.avi -pix_fmt yuv422p10le -vcodec ffv1 -an 10bit422output.avi

for r10k (10bit RGB)
ffmpeg -i input.avi pix_fmt gbrp10le -vcodec ffv1 -an 10bitRGBoutput.avi

(if you want to stream copy audio, use "-acodec copy" instead of "-an"

Wow, great! I am excited! I will start setting up and testing.

Yes, that's what it was. I have some stored in v210 and others stored in r10k (couldn't remember what it was called).

For my testing too.... would you know how to convert back?

10bit422output.avi -> back to uncompressed 'v210'

and

10bitRGBoutput.avi -> back to uncompressed 'r10k'

[poisondeathray]

same thing , just use -vcodec v210 or -vcodec r10k

If you're not planning on using the losslessly compressed files - and the plan is to compress, decompress just for storage purposes - why not use archiving software instead ? The potential benefits are better file management, built-in error correction and abilty to use recovery records

[JasonCA]

same thing , just use -vcodec v210 or -vcodec r10k

If you're not planning on using the losslessly compressed files - and the plan is to compress, decompress just for storage purposes - why not use archiving software instead ? The potential benefits are better file management, built-in error correction and ability to use recovery records

I'll eventually want to read the files to view the video too. However, explain what you mean by "archiving software"? Such as? You mean like a zip file? Or?

Also, we are talking about a lossless compression. What would be the tradeoffs as far as space and time (FFV1 lossless compressed vs "archiving software")?

My assumption, though I could be wrong, was that FFV1 lossless compressed may be better than a type of archival software? Or?

[poisondeathray]

same thing , just use -vcodec v210 or -vcodec r10k

If you're not planning on using the losslessly compressed files - and the plan is to compress, decompress just for storage purposes - why not use archiving software instead ? The potential benefits are better file management, built-in error correction and ability to use recovery records

I'll eventually want to read the files to view the video too. However, explain what you mean by "archiving software"? Such as? You mean like a zip file? Or?

Also, we are talking about a lossless compression. What would be the tradeoffs as far as space and time (FFV1 lossless compressed vs "archiving software")?

My assumption, though I could be wrong, was that FFV1 lossless compressed may be better than a type of archival software? Or?

Yes, I mean like a zip file

You can use FFV1 as a video, but it's heavily compressed and sluggish on the timeline (it's not a "fast" editing codec, like ut video codec, or huffyuv). You should be ok for SD resolution , however

Compression wise, FFV1 is very good, better than intra frame lossless codecs like lagarith, huffyuv in the 8bit scenario; it also uses inter frame (temporal) compression. But that's also why it's SLOOOOW to encode and decode. I'm not sure how it compares to 7-zip and the like , you can run some quick tests

The problem with FFV1 is not all NLE's will recognize it, and not necessarily correctly at 10bits either , so you might have to "decompress" it anyway (Whereas v210 is almost universally accepted, even on Mac's where it just needs to be re-wrapped in a MOV container) . It depends on your workflow and what software you are using

10bit RGB isn't used very much. Usually the jump is to 16bit RGB (I think FFV1's 10bit RGB implementation is actually "10bit in 16bit code" , so not native 10bit, so not as efficient)

[JasonCA]

Yes, I mean like a zip file

Ok, that's what you mean by "archival software"

You can use FFV1 as a video, but it's heavily compressed and sluggish on the timeline (it's not a "fast" editing codec, like ut video codec, or huffyuv). You should be ok for SD resolution , however

Great point. However, I don't mind the heavy compression. I am thinking for its use down the road. Probably by that time, I'd have a machine that is Intel machine with 32 cores Who knows right. In which case, this video will most likely be nothing to deal with.

I'm looking for heavy compression, convenient to view in the future, and lossless. Most importantly is for it to be tightly compressed as possible so I can get rid of these uncompressed files that are HUGE. 300GB for one file uncompressed for some.

Compression wise, FFV1 is very good, better than intra frame lossless codecs like lagarith, huffyuv in the 8bit scenario; it also uses inter frame (temporal) compression. But that's also why it's SLOOOOW to encode and decode. I'm not sure how it compares to 7-zip and the like , you can run some quick tests

Excellent! I like that it's heavily compressed...which is what makes me intrested. I was thinking that FFV1 might do better than say something like 7-zip considering FFV1 is fully optimized to deal with video...whereas 7-zip though great is not. But I don't know.

Yes, I plan on testing between 7-zip and FFV1...this will be my goal's for comparisons.

The problem with FFV1 is not all NLE's will recognize it, and not necessarily correctly at 10bits either , so you might have to "decompress" it anyway (Whereas v210 is almost universally accepted, even on Mac's where it just needs to be re-wrapped in a MOV container) . It depends on your workflow and what software you are using

In the worst case, I'd convert from FFV1 in the future to something that I could then deal better with in NLE's. As you know, we are heading towards 4K and what not. I'll have plenty of room to play since in comparison this is only SD stuff.

10bit RGB isn't used very much. Usually the jump is to 16bit RGB (I think FFV1's 10bit RGB implementation is actually "10bit in 16bit code" , so not native 10bit, so not as efficient)

The choice was 8 bit or 10 bit at the time. I opted for 10 bit for YUV 422 and then another time for 10 bit RGB 444. For film, I thought 10 bit RGB 444 would be better than YUV 422? However, the file sizes are bigger. I was thinking 10 bit RGB 444 would be more authentic to the original since there's no down-sampling of the chroma.

So, you are saying the jump is from 8bit to 16 bit ... a 2x jump as opposed to the 10 bit jump. Very interesting...I wonder why that is.

[poisondeathray]

Yes, I plan on testing between 7-zip and FFV1...this will be my goal's for comparisons.

Please post the testing results if you have time . Compression ratio and speed/encode times if you have the time

10bit RGB isn't used very much. Usually the jump is to 16bit RGB (I think FFV1's 10bit RGB implementation is actually "10bit in 16bit code" , so not native 10bit, so not as efficient)

The choice was 8 bit or 10 bit at the time. I opted for 10 bit for YUV 422 and then another time for 10 bit RGB 444. For film, I thought 10 bit RGB 444 would be better than YUV 422? However, the file sizes are bigger. I was thinking 10 bit RGB 444 would be more authentic to the original since there's no down-sampling of the chroma.

So, you are saying the jump is from 8bit to 16 bit ... a 2x jump as opposed to the 10 bit jump. Very interesting...I wonder why that is.

Back when film was used all the time (not digital acquisition) , digital scans were almost exclusively at 16bit RGB . Yes, intermediate workflows used 10bit RGB in the form of DPX, but the original scans were almost always 16bit

FFV1 's RGB implementation is not native RGB either, it's R, G, B encoded in separate channels as Y, Cb, Cr - or a "GBR" matrix - It's lossless, but again not as efficient (this is the same way x264 does it) . I think you'll find the RGB compression ratio not as good as it's YCbCr compression ratio

[JasonCA]

Please post the testing results if you have time . Compression ratio and speed/encode times if you have the time

Yes, I will do so once I begin the testing. Once I get a few conversions going, I'll be on my way to run different tests. I'm sure to find things along the way.

Back when film was used all the time (not digital acquisition) , digital scans were almost exclusively at 16bit RGB . Yes, intermediate workflows used 10bit RGB in the form of DPX, but the original scans were almost always 16bit

So, I'll have to assume my 8mm film scans were scanned at a true 10BIT RGB or above to make stroring the data in them worth it. I realize that the YUV colorspace is bigger than that of RGB. However, not all YUV translates into RGB. I am still trying to wrap my head around understanding that. That's why I opted for 10bit RGB 444 over 10bit YUV 422 since I wasn't sure. Plus, I thought that I would have issues translating the YUV video to RGB at the time.

FFV1 's RGB implementation is not native RGB either, it's R, G, B encoded in separate channels as Y, Cb, Cr - or a "GBR" matrix - It's lossless, but again not as efficient (this is the same way x264 does it) . I think you'll find the RGB compression ratio not as good as it's YCbCr compression ratio

I see, so you are saying that FFV1 doesn't excel at compressing RGB that well. They compress it separate channels as Y, Cb, CR....a lossless matrix...but not that efficient of a compression scheme. I'm assuming it's a full 4:4:4 Y, Cb, Cr to store the RGB.

It'll be interesting to see how well FFV1 can compress the uncompressed YUV (I believe they refer to this properly as YCbCr). 2x times? 4 times? I have no idea...it's why I'm curious and why I'll run some tests.

[poisondeathray]

It'll be interesting to see how well FFV1 can compress the uncompressed YUV (I believe they refer to this properly as YCbCr). 2x times? 4 times? I have no idea...it's why I'm curious and why I'll run some tests.

It depends on the type of content . Some types of content are more compressible than others

On average, you should expect ~ 1/2 size of uncompressed

I'm assuming it's a full 4:4:4 Y, Cb, Cr to store the RGB.

Yes it is , that's the only way that it can be lossless

Note - YCbCr 444 <=> RGB 444 normally isn't a lossless transformation. It's only storing the data as YCbCr (there is no actual colormodel or colorspace conversion occurring)

[vhelp]

what kind of 8mm are you talking about anyway ? was this from footage you shot with an 8mm camera or Hi8 or digital 8 ?

if this is home video, though i could be wrong, i doubt you shoot video at 10bit. 8bit is more like it.

you need to find out for certain if they were scanned at 10bit. not best to assume, because you would be waisting a lot of resources for nothing if not very minimal for 10bit archival.

[JasonCA]

So, I thought I would give this a quick go! I was very currious ...

v210 to FFV Example Test:

Original: 10BitYUV422_A.avi (file size: 410,215 KB)

Lossless Compression (v210->FFV): 10BitYUV422_B.avi (file size: 119,281 KB)

Back to Original (FFV->v210): 10BitYUV422_C.avi (file size: 410,079 KB)

Wow, that's like 1/4 the size for YUV video (at least for my test). So that's fantastic! Well, sort of ...

However, as you can see there is a difference of 136KB. Obviously, not the same.

So what remains is: how do I know it's lossless? Not sure how I can verify that it's giving me my original video back. For all I know, internally it's compressed it to 4:2:0 and then uncompressed back to 4:2:2. How do I know?

[JasonCA]

what kind of 8mm are you talking about anyway ? was this from footage you shot with an 8mm camera or Hi8 or digital 8 ?

Hello VHelp!

Super 8mm

if this is home video, though i could be wrong, i doubt you shoot video at 10bit. 8bit is more like it.

Film transfer to digital.

you need to find out for certain if they were scanned at 10bit. not best to assume, because you would be waisting a lot of resources for nothing if not very minimal for 10bit archival.

Yes, this is something I will do. As of right now, I have the data....so I want to see if I can at least compress it meanwhile...that obviously won't change. Assuming it wasn't scanned at 10bit, then storing 10bit uncompressed video data is a bit of waist. Otherwise, I'll have verify that they do 10bit scans or greater. For all I know it was scanned at 16bit and then down-converted to 10bit...as poisondeathray was mentioning. But to answer your question...I don't know for sure.

[JasonCA]

Thank you poisondeathray!!!!

I made some headway on this because of you!

10 Bit YUV 4:2:2 (v210) -> FFV1 lossless -> 10 Bit YUV 4:2:2 (v210):

First, 10 Bit YUV 4:2:2 (v210) -> FFV1 lossless

ffmpeg -i 10BitYUV422_210_Capture1.avi -pix_fmt yuv422p10le -vcodec ffv1 -c:a copy V210ToFFV.avi -f framemd5 -an V210ToFFV.framemd5

Results in:

Results in:

ffmpeg version N-59275-g9b195dd Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 21 2013 22:06:20 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 58.101 / 52. 58.101
  libavcodec     55. 45.103 / 55. 45.103
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.100 /  4.  0.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, avi, from '10BitYUV422_210_Capture1.avi':
  Duration: 00:00:15.02, start: 0.000000, bitrate: 223808 kb/s
    Stream #0:0: Video: v210 (v210 / 0x30313276), yuv422p10le, 720x486, 29.97 tb
r, 29.97 tbn, 29.97 tbc
[ffv1 @ 0000000002830e40] bits_per_raw_sample > 8, forcing coder 1
Output #0, avi, to 'V210ToFFV.avi':
  Metadata:
    ISFT            : Lavf55.22.100
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv422p10le, 720x486, q=2-31,
200 kb/s, 29.97 tbn, 29.97 tbc
Output #1, framemd5, to 'V210ToFFV.framemd5':
  Metadata:
    encoder         : Lavf55.22.100
    Stream #1:0: Video: rawvideo (Y3[10][10] / 0xA0A3359), yuv422p10le, 720x486,
 q=2-31, 200 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (v210 -> ffv1)
  Stream #0:0 -> #1:0 (v210 -> rawvideo)
Press [q] to stop, [?] for help
frame=   18 fps=0.0 q=0.0 q=0.0 size=    4056kB time=00:00:00.60 bitrate=55324.8
frame=   38 fps= 37 q=0.0 q=0.0 size=    8199kB time=00:00:01.26 bitrate=52971.2
frame=   57 fps= 37 q=0.0 q=0.0 size=   12199kB time=00:00:01.90 bitrate=52544.8
frame=   76 fps= 38 q=0.0 q=0.0 size=   16251kB time=00:00:02.53 bitrate=52497.0
frame=   95 fps= 38 q=0.0 q=0.0 size=   20360kB time=00:00:03.16 bitrate=52617.4
frame=  114 fps= 38 q=0.0 q=0.0 size=   24468kB time=00:00:03.80 bitrate=52695.4
frame=  133 fps= 38 q=0.0 q=0.0 size=   28591kB time=00:00:04.43 bitrate=52779.0
frame=  152 fps= 38 q=0.0 q=0.0 size=   32706kB time=00:00:05.07 bitrate=52827.3
frame=  171 fps= 38 q=0.0 q=0.0 size=   36895kB time=00:00:05.70 bitrate=52972.5
frame=  190 fps= 38 q=0.0 q=0.0 size=   41095kB time=00:00:06.33 bitrate=53102.6
frame=  209 fps= 38 q=0.0 q=0.0 size=   45335kB time=00:00:06.97 bitrate=53255.0
frame=  228 fps= 38 q=0.0 q=0.0 size=   49598kB time=00:00:07.60 bitrate=53407.6
frame=  247 fps= 37 q=0.0 q=0.0 size=   53851kB time=00:00:08.24 bitrate=53527.1
frame=  266 fps= 37 q=0.0 q=0.0 size=   58140kB time=00:00:08.87 bitrate=53662.1
frame=  282 fps= 37 q=0.0 q=0.0 size=   62868kB time=00:00:09.40 bitrate=54734.0
frame=  296 fps= 36 q=0.0 q=0.0 size=   67530kB time=00:00:09.87 bitrate=56012.3
frame=  310 fps= 36 q=0.0 q=0.0 size=   72190kB time=00:00:10.34 bitrate=57173.4
frame=  324 fps= 35 q=0.0 q=0.0 size=   76841kB time=00:00:10.81 bitrate=58227.4
frame=  338 fps= 35 q=0.0 q=0.0 size=   81502kB time=00:00:11.27 bitrate=59201.3
frame=  352 fps= 35 q=0.0 q=0.0 size=   86156kB time=00:00:11.74 bitrate=60092.5
frame=  366 fps= 34 q=0.0 q=0.0 size=   90799kB time=00:00:12.21 bitrate=60908.4
frame=  380 fps= 34 q=0.0 q=0.0 size=   95447kB time=00:00:12.67 bitrate=61667.3
frame=  394 fps= 34 q=0.0 q=0.0 size=  100103kB time=00:00:13.14 bitrate=62377.2
frame=  408 fps= 33 q=0.0 q=0.0 size=  104790kB time=00:00:13.61 bitrate=63057.6
frame=  422 fps= 33 q=0.0 q=0.0 size=  109680kB time=00:00:14.08 bitrate=63810.7
frame=  436 fps= 33 q=0.0 q=0.0 size=  114485kB time=00:00:14.54 bitrate=64467.4
frame=  449 fps= 32 q=0.0 q=0.0 size=  118934kB time=00:00:14.98 bitrate=65033.2
frame=  450 fps= 32 q=0.0 Lq=0.0 size=  119280kB time=00:00:15.01 bitrate=65077.
9kbits/s
video:734358kB audio:0kB subtitle:0 global headers:0kB muxing overhead -83.75719
5%

Bits per raw sample is 8?

So, I created a 'framemd5' file that I'll use to compare with when I convert it back.

Moving ON...so far so good!

Second, FFV1 lossless -> 10 Bit YUV 4:2:2 (v210):

ffmpeg -i V210ToFFV.avi -pix_fmt yuv422p10le -vcodec v210 -c:a copy FFVTov210.avi -f framemd5 -an FFVToV210.framemd5

Results in:

ffmpeg version N-59275-g9b195dd Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 21 2013 22:06:20 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 58.101 / 52. 58.101
  libavcodec     55. 45.103 / 55. 45.103
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.100 /  4.  0.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, avi, from 'V210ToFFV.avi':
  Metadata:
    encoder         : Lavf55.22.100
  Duration: 00:00:15.02, start: 0.000000, bitrate: 65077 kb/s
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv422p10le, 720x486, 29.97 tb
r, 29.97 tbn, 29.97 tbc
Output #0, avi, to 'FFVTov210.avi':
  Metadata:
    ISFT            : Lavf55.22.100
    Stream #0:0: Video: v210 (v210 / 0x30313276), yuv422p10le, 720x486, q=2-31,
200 kb/s, 29.97 tbn, 29.97 tbc
Output #1, framemd5, to 'FFVToV210.framemd5':
  Metadata:
    encoder         : Lavf55.22.100
    Stream #1:0: Video: rawvideo (Y3[10][10] / 0xA0A3359), yuv422p10le, 720x486,
 q=2-31, 200 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ffv1 -> v210)
  Stream #0:0 -> #1:0 (ffv1 -> rawvideo)
Press [q] to stop, [?] for help
frame=   24 fps=0.0 q=0.0 q=0.0 size=   21876kB time=00:00:00.80 bitrate=223783.
frame=   54 fps= 52 q=0.0 q=0.0 size=   49213kB time=00:00:01.80 bitrate=223752.
frame=   81 fps= 52 q=0.0 q=0.0 size=   73817kB time=00:00:02.70 bitrate=223743.
frame=  107 fps= 52 q=0.0 q=0.0 size=   97510kB time=00:00:03.57 bitrate=223739.
frame=  133 fps= 52 q=0.0 q=0.0 size=  121203kB time=00:00:04.43 bitrate=223737.
frame=  156 fps= 50 q=0.0 q=0.0 size=  142162kB time=00:00:05.20 bitrate=223735.
frame=  185 fps= 51 q=0.0 q=0.0 size=  168588kB time=00:00:06.17 bitrate=223734.
frame=  211 fps= 51 q=0.0 q=0.0 size=  192281kB time=00:00:07.04 bitrate=223733.
frame=  237 fps= 51 q=0.0 q=0.0 size=  215974kB time=00:00:07.90 bitrate=223732.
frame=  263 fps= 51 q=0.0 q=0.0 size=  239666kB time=00:00:08.77 bitrate=223732.
frame=  286 fps= 51 q=0.0 q=0.0 size=  260625kB time=00:00:09.54 bitrate=223731.
frame=  306 fps= 50 q=0.0 q=0.0 size=  278850kB time=00:00:10.21 bitrate=223731.
frame=  324 fps= 48 q=0.0 q=0.0 size=  295253kB time=00:00:10.81 bitrate=223731.
frame=  345 fps= 48 q=0.0 q=0.0 size=  314389kB time=00:00:11.51 bitrate=223730.
frame=  365 fps= 47 q=0.0 q=0.0 size=  332615kB time=00:00:12.17 bitrate=223730.
frame=  383 fps= 46 q=0.0 q=0.0 size=  349017kB time=00:00:12.77 bitrate=223730.
frame=  403 fps= 46 q=0.0 q=0.0 size=  367242kB time=00:00:13.44 bitrate=223730.
frame=  420 fps= 45 q=0.0 q=0.0 size=  382734kB time=00:00:14.01 bitrate=223730.
frame=  437 fps= 45 q=0.0 q=0.0 size=  398225kB time=00:00:14.58 bitrate=223730.
frame=  450 fps= 44 q=0.0 Lq=0.0 size=  410079kB time=00:00:15.01 bitrate=223733
.9kbits/s
video:1025156kB audio:0kB subtitle:0 global headers:0kB muxing overhead -59.9984
30%

Finaly, the results:

I compare the V210ToFFV.framemd5 to FFVToV210.framemd5 and did a binary compare...they are EXACTLY the same.

However, my resulting file is as follow:

• Original file: 10BitYUV422_210_Capture1.avi = 400MB (420,060,160 bytes)
• Converted to FFV lossless: V210ToFFV.avi = 116 MB (122,143,026 bytes)
• FFV lossless back to original: FFVTov210.avi = 400 MB (419,920,482 bytes)

However, take notice the byte size is NOT the same. They are close, but not exactly the same! Does the 'framemd5' being identical prove they converted correctly? And perhaps I should not worry about the small difference?

[JasonCA]

Ok, now I want to verify r10k lossless conversions...

10 Bit RGB 4:4:4 (r10k) -> FFV1 lossless -> 10 Bit RGB 4:4:4 (r10k):

First, 10 Bit RGB 4:4:4 (r10k) -> FFV1 lossless

ffmpeg -i 10BitRGB444_Full_B_Capture3.avi -pix_fmt gbrp10le -vcodec ffv1 -coder 1 -c:a copy r10kToFFV.avi -f framemd5 -an r10kToFFV.framemd5

Results In:

ffmpeg version N-59275-g9b195dd Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 21 2013 22:06:20 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 58.101 / 52. 58.101
  libavcodec     55. 45.103 / 55. 45.103
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.100 /  4.  0.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, avi, from '10BitRGB444_Full_B_Capture3.avi':
  Duration: 00:00:15.02, start: 0.000000, bitrate: 335640 kb/s
    Stream #0:0: Video: r10k (r10k / 0x6B303172), rgb48le(10 bpc), 720x486, 29.9
7 tbr, 29.97 tbn, 29.97 tbc
Output #0, avi, to 'r10kToFFV.avi':
  Metadata:
    ISFT            : Lavf55.22.100
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le, 720x486, q=2-31, 200
 kb/s, 29.97 tbn, 29.97 tbc
Output #1, framemd5, to 'r10kToFFV.framemd5':
  Metadata:
    encoder         : Lavf55.22.100
    Stream #1:0: Video: rawvideo (RGB0 / 0x30424752), rgb48le(10 bpc), 720x486,
q=2-31, 200 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (r10k -> ffv1)
  Stream #0:0 -> #1:0 (r10k -> rawvideo)
Press [q] to stop, [?] for help
frame=   10 fps=0.0 q=0.0 q=0.0 size=    3344kB time=00:00:00.33 bitrate=82088.7
frame=   20 fps= 19 q=0.0 q=0.0 size=    6666kB time=00:00:00.66 bitrate=81834.7
frame=   30 fps= 19 q=0.0 q=0.0 size=    9990kB time=00:00:01.00 bitrate=81759.1
frame=   40 fps= 18 q=0.0 q=0.0 size=   13437kB time=00:00:01.33 bitrate=82476.9
frame=   50 fps= 18 q=0.0 q=0.0 size=   16815kB time=00:00:01.66 bitrate=82565.6
frame=   60 fps= 18 q=0.0 q=0.0 size=   20211kB time=00:00:02.00 bitrate=82699.8
frame=   70 fps= 18 q=0.0 q=0.0 size=   23603kB time=00:00:02.33 bitrate=82784.8
frame=   79 fps= 18 q=0.0 q=0.0 size=   26666kB time=00:00:02.63 bitrate=82873.0
frame=   89 fps= 18 q=0.0 q=0.0 size=   30080kB time=00:00:02.96 bitrate=82977.1
frame=   99 fps= 18 q=0.0 q=0.0 size=   33479kB time=00:00:03.30 bitrate=83025.0
frame=  107 fps= 18 q=0.0 q=0.0 size=   37867kB time=00:00:03.57 bitrate=86887.3
frame=  112 fps= 17 q=0.0 q=0.0 size=   43623kB time=00:00:03.73 bitrate=95625.1
frame=  116 fps= 16 q=0.0 q=0.0 size=   48216kB time=00:00:03.87 bitrate=102049.
frame=  121 fps= 16 q=0.0 q=0.0 size=   53967kB time=00:00:04.03 bitrate=109502.
frame=  126 fps= 15 q=0.0 q=0.0 size=   59699kB time=00:00:04.20 bitrate=116325.
frame=  131 fps= 15 q=0.0 q=0.0 size=   65433kB time=00:00:04.37 bitrate=122632.
frame=  136 fps= 15 q=0.0 q=0.0 size=   71175kB time=00:00:04.53 bitrate=128488.
frame=  141 fps= 14 q=0.0 q=0.0 size=   76916kB time=00:00:04.70 bitrate=133928.

< omitted some of this >

frame=  388 fps= 10 q=0.0 q=0.0 size=  361572kB time=00:00:12.94 bitrate=228791.
frame=  393 fps= 10 q=0.0 q=0.0 size=  367310kB time=00:00:13.11 bitrate=229465.
frame=  398 fps= 10 q=0.0 q=0.0 size=  373060kB time=00:00:13.27 bitrate=230129.
frame=  403 fps=9.9 q=0.0 q=0.0 size=  378810kB time=00:00:13.44 bitrate=230777.
frame=  408 fps=9.9 q=0.0 q=0.0 size=  384498kB time=00:00:13.61 bitrate=231372.
frame=  413 fps=9.9 q=0.0 q=0.0 size=  390171kB time=00:00:13.78 bitrate=231943.
frame=  418 fps=9.9 q=0.0 q=0.0 size=  395833kB time=00:00:13.94 bitrate=232494.
frame=  423 fps=9.9 q=0.0 q=0.0 size=  401502kB time=00:00:14.11 bitrate=233036.
frame=  428 fps=9.9 q=0.0 q=0.0 size=  407138kB time=00:00:14.28 bitrate=233547.
frame=  433 fps=9.8 q=0.0 q=0.0 size=  412795kB time=00:00:14.44 bitrate=234057.
frame=  438 fps=9.8 q=0.0 q=0.0 size=  418450kB time=00:00:14.61 bitrate=234556.
frame=  443 fps=9.8 q=0.0 q=0.0 size=  424091kB time=00:00:14.78 bitrate=235035.
frame=  448 fps=9.8 q=0.0 q=0.0 size=  429741kB time=00:00:14.94 bitrate=235507.
frame=  450 fps=9.8 q=0.0 Lq=0.0 size=  432009kB time=00:00:15.01 bitrate=235698
.8kbits/s
video:1354633kB audio:0kB subtitle:0 global headers:0kB muxing overhead -68.1087
87%

Take notice above of "rgb48le(10 bpc)"

Second, FFV1 lossless -> 10 Bit RGB 4:4:4 (r10k):

ffmpeg -i r10kToFFV.avi -pix_fmt gbrp10le -vcodec r10k -c:a copy FFVTor10k.avi -f framemd5 -an FFVTor10k.framemd5

Results In:

ffmpeg version N-59275-g9b195dd Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 21 2013 22:06:20 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 58.101 / 52. 58.101
  libavcodec     55. 45.103 / 55. 45.103
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.100 /  4.  0.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, avi, from 'r10kToFFV.avi':
  Metadata:
    encoder         : Lavf55.22.100
  Duration: 00:00:15.02, start: 0.000000, bitrate: 235698 kb/s
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), gbrp10le, 720x486, 29.97 tbr,
29.97 tbn, 29.97 tbc
Incompatible pixel format 'gbrp10le' for codec 'r10k', auto-selecting format 'rg
b48le'
[swscaler @ 00000000026e7fe0] full chroma interpolation for destination format '
rgb48le' not yet implemented
Output #0, avi, to 'FFVTor10k.avi':
  Metadata:
    ISFT            : Lavf55.22.100
    Stream #0:0: Video: r10k (R10k / 0x6B303152), rgb48le, 720x486, q=2-31, 200
kb/s, 29.97 tbn, 29.97 tbc
Output #1, framemd5, to 'FFVTor10k.framemd5':
  Metadata:
    encoder         : Lavf55.22.100
    Stream #1:0: Video: rawvideo (G3[0][10] / 0xA003347), gbrp10le, 720x486, q=2
-31, 200 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (ffv1 -> r10k)
  Stream #0:0 -> #1:0 (ffv1 -> rawvideo)
Press [q] to stop, [?] for help
frame=   13 fps=0.0 q=0.0 q=0.0 size=   17775kB time=00:00:00.43 bitrate=335693.
frame=   30 fps= 29 q=0.0 q=0.0 size=   41012kB time=00:00:01.00 bitrate=335634.
frame=   45 fps= 29 q=0.0 q=0.0 size=   61515kB time=00:00:01.50 bitrate=335619.
frame=   59 fps= 29 q=0.0 q=0.0 size=   80652kB time=00:00:01.96 bitrate=335612.
frame=   73 fps= 28 q=0.0 q=0.0 size=   99788kB time=00:00:02.43 bitrate=335608.
frame=   90 fps= 29 q=0.0 q=0.0 size=  123025kB time=00:00:03.00 bitrate=335604.
frame=  105 fps= 29 q=0.0 q=0.0 size=  143528kB time=00:00:03.50 bitrate=335602.
frame=  113 fps= 27 q=0.0 q=0.0 size=  154463kB time=00:00:03.77 bitrate=335601.
frame=  118 fps= 25 q=0.0 q=0.0 size=  161298kB time=00:00:03.93 bitrate=335601.
frame=  126 fps= 24 q=0.0 q=0.0 size=  172233kB time=00:00:04.20 bitrate=335600.
frame=  131 fps= 22 q=0.0 q=0.0 size=  179067kB time=00:00:04.37 bitrate=335599.
frame=  139 fps= 22 q=0.0 q=0.0 size=  190002kB time=00:00:04.63 bitrate=335599.
frame=  144 fps= 21 q=0.0 q=0.0 size=  196837kB time=00:00:04.80 bitrate=335599.

< omitted some of this >

frame=  371 fps= 14 q=0.0 q=0.0 size=  507119kB time=00:00:12.37 bitrate=335593.
frame=  379 fps= 14 q=0.0 q=0.0 size=  518054kB time=00:00:12.64 bitrate=335593.
frame=  384 fps= 14 q=0.0 q=0.0 size=  524889kB time=00:00:12.81 bitrate=335593.
frame=  392 fps= 14 q=0.0 q=0.0 size=  535824kB time=00:00:13.07 bitrate=335593.
frame=  397 fps= 13 q=0.0 q=0.0 size=  542658kB time=00:00:13.24 bitrate=335592.
frame=  405 fps= 13 q=0.0 q=0.0 size=  553593kB time=00:00:13.51 bitrate=335592.
frame=  412 fps= 13 q=0.0 q=0.0 size=  563161kB time=00:00:13.74 bitrate=335592.
frame=  417 fps= 13 q=0.0 q=0.0 size=  569996kB time=00:00:13.91 bitrate=335592.
frame=  424 fps= 13 q=0.0 q=0.0 size=  579564kB time=00:00:14.14 bitrate=335592.
frame=  429 fps= 13 q=0.0 q=0.0 size=  586398kB time=00:00:14.31 bitrate=335592.
frame=  436 fps= 13 q=0.0 q=0.0 size=  595966kB time=00:00:14.54 bitrate=335592.
frame=  440 fps= 13 q=0.0 q=0.0 size=  601434kB time=00:00:14.68 bitrate=335592.
frame=  445 fps= 13 q=0.0 q=0.0 size=  608268kB time=00:00:14.84 bitrate=335592.
frame=  450 fps= 13 q=0.0 Lq=0.0 size=  615110kB time=00:00:15.01 bitrate=335596
.4kbits/s
video:1537734kB audio:0kB subtitle:0 global headers:0kB muxing overhead -59.9989
53%

You can see the warning messages above which was:

Incompatible pixel format 'gbrp10le' for codec 'r10k', auto-selecting format 'rg
b48le'
[swscaler @ 00000000026e7fe0] full chroma interpolation for destination format '
rgb48le' not yet implemented

The r10kToFFV.framemd5 is NOT identical to FFVTor10k.framemd5

So obviously, there is an issue!

Thoughts?

[poisondeathray]

Bits per raw sample is 8?

No, it's >8

[ffv1 @ 0000000002830e40] bits_per_raw_sample > 8, forcing coder 1

-coder 1 is automatically enabled and required when bpc is >8 . FFV1 supports 8,9,10, 16 bpc

However, take notice the byte size is NOT the same. They are close, but not exactly the same! Does the 'framemd5' being identical prove they converted correctly? And perhaps I should not worry about the small difference?

The byte difference could be accounted for in different AVI container headers, metadata. If you do a raw YUV dump for each video (input and encoded), they should be identical .

-vcodec rawvideo output.yuv

Or another method of testing is using difference testing in a program that decodes FFV1 at the correct bit depth



Not sure about the 10bit RGB case, it might not be implemented properly yet. I'll take a closer look

[poisondeathray]

RE: 10bit RGB

It's because ffmpeg doesn't support native 10bit RGB implementation . It supports 16bit RGB, and 8bit RGB for encode/decode directly. The 10bit RGB handling is really a "10bit in 16bit" implementation as mentioned earlier, hence the "rgb48le(10 bpc)"

FFV1 only uses GBRP for RGB encoding (the lossless RGB transform method described earlier; I actually described it incorrectly, it's not really Y'CbCr, it's GBR where G' = (2*G+B+R)/4, B' = B-G, R' = R-G) . Eitherway, the point is FFV1 doesn't encode RGB directly.

So in order for this to work with FFV1, gbrp16le must be implemented for encoding (it hasn't been yet, only for Input , not Output) . Either that, or ffmpeg needs a native 10bit RGB support (rgb30le or rgb30be) - unlikely this will happen. Or FFV1 needs to implement implement native RGB encoding (very unlikely)

ffmpeg -pix_fmts

I.... gbrp16be 3 48
I.... gbrp16le 3 48

It's not a high priority because 10bit RGB is not used very often, but the gbrp16le for output should be implmented sometime in the future, not sure when

[JasonCA]

No, it's >8

Ok, so it's greater than 8 bit encoding. Good!

-coder 1 is automatically enabled and required when bpc is >8 . FFV1 supports 8,9,10, 16 bpc

According the FFV1 Wiki, "FFV1, which stands for "FF video codec 1", is a lossless intra-frame video codec." Codec 1 and coder 1....so it would seem it's in a lossless mode I'm happy with that!

The byte difference could be accounted for in different AVI container headers, metadata. If you do a raw YUV dump for each video (input and encoded), they should be identical .

-vcodec rawvideo output.yuv

The following is the same files that we have been discussing in the recent posts #16 above https://forum.videohelp.com/threads/361133-Lossless-%2810-Bit-RGB-444%29-and-%2810-Bit-...=1#post2290389.

To be clear:

• 10BitYUV422_210_Capture1.avi is the TRUE original 10 Bit YUV 4:2:2 (v210) file
• V210ToFFV.avi (the generated FFV1 lossless video) is the result of having taken 10BitYUV422_210_Capture1.avi and converting it to FFV1 lossless.
• FFVToV210.avi is the result of taking the created FFV1 lossless file (originally created from 10BitYUV422_210_Capture1.avi) and then converting it back to the original V210 file (10BitYUV422_210_Capture1.avi).

I'm using these files since V210ToFFV.framemd5 and FFVToV210.framemd5 match as reported in Post #16!

First we take the true original and output it to raw YUV:

ffmpeg -i 10BitYUV422_210_Capture1.avi -vcodec rawvideo 10BitYUV422_210_Capture1_RawVideo.yuv

Next, we take the lossless FFV1 file and convert that to raw YUV:

ffmpeg -i V210ToFFV.avi -vcodec rawvideo V210ToFFV_RawVideo.yuv

Next, we take the V210 file that was created by using the FFV1 lossless file and covert it to raw YUV:

ffmpeg -i FFVToV210.avi -vcodec rawvideo FFVToV210_RawVideo.yuv

Now we compare:

1. [V210ToFFV_RawVideo.yuv vs 10BitYUV422_210_Capture1_RawVideo.yuv] = They Match!
2. [V210ToFFV_RawVideo.yuv vs FFVToV210_RawVideo.yuv] = They DO NOT Match!
3. [10BitYUV422_210_Capture1_RawVideo.yuv vs FFVToV210_RawVideo.yuv] = They DO NOT Match!

Comments:

#1: Taking the lossless FFV1 raw YUV file and ORIGINAL YUV file....they match! So far so good!
#2: Taking the lossless FFV1 raw YUV and comparing it to lossless file that was converted back to the original V210 file, they DO NOT match. Hmmm, so I can't seem to re-create the original V210 file.
#3: Comparing the Origianl to the file that was created by converting the lossless FFV1 back to the original V210 file, it doesn't compare.

Therefore, it would seem that the lossless FFV1 file is storing the original YUV file correctly. However, for whatever reason, I can't create the original 10 bit YUV 4:2:2 v210 original from it.

What are your thoughts?

Or another method of testing is using difference testing in a program that decodes FFV1 at the correct bit depth

What such program is this? Is it command line driven like FFMPEG? I am willing to try anything to correctly verify this.

Not sure about the 10bit RGB case, it might not be implemented properly yet. I'll take a closer look

I'll respond to this in your next thread and after I look at it more too

[poisondeathray]

Therefore, it would seem that the lossless FFV1 file is storing the original YUV file correctly. However, for whatever reason, I can't create the original 10 bit YUV 4:2:2 v210 original from it.

What are your thoughts?

I just ran some quick tests and everything , in all cases seems to match here with external md5 testing for v210 versions, yuv versions

You cannot create the original v210/AVI exactly because of the container and metadata differences e.g. it might say "blackmagic" or "AJA" instead of "ffmpeg" or "lavf" in the metadata writing application . e.g you can see in mediainfo there will be differences in the metadata. The important thing is the decoded image will be identical , so it is "lossless"

What are you using to test if it matches or not ?

What such program is this? Is it command line driven like FFMPEG? I am willing to try anything to correctly verify this.

I had more in mind any program that decodes correctly for difference layers. Or you can test the v210 variants in a NLE, after effects etc... just do amplifed differences and any minute differences will show up right away

EDIT: yes, just confirmed they match as well for v210 vs. (v210=>ffv1=>v210) difference testing .

EDIT#2: yes, just confirmed with SSIM and PSNR testing they match as well

EDIT#3: yes, just confirmed with ffmpeg's own internal per frame md5 testing - matches as well (I normally avoid using internal tests, like a certain codec's own SSIM/PSNR measurements because they might be biased)


That's 4 different tests saying the same thing


BTW, I did lots of similar tests over a year ago as well for v210, which is why I was certain it worked for 10bit YUV in my first reply


Could it be that your original isn't really v210 ?

[poisondeathray]

There was a ticket for ffv1 md5 difference, but it looks like it wasn't not reproducible, maybe a hardware issue
https://trac.ffmpeg.org/ticket/2866

Make sure you're using ffv1 1.3 (get a newer ffmpeg binary)

If you still get different results , then upload a sample




I've seen certain lossless codecs like lagarith decode with glitches under certain circumstances that are not repeatable . Some people, on certain configuration seem to have problems with lagarith for whatever reason. I haven't seen or heard it with ffv1

[JasonCA]

If you still get different results , then upload a sample

Here's my version which I just recently download once you confirmed ffmpeg can do lossless compression:

ffmpeg version N-59275-g9b195dd
built on Dec 21 2013 22:06:20 with gcc 4.8.2 (GCC)

Downloaded from: http://ffmpeg.zeranoe.com/builds/

Latest Zeranoe FFmpeg Build Version: git-9b195dd (2013-12-22)

ffmpeg-20131222-git-9b195dd-win64-static.7z

*** Uploaded Sample ***: 10BitYUV422_v210_10sec.avi

This is a 10 second 10 Bit YUV 4:2:2 v210 sample which you can download here: https://www.dropbox.com/s/xnsnbtx9xy7qxwo/10BitYUV422_v210_10sec.avi

VERIFYING V210 -> FFV Lossless -> v210 (Comparing .framemd5's):

ffmpeg -i 10BitYUV422_v210_10sec.avi -pix_fmt yuv422p10le -vcodec ffv1 -c:a copy 10BitYUV422_v210_10sec_V210ToFFV.avi -f framemd5 -an 10BitYUV422_v210_10sec_V210ToFFV.framemd5
ffmpeg -i 10BitYUV422_v210_10sec_V210ToFFV.avi -pix_fmt yuv422p10le -vcodec v210 -c:a copy 10BitYUV422_v210_10sec_FFVTov210.avi -f framemd5 -an 10BitYUV422_v210_10sec_FFVToV210.framemd5

Results in:

10BitYUV422_v210_10sec_FFVToV210.framemd5 vs 10BitYUV422_v210_10sec_V210ToFFV.framemd5 = Match!

Raw YUV Outputs:

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec rawvideo 10BitYUV422_v210_10sec_RawVideo.yuv
ffmpeg -i 10BitYUV422_v210_10sec_V210ToFFV.avi -vcodec rawvideo 10BitYUV422_v210_10sec_V210ToFFV_RawVideo.yuv
ffmpeg -i 10BitYUV422_v210_10sec_FFVTov210.avi -vcodec rawvideo 10BitYUV422_v210_10sec_FFVTov210_RawVideo.yuv

The only files that match:

10BitYUV422_v210_10sec_V210ToFFV_RawVideo vs 10BitYUV422_v210_10sec_RawVideo.yuv = Match!

So the same situation as last posts. The other raw YUV files DO NOT compare.

[poisondeathray]

EDIT: just a sec, there is a difference in the 1st frame... Let me get back to you

[JasonCA]

EDIT: just a sec, there is a difference in the 1st frame... Let me get back to you

Ok

[poisondeathray]

I don't have too much time tonight I'll have a closer look later, but I think it has to do with your particular file.

I believe it's related to the glitch in the beginning. I can't reproduce it with other videos

eg. If you cut the 1st part off and use that as the new input file, everything works with the raw yuv check , even on "new" source=>ffv1=>v210=>yuv

(e.g stream copy from 1 second onwards -vcodec copy -ss 00:00:01)

[JasonCA]

I don't have too much time tonight I'll have a closer look later, but I think it has to do with your particular file.

I believe it's related to the glitch in the beginning. I can't reproduce it with other videos

eg. If you cut the 1st part off and use that as the new input file, everything works with the raw yuv check , even on "new" source=>ffv1=>v210=>yuv

(e.g stream copy from 1 second onwards -vcodec copy -ss 00:00:01)

Added -ss 00:00:01, but apparently I'm not doing something correctly:

ffmpeg -i 10BitYUV422_v210_10sec.avi -pix_fmt yuv422p10le -vcodec ffv1 -c:a copy 10BitYUV422_v210_10sec_V210ToFFV.avi -ss 00:00:01 -f framemd5 -an 10BitYUV422_v210_10sec_V210ToFFV.framemd5 
ffmpeg -i 10BitYUV422_v210_10sec_V210ToFFV.avi -pix_fmt yuv422p10le -vcodec v210 -c:a copy 10BitYUV422_v210_10sec_FFVTov210.avi -ss 00:00:01 -f framemd5 -an 10BitYUV422_v210_10sec_FFVToV210.framemd5 

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec rawvideo -ss 00:00:01 10BitYUV422_v210_10sec_RawVideo.yuv
ffmpeg -i 10BitYUV422_v210_10sec_V210ToFFV.avi -vcodec rawvideo 10BitYUV422_v210_10sec_V210ToFFV_RawVideo.yuv
ffmpeg -i 10BitYUV422_v210_10sec_FFVTov210.avi -vcodec rawvideo 10BitYUV422_v210_10sec_FFVTov210_RawVideo.yuv

[poisondeathray]

You only need -ss for the 1st video. Once it's cut, that's your "new" source

e.g

ffmpeg -i "10BitYUV422_v210_10sec.avi" -vcodec copy -an -ss 00:00:01 cutsource.avi
ffmpeg -i cutsource.avi -vcodec ffv1 cutsource_ffv1.avi
ffmpeg -i cutsource_ffv1.avi -vcodec v210 cutsource_ffv1_v210.avi

ffmpeg -i cutsource.avi -vcodec rawvideo cutsource.yuv
ffmpeg -i cutsource_ffv1.avi -vcodec rawvideo cutsource_ffv1.yuv
ffmpeg -i cutsource_ffv1_v210.avi -vcodec rawvideo cutsource_ffv1_v210.yuv

cutsource.avi is the "new" source video. It's missing the 1st second
cutsource_ffv1.avi is the ffv1 version of it
cutsource_ffv1_v210.avi is the ffv1 version, encoded to v210

The same prefixes, but .yuv are the raw video versions

I don't think it's just a coincidence that the 1st frame was the only frame that differed in hash . That observation was why I cut the 1st second off to do more testing. Something is up with your original sample. I would test other samples from the same batch, because this problem doesn't occur with other v210 videos

[JasonCA]

You only need -ss for the 1st video. Once it's cut, that's your "new" source

e.g

ffmpeg -i "10BitYUV422_v210_10sec.avi" -vcodec copy -an -ss 00:00:01 cutsource.avi
ffmpeg -i cutsource.avi -vcodec ffv1 cutsource_ffv1.avi
ffmpeg -i cutsource_ffv1.avi -vcodec v210 cutsource_ffv1_v210.avi

ffmpeg -i cutsource.avi -vcodec rawvideo cutsource.yuv
ffmpeg -i cutsource_ffv1.avi -vcodec rawvideo cutsource_ffv1.yuv
ffmpeg -i cutsource_ffv1_v210.avi -vcodec rawvideo cutsource_ffv1_v210.yuv

cutsource.avi is the "new" source video. It's missing the 1st second
cutsource_ffv1.avi is the ffv1 version of it
cutsource_ffv1_v210.avi is the ffv1 version, encoded to v210

The same prefixes, but .yuv are the raw video versions

I don't think it's just a coincidence that the 1st frame was the only frame that differed in hash . That observation was why I cut the 1st second off to do more testing. Something is up with your original sample. I would test other samples from the same batch, because this problem doesn't occur with other v210 videos

Here is my modification:

ffmpeg -i "10BitYUV422_v210_10sec.avi" -vcodec copy -an -ss 00:00:01 cutsource.avi -f framemd5 -an -ss 00:00:01 cutsource.framemd5
ffmpeg -i cutsource.avi -vcodec ffv1 cutsource_ffv1.avi -f framemd5 -an cutsource_ffv1.framemd5
ffmpeg -i cutsource_ffv1.avi -vcodec v210 cutsource_ffv1_v210.avi -f framemd5 -an cutsource_ffv1_v210.framemd5

ffmpeg -i cutsource.avi -vcodec rawvideo cutsource.yuv
ffmpeg -i cutsource_ffv1.avi -vcodec rawvideo cutsource_ffv1.yuv
ffmpeg -i cutsource_ffv1_v210.avi -vcodec rawvideo cutsource_ffv1_v210.yuv

I wanted the frame md5 files created too. Now all the "framemd5" files compare.

I will run this against other samples to see if this is the case, I just ran it against the original ones that I first began posting in my #16th post. From the #16th post, if I cut the first 1second off, it works! So perhaps there is just something wrong with the 1st frame or two.

How can I be sure that this is a correct copy of the original:

ffmpeg -i "10BitYUV422_v210_10sec.avi" -vcodec copy -an -ss 00:00:01  cutsource.avi -f framemd5 -an -ss 00:00:01 cutsource.framemd5

For all I know, on the copy in, it's doing something funny internally before it outputs the "cutsource.avi". We are assuming now that "cutsource.avi" (minus the 1 second of course) is authentic to the original "10BitYUV422_v210_10sec.avi". Safe to assume?

I'm assuming that the "-vcodec copy" means that trully it's just lopping off 1 second from the original file, and outputting it to "cutsource.avi" and creating the "framemd5" file at the same time. You would believe that to be the case?

Let's assume all my files are like this...the 1st frame or two is wrong or corrupt. Well, I don't mind cutting 1 second off the begging to have a way to verify that it's doing what it should. If that is all that's happening then "cutsource.avi" (minus 1 second) is exactly the same as the original "10BitYUV422_v210_10sec.avi"

I'll have to try this on other video's to see.

But I can't tell you how great this is! If this reduces them by 1/2 or even 1/4th then that's fantastic!

I'll continue to look and play around with this. You helped me greatly by finding a way forward

[poisondeathray]

I will run this against other samples to see if this is the case, I just ran it against the original ones that I first began posting in my #16th post. From the #16th post, if I cut the first 1second off, it works! So perhaps there is just something wrong with the 1st frame or two.

Yes, you have to do some more testing on that batch of files . The entire batch might have issues. What was the capture workflow, and what devices and/or software were used ?

How can I be sure that this is a correct copy of the original:

ffmpeg -i "10BitYUV422_v210_10sec.avi" -vcodec copy -an -ss 00:00:01  cutsource.avi -f framemd5 -an -ss 00:00:01 cutsource.framemd5

For all I know, on the copy in, it's doing something funny internally before it outputs the "cutsource.avi". We are assuming now that "cutsource.avi" (minus the 1 second of course) is authentic to the original "10BitYUV422_v210_10sec.avi". Safe to assume?

I'm assuming that the "-vcodec copy" means that trully it's just lopping off 1 second from the original file, and outputting it to "cutsource.avi" and creating the "framemd5" file at the same time. You would believe that to be the case?

Yes I think that's what's happening but you can check the framecount or split the operation into 2 steps

It should be the same stream copy wise as well - but to test that assumption, you can check the per frame md5 values with the cut vs. non cut on the offset frames (adjust it by 1 second or ~29.97 frames or 30000/1001 exactly) to see if anything funky is going on . Or test ssim/psnr on the same offset frames.

e.g frame 30-32 compared to frame 0-2 of the original (obviously you'd do a full check , but just for demonstration purposes...)

0,         30,         30,        1,  1399680, 7fbec1904b86f63ca209d21acb6e8005
0,         31,         31,        1,  1399680, 5d459834e8b69b4fa79df1444c70176a
0,         32,         32,        1,  1399680, 9e8e62c5dd4176ba30787d135f6f65a0

0,          0,          0,        1,  1399680, 7fbec1904b86f63ca209d21acb6e8005
0,          1,          1,        1,  1399680, 5d459834e8b69b4fa79df1444c70176a
0,          2,          2,        1,  1399680, 9e8e62c5dd4176ba30787d135f6f65a0

But I can't tell you how great this is! If this reduces them by 1/2 or even 1/4th then that's fantastic!

I'll continue to look and play around with this. You helped me greatly by finding a way forward

BTW don't expect that much compression on "normal" content. A static test pattern will compression more than normal film content, both intra frame and interframe. A more typical ratio is 40-60% of the size