Lossless (10 Bit RGB 444) and (10 Bit YUV 422) Compression Codec's

[JasonCA]

I will run this against other samples to see if this is the case, I just ran it against the original ones that I first began posting in my #16th post. From the #16th post, if I cut the first 1second off, it works! So perhaps there is just something wrong with the 1st frame or two.

Yes, you have to do some more testing on that batch of files . The entire batch might have issues. What was the capture workflow, and what devices and/or software were used ?

How can I be sure that this is a correct copy of the original:

ffmpeg -i "10BitYUV422_v210_10sec.avi" -vcodec copy -an -ss 00:00:01  cutsource.avi -f framemd5 -an -ss 00:00:01 cutsource.framemd5

For all I know, on the copy in, it's doing something funny internally before it outputs the "cutsource.avi". We are assuming now that "cutsource.avi" (minus the 1 second of course) is authentic to the original "10BitYUV422_v210_10sec.avi". Safe to assume?

I'm assuming that the "-vcodec copy" means that trully it's just lopping off 1 second from the original file, and outputting it to "cutsource.avi" and creating the "framemd5" file at the same time. You would believe that to be the case?

Yes I think that's what's happening but you can check the framecount or split the operation into 2 steps

It should be the same stream copy wise as well - but to test that assumption, you can check the per frame md5 values with the cut vs. non cut on the offset frames (adjust it by 1 second or ~29.97 frames or 30000/1001 exactly) to see if anything funky is going on . Or test ssim/psnr on the same offset frames.

e.g frame 30-32 compared to frame 0-2 of the original (obviously you'd do a full check , but just for demonstration purposes...)

0,         30,         30,        1,  1399680, 7fbec1904b86f63ca209d21acb6e8005
0,         31,         31,        1,  1399680, 5d459834e8b69b4fa79df1444c70176a
0,         32,         32,        1,  1399680, 9e8e62c5dd4176ba30787d135f6f65a0

0,          0,          0,        1,  1399680, 7fbec1904b86f63ca209d21acb6e8005
0,          1,          1,        1,  1399680, 5d459834e8b69b4fa79df1444c70176a
0,          2,          2,        1,  1399680, 9e8e62c5dd4176ba30787d135f6f65a0

But I can't tell you how great this is! If this reduces them by 1/2 or even 1/4th then that's fantastic!

I'll continue to look and play around with this. You helped me greatly by finding a way forward

BTW don't expect that much compression on "normal" content. A static test pattern will compression more than normal film content, both intra frame and interframe. A more typical ratio is 40-60% of the size

I'll continue to play with this:

It's interesting to note the following:

framemd5's match, but source does not (source.avi vs source1.avi)

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec copy -an source.avi -f framemd5 -an source.framemd5
ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec v210 -an source1.avi -f framemd5 -an source1.framemd5

framemd5's match, but source does not (cutsource.avi vs cutsource1.avi)

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec copy -an -ss 00:00:01 cutsource.avi -f framemd5 -an -ss 00:00:01 cutsource.framemd5
ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec v210 -an -ss 00:00:01 cutsource1.avi -f framemd5 -an -ss 00:00:01 cutsource1.framemd5

Not sure why that is. All the framemd5's compare and match, the source files do not. However, they ALL play fine and report they are v210 files.

[poisondeathray]

It's interesting to note the following:

framemd5's match, but source does not (source.avi vs source1.avi)

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec copy -an source.avi -f framemd5 -an source.framemd5
ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec v210 -an source1.avi -f framemd5 -an source1.framemd5

framemd5's match, but source does not (cutsource.avi vs cutsource1.avi)

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec copy -an -ss 00:00:01 cutsource.avi -f framemd5 -an -ss 00:00:01 cutsource.framemd5
ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec v210 -an -ss 00:00:01 cutsource1.avi -f framemd5 -an -ss 00:00:01 cutsource1.framemd5

Not sure why that is. All the framemd5's compare and match, the source files do not. However, they ALL play fine and report they are v210 files.

Be more specific when you say "source doesn't match" - do you mean file (global) md5 ?

FFmpeg's per frame md5 decodes each frame to raw video and checks. You can see that in the log. That's why the per-frame md5 check shows the videos to be identical. Recall there are header and metadata container differences. Open up each in a hex editor and you will see they are very slightly different - but only at the beginning where the metadata is. This has NO impact on the decoded data, hence the per-frame md5 is the same, and the SSIM /PSNR are the same per frame or per average. But a file (global) md5 comparison or hex editor comparison might show differences

Do a raw yuv dump then compare a file md5 for each and you will see they are the same . Raw YUV has no metadata or tags or anything. It's "naked" .

The only way to get EXACTLY the same file is with archiving software. This means all metadata will be preserved, all tags, etc...

[JasonCA]

Be more specific when you say "source doesn't match" - do you mean file (global) md5 ?

That's why I put (source.avi vs source1.avi), but obviously not clear enough

FFmpeg's per frame md5 decodes each frame to raw video and checks. You can see that in the log. That's why the per-frame md5 check shows the videos to be identical. Recall there are header and metadata container differences. Open up each in a hex editor and you will see they are very slightly different - but only at the beginning where the metadata is. This has NO impact on the decoded data, hence the per-frame md5 is the same, and the SSIM /PSNR are the same per frame or per average. But a file (global) md5 comparison or hex editor comparison might show differences

I'm not seeing any SSIM/PSNR anywhere even when adding the -report to create a report. I only have the following in my framemd5:

stream#, dts,        pts, duration,     size, hash

Do a raw yuv dump then compare a file md5 for each and you will see they are the same . Raw YUV has no metadata or tags or anything. It's "naked" .

Yes, they compare.

I was just expecting the following to be equal:

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec copy -an source.avi -f framemd5 -an source.framemd5

Meaning, source.avi is binary equivalent to 10BitYUV422_v210_10sec.avi, but they are not. Remember, we are essentially making a COPY of the original source. So, it's matter of proving source.avi is TRULLY authentic to the original that it was COPIED from. However, like we have been seeing, the raw YUV files are equal. So, I'm already pretty convinced and happy. It looks to me that they are equal. You've help to confirm they are equal and help to convince me it is so too. So, I'm excited!

The only way to get EXACTLY the same file is with archiving software. This means all metadata will be preserved, all tags, etc...

Well that's the other question.... what metadata am I really losing? For instance, I know there's sometimes closed captioning embedded in the file. I wonder if that's maintained? Not that it matters for me....but it's just a question of what I am really loosing in the lossless FFV1 compression.

Yes, archiving software would make it for sure the same. But, in just general testing, I see it to be quite slow and not as compact as this.

So, I'm interested in using this FFV1 lossles compression, but just want to PROVE to myself what, if anything, I am loosing. If the video is exactly the same....that's all that really matters. As far as Metadata, I'm not sure how do know what metadata I'd be losing.

The nice thing also about using FFV1, is that all the files are still viewable. Using archival software...that's out of the question.

So, I am weighing the pro's and con's here.

Once fully confident that the quality output of video is lossless, I'll move on to conversion times.

I have to admit, I'm pretty excited by FFV1 from what I see so far.

[JasonCA]

One other thing I may try to do is compare the quality if possible between the two.

For example, I believe AviSynth has a way to compare two files.

Something like:

v1=AviSource("c:\original.avi")
v2=AviSource("c:\copy.avi")
Compare(v1, v2, "YUV","c:\compare.log")

Although, I also thought AviSynth only did 8bit. But, I could be wrong...I've mostly done 8bit work with AviSynth.

Something for me to try later.

Once I convert these files to lossless...that's it....I am deleting the originals since they are HUGE. So, I just want to make sure I do it right to where I don't kick myself in the future for not verifying things first.

The other issue is the need to find an alternative to handle the 10 Bit RGB r10k files.

But, I am very excited.

[poisondeathray]

Be more specific when you say "source doesn't match" - do you mean file (global) md5 ?

That's why I put (source.avi vs source1.avi), but obviously not clear enough

I meant in what manner. There are several different methods of comparing. You were clear that they matched with per-frame md5. Not so clear about how they differed

I'm not seeing any SSIM/PSNR anywhere even when adding the -report to create a report. I only have the following in my framemd5:

I mean using an external program. Remember I checked with a bunch of different methods? Unfortunately the free version of msu vqmt is limited to 8bit as well, you need the pro version

Like you, I'm paranoid as well .... at least for things that are important. So I did a battery of tests over a year ago to convince myself as well. I hate using the same tool for checking (bias)

I was just expecting the following to be equal:

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec copy -an source.avi -f framemd5 -an source.framemd5

Meaning, source.avi is binary equivalent to 10BitYUV422_v210_10sec.avi, but they are not. Remember, we are essentially making a COPY of the original source. So, it's matter of proving source.avi is TRULLY authentic to the original that it was COPIED from. However, like we have been seeing, the raw YUV files are equal. So, I'm already pretty convinced and happy. It looks to me that they are equal. You've help to confirm they are equal and help to convince me it is so too. So, I'm excited!

Again, it shouldn't be, nor should you expect a binary comparison to be equal . Even look at mediainfo (view=>text) . It will show "encoded by" using lavf or ffmpeg. An original capture won't show that. That alone will change any global checksum or file comparison

The only way to get EXACTLY the same file is with archiving software. This means all metadata will be preserved, all tags, etc...

Well that's the other question.... what metadata am I really losing? For instance, I know there's sometimes closed captioning embedded in the file. I wonder if that's maintained? Not that it matters for me....but it's just a question of what I am really loosing in the lossless FFV1 compression.

CC is usually not maintained, and definitely not maintained when you use -vcodec something (other than copy). You usually lose everything else.

It's probably not important for this project, but professional formats often have timecode data (times, dates), exif data, user data (scene metatdata), lens data , including CA correction data , etc... Sometimes they are embedded as metadata, sometimes as another stream, sometimes it's within another folder in the directory structure

"Lossless" video compression only means video . The video is decompressed before it's re-compressed. Any accessory streams, any container metadata is often lost. (But there are ways to copy some of the meta data, even with ffmpeg)

I also thought AviSynth only did 8bit. But, I could be wrong...I've mostly done 8bit work with AviSynth.

Yes, 8bit only . But it can be used to detect lossless vs. not lossless because the truncation from 10=>8bits is done in the same manner .

The other issue is the need to find an alternative to handle the 10 Bit RGB r10k files.

Not sure. FFV1 won't be truly lossless for that it until at least 16bit GBRP gets implemented into ffmpeg

You mentioned sheervideo in the 1st post. I tested that about 3-4 years ago. It had problems on the PC . I don't know if it's changed but it's more geared towards macs, and MOV wrapped . If you test it and it works out for you, can you summarize your findings ?

[JasonCA]

Earlier, you were talking about PSNR and SSIM, but I wasn't seeing the PSNR show anywhere before. Now, I figured out how to get the PSRN with ffmpeg to show which is helpful. But, I just want to make sure I am reading this correctly ...

ffmpeg -i 10BitYUV422_v210_10sec.avi -pix_fmt yuv422p10le -vcodec ffv1 10BitYUV422_v210_10sec_ffv1.avi -vstats -psnr

Results in:

ffmpeg version N-59275-g9b195dd Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec 21 2013 22:06:20 with gcc 4.8.2 (GCC)
  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-av
isynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnutls --enab
le-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetyp
e --enable-libgsm --enable-libilbc --enable-libmodplug --enable-libmp3lame --ena
ble-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-l
ibopus --enable-librtmp --enable-libschroedinger --enable-libsoxr --enable-libsp
eex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aa
cenc --enable-libvo-amrwbenc --enable-libvorbis --enable-libvpx --enable-libwavp
ack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 58.101 / 52. 58.101
  libavcodec     55. 45.103 / 55. 45.103
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     4.  0.100 /  4.  0.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, avi, from '10BitYUV422_v210_10sec.avi':
  Duration: 00:00:10.01, start: 0.000000, bitrate: 223817 kb/s
    Stream #0:0: Video: v210 (v210 / 0x30313276), yuv422p10le, 720x486, 29.97 tb
r, 29.97 tbn, 29.97 tbc
[ffv1 @ 0000000000301240] bits_per_raw_sample > 8, forcing coder 1
Output #0, avi, to '10BitYUV422_v210_10sec_ffv1.avi':
  Metadata:
    ISFT            : Lavf55.22.100
    Stream #0:0: Video: ffv1 (FFV1 / 0x31564646), yuv422p10le, 720x486, q=2-31,
200 kb/s, 29.97 tbn, 29.97 tbc
Stream mapping:
  Stream #0:0 -> #0:0 (v210 -> ffv1)
Press [q] to stop, [?] for help
frame=   21 fps=0.0 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=    4885kB time=00:0
frame=   43 fps= 42 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=    9478kB time=00:0
frame=   65 fps= 43 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   14065kB time=00:0
frame=   88 fps= 43 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   18857kB time=00:0
frame=  111 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   23648kB time=00:0
frame=  134 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   28438kB time=00:0
frame=  157 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   33235kB time=00:0
frame=  180 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   38037kB time=00:0
frame=  203 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   42852kB time=00:0
frame=  226 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   47659kB time=00:0
frame=  249 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   52469kB time=00:0
frame=  272 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   57268kB time=00:0
frame=  295 fps= 44 q=0.0 PSNR=Y:inf U:inf V:inf *:inf size=   62056kB time=00:0
frame=  300 fps= 44 q=0.0 LPSNR=Y:inf U:inf V:inf *:inf size=   63098kB time=00:
00:10.01 bitrate=51638.5kbits/s
video:63086kB audio:0kB subtitle:0 global headers:0kB muxing overhead 0.020122%

Just so I know I am looking at this correctly, "PSNR=Y:inf U:inf V:inf *:inf " has the value 'inf' which I take means it's FULL quality in all the Y, U, and V... and therefore lossless. Otherwise, I would expect a value that varies I would imagine.

And the vstat file looks like:

frame=     1 q= 0.0 PSNR=    inf f_size= 724445 s_size=      707kB time= 0.033 br= 173693.1kbits/s avg_br= 173693.1kbits/s type= I
frame=     2 q= 0.0 PSNR=    inf f_size= 213149 s_size=      916kB time= 0.067 br= 51104.7kbits/s avg_br= 112398.9kbits/s type= I
frame=     3 q= 0.0 PSNR=    inf f_size= 212960 s_size=     1124kB time= 0.100 br= 51059.3kbits/s avg_br= 91952.4kbits/s type= I

< excess omitted >

frame=   298 q= 0.0 PSNR=    inf f_size= 212257 s_size=    62671kB time= 9.943 br= 50890.8kbits/s avg_br= 51632.7kbits/s type= I
frame=   299 q= 0.0 PSNR=    inf f_size= 212208 s_size=    62878kB time= 9.977 br= 50879.0kbits/s avg_br= 51630.1kbits/s type= I
frame=   300 q= 0.0 PSNR=    inf f_size= 212870 s_size=    63086kB time= 10.010 br= 51037.8kbits/s avg_br= 51628.2kbits/s type= I

Here, "PSNR= inf" is the overall PSNR for each frame. Again, this is reporting lossless too...well...if I am understanding this correctly.

I wasn't able to see SSIM anywhere though. My understanding is SSIM is better measure of quality in the overall frames.

Still haven't had a chance to try the AviSynth compare to see what the output there shows for fun.

[JasonCA]

I meant in what manner. There are several different methods of comparing. You were clear that they matched with per-frame md5. Not so clear about how they differed

Yes, I realize that. In trying to convey my thoughts to you, I missed out on some detail. That happens now and then It wasn't intentional. I try to provide all the information...but I'm bound to make mistakes somewhere.

I mean using an external program. Remember I checked with a bunch of different methods? Unfortunately the free version of msu vqmt is limited to 8bit as well, you need the pro version

Oh I see now, you were using another program!

You are right, I was just looking at that too! You are talking about the MSU Video Quality Measurement Tool. Yes, because we are dealing with 10 bit, we would need the pro version to check the quality. And to do that, it's like $1000 (ouch)! If I were a company, then sure. But, for my own personal use, that's a bit high! lol

Although, I should try it too since it does 8bit.

Like you, I'm paranoid as well .... at least for things that are important. So I did a battery of tests over a year ago to convince myself as well. I hate using the same tool for checking (bias)

Yes, you would think I'm being a bit anal, but because i am serious about really using this for archiving, I want to make sure I know that it's working correctly (knowing I said this already). You had all this work to get this stuff transferred, the last thing I want to do is convert the uncompressed files to FFV1 and then later down the road find out that, for example, it was converted from 10 bit and stored compressed as 8 bit. Then it's just irreversible!

But to your point, yes, so you understand my somewhat frantic state since you've been there yourself running such tests in years past! lol.

So FFV1 and MSU are interrelated?

Again, it shouldn't be, nor should you expect a binary comparison to be equal . Even look at mediainfo (view=>text) . It will show "encoded by" using lavf or ffmpeg. An original capture won't show that. That alone will change any global checksum or file comparison

It would make it easier if you could do a binary compare! But yes, I know what you are saying, there is no way that you could considering the mediainfo embedded would be different. As you said, lavf vs ffmpeg...or otherwise.

CC is usually not maintained, and definitely not maintained when you use -vcodec something (other than copy). You usually lose everything else.

Thankfully, in my case there is no CC or even audio...just video. So that makes it easier to decide in using this as means for archive. It's why I've not said anything about audio too...it's not been a concern at least for me. Otherwise, i'd be concerned that I would storing the audio correctly too.

It's probably not important for this project, but professional formats often have timecode data (times, dates), exif data, user data (scene metatdata), lens data , including CA correction data , etc... Sometimes they are embedded as metadata, sometimes as another stream, sometimes it's within another folder in the directory structure

"Lossless" video compression only means video . The video is decompressed before it's re-compressed. Any accessory streams, any container metadata is often lost. (But there are ways to copy some of the meta data, even with ffmpeg)

Not not really important for me. Although, I wish there was a nice way to know what was really in the file. Like, how would you know there is CC or even other stuff? Hopefully some application would give you a glimpse....but, I don't know of a 'goto' type of app for just examining the metadata to see what's really inside.

Yes, 8bit only . But it can be used to detect lossless vs. not lossless because the truncation from 10=>8bits is done in the same manner .

I like AviSynth. Hopefully in the future they'll expand it to be beyond 8bit only. But, great, that's what I'm intrested in running the video through the AviSynth 'compare()' function to see what that outputs too. I'll soon attempt that. Although, if I am understanding it correctly, ffmpeg is showing PSNR = inf...which I think is good!

Not sure. FFV1 won't be truly lossless for that it until at least 16bit GBRP gets implemented into ffmpeg

Which doesn't help for storing 10bit . I'd be storing 10bit in 16bit lossless.... and break even perhaps in file size? haha But this means ffmpeg will not help me with storing the r10k in a lossless format. So, I'll have to find an alternative for those.

You mentioned sheervideo in the 1st post. I tested that about 3-4 years ago. It had problems on the PC . I don't know if it's changed but it's more geared towards macs, and MOV wrapped . If you test it and it works out for you, can you summarize your findings ?

I've not attempted SheerVideo yet since you've confirmed that ffmpeg can do lossless v210. Since then, I've been here investigating ffmpeg for the first time . I owe my thanks to you!

But, I've also not attempted SheerVideo because you need an application to use it. And yes, it's wrapped in a MOV container from what I've read too. I can't, for example use SheerVideo from the command prompt like you can with ffmpeg. Command prompt is what makes ffmpeg so powerful and simple at the same time.

The other problem is that most NLE's convert to RGB...so then there's the issue of knowing if your video is being converted before it even makes it to the SheerVideo codec.

SheerVideo has a trial, so I may still play around with it. If so, I'll be sure to post whatever I find

[poisondeathray]

Yes, you would think I'm being a bit anal, but because i am serious about really using this for archiving, I want to make sure I know that it's working correctly (knowing I said this already). You had all this work to get this stuff transferred, the last thing I want to do is convert the uncompressed files to FFV1 and then later down the road find out that, for example, it was converted from 10 bit and stored compressed as 8 bit. Then it's just irreversible!

I was certain it worked for 10bit 422 YCbCr (and after the last few tests, I'm still certain) . But that is also why I raised the option of using archiving software if you want completely reversible (everything, including metatdata preservation)

So FFV1 and MSU are interrelated?

Not related.

FFV1 is a FFMpeg project, MSU = Moscow State University

Not not really important for me. Although, I wish there was a nice way to know what was really in the file. Like, how would you know there is CC or even other stuff? Hopefully some application would give you a glimpse....but, I don't know of a 'goto' type of app for just examining the metadata to see what's really inside.

mediainfo (view=>text) will give you quite a bit of info. It does miss some stream types, and makes a few mistakes, but overall it's probably the "best"

ffmpeg -i , will give you some data as well

Also (not for film scans) but for some other formats, sometimes the data is held in other accessory files (not in the same container)

But, I've also not attempted SheerVideo because you need an application to use it. And yes, it's wrapped in a MOV container from what I've read too. I can't, for example use SheerVideo from the command prompt like you can with ffmpeg. Command prompt is what makes ffmpeg so powerful and simple at the same time.

Yes, it was done through the quicktime API before. Basically any application that can encoded "official" quicktime had access to it , incl. quicktime pro

You can ask if they have command line access

The other problem is that most NLE's convert to RGB...so then there's the issue of knowing if your video is being converted before it even makes it to the SheerVideo codec.

Yes, the problem with any program is handling and potential compatibility issues. I mentioned it earlier - some truncate 10bit FFV1 to 8bits , others can't even read it. Some apply unwanted colorspace conversions.


Another option is 10bit x264 with lossless settings. It does 10bit YCbCr 422 correctly, it does 8bit RGB correctly (with the same GBRP method) - but last time I checked the 10bit RGB lossess implementation had problems

[JasonCA]

You can ask if they have command line access

Yes, I will ask them for that

Yes, the problem with any program is handling and potential compatibility issues. I mentioned it earlier - some truncate 10bit FFV1 to 8bits , others can't even read it. Some apply unwanted colorspace conversions.

Right. Hard to tell what's happening in a NLE as you said. That's why I like ffmpeg for this.

Another option is 10bit x264 with lossless settings. It does 10bit YCbCr 422 correctly, it does 8bit RGB correctly (with the same GBRP method) - but last time I checked the 10bit RGB lossess implementation had problems

Please tell me more about this. ffmpeg does lossless 10bit YCbCr 422 through x264? Or what are you refering to that does this? I'll have to try that...hopefully it honors the same resolution size.

By the way, I tried the MediaInfo for the files. Once I convert the video to FFV1 and then back to v210, if you look, it is very similar to the ORIGINAL Pretty nice .

[poisondeathray]

Please tell me more about this. ffmpeg does lossless 10bit YCbCr 422 through x264? Or what are you refering to that does this? I'll have to try that...hopefully it honors the same resolution size.

No, x264.exe separately (it's a command line AVC encoder), with a compiled binary with 10bit support

ffmpeg has -vcodec libx264, but it usually isn't compiled with 10bit support (unless you do it yourself)

[JasonCA]

No, x264.exe separately (it's a command line AVC encoder), with a compiled binary with 10bit support

ffmpeg has -vcodec libx264, but it usually isn't compiled with 10bit support (unless you do it yourself)

Would you happen to know the syntax to do lossless with x264.exe? For example, using the '10BitYUV422_v210_10sec.avi' as an example? I'll give that a try too and see what the difference in size and time is between FFV1.

[poisondeathray]

No, x264.exe separately (it's a command line AVC encoder), with a compiled binary with 10bit support

ffmpeg has -vcodec libx264, but it usually isn't compiled with 10bit support (unless you do it yourself)

Would you happen to know the syntax to do lossless with x264.exe? For example, using the '10BitYUV422_v210_10sec.avi' as an example? I'll give that a try too and see what the difference in size and time is between FFV1.

You should be aware that x264 has a gazillion switches that can affect speed vs. compression, even in lossless mode . You would use a different setting for different scenarios . eg. for editing you would probably use intra (all I-frames), with low latency settings, but this means less compressed. For archiving / pure compression you might use slow compression settings with a large GOP interval
http://mewiki.project357.com/wiki/X264_Settings

In short, there is a range of speed vs. compression settings that you an use. It might be 50x slower or 10x faster depending on the settings you use. But I would start with using the --preset values for ease of use

You need a 10bit support x264 binary . You can find some here

http://komisar.gin.by/

eg

x264_10 --demuxer lavf --profile high444 --qp 0 --input-depth 10 --input-csp i422 --output-csp i422 -o x264_10_output.mkv v210_input.avi

[JasonCA]

You should be aware that x264 has a gazillion switches that can affect speed vs. compression, even in lossless mode . You would use a different setting for different scenarios . eg. for editing you would probably use intra (all I-frames), with low latency settings, but this means less compressed. For archiving / pure compression you might use slow compression settings with a large GOP interval
http://mewiki.project357.com/wiki/X264_Settings

Is this all for lossless? Lossless, but different amount of compression which may have a speed impact?

In short, there is a range of speed vs. compression settings that you an use. It might be 50x slower or 10x faster depending on the settings you use. But I would start with using the --preset values for ease of use

Good to know, I'll have to play around with it and post my findings.

You need a 10bit support x264 binary . You can find some here

http://komisar.gin.by/

What about VideoLan's x264: http://www.videolan.org/developers/x264.html

Are there different flavors? Why are there so many flavors?

There are some here too: http://www.x264.nl/x264_main.php

You may be intrested in the PDF's. I'll probably take a look at those...seems like an intresting read.

eg

x264_10 --demuxer lavf --profile high444 --qp 0 --input-depth 10 --input-csp i422 --output-csp i422 -o x264_10_output.mkv v210_input.avi

Is this a lossless compression example?

[poisondeathray]

Yes, anytime --crf or --qp = 0 that's lossless mode. The higher the value, the higher the quantizer, thus the lower the quality. At zero, it's lossless (if used in the same colorspace and bitdepth).

Yes, there are many different x264 "flavors" ; some binaries are compiled with other filters, other decoders, other functions like audio, or patches e.g. a film grain optimization patch

The same can be said for ffmpeg or ffmbc, there are different versions that can have different things compiled

[JasonCA]

Yes, anytime --crf or --qp = 0 that's lossless mode. The higher the value, the higher the quantizer, thus the lower the quality. At zero, it's lossless (if used in the same colorspace and bitdepth).

Yes, there are many different x264 "flavors" ; some binaries are compiled with other filters, other decoders, other functions like audio, or patches e.g. a film grain optimization patch

The same can be said for ffmpeg or ffmbc, there are different versions that can have different things compiled

Thanks poisondeathray, this seems to work:

x264-10b-r2345-f0c1c53.exe --demuxer lavf --profile high444 --qp 0 --input-depth 10 --input-csp i422 --output-csp i422 -o x264_10_output.mkv 10BitYUV422_v210_10sec.avi

Excellent! It is about 2mb smaller than FFV1 it would seem.

And how do I get it back to v210 uncompressed?

[poisondeathray]

You can use ffmpeg to convert it to v210 . x264 doesn't encode to other video formats

That example command line was using "--preset medium" settings, since no preset was specified. You can try for slighter higher compression, but drastically slower encoding speed with something like "--preset veryslow"

In 8bit lossless YUV, usually ffv1 offers better lossless compression on most types of content than 8bit x264 in lossless mode (not that you care about 8bit right now, I'm just throwing it out there)

Also keep in mind a static test pattern will compress VERY differently that "normal" content.

[JasonCA]

You can use ffmpeg to convert it to v210 . x264 doesn't encode to other video formats

That example command line was using "--preset medium" settings, since no preset was specified. You can try for slighter higher compression, but drastically slower encoding speed with something like "--preset veryslow"

In 8bit lossless YUV, usually ffv1 offers better lossless compression on most types of content than 8bit x264 in lossless mode (not that you care about 8bit right now, I'm just throwing it out there)

Also keep in mind a static test pattern will compress VERY differently that "normal" content.

I will keep that in mind that a static test pattern will compress differently. Right now, I just want to get the method down. I'll then fully try to encode some of them to see what the differences are. But, that will probably take a few hours as you can imagine since they are huge!

I tried this:

ffmpeg -i 10BitYUV422_v210_10sec.avi -f framemd5 -an source.framemd5 -vstats -psnr
x264-10b-r2345-f0c1c53.exe --demuxer lavf --profile high444 --qp 0 --input-depth 10 --input-csp i422 --output-csp i422 --preset veryslow -o x264_10_output2.mkv 10BitYUV422_v210_10sec.avi
ffmpeg -i x264_10_output2.mkv -vcodec v210 source_x264_v210.avi -f framemd5 -an source_x264_v210.framemd5 -vstats -psnr

The resulting source.framemd5 vs source_x264_v210.framemd5 do not match for any of the hashes between the two framemd5's. I would have thought they would on a per frame basis.

[poisondeathray]

Did you remember to cut off the first frame or are you still using the damaged/problematic sample ?

hashes match here on other videos . (I deleted yours and don't feel like downloading it again)

[JasonCA]

Did you remember to cut off the first frame or are you still using the damaged/problematic sample ?

hashes match here on other videos . (I deleted yours and don't feel like downloading it again)

Using the following version of x264.exe:

x264 0.135.2345 f0c1c53
(libswscale 2.1.2)
(libavformat 55.2.0)
built on Aug  3 2013, gcc: 4.6.3
configuration: --bit-depth=10 --chroma-format=all
x264 license: GPL version 2 or later
libswscale/libavformat license: LGPL version 2.1 or later

Which I downloaded here: http://download.videolan.org/pub/x264/binaries/win64/x264-10b-r2345-f0c1c53.exe


Cut off the 1st 5 seconds

ffmpeg -i 10BitYUV422_v210_10sec.avi -vcodec copy -pix_fmt yuv422p10le -an -ss 00:00:05 cutsource.avi -f framemd5 -an -ss 00:00:05 cutsource.framemd5 -vstats -psnr

Create Lossless x264 file (v210 -> mp4)

x264-10b-r2345-f0c1c53.exe --demuxer lavf --profile high444 --qp 0 --input-depth 10 --input-csp i422 --output-csp i422 --preset veryslow -o cutsource_x264.mkv cutsource.avi

Create Lossless x264 file (mp4 -> v210)

ffmpeg -i cutsource_x264.mkv -vcodec v210 -pix_fmt yuv422p10le cutsource_x264_v210.avi -f framemd5 -an cutsource_x264_v210.framemd5 -vstats -psnr

Dump Raw YUV

ffmpeg -i cutsource.avi -vcodec rawvideo cutsource.yuv
ffmpeg -i cutsource_x264.mkv -vcodec rawvideo cutsource_x264.yuv
ffmpeg -i cutsource_x264_v210.avi -vcodec rawvideo cutsource_x264_v210.yuv

Like what we did with FFV1, I expect the following:

• All framemd5's are equal (in this case with x264, none are equal)
• If they framemd5's where equal, like they were for FFV1, then I'd expect all the YUV files to be binarily equal as they where with FFV1
• I don't expect the cutsource.avi to be equal to cutsource_x264_v210.avi (as they were not equal for FFV1 either). It's why we were relying on the YUV's to be equal since as we understood the YUV files to be the "naked" outputs of it

So, using x264, the framemd5's did not match. So of course the YUV's wouldn't match either (and they did not match).

Yes, I tried this by just cutting off 1 second too.... same thing... they didn't match.

After you questioned if I took off the 1st second, I took the time (plus ) and went back to verify everything we did with FFV1 and don't believe i made a mistake .

[poisondeathray]

They all match here on different videos, as does the global binary md5 check for YUV dump. If NONE of the per frame md5's, that means something is completely wrong, or maybe frames are offset

Post your x264 log, maybe it hasn't been compiled with 10bit support properly and has done something else to it (maybe lavf decoder wasn't feed 10bit properly)

Or try a different x264 binary, like the one I linked to which is known to work properly

[JasonCA]

They all match here on different videos, as does the global binary md5 check for YUV dump. If NONE of the per frame md5's, that means something is completely wrong, or maybe frames are offset

Post your x264 log, maybe it hasn't been compiled with 10bit support properly and has done something else to it (maybe lavf decoder wasn't feed 10bit properly)

Or try a different x264 binary, like the one I linked to which is known to work properly

Very Interesting!

So, I replaced VideoLan.org's x264 with this version that you suggested: http://komisar.gin.by/old/2377/x264.2377.10bit.x86_64.exe

Re-ran the last test, and it worked fine! All the framemd5's and YUV's compared!

It must mean we are going to have a good 2014!!!

Thank you poisondeathray !

Avoid VideoLang.org's x264!

[poisondeathray]

Can you post the log file anyway? That will help track down what went wrong . Maybe help them to address the issue

[JasonCA]

Can you post the log file anyway? That will help track down what went wrong . Maybe help them to address the issue

As you have requested !

Using the good one:

x264.2377.10bit.x86_64.exe --demuxer lavf --profile
high444 --qp 0 --input-depth 10 --input-csp i422 --output-csp i422 --preset very
slow -o cutsource_x264.mkv cutsource.avi
lavf [info]: 720x486p 0:1 @ 0/0 fps (vfr)
resize [warning]: converting from yuv422p10le to yuv422p16le
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
x264 [info]: profile High 4:4:4 Predictive, level 4.0, 4:2:2 10-bit
x264 [info]: frame I:2     Avg QP: 0.00  size:204824
x264 [info]: frame P:148   Avg QP: 0.00  size:192764
x264 [info]: mb I  I16..4: 46.2% 24.3% 29.5%
x264 [info]: mb P  I16..4:  2.6% 63.3%  3.3%  P16..4:  8.3%  8.4% 14.2%  0.0%  0
.1%    skip: 0.0%
x264 [info]: 8x8 transform intra:90.3% inter:87.6%
x264 [info]: coded y,uvDC,uvAC intra: 98.4% 89.1% 89.1% inter: 98.5% 95.8% 95.8%

x264 [info]: i16 v,h,dc,p:  4%  0% 87%  8%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  5%  3% 66%  6%  3%  3%  3%  5%  6%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 15%  3% 60%  5%  3%  3%  2%  4%  4%
x264 [info]: i8c dc,h,v,p:  6% 91%  3%  0%
x264 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x264 [info]: ref P L0: 11.9% 21.8%  2.8% 12.8%  1.5%  9.6%  1.3%  6.8%  1.0%  5.
9%  1.1%  5.7%  1.6%  7.6%  2.1%  6.5%
x264 [info]: kb/s:46255.72

encoded 150 frames, 6.46 fps, 46256.62 kb/s

The problematic VideoLan.org's version :

x264-10b-r2345-f0c1c53.exe --demuxer lavf --profile
high444 --qp 0 --input-depth 10 --input-csp i422 --output-csp i422 --preset very
slow -o cutsource_x264.mkv cutsource.avi
lavf [info]: 720x486p 0:1 @ 30000/1001 fps (vfr)
resize [warning]: converting from yuv422p10le to yuv422p16le
x264 [info]: using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2
x264 [info]: profile High 4:4:4 Predictive, level 4.0, 4:2:2 10-bit
x264 [info]: frame I:2     Avg QP: 0.00  size:220425
x264 [info]: frame P:148   Avg QP: 0.00  size:205579
x264 [info]: mb I  I16..4: 30.3% 32.2% 37.5%
x264 [info]: mb P  I16..4:  0.9% 54.1%  2.4%  P16..4: 12.5% 12.1% 18.0%  0.0%  0
.0%    skip: 0.0%
x264 [info]: 8x8 transform intra:92.9% inter:92.6%
x264 [info]: coded y,uvDC,uvAC intra: 98.8% 93.8% 93.8% inter: 97.9% 94.7% 94.6%

x264 [info]: i16 v,h,dc,p:  8%  0% 85%  6%
x264 [info]: i8 v,h,dc,ddl,ddr,vr,hd,vl,hu:  6%  1% 66%  6%  3%  3%  3%  5%  6%
x264 [info]: i4 v,h,dc,ddl,ddr,vr,hd,vl,hu: 17%  3% 59%  5%  3%  3%  2%  3%  4%
x264 [info]: i8c dc,h,v,p:  4% 81% 14%  0%
x264 [info]: Weighted P-Frames: Y:0.0% UV:0.0%
x264 [info]: ref P L0: 14.0% 19.7%  4.0% 11.6%  2.2%  8.7%  1.9%  6.2%  1.4%  5.
5%  1.5%  5.6%  2.2%  7.0%  2.7%  5.9%
x264 [info]: kb/s:49337.14

encoded 150 frames, 5.98 fps, 49338.03 kb/s

[poisondeathray]

Observations RE: videolan's 10bit x264 version:
1) If you send v210 through ffmpeg using a raw pipe to videolan's x264 version , it works ok , md5's check (thereby bypassing --demuxer lavf)
2) If you use raw YUV video input, it also works ok (also bypassing --demuxer lavf)

So the encoder portion appears to work correctly, it's that specific compiled version of lavf demuxer 's handling of v210 input, or that version of x264's handling of lavf input that has a problem . I checked that version's --fullhelp in case some switches were slightly different , and tested a few different combos , but direct v210 input seems b0rked . (It's not completely screwed up as in no picture or green/black frame - the decoded picture looks fine to the naked eye, but it's just not lossless)

I would just stick with komisar's build's. They are very stable and trusted, I've used them for years (Even though videolan's shoud be considered the "official" one)

[poisondeathray]

Oh.. and BTW, even on 10bit "normal" content, ffv1 was ~3% larger for the 10bit422 cases on several samples than x264 10bit using the default medium preset,. The compression was usually between 40-50% of the original v210 size

[JasonCA]

I've been spending time testing this all the way through....

If you recall, I create framemd5's.

• Create a framemd5 only on the source
• Create a framemd5 for the FFV1 when going from v210 -> FFV1 (you can see this in prior posts)
• Create a framemd5 going from FFV1 -> v210.

Going from v210->FFV1 the framemd5's are matching. That's great!

However, going from FFV1 -> v210 which is back to the original, I'm getting corrupt frames now and then on the decompress. How did I know there was corruption? I compared the FFV1->v210 framemd5 to either the original source (framemd5) or the v210->FFV1 framemd5 and they didn't match. Checked the frames and saw corruption!

Here's an example of a bad frame:



So:

• Original v210 -> FFV1 framemd5's match between the two. So the FFV1 lossless should be correct.
• Lossless FFV1 -> v210 the framemd5's MOSTLY match, but now and then I get a bad frame or 5 in a row.

Right now, I'm decompressing again to see if the blip happens to be in the same spot.



Glad I've not deleted my original's yet!

[poisondeathray]

• Original v210 -> FFV1 framemd5's match between the two. So the FFV1 lossless should be correct.
• Lossless FFV1 -> v210 the framemd5's MOSTLY match, but now and then I get a bad frame or 5 in a row.

Right now, I'm decompressing again to see if the blip happens to be in the same spot.

Any new info ? Is it random or repeatable ?

That screenshot is remeniscent of reported "random" lagarith glitches that I mentioned earlier. There was speculation that it was due to floating point calculations, but that was debunked by the author. Basically nobody knows why it happens for the lagarith case

Those observations don't make sense, because FFV1's framemd5's are decoded and converted to raw YUV in order for them to be measured . That suggests the problem isn't encoding or decoding of FFV1. It suggests the problem is encoding v210. Eitherway, I've never heard or seen anything like that from FFV1

Is it possible there are more problems with your samples ? Certainly that 1st sample had a problem on the 1st frame.

FFV1 v1.3 actually has a multithreading -slice and -slicecrc option and several other poorly documented options . Not sure what they do or how to use them properly . Maybe the -slicecrc gives another layer of confidence, not sure

[JasonCA]

Any new info ? Is it random or repeatable ?

That screenshot is remeniscent of reported "random" lagarith glitches that I mentioned earlier. There was speculation that it was due to floating point calculations, but that was debunked by the author. Basically nobody knows why it happens for the lagarith case

Those observations don't make sense, because FFV1's framemd5's are decoded and converted to raw YUV in order for them to be measured . That suggests the problem isn't encoding or decoding of FFV1. It suggests the problem is encoding v210. Eitherway, I've never heard or seen anything like that from FFV1

Is it possible there are more problems with your samples ? Certainly that 1st sample had a problem on the 1st frame.

FFV1 v1.3 actually has a multithreading -slice and -slicecrc option and several other poorly documented options . Not sure what they do or how to use them properly . Maybe the -slicecrc gives another layer of confidence, not sure

Since we last chatted, in the background I've been converting my uncompressed v210 files to FFV1. So, it's taken a good few days to allow my machines to convert some of the files over since they are quite huge and since I have a lot of them. I've stuck with FFV1 since it's all inclusive in ffmpeg. As you have made note, you don't get that much additional compression using x264.

In answering your question, I've not been able to re-create the bad frames on decompression from FFV1 back to v210. For whatever reason, I did get some bad frames when I made my last post. But, then I decompressed from the SAME EXACT FFV1 file and there were no bad frames. Kind of a . I thought during that post, that I was faced with a new issue or problem with ffmpeg. But, that doesn't seem to be the case.

I remember when decompressing and ended up with bad frames, I was doing a lot of other heavy multi-tasking stuff on my system. The only thing I can think of is that on decompression, if you somehow impede the decompression (starve ffmpeg or stall it), that it may throw in a bad frame or two? In all other tests, I've purposefully decompressed on a system that was not interrupted by anything else. And since then, I've not been able to re-create bad frames on decompression from FFV1->v210. It's been working flawlessly! The framemd5's are helpful at ensuring that everything is working as it should.

On a related issue to my original post, I've been trying to find a solution to my 10 Bit RGB 444 r10k problem. So far, both ffmpeg and x264 seem to be able to compress from r10k (10 bit RGB 444) to either FFV1 (in the case of ffmpeg) or mkv (in the case of x264). However, the problem is decompressing back to r10k. ffmpeg reports the same issue with the FFV1 or mkv files on decompression (I made mention to this on post #17 https://forum.videohelp.com/threads/361133-Lossless-%2810-Bit-RGB-444%29-and-%2810-Bit-...=1#post2290391):

full chroma interpolation for destination format 'rgb48le' not yet implemented

So, I'm trying to find an alternative decompressor. But no luck with that so far. I'm using a windows based "zeranoe" ffmpeg build. I thought perhaps if I found a Linux version of ffmpeg, that maybe it would have the chroma support implemented. But, I've not investigated this further. However, I have contacted those that create the windows "zeranoe" build and am waiting for someone to provide some feedback. Soon as I get that, I'll be sure to post the update here.

In summary, things are going great with the uncompressed v210 to lossless FFV1 using ffmpeg! So I've been enjoying getting those converted over. But, so far, it seems I'm stuck with the huge r10k files until I find a solution to decompress those. The only solution is archival software! But, it seems that can take roughly 20 hours per file on the tightest compression method. Whereas, like when compressing v210, it's practically real-time (emmmm, so maybe about 2 hours). The biggest benefit to FFV1 is not only the compression, but the fact I can still view the video files which is a nice plus .

[poisondeathray]

full chroma interpolation for destination format 'rgb48le' not yet implemented

So, I'm trying to find an alternative decompressor. But no luck with that so far. I'm using a windows based "zeranoe" ffmpeg build. I thought perhaps if I found a Linux version of ffmpeg, that maybe it would have the chroma support implemented. But, I've not investigated this further. However, I have contacted those that create the windows "zeranoe" build and am waiting for someone to provide some feedback. Soon as I get that, I'll be sure to post the update here.

Not implemented in ffmpeg linux . Zeranoe just complies it for windows, he has nothing to do with the actual libraries. One of the changes in post #19 has to be done in order for it to work in ffmpeg

In summary, things are going great with the uncompressed v210 to lossless FFV1 using ffmpeg! So I've been enjoying getting those converted over. But, so far, it seems I'm stuck with the huge r10k files until I find a solution to decompress those. The only solution is archival software! But, it seems that can take roughly 20 hours per file on the tightest compression method. Whereas, like when compressing v210, it's practically real-time (emmmm, so maybe about 2 hours). The biggest benefit to FFV1 is not only the compression, but the fact I can still view the video files which is a nice plus .

BTW, you can play videos in .rar format (winrar archive) with various video players eg. vlc, kmplayer, many others. They have rar file parsers

[JasonCA]

full chroma interpolation for destination format 'rgb48le' not yet implemented

So, I'm trying to find an alternative decompressor. But no luck with that so far. I'm using a windows based "zeranoe" ffmpeg build. I thought perhaps if I found a Linux version of ffmpeg, that maybe it would have the chroma support implemented. But, I've not investigated this further. However, I have contacted those that create the windows "zeranoe" build and am waiting for someone to provide some feedback. Soon as I get that, I'll be sure to post the update here.

Not implemented in ffmpeg linux . Zeranoe just complies it for windows, he has nothing to do with the actual libraries. One of the changes in post #19 has to be done in order for it to work in ffmpeg

In summary, things are going great with the uncompressed v210 to lossless FFV1 using ffmpeg! So I've been enjoying getting those converted over. But, so far, it seems I'm stuck with the huge r10k files until I find a solution to decompress those. The only solution is archival software! But, it seems that can take roughly 20 hours per file on the tightest compression method. Whereas, like when compressing v210, it's practically real-time (emmmm, so maybe about 2 hours). The biggest benefit to FFV1 is not only the compression, but the fact I can still view the video files which is a nice plus .

BTW, you can play videos in .rar format (winrar archive) with various video players eg. vlc, kmplayer, many others. They have rar file parsers

Just a shout out to poisondeathray from the past to say Hello ! Have a Merry Christmas too! It's one year later now since I last started this thread. I just thought I would let you know I put a question out most recently on the mailing list:

https://ffmpeg.org/pipermail/ffmpeg-user/2014-December/024722.html

I wanted to see if they have made any improvements. Gave it a quick try, and I would say no. FFMpeg Still doesn't support r10k. Perhaps they'll have something to say though. We'll see.