About a year ago, I made several lossless temporary files for a movie I've been working on for a looong time. Now that said movie is considered finished {*}, in the process of deleting most of those files, I made a quick test (for peace-of-mind's sake, or for O.C.D.-ness' sake depending on the P.O.V.), to verify if the compression of a simple Avisynth YUV to RGB conversion of the same file (created then because the automatic colorspace conversion in the Magix editor produced unpredictable results) would produce the same output, or if the files would be different, and to what extent — for instance I've noticed that the uncompressed audio extraction from the same AVI file with VirtualDub could produce very different WAV files (almost always when using a different version of VD but sometimes also when using the same version), although the audio content is supposedly the same (I may create a dedicated thread to investigate that similar issue).
That script was used (created just before the AVI Lagarith file in question was rendered on 2019-01-15 and not modified since) :
To my surprise, the new AVI Lagarith file turned out to be significantly larger, by about 5MB (out of 23.2GB – it's a small difference but it's definitely more than what could be attributed to a small variation in the header or the container's structures). Then I used the WriteFileIf method learned in this thread : with a detection threshold of LumaDifference > 0.0, every single frame was detected as different !Code:LWLibavVideoSource("20131224_145353.m2ts", threads=1) ConvertToRGB(matrix="Rec709")
Then I used the Subtract method learned in that same thread...Code:VID1 = AVISource("[Lagarith AVI created on 2019-01-15") VID2 = AVISource("[Lagarith AVI created on 2020-01-20") VID1 WriteFileIf(last, "WriteFileIf.txt", "LumaDifference(VID1, VID2) > 0.0", "current_frame")
...and observed this, for every frame (here frame 0) :Code:Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
Small red, blue and green dots moving across the picture. So it would seem to imply that the colorspace conversion was not quite the same the second time around. How can this be explained ?
In the mean time the only thing that has changed that would be relevant that I can think of is that I updated Avisynth. But such a basic thing as a RGB conversion with a specific conversion matrix shouldn't change over time and updates, right ? Is there something I'm missing here ?
I did the compression today with VirtualDub2 x86 build 43073, which is the version most likely used originally. I tried in “Fast recompress” mode and in “Full processing” mode : the files generated have the exact same size and are mostly identical when examined with a hexadecimal editor, although some small areas are different.
The 2019-01-15 file has a size of 24985394560.
The 2020-01-20 file has a size of 24990687420.
VirtualDub2 (“Decode format”) or Avisynth Info() report both files as being RGB32. I don't know what else to look for.
{*} “There are some minor adjustments that could theoretically be made, but at this stage, [...] it's easy to end up chasing an unattainable perfection. Once you hit diminishing returns, completed work trumps perfect work.”
— quoted from this article
“A work is finished when we can no longer improve it, although we know that it is insufficient and incomplete. We are so saturated that we no longer have the courage to add a single comma, however crucial it may be. What decides of the progress and level of completion of a work is certainly not a pursuit of art or truth, it's mere fatigue, and, even more so, disgust.”
— Emil Cioran
Try StreamFab Downloader and download from Netflix, Amazon, Youtube! Or Try DVDFab and copy Blu-rays! or rip iTunes movies!
+ Reply to Thread
Results 1 to 22 of 22
Thread
-
Last edited by abolibibelot; 20th Jan 2020 at 18:37.
-
"lossless" avi files are visually lossless but not perfectly lossless. you can't get back the same file going back the other way. and from the statement "version most likely used" it could just be an updated codec was used. 5mb out of a 25gb file is a small margin of change.
--
"a lot of people are better dead" - prisoner KSC2-303 -
In this case it's not “visually lossless” if a difference in luma / chroma levels, however small, can be detected (albeit not “visually”, strictly speaking).
The Lagarith codec itself hasn't been updated in the mean time, the only uncertainty is the version of VirtualDub2 used for the rendering of the exact same simple RGB conversion script, which should not change anything, at least "visually". -
you don't understand the term "visually". it means with your eyes, not an algorithm. no one ever claimed you could render to a "lossless" codec and then render it back to the exact same file it started as, which would be truly "lossless".
--
"a lot of people are better dead" - prisoner KSC2-303 -
-
What about lsmash version ? same/different ?
you don't understand the term "visually". it means with your eyes, not an algorithm.
Conversely, when I noticed that two WAV files extracted from the same AVI file could be different (as mentioned above as a side note), I used the method proposed by “jagabo” in the aforementioned thread (“Audacity : Load two tracks, invert one, merge two tracks into one, amplify. If they're identical the result will be silence.”) — and contrary to what I observe here there was no difference (the result was complete silence), so in this case, even though the binary content is vastly different, the actual audio samples are identical.
(Here the values that are different are all different by a +1 hexadecimal increment, I don't know what that means.) -
lagarith has options that might not have been selected the same way for the separate encodes a year apart that would have induced small changes. try using it twice with the same things checked and then test.
--
"a lot of people are better dead" - prisoner KSC2-303 -
As a test, I removed the ConvertToRGB command, exported a small 100 frames sample (thus in YUV colorspace), then re-made the Subtract analysis with each RGB conversion against that file (the YUV Lagarith sample) converted with the same ConvertToRGB command within the test script.
Code:VID1 = AVISource("[direct Lagarith conversion 100 frames sample].avi").ConvertToRGB(matrix="Rec709") VID2 = AVISource("[2019-01-15 Lagarith RGB conversion].avi").Trim(0,100) Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
Code:VID1 = AVISource("[direct Lagarith conversion 100 frames sample].avi").ConvertToRGB(matrix="Rec709") VID2 = AVISource("[2020-01-20 Lagarith RGB conversion].avi").Trim(0,100) Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
So, apparently the new conversion is lossless, the former one was not. Is there a reasonable explanation ? Another test I can make ? -
sorry but that's a meaningless test. under 3 seconds. they could all be just black. unless you run the test on the same entire video as the first one it's worthless.
--
"a lot of people are better dead" - prisoner KSC2-303 -
I suspect you updated LWlibavVideoSource() and it is putting out a chroma subsampling or different clamping between then and now. Or maybe you updated AviSynth and the chroma upsampling algorithm has changed.
Last edited by jagabo; 20th Jan 2020 at 21:50.
-
lagarith has options that might not have been selected the same way for the separate encodes a year apart that would have induced small changes. try using it twice with the same things checked and then test.
As you can see, the differences are limited to small areas of the frames, does this give a clue as to what happened ?
Also tried Lagarith's YV12 mode (the resulting file has a size reduced by about 50%), the result is quite funky :
-
@aedipus
sorry but that's a meaningless test. under 3 seconds. they could all be just black. unless you run the test on the same entire video as the first one it's worthless.
@jagabo
I suspect you updated LWlibavVideoSource() and it is putting out a chroma subsampling or different clamping between then and now.
Or maybe you updated AviSynth and the chroma upsampling algorithm has changed.Last edited by abolibibelot; 20th Jan 2020 at 22:15.
-
-
I seem to recall there was a fix to some minor YUV/RGB conversion bug(s) in AvSynth a few years (?) ago. Maybe chroma placement issues? In any case, it's much more likely that the problem is in AviSynth rather than in Lagarith. There is no absolute "right" way to convert RGB/YUV, or 4:2:2/4:2:0 chroma subsampling.
-
@jagabo
I seem to recall there was a fix to some minor YUV/RGB conversion bug(s) in AvSynth a few years (?) ago. Maybe chroma placement issues?
There is no absolute "right" way to convert RGB/YUV, or 4:2:2/4:2:0 chroma subsampling.
@poisondeathray
If all versions are the same x86 (or x64) , and it's currently working (this indicates registry keys are correct), it's just a .dll swap (avisynth.dll). Just copy/paste
Windows/System32 for x64
Windows/SysWOW64 for x86
But... isn't it the other way around (System32 / SysWOW64) ? -
Nope, System32 is for x64 , SysWOW64 is for x86
https://www.samlogic.net/articles/32-64-bit-windows-folder-x86-syswow64.htm
And that's all I do to swap versions the last few years
You can double check with version() to see what version is actually being used . ( If you have some portable installation or something like megui, not sure how that works, or which version is being used or gets "priority") -
Here's the result, with the same test as before :
Code:Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
=> no difference
– VID1 generated with Avisynth+ 3.4.0.0, VID2 generated with Avisynth+ r1576
=> same weirdness as above
Also the Avisynth+ 3.4 and Avisynth 2.6 files have the same size (62223220 bytes for 100 frames) although their binary content is different in places (but identical in other places) ; while the Avisynth+ r1576 file is slightly smaller (62208464 bytes for 100 frames).
And so indeed there was an issue, or at least a discrepancy (?), in the YUV to RGB conversion in Avisynth+ r1576, which based on those results was the version installed when I exported that file a year ago, and in all likelyhood all the others which were actually used for the editing of this movie. Does that mean that all those files are wrong ? To be honest, when watching screenshots from the same frame from the Avisynth+ 3.4 conversion and the Avisynth+ r1576 conversion back and forth, I can't see any difference...Last edited by abolibibelot; 24th Jan 2020 at 23:34. Reason: the screenshots were in reversed order
-
That difference using subtract and levels massively amplifies the differences. Just plain difference is probably difficult to see. And just single frames very difficult too. It's probably negligible for most scenarios
Not sure which specific change since r1576 , but you can look at the changelog and commits to track the difference
There are dozens of variations just within avisynth on how to do the YUV to RGB conversion. If you look at just the scaling algorithm part of the conversion, there slightly different implementations of say "bicubic" or "bilinear" or "whatever" algorithms. That's part of what you're doing when going from YUV 4:2:0 to RGB, you're resizing the chroma or U,V Planes
But you could argue internal (default) avisynth resizers are technically "incorrect", or different than convention. Avisynth internal resizers assume "MPEG1" 4:2:0 chroma placement , which is sited in the center. By convention, most video formats use "MPEG2" 4:2:0, which is "left" placement. (There are exceptions like DV, UHD placement, etc..If you look at the dither tools docs, there are illustrations about the different placements) .
But as a result - the chroma is shifted right when upsizing, and left when downsizing. If you use point resize ("nearest neighbor") for the chroma up/down in multiples of 2 , "n" times - it's theoretically lossless; you're supposedly duplicating and dropping the same pixels you just duplicated. But it's not in avisynth using default settings because of this - you get chroma shifting worse each generation. Of course there are various workarounds you can use to shift the chroma, or the avsresize plugin (zimg/z.lib library) does this operation like other programs -
Thanks for the added explanations. I'll have to dig deeper to understand them fully, though.
I did another test : imported the source video into the NLE (Magix Video Deluxe 2016), and exported the first 4s in Lagarith straight away with no processing. Then used the same Subtract command on this file against the one generated by Avisynth+ 3.4 and ConvertToRGB(matrix="Rec709"). Result — the differences are much more pronounced :
Same test with ConvertToRGB(matrix="Rec601") :
(If re-importing the exported file into the NLE the colors are correct, and the file generated with Avisynth and ConvertToRGB(matrix="Rec601") has different shades of reds and green, as it should be, so at least the correct conversion matrix is used when exporting from the NLE.)
A stranger thing yet is that darker strip on the right edge of both screenshots ; I verified, it appears on all frames. What could possibly explain this ?
(See sample videos. That was grandma holding a bottle of milk six years ago. How spooky is that.)
That difference using subtract and levels massively amplifies the differences.
But as a result - the chroma is shifted right when upsizing, and left when downsizing. If you use point resize ("nearest neighbor") for the chroma up/down in multiples of 2 , "n" times - it's theoretically lossless; you're supposedly duplicating and dropping the same pixels you just duplicated. But it's not in avisynth using default settings because of this - you get chroma shifting worse each generation. Of course there are various workarounds you can use to shift the chroma, or the avsresize plugin (zimg/z.lib library) does this operation like other programs
Is there any justification for this behaviour to be different in Avisynth from what it is in “other programs” ? What common video processing utilities are known to perform resizing operations based on the conventional “MPEG2” 4:2:0 chroma placement ? Perhaps VirtualDub for instance ?Last edited by abolibibelot; 25th Jan 2020 at 03:43.
-
Those may be edge effect of the chroma resizing algorithm.
The luma values between 127 and 129 are stretched to 0 to 255. So a single bit less than 128 becomes 0, a single bit more than 128 becomes 255. Basically this is a very extreme contrast stretch.
All programs have to perform chroma upsampling when converting YUV 4:2:0 to RGB, and chroma downsampling between RGB and YUV 4:2:0. Remember, SD ITU YUV 4:2:0 has a 720x480 luma channel and two 360x240 chroma channels, RGB has 720x480 R, G, and B channels. On conversion from YUV to RGB the chroma channels have to be upscaled from 360x240 to 720x480 to provide U and V values for each Y -- then each YUV value is converted to an RGB value. On conversion from RGB to YUV 4:2:0 you have to do the opposite, downscale the U and V channels. Different programs may use different scaling algorithms and assume different chroma placement. -
The luma values between 127 and 129 are stretched to 0 to 255. So a single bit less than 128 becomes 0, a single bit more than 128 becomes 255. Basically this is a very extreme contrast stretch.
All programs have to perform chroma upsampling when converting YUV 4:2:0 to RGB, and chroma downsampling between RGB and YUV 4:2:0. Remember, SD ITU YUV 4:2:0 has a 720x480 luma channel and two 360x240 chroma channels, RGB has 720x480 R, G, and B channels. On conversion from YUV to RGB the chroma channels have to be upscaled from 360x240 to 720x480 to provide U and V values for each Y -- then each YUV value is converted to an RGB value. On conversion from RGB to YUV 4:2:0 you have to do the opposite, downscale the U and V channels. Different programs may use different scaling algorithms and assume different chroma placement.
Also, to complete the verification, I re-exported the whole video in Lagarith with Avisynth+ r1576, and this time the size is exactly identical (24985394560), although the AVI files are not identical all along (but they're not even identical between two files exported right after one another, as I indicated yesterday).
Similar Threads
-
AVS script compression failure in VirtualDub2 / Lagarith RGB
By abolibibelot in forum Video ConversionReplies: 19Last Post: 20th Dec 2018, 03:07 -
Inaccurate YUV -> RGB Conversion.
By chris319 in forum Video ConversionReplies: 151Last Post: 2nd Dec 2018, 18:58 -
DV to Lagarith conversion help
By robmausser in forum Video ConversionReplies: 7Last Post: 19th Aug 2017, 23:27 -
If YPbPr is almost always converted from RGB, why does it exist at all?
By CursedLemon in forum Video ConversionReplies: 19Last Post: 17th Oct 2016, 16:32 -
Smart render. Lagarith > Vegas(edit) > Lagarith.
By ValentineStone in forum Video ConversionReplies: 11Last Post: 5th Oct 2016, 14:31