VideoHelp Forum

Try DVDFab and download streaming video, copy, convert or make Blu-rays,DVDs! Download free trial !
+ Reply to Thread
Results 1 to 22 of 22
Thread
  1. About a year ago, I made several lossless temporary files for a movie I've been working on for a looong time. Now that said movie is considered finished {*}, in the process of deleting most of those files, I made a quick test (for peace-of-mind's sake, or for O.C.D.-ness' sake depending on the P.O.V.), to verify if the compression of a simple Avisynth YUV to RGB conversion of the same file (created then because the automatic colorspace conversion in the Magix editor produced unpredictable results) would produce the same output, or if the files would be different, and to what extent — for instance I've noticed that the uncompressed audio extraction from the same AVI file with VirtualDub could produce very different WAV files (almost always when using a different version of VD but sometimes also when using the same version), although the audio content is supposedly the same (I may create a dedicated thread to investigate that similar issue).
    That script was used (created just before the AVI Lagarith file in question was rendered on 2019-01-15 and not modified since) :
    Code:
    LWLibavVideoSource("20131224_145353.m2ts", threads=1)
    ConvertToRGB(matrix="Rec709")
    To my surprise, the new AVI Lagarith file turned out to be significantly larger, by about 5MB (out of 23.2GB – it's a small difference but it's definitely more than what could be attributed to a small variation in the header or the container's structures). Then I used the WriteFileIf method learned in this thread : with a detection threshold of LumaDifference > 0.0, every single frame was detected as different !
    Code:
    VID1 = AVISource("[Lagarith AVI created on 2019-01-15")
    VID2 = AVISource("[Lagarith AVI created on 2020-01-20")
    VID1
    WriteFileIf(last, "WriteFileIf.txt", "LumaDifference(VID1, VID2) > 0.0", "current_frame")
    Then I used the Subtract method learned in that same thread...
    Code:
    Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
    ...and observed this, for every frame (here frame 0) :
    Click image for larger version

Name:	20131224_145353 [conversion RGB sans pré-traitement].avi - Subtract - F0.png
Views:	33
Size:	31.9 KB
ID:	51574
    Small red, blue and green dots moving across the picture. So it would seem to imply that the colorspace conversion was not quite the same the second time around. How can this be explained ?
    In the mean time the only thing that has changed that would be relevant that I can think of is that I updated Avisynth. But such a basic thing as a RGB conversion with a specific conversion matrix shouldn't change over time and updates, right ? Is there something I'm missing here ?
    I did the compression today with VirtualDub2 x86 build 43073, which is the version most likely used originally. I tried in “Fast recompress” mode and in “Full processing” mode : the files generated have the exact same size and are mostly identical when examined with a hexadecimal editor, although some small areas are different.
    The 2019-01-15 file has a size of 24985394560.
    The 2020-01-20 file has a size of 24990687420.
    VirtualDub2 (“Decode format”) or Avisynth Info() report both files as being RGB32. I don't know what else to look for.


    {*} “There are some minor adjustments that could theoretically be made, but at this stage, [...] it's easy to end up chasing an unattainable perfection. Once you hit diminishing returns, completed work trumps perfect work.”
    — quoted from this article
    “A work is finished when we can no longer improve it, although we know that it is insufficient and incomplete. We are so saturated that we no longer have the courage to add a single comma, however crucial it may be. What decides of the progress and level of completion of a work is certainly not a pursuit of art or truth, it's mere fatigue, and, even more so, disgust.”
    — Emil Cioran
    Last edited by abolibibelot; 20th Jan 2020 at 18:37.
    Quote Quote  
  2. aBigMeanie aedipuss's Avatar
    Join Date
    Oct 2005
    Location
    666th portal
    Search Comp PM
    "lossless" avi files are visually lossless but not perfectly lossless. you can't get back the same file going back the other way. and from the statement "version most likely used" it could just be an updated codec was used. 5mb out of a 25gb file is a small margin of change.
    --
    "a lot of people are better dead" - prisoner KSC2-303
    Quote Quote  
  3. In this case it's not “visually lossless” if a difference in luma / chroma levels, however small, can be detected (albeit not “visually”, strictly speaking).
    The Lagarith codec itself hasn't been updated in the mean time, the only uncertainty is the version of VirtualDub2 used for the rendering of the exact same simple RGB conversion script, which should not change anything, at least "visually".
    Quote Quote  
  4. aBigMeanie aedipuss's Avatar
    Join Date
    Oct 2005
    Location
    666th portal
    Search Comp PM
    you don't understand the term "visually". it means with your eyes, not an algorithm. no one ever claimed you could render to a "lossless" codec and then render it back to the exact same file it started as, which would be truly "lossless".
    --
    "a lot of people are better dead" - prisoner KSC2-303
    Quote Quote  
  5. What about lsmash version ? same/different ?


    Originally Posted by aedipuss View Post
    you don't understand the term "visually". it means with your eyes, not an algorithm. no one ever claimed you could render to a "lossless" codec and then render it back to the exact same file it started as, which would be truly "lossless".
    He' s not rendering back to exact same file with original compression ; The "lossless" video term refers to decoded, uncompressed state
    Quote Quote  
  6. What about lsmash version ? same/different ?
    The script (I omitted that line above) points to the LSMASHSource.dll in MeGUI's “tools” subfolders, and according to the timestamps it hasn't been updated since 2017-02-24 (although the “lsmash” folder appears to have been modified on 2019-10-30, which seems to be when I last used MeGUI).

    you don't understand the term "visually". it means with your eyes, not an algorithm.
    Unless I completely misunderstood the whole purpose of that test, it's meant to systematically report actual differences in the video content, even differences that are too small to be detected by comparing two files “visually”, but those differences are still real, “visual” for lack of a better word (as opposed to differences in the binary structure of the file which would not affect the way pixels are computed and rendered on the monitor), and should not exist between two files converted losslessly with the exact same parameters.

    Conversely, when I noticed that two WAV files extracted from the same AVI file could be different (as mentioned above as a side note), I used the method proposed by “jagabo” in the aforementioned thread (Audacity : Load two tracks, invert one, merge two tracks into one, amplify. If they're identical the result will be silence.”) — and contrary to what I observe here there was no difference (the result was complete silence), so in this case, even though the binary content is vastly different, the actual audio samples are identical.
    Click image for larger version

Name:	GCarlin Doin' it again - WAV re-created 20181207 totally different, yet equal audio content.png
Views:	14
Size:	143.6 KB
ID:	51577
    (Here the values that are different are all different by a +1 hexadecimal increment, I don't know what that means.)
    Quote Quote  
  7. aBigMeanie aedipuss's Avatar
    Join Date
    Oct 2005
    Location
    666th portal
    Search Comp PM
    lagarith has options that might not have been selected the same way for the separate encodes a year apart that would have induced small changes. try using it twice with the same things checked and then test.
    --
    "a lot of people are better dead" - prisoner KSC2-303
    Quote Quote  
  8. As a test, I removed the ConvertToRGB command, exported a small 100 frames sample (thus in YUV colorspace), then re-made the Subtract analysis with each RGB conversion against that file (the YUV Lagarith sample) converted with the same ConvertToRGB command within the test script.

    Code:
    VID1 = AVISource("[direct Lagarith conversion 100 frames sample].avi").ConvertToRGB(matrix="Rec709")
    VID2 = AVISource("[2019-01-15 Lagarith RGB conversion].avi").Trim(0,100)
    Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
    Click image for larger version

Name:	20131224_145353 [conversion RGB sans pré-traitement].avi -- 20190115 vs fichier converti sans tr.png
Views:	14
Size:	33.4 KB
ID:	51578

    Code:
    VID1 = AVISource("[direct Lagarith conversion 100 frames sample].avi").ConvertToRGB(matrix="Rec709")
    VID2 = AVISource("[2020-01-20 Lagarith RGB conversion].avi").Trim(0,100)
    Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
    Click image for larger version

Name:	20131224_145353 [conversion RGB sans pré-traitement].avi -- 20200120 vs fichier converti sans tr.png
Views:	15
Size:	20.8 KB
ID:	51579

    So, apparently the new conversion is lossless, the former one was not. Is there a reasonable explanation ? Another test I can make ?
    Quote Quote  
  9. aBigMeanie aedipuss's Avatar
    Join Date
    Oct 2005
    Location
    666th portal
    Search Comp PM
    sorry but that's a meaningless test. under 3 seconds. they could all be just black. unless you run the test on the same entire video as the first one it's worthless.
    --
    "a lot of people are better dead" - prisoner KSC2-303
    Quote Quote  
  10. I suspect you updated LWlibavVideoSource() and it is putting out a chroma subsampling or different clamping between then and now. Or maybe you updated AviSynth and the chroma upsampling algorithm has changed.
    Last edited by jagabo; 20th Jan 2020 at 21:50.
    Quote Quote  
  11. lagarith has options that might not have been selected the same way for the separate encodes a year apart that would have induced small changes. try using it twice with the same things checked and then test.
    I tried exporting the original script in “Fast recompress” mode in Lagarith's “RGBA” and “RGB” mode, then in “Full processing” mode again in Lagarith's “RGBA” and “RGB” mode, and did again the Subtract test against the test file mentioned above (created with no colorspace conversion and then converted to RGB within the test script), for all 4 tests the result was : all frames are grey, meaning, no “visual” difference at all, meaning, perfectly lossless (even though the files are almost completely different with regards to their binary content, which is — also — quite puzzling).
    As you can see, the differences are limited to small areas of the frames, does this give a clue as to what happened ?

    Also tried Lagarith's YV12 mode (the resulting file has a size reduced by about 50%), the result is quite funky :
    Click image for larger version

Name:	20131224_145353 -- test Lagarith YV12 vs fichier converti sans traitement.png
Views:	13
Size:	785.9 KB
ID:	51581
    Quote Quote  
  12. @aedipus
    sorry but that's a meaningless test. under 3 seconds. they could all be just black. unless you run the test on the same entire video as the first one it's worthless.
    With one test, every single frame is different (with those dancing dots of vivid colors), with the other, all 100 frames are perfectly identical (result of Subtract is all grey), so it seems pretty conclusive to me. (In the first test I only examined 40 frames or so — not the entire 40365 frames of the whole 26m54s video... I'm not that crazy... yet...)

    @jagabo
    I suspect you updated LWlibavVideoSource() and it is putting out a chroma subsampling or different clamping between then and now.
    I made a backup of the whole MeGUI directory on 2017-12-15 (way before the problematic video was created), just checked it : the CRC32 of LSMASHSource.dll matches that of the current file (which is consistent with the created date of that file : 2017-02-24 hence before the backup).

    Or maybe you updated AviSynth and the chroma upsampling algorithm has changed.
    I did update Avisynth as stated in the first post, now using Avisynth+ 3.4, then I was using either Avisynth 2.60, or Avisynth+ r1576.
    Last edited by abolibibelot; 20th Jan 2020 at 22:15.
    Quote Quote  
  13. Would it be possible to do a test with the former Avisynth version, without messing with the current install ?
    Quote Quote  
  14. Originally Posted by abolibibelot View Post
    Would it be possible to do a test with the former Avisynth version, without messing with the current install ?
    If all versions are the same x86 (or x64) , and it's currently working (this indicates registry keys are correct), it's just a .dll swap (avisynth.dll). Just copy/paste

    Windows/System32 for x64

    Windows/SysWOW64 for x86
    Quote Quote  
  15. I seem to recall there was a fix to some minor YUV/RGB conversion bug(s) in AvSynth a few years (?) ago. Maybe chroma placement issues? In any case, it's much more likely that the problem is in AviSynth rather than in Lagarith. There is no absolute "right" way to convert RGB/YUV, or 4:2:2/4:2:0 chroma subsampling.
    Quote Quote  
  16. @jagabo
    I seem to recall there was a fix to some minor YUV/RGB conversion bug(s) in AvSynth a few years (?) ago. Maybe chroma placement issues?
    Would what you seem to recall be consistent with what is observed here ?

    There is no absolute "right" way to convert RGB/YUV, or 4:2:2/4:2:0 chroma subsampling.
    Really ? I thought that it was a well established standard, although I don't quite understand the technical details.


    @poisondeathray
    If all versions are the same x86 (or x64) , and it's currently working (this indicates registry keys are correct), it's just a .dll swap (avisynth.dll). Just copy/paste
    Windows/System32 for x64
    Windows/SysWOW64 for x86
    Alright, I'll try that.
    But... isn't it the other way around (System32 / SysWOW64) ?
    Quote Quote  
  17. Originally Posted by abolibibelot View Post


    @poisondeathray
    If all versions are the same x86 (or x64) , and it's currently working (this indicates registry keys are correct), it's just a .dll swap (avisynth.dll). Just copy/paste
    Windows/System32 for x64
    Windows/SysWOW64 for x86
    Alright, I'll try that.
    But... isn't it the other way around (System32 / SysWOW64) ?
    Nope, System32 is for x64 , SysWOW64 is for x86

    https://www.samlogic.net/articles/32-64-bit-windows-folder-x86-syswow64.htm

    And that's all I do to swap versions the last few years

    You can double check with version() to see what version is actually being used . ( If you have some portable installation or something like megui, not sure how that works, or which version is being used or gets "priority")
    Quote Quote  
  18. Here's the result, with the same test as before :
    Code:
    Subtract(VID1, VID2).Levels(127, 1, 129, 0, 255)
    – VID1 generated with Avisynth+ 3.4.0.0, VID2 generated with Avisynth 2.6.0.6
    Click image for larger version

Name:	Subtract Avisynth+ 3.4 ~ Avisynth 2.6.png
Views:	9
Size:	3.7 KB
ID:	51655
    => no difference

    – VID1 generated with Avisynth+ 3.4.0.0, VID2 generated with Avisynth+ r1576
    Click image for larger version

Name:	Subtract Avisynth+ 3.4 ~ Avisynth+ r1576.png
Views:	9
Size:	14.1 KB
ID:	51654
    => same weirdness as above

    Also the Avisynth+ 3.4 and Avisynth 2.6 files have the same size (62223220 bytes for 100 frames) although their binary content is different in places (but identical in other places) ; while the Avisynth+ r1576 file is slightly smaller (62208464 bytes for 100 frames).

    And so indeed there was an issue, or at least a discrepancy (?), in the YUV to RGB conversion in Avisynth+ r1576, which based on those results was the version installed when I exported that file a year ago, and in all likelyhood all the others which were actually used for the editing of this movie. Does that mean that all those files are wrong ? To be honest, when watching screenshots from the same frame from the Avisynth+ 3.4 conversion and the Avisynth+ r1576 conversion back and forth, I can't see any difference...
    Last edited by abolibibelot; 24th Jan 2020 at 23:34. Reason: the screenshots were in reversed order
    Quote Quote  
  19. That difference using subtract and levels massively amplifies the differences. Just plain difference is probably difficult to see. And just single frames very difficult too. It's probably negligible for most scenarios

    Not sure which specific change since r1576 , but you can look at the changelog and commits to track the difference



    There are dozens of variations just within avisynth on how to do the YUV to RGB conversion. If you look at just the scaling algorithm part of the conversion, there slightly different implementations of say "bicubic" or "bilinear" or "whatever" algorithms. That's part of what you're doing when going from YUV 4:2:0 to RGB, you're resizing the chroma or U,V Planes

    But you could argue internal (default) avisynth resizers are technically "incorrect", or different than convention. Avisynth internal resizers assume "MPEG1" 4:2:0 chroma placement , which is sited in the center. By convention, most video formats use "MPEG2" 4:2:0, which is "left" placement. (There are exceptions like DV, UHD placement, etc..If you look at the dither tools docs, there are illustrations about the different placements) .

    But as a result - the chroma is shifted right when upsizing, and left when downsizing. If you use point resize ("nearest neighbor") for the chroma up/down in multiples of 2 , "n" times - it's theoretically lossless; you're supposedly duplicating and dropping the same pixels you just duplicated. But it's not in avisynth using default settings because of this - you get chroma shifting worse each generation. Of course there are various workarounds you can use to shift the chroma, or the avsresize plugin (zimg/z.lib library) does this operation like other programs
    Quote Quote  
  20. Thanks for the added explanations. I'll have to dig deeper to understand them fully, though.

    I did another test : imported the source video into the NLE (Magix Video Deluxe 2016), and exported the first 4s in Lagarith straight away with no processing. Then used the same Subtract command on this file against the one generated by Avisynth+ 3.4 and ConvertToRGB(matrix="Rec709"). Result — the differences are much more pronounced :
    Click image for larger version

Name:	Subtract Avisynth+ 3.4 ~ export MVD.png
Views:	8
Size:	278.3 KB
ID:	51657
    Same test with ConvertToRGB(matrix="Rec601") :
    Click image for larger version

Name:	Subtract Avisynth+ 3.4 (Rec601) ~ export MVD.png
Views:	9
Size:	234.0 KB
ID:	51658
    (If re-importing the exported file into the NLE the colors are correct, and the file generated with Avisynth and ConvertToRGB(matrix="Rec601") has different shades of reds and green, as it should be, so at least the correct conversion matrix is used when exporting from the NLE.)
    A stranger thing yet is that darker strip on the right edge of both screenshots ; I verified, it appears on all frames. What could possibly explain this ?
    (See sample videos. That was grandma holding a bottle of milk six years ago. How spooky is that.)

    That difference using subtract and levels massively amplifies the differences.
    What does this Subtract command do, in plain terms ? (The “Levels” part in particular, I don't quite understand what the 127 and 129 values mean here.)

    But as a result - the chroma is shifted right when upsizing, and left when downsizing. If you use point resize ("nearest neighbor") for the chroma up/down in multiples of 2 , "n" times - it's theoretically lossless; you're supposedly duplicating and dropping the same pixels you just duplicated. But it's not in avisynth using default settings because of this - you get chroma shifting worse each generation. Of course there are various workarounds you can use to shift the chroma, or the avsresize plugin (zimg/z.lib library) does this operation like other programs
    Wow... it seems like, for each new little trick I learn, there are a gazillion of intricacies and obfuscating SNAFUs to discover and take into account... é_è
    Is there any justification for this behaviour to be different in Avisynth from what it is in “other programs” ? What common video processing utilities are known to perform resizing operations based on the conventional “MPEG2” 4:2:0 chroma placement ? Perhaps VirtualDub for instance ?
    Last edited by abolibibelot; 25th Jan 2020 at 03:43.
    Quote Quote  
  21. Originally Posted by abolibibelot View Post
    A stranger thing yet is that darker strip on the right edge of both screenshots ; I verified, it appears on all frames. What could possibly explain this ?
    Those may be edge effect of the chroma resizing algorithm.

    Originally Posted by abolibibelot View Post
    That difference using subtract and levels massively amplifies the differences.
    What does this Subtract command do, in plain terms ? (The “Levels” part in particular, I don't quite understand what the 127 and 129 values mean here.)
    The luma values between 127 and 129 are stretched to 0 to 255. So a single bit less than 128 becomes 0, a single bit more than 128 becomes 255. Basically this is a very extreme contrast stretch.

    Originally Posted by abolibibelot View Post
    Is there any justification for this (chroma upsampling) behaviour to be different in Avisynth from what it is in “other programs” ? What common video processing utilities are known to perform resizing operations based on the conventional “MPEG2” 4:2:0 chroma placement ? Perhaps VirtualDub for instance ?
    All programs have to perform chroma upsampling when converting YUV 4:2:0 to RGB, and chroma downsampling between RGB and YUV 4:2:0. Remember, SD ITU YUV 4:2:0 has a 720x480 luma channel and two 360x240 chroma channels, RGB has 720x480 R, G, and B channels. On conversion from YUV to RGB the chroma channels have to be upscaled from 360x240 to 720x480 to provide U and V values for each Y -- then each YUV value is converted to an RGB value. On conversion from RGB to YUV 4:2:0 you have to do the opposite, downscale the U and V channels. Different programs may use different scaling algorithms and assume different chroma placement.
    Quote Quote  
  22. The luma values between 127 and 129 are stretched to 0 to 255. So a single bit less than 128 becomes 0, a single bit more than 128 becomes 255. Basically this is a very extreme contrast stretch.
    Alright, and so when subtracting, the values are calculated in reference to 128, correct ? (Otherwise 0 minus 0 would be 0 and those pixels would be black instead of grey.)

    All programs have to perform chroma upsampling when converting YUV 4:2:0 to RGB, and chroma downsampling between RGB and YUV 4:2:0. Remember, SD ITU YUV 4:2:0 has a 720x480 luma channel and two 360x240 chroma channels, RGB has 720x480 R, G, and B channels. On conversion from YUV to RGB the chroma channels have to be upscaled from 360x240 to 720x480 to provide U and V values for each Y -- then each YUV value is converted to an RGB value. On conversion from RGB to YUV 4:2:0 you have to do the opposite, downscale the U and V channels. Different programs may use different scaling algorithms and assume different chroma placement.
    Alright, and so the conversion matrixes Rec709 / Rec601 (which are apparently a precisely defined standard) are only used for the second step of the process, after the upscaling, which according to those explanations is (legitimately) variable to some extent between different programs ?


    Also, to complete the verification, I re-exported the whole video in Lagarith with Avisynth+ r1576, and this time the size is exactly identical (24985394560), although the AVI files are not identical all along (but they're not even identical between two files exported right after one another, as I indicated yesterday).
    Quote Quote  



Similar Threads