Premiere export to lossless is not lossless

[Dogway]

They passed the hash test (also visual test), check ffms2 in PM, maybe it's that. I'm certainly intrigued by this, it's like a nice flick, haha

if it's not that let's assume is xmas magick lol

[poisondeathray]

And we haven't even got to the part about your original question yet...

Yes, differences detected with your ffms2 version. I made sure to delete index files first, close all avisynth applications like vdub, avspmod, anything that might have an avs process open, before restarting (to re-initialize avisynth start up)

Something screwy going on , but I'm still sure there are differences between those files

[Dogway]

Post screenshots of frame 25 for example. I will post mine here, use:
ConvertToRGB(matrix="Rec601", interlaced=true)

source...............vdub

[poisondeathray]

Your screenshots match

Mine don't. Taken with avspmod

ffms2 and converttorgb24(matrix="rec601", interlaced=true)

[poisondeathray]

Try ffmpeg psnr on your files

ffmpeg -i "cacho01 vdub.avi" -vcodec rawvideo -vf "movie=chunky.avi,setpts=PTS-STARTPTS[main];[main][ref]psnr="stats_file=chunky_vs_vdub.log" [out]" -f rawvideo -y /NUL

[Dogway]

poisondeathray, the only disparities found in the videos are quantization blocks, it's the type of difference you find for decoding. I think ffmpeg is applying internal deblocking to the DVCPRO file, test it by converting the DVCPRO file to Lagarith and repeat the process. Maybe it's that (or maybe not).

[poisondeathray]

poisondeathray, the only disparities found in the videos are quantization blocks, it's the type of difference you find for decoding. I think ffmpeg is applying internal deblocking to the DVCPRO file, test it by converting the DVCPRO file to Lagarith and repeat the process. Maybe it's that (or maybe not).

I'll try it, but it agrees completely with avisynth results, even the misordered 1st frame (large jump in MSE value)

How did you want me to convert the dvsd file to lagarith ? obviously not by ffmpeg. How did you want me to decode the dvsd file ?

[Dogway]

I use vdub, fast recompress to lagarith (YV12), I think the decoder is my system defaults (LAV Video).

edit: in your screenshots the one different to mines is the source one, so it tell a bit it might be a decoding issue.

Here is my log.

[poisondeathray]

There are errors in your log, some frames don't match

And here is mine, when re-encoded with lagarith, to test with ffmpeg psnr

(ffms2 to load the dvsd file, loaded into vdub, with "AVI files of type" in the drop down menu - this is important if you have the ffinputdriver installed, video=>fast recompreess, lagarith, video=>save as avi)

That's another thing - if you have the vdub ffinputdriver installed, you can get messed up avs results unless you select from the drop down box "files of type" in the open file dialog box AVIFile input driver, otherwise it will open with the ffinputdriver - there are problems using avs with ffinput driver . File=> File information should say AVI information, not FFMpeg information. That's how you know it' s using the proper input method in vdub

[poisondeathray]

How did you load the original dvsd video into vdub before encoding to lagarith ? Directly as an AVI ? Try it now, and use file=>file information, what does it say ?

[Dogway]

That's interesting, I don't know if it's the culprit but we are learning a buttload of things.
For these videos with no filtering I used to load as is, but for this case I certainly used avisynth to trim(). let me try your workflow.

btw, what should I try?

[poisondeathray]

Well everything seems to be working fine here. I've done literally thousands of these sorts of tests with multiple types of check. I'm certain everything I've said is correct.

If you load an AVI video directly into vdub, you can get different results, because it depends on the settings (do you have "prefer internal decoders" checkmarked in the options?), and the VFW decoder installed . To check, you use file=>file information and it should list the current VFW decoder being used to open the file

If you load an .avs in vdub, normally it should work ok, if you use video=>fast recompress. HOWEVER, there are documentated issues with the ffinputdriver and avs. So you need to use that procedure above, or remove the ffinputdriver before loading an .avs into vdub. So if you had ffinputdriver installed, try encoding lagarith again with the original file using that procedure above

[Dogway]

The problem is that in this situation I don't think I have used vdub at all. I encoded lagarith from Premiere, and decoded, splitted and encoded source from Premiere. I'm not sure what I should be testing. Maybe I should do everything again from scratch?

[poisondeathray]

The problem is that in this situation I don't think I have used vdub at all. I encoded lagarith from Premiere, and decoded, splitted and encoded source from Premiere. I'm not sure what I should be testing. Maybe I should do everything again from scratch?

Your original question was about premiere, but we deviated from that because I noticed none of your files matched

Your premiere one for sure didn't match (naked eye can tell), but you would have expected the vdub one to match (I though you used lagarith in vdub to encode that?)

If you can't get a baseline on what is lossless, and what isn't lossless, then it' s hard to test later on when it really matters . Once you have that sorted out, then you might want to revisit premiere

There is a lot of "weirdness" going on with your results, I would even do some hardware checks like RAM, system, stress testing to rule out hardware issues

[Dogway]

If you can't get a baseline on what is lossless, and what isn't lossless, then it' s hard to test later on when it really matters . Once you have that sorted out, then you might want to revisit premiere

There is a lot of "weirdness" going on with your results, I would even do some hardware checks like RAM, system, stress testing to rule out hardware issues

You sound stressed man, I'm pretty sure it's your end what is wrong and the reason is very easy, do you think that a failed workflow would take two totally different videos and make them match 100%? clearly not. A failed workflow would create two different results, moreover, I just redid the test from scratch with your suggestions and it's fine here, both videos match.

New Tests 80Mb:
DV PAL Passthrough Premiere
Lagarith Premiere (this is glitched by Premiere, OP issue)
Lagarith VDub*

* this was encoded using the first chunk loaded in avs with ffms2 and in vdub, with the AVIFile input.

My system is very new 1 year only. And I wouldn't rely on ffmpeg, I have had nightmares before with it, it's a sack full of bugs.

[poisondeathray]

If you can't get a baseline on what is lossless, and what isn't lossless, then it' s hard to test later on when it really matters . Once you have that sorted out, then you might want to revisit premiere

There is a lot of "weirdness" going on with your results, I would even do some hardware checks like RAM, system, stress testing to rule out hardware issues

You sound stressed man, I'm pretty sure it's your end what is wrong and the reason is very easy, do you think that a failed workflow would take two totally different videos and make them match 100%? clearly not. A failed workflow would create two different results, moreover, I just redid the test from scratch with your suggestions and it's fine here, both videos match.

New Tests 80Mb:
DV PAL Passthrough Premiere
Lagarith Premiere (this is glitched by Premiere, OP issue)
Lagarith VDub*

* this was encoded using the first chunk loaded in avs with ffms2 and in vdub, with the AVIFile input.

My system is very new 1 year only. And I wouldn't rely on ffmpeg, I have had nightmares before with it, it's a sack full of bugs.

What do you think is "wrong" on this end?

Trust me, I've been doing this sort of thing for a long time.

Did you understand what my re-encode test demonstrated? It's taking whatever that "chunky.avi" is that I have here, even if there are errors, and converting it to lagarith, a lossless file. So even if there are errors in that file, they will be reproduced. That is the definition of lossless. Those results match . With multiple checks

You original ones didn't. Even your own check with ffmpeg showed this. Even subtact showed this on your computer. Middle grey is 128, not 137. Don't tell me 256/2 = 137 ? At the very least, something is wong with your avisynth install. Try subtract(a,a) on any video, and if you get 137 something is wrong. Hell , don't even use a video, use colorbars()

It doesn't matter if you dislike ffmpeg, it agrees with avisynth results and other testing as well. I'm not "relying" on it, it's just used as another level of checking. Do you disagree with something because you don't like what it says ? Just like avisynth is another level of checking , but shouldn't be relied on alone. 3 independent checks all agree you're telling me that's wrong ? Have someone else check the results if you don't believe me. I'm certain your original 3 files are not equivalent.

So if you repeated the tests, and everything matches (even with "bad" ffmpeg ), then everything is fine I guess...

[Cornucopia]

+1 on pdr's assessment.

Scott

[Dogway]

I wouldn't trust any program for granted, avisynth is full of bugs and inconsistencies, many old legacy functions haven't been removed just for the sake of backward compatibility, if you want to be exact this day you use masktools2. You don't use Levels() you use SmoothLevels(). Everything else there be dragons.

Do you know what Levels(127, 1, 129, 0, 255) is doing?

coring = true/false (true by default): When set to true: input luma is clamped to [16,235] (and the chroma to [16,240]), this result is *scaled* from [16,235] to [0,255], the conversion takes place according to the formula above, and output is *scaled* back from [0,255] to [16,235]. When set to false, the conversion takes place according to the formula above.

Good luck trying to reason that.

Trust me when I say programs are made by flawed developers, I've seen it all:

ffmpeg:
http://forum.doom9.org/showthread.php?p=1520192#post1520192
http://forum.doom9.org/showthread.php?p=1528626#post1528626
http://forum.doom9.org/showthread.php?t=159842 (problem was not addgrainC at all)

My painful findings:

ffms2
r.473 - seems all good (old good version)
r.533 - all good although a bit slower seeking than r.473
r.588 - "memory can't be read" random crashes
v.16onwards - random crashes on loading/seeking
r.683 - FFV1 decoding not working
r.722 - Indexing is bugged
r.725 - out of bounds memory, and no functions highlighting
http://forum.doom9.org/showthread.php?p=1520620#post1520620
http://forum.doom9.org/showthread.php?p=1520192#post1520192

And my last finding; latest ffms2 version ffms2_r936c59-avs_vsp matches your flawed output by 99% and obviously much less for my "correct" version. What this means is that your perfect decoder ffms2_r936+55-avs+vsp differs with ffms2_r936c59-avs_vsp by a 1% (by eye), now who are you going to trust, a decoder (my ffms2 version) that matches that of Premiere's, or 2 ffms2 versions in a row that don't even match between them?

Credit must be earned, don't give ffms2 devs too much of that.

[poisondeathray]

I wouldn't trust any program for granted, avisynth is full of bugs and inconsistencies, many old legacy functions haven't been removed just for the sake of backward compatibility, if you want to be exact this day you use masktools2. You don't use Levels() you use SmoothLevels(). Everything else there be dragons.

Do you know what Levels(127, 1, 129, 0, 255) is doing?

coring = true/false (true by default): When set to true: input luma is clamped to [16,235] (and the chroma to [16,240]), this result is *scaled* from [16,235] to [0,255], the conversion takes place according to the formula above, and output is *scaled* back from [0,255] to [16,235]. When set to false, the conversion takes place according to the formula above.

Good luck trying to reason that.

Do you understand why levels is even used here ? It's only for amplified differeces, in case there were mild differences, it can be difficult to see. It's not involved in the math to detect differences. It's only amplifing the differences. If it makes you feel better , use smoothlevels()

Trust me when I say programs are made by flawed developers, I've seen it all:

ffmpeg:
http://forum.doom9.org/showthread.php?p=1520192#post1520192
http://forum.doom9.org/showthread.php?p=1528626#post1528626
http://forum.doom9.org/showthread.php?t=159842 (problem was not addgrainC at all)

My painful findings:

ffms2
r.473 - seems all good (old good version)
r.533 - all good although a bit slower seeking than r.473
r.588 - "memory can't be read" random crashes
v.16onwards - random crashes on loading/seeking
r.683 - FFV1 decoding not working
r.722 - Indexing is bugged
r.725 - out of bounds memory, and no functions highlighting
http://forum.doom9.org/showthread.php?p=1520620#post1520620
http://forum.doom9.org/showthread.php?p=1520192#post1520192

And my last finding; latest ffms2 version ffms2_r936c59-avs_vsp matches your flawed output by 99% and obviously much less for my "correct" version. What this means is that your perfect decoder ffms2_r936+55-avs+vsp differs with ffms2_r936c59-avs_vsp by a 1% (by eye), now who are you going to trust, a decoder (my ffms2 version) that matches that of Premiere's, or 2 ffms2 versions in a row that don't even match between them?

Credit must be earned, don't give ffms2 devs too much of that.

I'm fully aware of bugs in ffmpeg and ffms2 and other programs. In case you haven't noticed - I have mild paranoia when it comes to workflows. I use multiple checks with multiple methods on multiple systems. Why did you think I didn't blindly trust your results ? Why do you think I offered another method of testing ? That's how I know these things that I'm sharing with you. I've been around the block more than once . Remember, I tested with "your" ffms2 version as well - same thing it doesn't match either

[Dogway]

For levels() to match smoothlevels() Photoshop levels() or any other kind of standard levels you should use coring=false. For your purpose I would use SmoothCurves to be honest.

I used to do hundreds of SSIM and PSNR tests back then until I found that the program you based all your findings had such blatant bugs that their developers wouldn't want to fix, do you think I am going to support such program? not on a political level, but on a bet my money level. I used the avs plugins more though.

Another point to note is that for a bug to be present it makes more sense to be omnipresent than only at times as in my case and in varying degrees which makes little sense. I know you tested my ffms2, you said it's still different, but what you haven't yet done is do a test by yourself as I did, trying to get same results from ffms2 and Premiere decoding, maybe they won't match in your system either and that will tell you you have a problem. It would be rare that in my case I have a bug that plagues both Premiere and ffms2, none the less I don't take things for granted either, so I'm going to boot into my XP partition with pretty much all different configuration, and do a brief check.

[Cornucopia]

You shouldn't be using Photoshop for these comparisons if they are YUV encoded, as Pshop does not have a YUV-colorspace. IOW, ALL processing done in Pshop is being converted (either at input, or your are converting to RGB when creating the still you are using to compare). This conversion is likely muddying the waters.

...If I misunderstood, it's because you mentioned Pshop above.

Scott

[poisondeathray]

For levels() to match smoothlevels() Photoshop levels() or any other kind of standard levels you should use coring=false. For your purpose I would use SmoothCurves to be honest.

Yes, I know that obviously, but you don't seem to understand WHY it's being used here.

Levels is not matching anything - it's not supposed to. It's used to amplify the differences, to see with your naked eyes. You can use another method of amplification if you want . This is standard procedure for difference testing in other programs so you can "see" with bare eyes

I used to do hundreds of SSIM and PSNR tests back then until I found that the program you based all your findings had such blatant bugs that their developers wouldn't want to fix, do you think I am going to support such program? not on a political level, but on a bet my money level. I used the avs plugins more though.

Another point to note is that for a bug to be present it makes more sense to be omnipresent than only at times as in my case and in varying degrees which makes little sense. I know you tested my ffms2, you said it's still different, but what you haven't yet done is do a test by yourself as I did, trying to get same results from ffms2 and Premiere decoding, maybe they won't match in your system either and that will tell you you have a problem. It would be rare that in my case I have a bug that plagues both Premiere and ffms2, none the less I don't take things for granted either, so I'm going to boot into my XP partition with pretty much all different configuration, and do a brief check.

Forget about bugs with open source ware just for a second. I've tested this with several "professional" programs and several decoders - same results. e.g. Drop those 2 files into after effects, there are differences

[Dogway]

I use photoshop for various reasons, it's easy to zoom in, pan, I can zoom much more, I can do comparisons of more than 2 sources with layers, and it's easy to compare and undo the comparison (F2 and F3 action respectively). It doesn't matter if the yuv to rgb conversion is crap, they are the same crap so testbed doesn't change, I'm only spotting differences.

I just tested on XP, and I get the same behaviours than on 7 x64. The old ffms2 is same as the one decoded by Premiere, and the more recent ones not. If we are over with the I'm wrong you are wrong, we can try to hunt for the bug, test some more ffms2 and avs versions and check what is happening. Tomorrow, today is a day for me.

I do understand enough to know you have better alternatives:

SmoothCurve(\
ycurve="0-0;127-0;129-255;255-255",\
interp=10,debug=false,limiter=false)

SmoothLevels(127, 1, 129, 0, 255)

mt_lutxy(a,b,"x y - abs 3 *",u=-128,v=-128)

mt_lutxy(a,b,"x y - 128 - abs ",u=-128,v=-128)

mt_makediff(a,b,U=128,v=128).AmplifyFunctionHere()

Good night.

[poisondeathray]

Maybe we need some fresh eyes / computers ? Maybe I have the same wonky bug that's affecting 2 different computers and multiple programs and decoder. (Hey, unlikely, but possible...)

I'll ask a few people to check the files for differences in post 30
https://forum.videohelp.com/threads/369232-Premiere-export-to-lossless-is-not-lossless?...=1#post2364932

Scott, if you have time, would you be able to check those for differences ? I'll ask a few others as well

[poisondeathray]

If we are over with the I'm wrong you are wrong, we can try to hunt for the bug, test some more ffms2 and avs versions and check what is happening. Tomorrow, today is a day for me.

Do you understand this IS hunting for the bug. If we cannot agree whether or not something is "lossless", that a BIG problem don't you think? How do you go on with the rest of the workflow and testing premiere ?

I do understand enough to know you have better alternatives:

SmoothCurve(\
ycurve="0-0;127-0;129-255;255-255",\
interp=10,debug=false,limiter=false)

SmoothLevels(127, 1, 129, 0, 255)

mt_lutxy(a,b,"x y - abs 3 *",u=-128,v=-128)

mt_lutxy(a,b,"x y - 128 - abs ",u=-128,v=-128)

mt_makediff(a,b,U=128,v=128).AmplifyFunctionHere()

Good night.

That's nice for usage on color correcting etc.. on footage , but not for testing. It's going to be slower.

Listen, if there is a difference, just amplify it .That's the gist of it. If they are bit for bit identical, no difference will be amplified

[jagabo]

I downloaded the two files in post # 30, opened them with ffVideoSource() and ConvertToRGB(matrix="Rec601", interlaced=true) in VirutalDubMod and exported frame 25 as PNG. I got the same images that poisondeathray in post #34. Ie, the two files are not identical. The differences are primarily in the Y channel (very very little in the U and V channels) so this is not a chroma downsampling/upsampling issue or a DV vs. MPEG chroma subsampling issue.

If I open chunky.avi in VirtualDub with Cedocida set to output YV12 only, and compress (fast recompress) with Lagarith in YV12 mode, the result is slightly different than when using AviSynth to open the AVI file instead. So there's some difference between VirtualDub's import of chunky.avi and AviSynth's (whether using AviSource() or ffVideoSource() which are identical). But it's not as big as the difference between chunky.avi and cacho01 vdub.avi. I've found no way to duplicate what's going on in cacho01 vdub.avi.

I haven't read the entire thread yet so I may be a little behind the times...

[poisondeathray]

Thanks for checking jagabo (and thanks to Scott as well for checking in PM, who also detected differences)

I'm confused as to how we are getting such different results, even when I switch and use the same ffms2.dll that dogway was using. Clearly his screenshots of frame 25 are identical

For reference, I'm using the most recent avisynth 2.6 alpha. I too checked cedocida in forced YV12 mode, as well as mainconcept's decoder

[Cornucopia]

I think the burden of proof ball is in his court right now. Plus, contrary to what he seems to imply, he has holes in his knowledge of YUV vs. RGB, whether decoders are built-in or plugins (and lack of trust of coders), DV/DVCam/DVCPro vs. DVCPro50/etc., P2 (which is SUPPOSED to be in MXF format, not the AVI that his copy somehow is), and 32kHz somehow being a proper default for DV.
So I'm taking all this with a big grain of salt.

Premiere has had no trouble working with solely SD DV material since the late '90s.

Scott

[poisondeathray]

He has "normal" PAL DV, the fourcc is dvsd . There is no question that the premiere export was b0rked and levels clamped, but let's leave that aside for now

We need to at least start with a common agreement that something is "lossless" or not - that was part of the original topic in the first place. If anything, the vdub version should have been lossless

dogway is no "newbie" - he's been around a long time at doom9/10, and contributes a lot with avisynth help / smdegrain

"weird" results can certainly plague buggy programs, but the chance of it affecting so many programs and 4 different computers, different people is low.

My money was on a corrupted download, but he said the hashes matched. So I'm still suspecting some system problem. I doesn't matter how "new" a computer is - hardware problems can occur anytime

[Dogway]

First and foremost, this is not a straight lossless discussion, this is a DV decoding discussion.
I'm very procedural when it comes to systems, I build my own OS images, I know what I have or what I'm missing. I know what I install and what is running in my system. Just in case, letting you know that I wasn't handed a PC yesterday.

For all the people getting differences let me please remind you this post of me above, I also do get different outputs if you decode with latest ffms2 builds.

Carrying on with my benchmark, I get no differences in vdub using either the AVIInput driver or the FFMpeg when loading avs scripts.

Getting different results in different programs is normal because they use different renderers, different color format conversion tables or even quantization matrices, I think I have to remind you that you are not going to get the same decoded output of a jpeg image when loaded in avisynth/vdub and in Photoshop because loose standards. This is what happens specially with old codecs. So you need to keep your testbed, in my case vdub.

What is good to know is that there is a pattern in my results (blue layers), I wouldn't say that a failed benchmarking manages to get so different decoders as ffdshow, LAV, Premiere and the old ffms2 to match.
The rest of the colored layers are very similar, and all of them except the new ffms2 builds only differ in chroma (as I said, different color conversion implementations, etc). For me this is a solid result to trust, I would rather trust four core program matches than a stranded ffms2 implementation of DV decoding.

The only mystery going on is why you get same results on all ffms2 builds.

Benchmarks in PSD (10Mb)