Premiere export to lossless is not lossless

[poisondeathray]

Ok now we are getting somewhere with systematic testing

While it's true that different decoders may decoder differently (in theory they are not supposed to, but it happens), e.g. different chroma treatment mpeg2 vs. dv placement. Left aligned vs. center etc.. But the exact same decoder, same revision should decode the same way, shouldn't it ? I still can't explain different results with the same decoder, the same ffms2.dll that you supplied. I uploaded it below

Getting different results in different programs is normal because they use different renderers, different color format conversion tables or even quantization matrices, I think I have to remind you that you are not going to get the same decoded output of a jpeg image when loaded in avisynth/vdub and in Photoshop because loose standards. This is what happens specially with old codecs. So you need to keep your testbed, in my case vdub.

Exactly. You're unnecessarily complicating things by converting to RGB, and using different renderers. This introduces issues such as upsampling method, chroma placement, matrix differences. Lossless test should test the raw yuv output from the decoder directly. It's the KISS principle. At minimum if you 're going to convert to RGB, the same upsampling method, same method of converting to RGB, same screenshot method should be used. If you take a screenshot in VLC, it's not going to look the same as vdub.

The only mystery going on is why you get same results on all ffms2 builds.

I don't get the same results on all ffms2 builds (and I didn't test them all). I get the same results using the same build, not inconsistent results using the same build. Big difference

Nor can I explain how subtract gives you 137 ?

Another way of saying all this - I cannot replicate your results. Something buggy is going on, and I think it's your setup unless you have better explanation ?

And try not to work too hard ... It's Happy New Year time

[Dogway]

Exactly. You're unnecessarily complicating things by converting to RGB, and using different renderers.

If I am going to say that different programs/renderers yield different results I need to prove that, those apparently "unrelated" tests are thrown there to precisely prove that they are non-relevant due to using different renderers, would you believe me if I only mentioned it without proofs?

RGB conversion is not explicitly declared in any of my tests, it's implicitly done in the video driver, everything you see in your monitor is RGB, my video driver also converts that to my monitor profile. The avs scripts only contain ffvideosource("path").

I don't get the same results on all ffms2 builds

And that sir is very fishy, which one is correct then? you need to compare one against each other and more importantly against other decoders, then unless you are a mathematician or programmer the most we can do is keep those ones that are more consistent with results. I'm done with tests (your/my uploaded ffms2 is the same btw), but you can run some if you want, I will do some editing since DV output from Premiere seems fine, in that regard OP is solved. Don't worry for work, I'm a workaholic plus it's a Xmas present :)

Nor can I explain how subtract gives you 137 ?

Another way of saying all this - I cannot replicate your results. Something buggy is going on, and I think it's your setup unless you have better explanation ?

You keep talking about subtract over and over in a way to discredit me and make your argument true that my system is bugged.
Read documentation:

About offset of luma range:

For YUV formats the valid Y range is from 16 to 235 inclusive and subtract takes this into account. This means that the following script

Subtract(any_clip, any_clip)

will result in a grey clip with luma = 126. For those that require a subtract function for pc_range YUV data use Overlay:

#Overlay(any_clip, any_clip, mode="Difference", pc_range=true) # grey clip with luma = 128
Overlay(clip1, clip2, mode="Difference", pc_range=true)

So you shouldn't expect a base of 128 neither using subtract() nor after using Levels(127, 1, 129, 0, 255).

subtract(a,a) -> base of 126
after levels() -> 126 goes to 134
to PC levels RGB -> 134 converts to 137

if you want a base of 128, use the overlay workaround above, or as I prefer:

mt_lutxy(a,a," x y - 128 + abs ",u=-128,v=-128)

and to pump differences without losing 128 as a base (levels() with coring will raise it to 218, test it), add:

Levels(127, 1, 129, 0, 255, coring=false)
or as I prefer:
SmoothLevels(127, 1, 129, 0, 255,dither=-1)

[poisondeathray]

I don't get the same results on all ffms2 builds

And that sir is very fishy, which one is correct then? you need to compare one against each other and more importantly against other decoders, then unless you are a mathematician or programmer the most we can do is keep those ones that are more consistent with results. I'm done with tests (your/my uploaded ffms2 is the same btw), but you can run some if you want, I will do some editing since DV output from Premiere seems fine, in that regard OP is solved. Don't worry for work, I'm a workaholic plus it's a Xmas present

You only quoted part of the sentence - To clarify - What I'm saying is I didn't test them all, so I cannot conclude that I get the same results or not . I only get the same results on the same ffms2 build, which was different than what you were getting. I would have expected to get the same results as you with the same software/decoder etc....

You keep talking about subtract over and over in a way to discredit me and make your argument true that my system is bugged.

Where did you get that idea? I'm not trying to "discredit" you - I'm trying to make sense of the inconsistencies in these observations.

Read documentation:

About offset of luma range:

For YUV formats the valid Y range is from 16 to 235 inclusive and subtract takes this into account. This means that the following script

Subtract(any_clip, any_clip)

will result in a grey clip with luma = 126. For those that require a subtract function for pc_range YUV data use Overlay:

#Overlay(any_clip, any_clip, mode="Difference", pc_range=true) # grey clip with luma = 128
Overlay(clip1, clip2, mode="Difference", pc_range=true)

So you shouldn't expect a base of 128 neither using subtract() nor after using Levels(127, 1, 129, 0, 255).

subtract(a,a) -> base of 126
after levels() -> 126 goes to 134
to PC levels RGB -> 134 converts to 137

if you want a base of 128, use the overlay workaround above, or as I prefer:

mt_lutxy(a,a," x y - 128 + abs ",u=-128,v=-128)

and to pump differences without losing 128 as a base (levels() with coring will raise it to 218, test it), add:

Levels(127, 1, 129, 0, 255, coring=false)
or as I prefer:
SmoothLevels(127, 1, 129, 0, 255,dither=-1)

Thanks for the explanation about the mysterious "134" and "137". I was wrong .

And Subtract(a,a) does indeed return Y=126, Cb=128, Cr=128 , even on a full range clip, or limited range clip

mt_lutxy(a,a," x y - 128 + abs ",u=-128,v=-128)

This is for Y only, correct?

Do you happen to have one for Y, U, V ?

While using Overlay() above would work, I recall gavino or someone saying it was much less efficient than using masktools to do it

[Dogway]

You only quoted part of the sentence - To clarify - What I'm saying is I didn't test them all, so I cannot conclude that I get the same results or not . I only get the same results on the same ffms2 build, which was different than what you were getting. I would have expected to get the same results as you with the same software/decoder etc....

We are way past lossless-DV comparison. We are debugging only DV decoding in ffms2.
I get different results decoding DV sources across different ffms2 builds. That's the point and what is really fishy. I thought we came to terms that Lagarith decoding is not a problem (we get same results as tested with the screenshots).

Interesting, but you don't have a limited range clip (recall you had overbrights that the premiere encoded version clamped)

Both Subtract(a,a) and Overlay(a,a, mode="Difference", pc_range=true) should return 128 , shouldn't it ? It certainly does here

Analog inputs usually record some overshoot in the brights and even on the darks as a kind of cushion but that doesn't make it PC levels, and it's unrelated in this example.
colorbars(pixel_type="yv12")
subtract(last,last)
this will give you 126 grey, if you use RGB colorbars you will get 128 though. Subtract doesn't understand PC Levels YUV so no matter how you put it, as documentation states using subtract always outputs 126 grey for YUV sources.

[poisondeathray]

I made some edits before your reply, sorry. Can you answer the last one about a modified masktools script ? Thanks

To be clear, with subtract in YUV, I can confirm it is Y=126,U=128,V=128 with both PC or Rec levels, and it makes sense with that explanation

ie. both these return the same thing

a=colorbars().converttoyv12(matrix="pc.601")
subtract(a,a)

a=colorbars().converttoyv12(matrix="rec601")
subtract(a,a)

[Dogway]

mt_ stands for multithreading, so it should be faster, more optimized.

with chroma you use:
mt_lutxy(a,b," x y - 128 + abs ",u=3,v=3)
the problem is that chroma changes are very faint and obscured by the more prominent luma differences.

I recommend you one script I posted on first page, convert chroma to luma, append to video, then compare.

a=stackhorizontal(a,stackvertical(Utoy(a),Vtoy(a)))
b=stackhorizontal(b,stackvertical(Utoy(b),Vtoy(b)))

mt_lutxy(a,b," x y - 128 + abs ",u=-128,v=-128)

or the faster

mt_makediff(a,b,U=128,v=128)

[poisondeathray]

Ok thanks. Sometimes you might want to look at U, or V separately and I suppose you could use UtoY, VtoY, it' s nice to have a few options