compare compression of two frames

[mathmax]

Hello

I have two different versions of a single video. The sources are streaming and the frames are randomly compressed... sometimes the frame of the clip 1 is better quality, sometimes the frame of clip 2 is better.

Having two versions of a single picture, is there an automatic method to detect which one is the less compressed?

Ideally, I wonder if it is possible to "deduce" a better quality picture from each couple of frame.. Maybe some areas are more or less well conserved in each frame and it's possible to combine them to get a higher quality picture..

[poisondeathray]

I have two different versions of a single video. The sources are streaming and the frames are randomly compressed... sometimes the frame of the clip 1 is better quality, sometimes the frame of clip 2 is better.

Having two versions of a single picture, is there an automatic method to detect which one is the less compressed?

Ideally, I wonder if it is possible to "deduce" a better quality picture from each couple of frame.. Maybe some areas are more or less well conserved in each frame and it's possible to combine them to get a higher quality picture..

Are they matched up ? Frames are identical timestamps ? (eg. frame 100 in video #1 @ time x = frame 100 in video #2 @ time x) or is it VFR with different time stamps ? Are there any frame drops during recording ?

Did they use the same recorder / encoding or compression method ?



Probably not ; because you don't have an "original" to compare with. Otherwise you can use a metric like PSNR or SSIM if the frames matched up

There is no way an automatic algorithm can detect what "looks better" because a lot of it is subjective . Even "objective" measures like PSNR or SSIM deviate from human perception

Less compressed or larger filesize doesn't always equate to "better quality" . If one used better compression for example, then the relationship won't hold. A smaller coded frame size for a specific maybe a lot better quality (e.g. x264 encoded b-frames are high quality and very tiny, but some other AVC encoders are far far worse despite being 3-4x the size)

So if one used different types of compression, different bitrates , different settings, prefiltering, etc.... I'm sure you get the idea - very difficult to compare "automatically"

[mathmax]

The same compression was used.. both videos were captured from straming and they are both VFR with different time stamps, so the frames don't necessarily match (some are identical but not compressed with the same strength... and some frames are only in one clip).
Both files are .mkv containing h.264 file.. In case the same algorithm of compression was used, can I deduce the quality directly from the weight of the image?

Otherwise you can use a metric like PSNR or SSIM if the frames matched up

I guess it's better than comparing the weight of the frames... how can I use these metrics in avisynth?

Also I would need a function that compares two frame and return a value indicating if the frames match. I guess it's difficult because I want to recognize two pictures that come from the same original frame but the compression differ..

[poisondeathray]

The same compression was used.. both videos were captured from straming and they are both VFR with different time stamps, so the frames don't necessarily match (some are identical but not compressed with the same strength... and some frames are only in one clip).

That's a big problem if they don't match . Even if you convert to CFR, they probably still won't match. (inserts are spaced differently)

Both files are .mkv containing h.264 file.. In case the same algorithm of compression was used, can I deduce the quality directly from the weight of the image?

Not necessarily. You have a better chance but it doesn't necessarily correlate exactly (higher coefficient of correlation, but not perfect)

Otherwise you can use a metric like PSNR or SSIM if the frames matched up

I guess it's better than comparing the weight of the frames... how can I use these metrics in avisynth?

You need something to compare to (the original) , which you probably don't have . They measure difference from original.

http://avisynth.org/mediawiki/Compare
http://forum.doom9.org/showthread.php?p=370635

MSU has a video quality measurement tool too MSU VQMT . But again, you need the original, also framecount has to be identical (you don't have that with offset VFR clips)

Also I would need a function that compares two frame and return a value indicating if the frames match. I guess it's difficult because I want to recognize two pictures that come from the same original frame but the compression differ..

I think this is possible, but you're going to get a lot of errors because of compression differences

Some of the mvtools derived functions or dupe detection scripts can be modified . Ask at Doom9

[mathmax]

That's a big problem if they don't match . Even if you convert to CFR, they probably still won't match. (inserts are spaced differently)

some frames match... of course I only want to compare the frames that match. The problem is to detect which frames match...
But I think I can know this through the timecodes, what do you think?

Assuming that the original file is 29.97 fps, that would mean about 32ms per frame. So in the timecodes, if I have:

0
73
100
..

I know that the second image will display for 73ms and then the next will display for 37ms, which means that from the frames number 1,2,3,4 of the original video, the frame number 2 was dropped.

if on the second timecode I have:

0
37
100
..

I can deduce the same way that the frame number 3 was dropped.

So of course I can't compare the frame number 2 and 3, but still the frame number 1 and 4 match and differ only by the compression strength..

You need something to compare to (the original) , which you probably don't have . They measure difference from original.

http://avisynth.org/mediawiki/Compare
http://forum.doom9.org/showthread.php?p=370635

MSU has a video quality measurement tool too MSU VQMT . But again, you need the original, also framecount has to be identical (you don't have that with offset VFR clips)

Is there no tools to measure the quality of compression between two compressed frame?

I think this is possible, but you're going to get a lot of errors because of compression differences

Some of the mvtools derived functions or dupe detection scripts can be modified . Ask at Doom9

These function will compare two frames and return a value that indicates how they match, right? Maybe I shouldn't work on the frame directly but apply a filter before.. For example, if the compression affect more the colors than the shapes, it would certainly be more relevant to work on Black & White picture or even on the edges only...

[mathmax]

Also maybe I could deduce the quality of compression from the timescodes..
The more a frame is displayed long, the more the connection was slow when it was broadcasted and I guess the more the image was compressed too. Do you think this is a good correlation?

[poisondeathray]

That's a big problem if they don't match . Even if you convert to CFR, they probably still won't match. (inserts are spaced differently)

some frames match... of course I only want to compare the frames that match. The problem is to detect which frames match...
But I think I can know this through the timecodes, what do you think?

Maybe, but some recorders not only drop frames, some insert frames (dupes). This adds other complications

So of course I can't compare the frame number 2 and 3, but still the frame number 1 and 4 match and differ only by the compression strength..

Even if you could match them up, you have no way of measuring quality. Coded frame size would be a decent correlation but far from perfect.

Is there no tools to measure the quality of compression between two compressed frame?

No , it's impossible. Even if have the original, the measures of "quality" such as SSIM and PSNR are not accurate. They are rough correlations

You can measure the size or relative compression, but quality is impossible to measure accurately, except with your eyes

I think this is possible, but you're going to get a lot of errors because of compression differences

Some of the mvtools derived functions or dupe detection scripts can be modified . Ask at Doom9

These function will compare two frames and return a value that indicates how they match, right? Maybe I shouldn't work on the frame directly but apply a filter before.. For example, if the compression affect more the colors than the shapes, it would certainly be more relevant to work on Black & White picture or even on the edges only...[/QUOTE]

Not sure, try it out or ask at D9

Also maybe I could deduce the quality of compression from the timescodes..
The more a frame is displayed long, the more the connection was slow when it was broadcasted and I guess the more the image was compressed too. Do you think this is a good correlation?

I doubt it, because timecodes might be independent of how it was encoded.

For example, you might expect a frame displayed longer to be higher quality, because higher average bitrate over time. But it depends what method of rate control and what settings the encoder was using. New frames require more bitrate (coding the difference between frames), therefore lowering the average quality

There are too many variables. I don't think you will get a satisfactory "automatic" method.

[mathmax]

Maybe, but some recorders not only drop frames, some insert frames (dupes). This adds other complications

That means exact same frames are duplicated, right? In this case, I should be able to locate them...

Even if you could match them up, you have no way of measuring quality. Coded frame size would be a decent correlation but far from perfect.

if it's a decent correlation, it might be enough.. I'm not asking perfection after all. But how to compare the frame size in a .264 file? Should I extract the frames? How can I do this?

I doubt it, because timecodes might be independent of how it was encoded.

For example, you might expect a frame displayed longer to be higher quality, because higher average bitrate over time. But it depends what method of rate control and what settings the encoder was using. New frames require more bitrate (coding the difference between frames), therefore lowering the average quality

There are too many variables. I don't think you will get a satisfactory "automatic" method.

I thought it was encoded on the fly and the timecodes were created at the same time.. I thought that that was directly correlated with the bandwidth while broadcasting.. which means the frames should be more compressed and the framerate should be more low when the connection is not good. So I hoped to deduce the quality from the timecodes this way... of course maybe not on a single frame but at least on a set of frames..
For example if 10 frames are displayed in 1 sec
and then 20 frames are displayed in 1 sec, probably the 20 frames are better quality than the 10 first frames, aren't they?

[poisondeathray]

if it's a decent correlation, it might be enough.. I'm not asking perfection after all. But how to compare the frame size in a .264 file? Should I extract the frames? How can I do this?

You can use a stream analyzer, like streameye, h264visa

A free way to do this is ffdshow's OSD, where you can checkmark framenumber, frametype, coded frame size, timestamps etc... and output to a .csv file

I thought it was encoded on the fly and the timecodes were created at the same time.. I thought that that was directly correlated with the bandwidth while broadcasting.. which means the frames should be more compressed and the framerate should be more low when the connection is not good. So I hoped to deduce the quality from the timecodes this way... of course maybe not on a single frame but at least on a set of frames..

Clarify what you have. Is it a download of the direct transmission or a stream recording ? (capped and then re-encoded)

For example if 10 frames are displayed in 1 sec
and then 20 frames are displayed in 1 sec, probably the 20 frames are better quality than the 10 first frames, aren't they?

No, it depends on the rate control and method of encoding used, encoder used, settings used many variables

If that 1 second was allocated the same bitrate, then chances are the 10 frames would be higher quality. (Think of it as each frame in the 20 frame video gets less bitrate because it's spread out thinner) . If that 1 second was allocated different bitrates, you cannot tell

[mathmax]

You can use a stream analyzer, like streameye, h264visa

A free way to do this is ffdshow's OSD, where you can checkmark framenumber, frametype, coded frame size, timestamps etc... and output to a .csv file

Looks good, I have to try that.. and check it is the size comparison is relevant in my case. Thank you for the advice

Clarify what you have. Is it a download of the direct transmission or a stream recording ? (capped and then re-encoded)

it was caught on ustream with a software... but I think the software caught directly the .flv sent to the flash player on the client side. And the software used to encode and broadcast on the server side was ustream producer

No, it depends on the rate control and method of encoding used, encoder used, settings used many variables

If that 1 second was allocated the same bitrate, then chances are the 10 frames would be higher quality. (Think of it as each frame in the 20 frame video gets less bitrate because it's spread out thinner) . If that 1 second was allocated different bitrates, you cannot tell

Seems logical.. but I wonder why the encoder decides to modulate the framerate? Would it be according to the motion (less motion = less frames needed)? I thought it would be only according to the bandwidth and if the bandwidth suddenly got thiner, the encoder would decide both to compress more and decrease the framerate..

[poisondeathray]

If this was the same broadcast, but 2 different files, then it sounds like it was capped differently from rtsp stream (not downloaded from the source, like a progressive http stream which would be identical to original file quality)

I'm not sure what the specifics are on how ustream works

I wonder why the encoder decides to modulate the framerate? Would it be according to the motion (less motion = less frames needed)?

Yes, usually a stream recorder will drop frames if it can't keep up (it can be transmission side, or receiving side issue in terms of bandwidth, or CPU usage) . And yes, less motion will require fewer frames

[mathmax]

If this was the same broadcast, but 2 different files, then it sounds like it was capped differently (not downloaded from the source, which would be identical to original file quality)

no... the same footage was broadcasted at two different times.. so I have two recorded files.. that's why I can eliminate the "motion factor" if I take into account the two timecodes...

So if a part of the timecode1 as more frames than the corresponding part of the timecode 2, that would mean the bandwith was better on the first broadcast... and thus, there might be chance that the frames are less compressed to..

[mathmax]

I allow myself to move up the thread, in case you didn't see my last post poisondeathray

[poisondeathray]

So if a part of the timecode1 as more frames than the corresponding part of the timecode 2, that would mean the bandwith was better on the first broadcast... and thus, there might be chance that the frames are less compressed to..

Not necessarily ; it depends on how it was recorded (bitrate especially), settings used etc...

If both have the same average bitrate, but one has more frames, then the average quality of the frames should be lower, if everything else is the same (that's the problem however, rarely is "everything else the same")

[mathmax]

So if a part of the timecode1 as more frames than the corresponding part of the timecode 2, that would mean the bandwith was better on the first broadcast... and thus, there might be chance that the frames are less compressed to..

Not necessarily ; it depends on how it was recorded (bitrate especially), settings used etc...

If both have the same average bitrate, but one has more frames, then the average quality of the frames should be lower, if everything else is the same (that's the problem however, rarely is "everything else the same")

mmm... for one broadcast every clients get the same stream (same framerate and same compression)... the way the file is encoded only depends on the bandwidth on the server side, right?
And on the server side, the same software was used, with the same settings...

[poisondeathray]

mmm... for one broadcast every clients get the same stream (same framerate and same compression)... the way the file is encoded only depends on the bandwidth on the server side, right?
And on the server side, the same software was used, with the same settings...

But the client side determines the final quality. This isn't a HTTP progressive download. It sounds like RTSP stream

So if the client uses a lossless format, you get better quality. If you use lossy compression, lower quality . If you use higher bitrate, you get higher quality. You're recording the stream with a stream recorder (re-encoding it) , not downloading it via HTTP. That's why the settings matter

If it was a HTTP progressive download, every copy would be identical as the original upload, with no dropped frame, no duplicate frames (assuming the uploaded version didn't have any)

[mathmax]

I see.. I didn't know well about that..
I thought that the flv caught by the stream recorder was the one directly sent to the client and read by the flash player... but if it's reencoded in real time on the client side, of course the problem is more tricky...

And now, what about the inserted frames (dupes).. are they the exact copy of the preceding frame with same compression?

[poisondeathray]

I thought that the flv caught by the stream recorder was the one directly sent to the client and read by the flash player... but if it's reencoded in real time on the client side, of course the problem is more tricky...

That's what most "stream recorders" do. They re-encode the video . So if you have bandwidth issue (up or down), or cpu issue, then you drop frames - that's probably why you have different versions

e.g. but if you download a normal youtube video (using http) , it's identical to the file on youtube right ? That's HTTP download, not a stream recording

And now, what about the inserted frames (dupes).. are they the exact copy of the preceding frame with same compression?

No - they are usually slightly different because encoder encodes difference between frames with long gop compression, but isn't perfect . You can use difference masks to determine if they are exact copy

[mathmax]

A free way to do this is ffdshow's OSD, where you can checkmark framenumber, frametype, coded frame size, timestamps etc... and output to a .csv file

I couldn't understand how to use ffdshow to get the size of the frames in a csv..
First, I'm not sure whether ffdshow's OSD is ffdshow (the decoder).. is it another complementary tool?
I went in the configuration of the video decoder DXVA.. I could see that some infos can be displayed on the screen, but I couldn't find the "frame size" in the list.. on the same tab, there is indeed an option to save as CSV. But I don't know if I'm at the right place nor if I'm using the right tool...

You can use difference masks to determine if they are exact copy

Using the overlay() function with the "Difference" mode in avisynth, you mean? And then, how do I get a value that indicates how much they match?

[poisondeathray]

ffdshow the decoder

1) checkmark osd in the left column to enable it

2) checkmark the options you want eg. frame number, coded frame size, etc... The option you want is "coded frame size"

3) checkmark "save to" box, and specify the path where you want the csv file

4) push apply & ok

5) play the video in directshow player like mpc , that has ffdshow configured to decode that particular video

You can use difference masks to determine if they are exact copy

Using the overlay() function with the "Difference" mode in avisynth, you mean? And then, how do I get a value that indicates how much they match?

I mean visually using masks, not numerical value ; but you might be able to write a function that returns a value with masktools, not sure - ask at doom9

You can use one of the dupe detection functions in avisynth if you want locate them

[mathmax]

Sorry for my late answer.. I could only work on the project tonight.

ffdshow the decoder
1) checkmark osd in the left column to enable it

2) checkmark the options you want eg. frame number, coded frame size, etc... The option you want is "coded frame size"

3) checkmark "save to" box, and specify the path where you want the csv file

4) push apply & ok

5) play the video in directshow player like mpc , that has ffdshow configured to decode that particular video

That works perfectly Just a question: what is "Frame mean quantizer"? It seems that its value is inversely proportional to the frame size... I wonder why..

I could find some avisynth filter to detect the dups, thank you

[poisondeathray]

That's correct, lower quantizer means higher bitrate

frame mean quantizer is just the average for the frame

It's actually not that important, because h264 uses quantizers on a macroblock basis, not frame basis

For what you are doing, "coded frame size" is probably the closest indicator if they were recorded using similar parameters