? Objective Video Quality Comparison App

[Synetech]

Hi,

I’m trying to compress the videos from my digital camera with XviD because they take up way too much space in the camera’s native MJPEG format.

What I need is a way to compare the quality of the compressed video to that of the original. I am looking for an app that can compare the two vids and give me some sort of easy to use and understand rating of whether or not the compressed one is good enough.

No, I can’t just look at them both and do an A/B(/X) comparison because it would take forever. I need a way of automating the process (objective comparison), and if possible I would like the app to have a batch compare function.


I have only been able to find three such apps: MSU’s Video Quality Measurement Tool, Georgios Diamantopoulos’ Video Quality Studio, and Semaca’s VQLab.

VQMT seems pretty good and has a bunch of different metrics, but it doesn’t seem to display the results after the comparison and I can’t find any useful information on interpreting the CSV file.

VQS crashes outright and I can’t get it to work at all.

VQLab is a commercial app, seems to only support a couple of metrics, and it too has no helpful information on interpreting its results.


Does anyone know of a good object video quality compare app?


Thanks a lot.

[jagabo]

Use constant quality encoding (aka constant quantizer, target quantizer, etc). Run a few experiments at different quality levels to determine what you find acceptable. Use that setting for all your encoding. Every video you encode will have the same quality (relative to the source).

With Xvid I usually use a Target Quantizer of 3. 2 for really critical material.

[Synetech]

Thanks for the suggestion, but I am trying to find a way to objectively compare the original and the encoded videos. Like I said, I don’t want to do an subjective A/B(/X) comparison.

[45tripp]

i think jagabo is saying you'll reach your objective faster if you forget the objective

anyway,
you seem to have mentioned most tools.
another one is Elecard's streameye toolset, to get metric comparisons.

and avisynth.
there's an ssim and psnr plug.


tripp

[jagabo]

Unfortunately, when you measure something with many dimensions no single number will tell you all there is to know. This is especially true with something as subjective as video. Here's an example:

Say I develop a new compression codec. After decompression every pixel is exactly the same as the source except they are all darker by 1 unit (out of 255). Now I develop another codec. After decompression every pixel is exactly the same except one pixel right at the center of the frame flickers between full black and full white.

Say we use a video quality measurement program that simply sums up all the errors per frame. Say the frame size is 720x480. The first video gets a error rating of 345,600 (345,600 pixels, off by one each). The second video only has one pixel that's off by 255 at most. So the program gives the video an error rating of 255.

In practice nobody will see any difference between the source video and the first compressed video. But the second video will have this blinking dot right in the middle of the picture. Most people will say the second video is worse than the first, even though the "objective" rating is far better.

If you're developing your own encoder, or comparing several different encoders, having some kind of quick metric helps but there is no substitute for visual inspection.

Regarding how long it will take you to figure out what quantizer or quality setting you want to use: It won't take long at all. Encode a few short difficult clips at a few different quantizers. Look at the results as enlarged still frames and at normal playback speed on your target playback device. You don't have to do this for every video you encode. This is constant quality encoding -- everything you encode will have the same quality (relative to the source) regardless of frame size, frame rate, or the amount of detail or motion in the frames. Once you've discovered what quality is acceptable for a particular encoder you can always use that same setting.

[Synetech]

you seem to have mentioned most tools.
another one is Elecard's streameye toolset, to get metric comparisons.

and avisynth.
there's an ssim and psnr plug.

That’s the problem. There seem to be very few. I’ll check out StreamEye. Oh, and I’ve got several AviSynth scripts to do various types of comparisons. Of course I need to find a way of making the results easy to interpret.

Unfortunately, when you measure something with many dimensions no single number will tell you all there is to know. This is especially true with something as subjective as video.

That’s why I’m looking for a good tool that can perform various analyses on a video and give a rating.

Regarding how long it will take you to figure out what quantizer or quality setting you want to use: It won't take long at all. Encode a few short difficult clips at a few different quantizers. Look at the results as enlarged still frames and at normal playback speed on your target playback device. You don't have to do this for every video you encode. This is constant quality encoding -- everything you encode will have the same quality (relative to the source) regardless of frame size, frame rate, or the amount of detail or motion in the frames. Once you've discovered what quality is acceptable for a particular encoder you can always use that same setting.

That sounds feasible; I’ll try that out. I usually use XviD’s default of Q4, and sometimes up it to Q3 for things that I cannot reproduce.

The thing is that even with Q0, the videos are discernibly different. For example I have a video with grass and stuff in it, and the compressed one is clearly more smoothed out.

[jagabo]

[quote="Synetech"]

That sounds feasible; I’ll try that out. I usually use XviD’s default of Q4, and sometimes up it to Q3 for things that I cannot reproduce.

I use Q=3 for most video, Q=2 for critical things.

The thing is that even with Q0, the videos are discernibly different. For example I have a video with grass and stuff in it, and the compressed one is clearly more smoothed out.

Q=1 is the lowest you can go. And no VBR or CBR encode will ever get better than that. All VBR and CBR do is vary the Q for each frame to meet the requested bitrate. The lowest Q they can use is 1. So a CQ encode at Q=1 (and don't use B frames as discussed earlier) is the best Xvid will ever get.

[Synetech]

What I meant by Q0 was that in the XviD dialog you can enter zero and it will take it and remember it. I assume that it just treats it like Q1 during encodes, but it does show zero the next time that you open the config dialog.

[jagabo]

What I meant by Q0 was that in the XviD dialog you can enter zero and it will take it and remember it.

I never noticed that before. I just ran a test, encoded a Q=0 and Q=1. The resulting files were bit-for-bit identical.

[manono]

The thing is that even with Q0, the videos are discernibly different. For example I have a video with grass and stuff in it, and the compressed one is clearly more smoothed out.

I hope you don't have any post-processing - deblocking and the like - turned on. And I hope you're using an AviSynth script to frameserve into VDub(Mod), and are encoding using Fast Recompress. If you're going to get particular about how they look, you may as well learn how to create the XviDs properly.

Try a decent quantisation matrix and you can make them transparent to the source - you won't be able to tell the difference. Both of the 2 commonly used matrices - MPEG and H.263 - have their drawbacks.

[mpack]

Hi,
What I need is a way to compare the quality of the compressed video to that of the original. I am looking for an app that can compare the two vids and give me some sort of easy to use and understand rating of whether or not the compressed one is good enough.

Video quality is notoriously subjective / difficult to quantify. One common measure is MSE (mean squared error), which as it sounds is the average squared difference between before and after. However this fails to capture what human beings perceive as significant errors, eg. as someone else points out, "noisy" features like grass and hair get smoothed without affecting the average much. Another common failure is loss of detail around the eyes on a human face, making the face look weird or make it hard to read expression: average measures don't take into account the higher importance of relatively small features in the image.

[jagabo]

The thing is that even with Q0, the videos are discernibly different.

If you look closely enough, all lossy codecs produce discernibly different results. If you want a video that's not discernibly different, use a lossless codec like Lagarith or HuffYUV. Of course, it makes no sense to do this with MJPEG files. All you'll do is make bigger files.