Quality Comparison of optimized MP4 videos

[avzabr]

Hello,
Sorry if i have raised the topic which already exists..I have checked the forum - but have not found the answer.

The question is the following:
I have the MSU VQMT tool which compares the quality of the original and the optimized videos by checking the same frames of each video. And i am faced with the problem of comparison mp4 videos with Variable Frame rate. I use the Avisynth script to open mp4 and i can see that program compares the wrong frames: frame of original videos with frame of optimized video, but several seconds later or prior. Therefore i get the wrong results of tool.

So the question is:
How to open the Original and Optimized variants of MP4 videos with Avisynth script to compare the right frames?
The following cases could be:
1) The VBR mp4 is optimized to the VBR mp4
2) The CRF mp4 is optimized to the CRF mp4
3) The VBR mp4 is optimized to the CRF mp4

I assume that for the 1st case files should be opened in the following way:
clip = DirectShowSource("D:\Qualitytest\Test\orig\00001-LUjn3RpkcKY.mp4", converfps=true)
return clip
But original and optimized frames are very shifted.

For the second case:
clip = DirectShowSource("D:\Qualitytest\MO2.3.0.2HO2\NewS cript\orig\00001-LUjn3RpkcKY.mp4")
framePerSecond=clip.FrameRate
clip1 = DirectShowSource("D:\Qualitytest\MO2.3.0.2HO2\NewS cript\orig\00001-LUjn3RpkcKY.mp4",fps = framePerSecond)
return clip1
The original and optimized frames are almost the same, but not exacly

For the 3rd case i even do not know what to do..

PS: I do not know in which way videos have been optimized. But need to compare the right frames and get the results..

Does anybody have any ideas?

[poisondeathray]

Usually it's not possible

A VFR encode of the same video will have fewer frames than the original version

Even if you tried to convert it to CFR, it will be different than the original (inserted duplicate frames won't be in exact location)

The following cases could be:
1) The VBR mp4 is optimized to the VBR mp4
2) The CRF mp4 is optimized to the CRF mp4
3) The VBR mp4 is optimized to the CRF mp4

I'm assuming you mean VFR and CFR, because there should be no problem with CRF and VBR is you use a frame accurate source filter (e.g. try ffms2 with seekmode=0)

[avzabr]

Sorry for mistake: i mean VFR and CFR... Variable Frame Rate and Constant Frame Rate
What is the best practice to open videos in cases 1-3? If it is not possible to compare exactly the right frames - how can i open videos to achieve almost the same pictures in the same frames (e.g. to get almost the same picture in frame 1000 of optimized and origin video).
It is not necessary to compare all frames of video (could truncate, add frames), but need to compare frames that contain the same picture.

Thanks,

[poisondeathray]

You would have check on a case by case basis

It depends how variable the video was, eg if there are a lot of frames missing. Sometimes if it is minimally variable, the correlation will be high

You can try converting to CFR (as you already have) with DirectShowSource and convertfps=true , but still not always reliable

Eitherway, PSNR, SSIM or similar metrics will be useless in those cases on a large scale where there is deviation . The plotted graph can still tell you where there are frame mismatches (there will be large jumps)

[avzabr]

If i have the 1st case:
Original video:
Frame rate mode : Variable
Frame rate : 30.000 fps
Minimum frame rate : 29.412 fps
Maximum frame rate : 30.303 fps
Optimized video:
Frame rate mode : Variable
Frame rate : 30.000 fps
Minimum frame rate : 29.412 fps
Maximum frame rate : 30.303 fps

When will i get the closest to real VQM value most likely:
a) Both videos: DirectShowSource("file", convertfps=true ) //just use the convertfps=true without the specifying of fps
b) Both videos: DirectShowSource("file", fps=30.000, convertfps=true ) // As i understand convert to average frame rate
c) Both videos: DirectShowSource("file", fps=29.412, convertfps=true ) // Convert to Minimum frame rate
d) Both videos: DirectShowSource("file", fps=30.303, convertfps=true ) // Convert to Maximum frame rate
e) Both videos: DirectShowSource("file", fps=119.88, convertfps=true ) // Convert to 119.88
f) Original video DirectShowSource("file", fps=<Optimzed video fps>, convertfps=true ) // Convert both videos to the average frame rate of optimized video
Optimized video DirectShowSource("file", fps=<Optimzed video fps>, convertfps=true )
g) Any other variant? =)

Thanks

[poisondeathray]

my guess would be (b) , but you would have to try all of them

These VQM numbers are not necessarily a good indicator of "quality" - there is often a significant deviation between subjective perceptions of "quality" and these metrics

[avzabr]

Thanks a lot for your help,

I will try with different avisynth script to find out what is the best practice and post the results.
If you get any other suggestions please let me know.