Cannot compare lossless with MPEG-2 encodes: Frames aren't lining up!

[Mini-Me]

I have some Huffyuv captures of home movies that I'm converting to MPEG-2 and x264 for DVD and computer files, respectively. I haven't gotten to the x264 encodes yet, but I'm using HCEnc for my MPEG-2 encodes. Since I'm new to encoding, I decided to try comparing my MPEG-2 encodes with the lossless captures to see what I'm losing, whether I need to tweak settings, etc. The problem is, I cannot get the frames to line up for a proper comparison.

For instance, if I use the following Avisynth script to flick between frames, the files drift out of sync and make comparisons useless, because they're comparing totally different frames (sometimes up to a couple seconds apart):

lossless_name = "path.avi"
encode_name = "path.mpg"
lossless = AviSource(lossless_name)
encode = ConvertToYUY2(DirectShowSource(encode_name))
return Interleave(lossless.Subtitle(lossless_name), encode.Subtitle(encode_name))

I looked online, and apparently DirectShow isn't frame-accurate...but Haali's replacement, DSS2, is supposed to be frame-accurate. So, I tried that:

LoadPlugin("C:\Program Files\Haali\MatroskaSplitter\avss.dll")
lossless_name = "path.avi"
encode_name = "path.mpg"
lossless = DSS2(lossless_name)
encode = ConvertToYUY2(DSS2(encode_name))
return Interleave(lossless.Subtitle(lossless_name), encode.Subtitle(encode_name))

The problem is, even this didn't work! I've also tried stacking the clips side-by-side, with the differences and magnified differences underneath, but the same out-of-sync issue still invalidates the comparison. It also shows up in the MSU Video Quality Measurement Tool, making objective metrics just as useless.

I double-checked the files with Gspot, and they do have the same framerate and number of frames, so it's not like HCEnc totally screwed up or anything. I've even tried multiple MPEG-2 decoders with no difference on the synchronization issues. (That said getting rid of my nVidia Purevideo codec and making DirectShow use ffdshow/libavcodec did help with something else: The Purevideo decoder was doing some deinterlacing that I couldn't disable, which also threw off comparisons, and for some reason it softened the image as well.)

I should mention that I do NOT get this problem when I compare two .avi files with each other...only when one of them is an .mpg, and presumably only when some form of DirectShow is being used. So, does anyone know what's going on and how to fix it?

Update: I just discovered the FFmpegSource project and the FFVideoSource loader, and it helps a LOT playing straight through. The video still get out of sync if I seek, but otherwise the issue seems greatly minimized. The MSU Video Quality Measurement Tool can't use it AFAIK, but at least the Avisynth comparisons are more useful.

[manono]

Me, I'd make a D2V project file on the MPEG-2 video and then use MPEG2Source on it

[Mini-Me]

Awesome tip, manono! I just got DGMPGDec (what a mouthful) and did exactly as you said. Using MPEG2Source makes all of my Avisynth comparisons work perfectly, even if I seek randomly. It's a somewhat clumsy process, but it works. So far, I can't even get it to mess up if I try. You're a lifesaver!

Scratch this part:
[The only area I'm left wanting is in getting objective metrics. Even SSIM is imperfect, but it's nice for a second opinion about what scenes might be worth inspecting more closely by eye. By any chance, would you know if there would be any hacky way to trick MSU Video Quality Measurement Tool into recognizing and loading the d2v?]

EDIT: I'm a damn idiot. MSU VQMT uses automatically generated Avisynth scripts to load in MPEG files. Those scripts result in frame inaccuracy, and I was fretting about the program not letting me customize the script to load the video file with...but since the program uses Avisynth, I should have known that I could just load in an arbitrary avs script as the video file itself. Hah. Basically, your post just fixed all of my problems with this.

Now that I've gotten a chance to properly compare my encode to my source, I have to say that I'm quite impressed with HCEnc's performance, too.

[jagabo]

You can also try something like:

# If Videos start at different frames use frameadjust to align them
frameadjust=0
name1="video1.avi"
name2="video2.avi"

v1 = AviSource(name1)
v2 = AviSource(name2).trim(frameadjust,0)
sub = v1.subtract(v2)
substrong = sub.levels(112,1,144,0,255)

StackVertical(StackHorizontal(v1.subtitle(name1),v 2.subtitle(name2)),StackHorizontal(sub.subtitle("D ifference"),substrong.subtitle("Difference amplified 8x")))

That subtracts one image from the other and shows the differences, and the differences amplified. Obviously, you'll have to change the AviSource() to Mpeg2Source().

[El Heggunte]

<SNIP>
and apparently DirectShow isn't frame-accurate...but Haali's replacement, DSS2, is supposed to be frame-accurate.
<SNIP>

Generally speaking, that should be regarded as "anti-MS trolling"

IF the used splitter and/or decoder is not decent enough,
surely *DirectShowSource + convertfps=true* will not be frame-accurate.

OTOH, often DSS2 will either drop or duplicate the last frame of a
video sequence, and possibly even Haali doesn't know why.

[poisondeathray]

Update: I just discovered the FFmpegSource project and the FFVideoSource loader, and it helps a LOT playing straight through. The video still get out of sync if I seek, but otherwise the issue seems greatly minimized. The MSU Video Quality Measurement Tool can't use it AFAIK, but at least the Avisynth comparisons are more useful.

MSU can use it or any avs script

Use seekmode=0 for frame accurate seeks (but slower)

If DSS2() is missing the last frame you can align the clips by using Trim()

You can use PSNR and SSIM in avisynth as well (it has plugins)


You should do some reasearch on what PSNR and SSIM really mean in terms of "quality". There is a lot of discussion on this topic. I wouldn't blindly use it and assume quality is better if you score higher. They are objective measurements, but often do not correlate with human percieved quality, especially when encoders use psy optimizations. For example you can tune some encoders to give higher PSNR and SSIM scores, but the image quality to the human eye is actually worse.

[Mini-Me]

You can also try something like:

# If Videos start at different frames use frameadjust to align them
frameadjust=0
name1="video1.avi"
name2="video2.avi"

v1 = AviSource(name1)
v2 = AviSource(name2).trim(frameadjust,0)
sub = v1.subtract(v2)
substrong = sub.levels(112,1,144,0,255)

StackVertical(StackHorizontal(v1.subtitle(name1),v 2.subtitle(name2)),StackHorizontal(sub.subtitle("D ifference"),substrong.subtitle("Difference amplified 8x")))

That subtracts one image from the other and shows the differences, and the differences amplified. Obviously, you'll have to change the AviSource() to Mpeg2Source().

Yup, I do that too. That's what I meant in the OP when I said, "I've also tried stacking the clips side-by-side, with the differences and magnified differences underneath,..." I originally hacked together a less useful version of my own, but I stumbled upon yours that included the magnified difference, and I've been using that ever since (along with interleaving and metrics). It was frustrating though when even the regular difference showed an "embossed" version of the entire picture, since everything was a few frames off. Now, that filter is the first one I'm using for getting a broad idea of differences!

[Mini-Me]

Update: I just discovered the FFmpegSource project and the FFVideoSource loader, and it helps a LOT playing straight through. The video still get out of sync if I seek, but otherwise the issue seems greatly minimized. The MSU Video Quality Measurement Tool can't use it AFAIK, but at least the Avisynth comparisons are more useful.

MSU can use it or any avs script

Use seekmode=0 for frame accurate seeks (but slower)

If DSS2() is missing the last frame you can align the clips by using Trim()

Interestingly, even DSS2 was way off for me. It wasn't about missing the last frame, or the last three, or just the first two, but more about getting up to a couple seconds out of sync with the avi. That's a lot of frames. FFmpegSource did a lot better, and it tended to play all the way through just fine, but it was still a few frames off when I tried to seek. MPEG2Source with a .d2v has no such issues for me though. Of course, maybe it's not a fair comparison, since I haven't used a .d2v with any of the other loaders.

You can use PSNR and SSIM in avisynth as well (it has plugins)

That's true, and I was going to go that route until I found MSU VQMT. What I like about it is that I can test a much larger number of metrics and find potential problem frames on a graph. It's no substitute for eyeball comparisons, but at the moment it seems a lot more convenient than using Avisynth for metrics.

You should do some reasearch on what PSNR and SSIM really mean in terms of "quality". There is a lot of discussion on this topic. I wouldn't blindly use it and assume quality is better if you score higher. They are objective measurements, but often do not correlate with human percieved quality, especially when encoders use psy optimizations. For example you can tune some encoders to give higher PSNR and SSIM scores, but the image quality to the human eye is actually worse.

I understand. I've been doing a lot of reading on this forum about that topic, and I know to trust my eyes better as the final arbiter. The problem is, I just don't always notice artifacts immediately by eye (although using Avisynth, subtract, and magnification helps a LOT). Sometimes I end up seeing them way down the line, and once you see something, you can't unsee it. That's why I like to use metrics to give me a low-effort second opinion about what areas might be problem areas, when I might not have initially noticed them on my own. As far as the metrics themselves go, I hear SSIM is far better than PSNR for corresponding with the differences humans perceive in typical scenes, even though certain optimizations undermine it while providing better visual quality in reality.

This is off topic now, but since you're in my thread, I wanted to ask you something: I skimmed through the "high quality mpeg-2 encoding" thread, and I noticed that your x264 encodes are superb. What do you suggest I read to learn how to use x264 properly?

[poisondeathray]

What do you suggest I read to learn how to use x264 properly?

Unfortunately, there aren't any guides. You can learn about the settings by reading about them
( http://mewiki.project357.com/wiki/X264_Settings ) ; but only playing with the settings , getting comfortable with them and seeing what they do in different situations, sources, bitrate ranges etc..., can you familiarize yourself with how an encoder will react. To start out, just keep it simple by using the preset system

In reality, the end quality of an encode is usually more affected by pre-filtering and preparation of the source than the encoding settings of an encoder.

That's why I like to use metrics to give me a low-effort second opinion about what areas might be problem areas, when I might not have initially noticed them on my own.

That's fine, but how does that affect your management ? You're not segment re-encoding or adjusting individual frames are you ? Let's say there's a bad I-frame on frame 456. What are you going to do about it ?

[Mini-Me]

What do you suggest I read to learn how to use x264 properly?

Unfortunately, there aren't any guides. You can learn about the settings by reading about them
( http://mewiki.project357.com/wiki/X264_Settings ) ; but only playing with the settings , getting comfortable with them and seeing what they do in different situations, sources, bitrate ranges etc..., can you familiarize yourself with how an encoder will react. To start out, just keep it simple by using the preset system

In reality, the end quality of an encode is usually more affected by pre-filtering and preparation of the source than the encoding settings of an encoder.

That's why I like to use metrics to give me a low-effort second opinion about what areas might be problem areas, when I might not have initially noticed them on my own.

That's fine, but how does that affect your management ? You're not segment re-encoding or adjusting individual frames are you ? Let's say there's a bad I-frame on frame 456. What are you going to do about it ?

It depends on my target format. For my x264 encodes, I will have the luxury of deciding my own bitrates and filesizes, so I can adjust the bitrate to suit the quality I want. For a given bitrate though, I still want to obtain the best quality reasonably possible.

I have less flexibility for my MPEG-2 encodes, since they're destined for two hour DVD's and therefore limited to about 5mbps average. There, the only thing I can really do is toy with settings, but I don't want to waste the time if there are no problem areas to fix in the first place.

[jagabo]

For my x264 encodes, I will have the luxury of deciding my own bitrates and filesizes, so I can adjust the bitrate to suit the quality I want. For a given bitrate though, I still want to obtain the best quality reasonably possible.

Use CRF (constant quality) encoding instead of bitrate based encoding. It's faster and always get the quality you ask for.

[lordsmurf]

Use the ffmpeg loader, not DSS.

[Mini-Me]

For my x264 encodes, I will have the luxury of deciding my own bitrates and filesizes, so I can adjust the bitrate to suit the quality I want. For a given bitrate though, I still want to obtain the best quality reasonably possible.

Use CRF (constant quality) encoding instead of bitrate based encoding. It's faster and always get the quality you ask for.

I actually haven't started messing with x264 yet, so I'm talking out of my ass a bit, but isn't CRF only second best? I remember reading somewhere that CRF is single pass, and it's the best single pass method, but it's not as good as slower multi-pass methods. Is that true?

[poisondeathray]

I actually haven't started messing with x264 yet, so I'm talking out of my ass a bit, but isn't CRF only second best? I remember reading somewhere that CRF is single pass, and it's the best single pass method, but it's not as good as slower multi-pass methods. Is that true?

No.

If you did a crf encode and it ended up at whatever bitrate, then used 2pass using same settings to achieve the same bitrate as the crf encode, they would be nearly identical.

[jagabo]

Basically, they are two sides of the same coin. With constant quality encoding you pick the quality and the encoder uses whatever bitrate is necessary to deliver that quality at each frame. With bitrate based encoding you pick the bitrate (and hence file size) and the encoder delivers whatever quality it can for that bitrate. The only reason to use bitrate based encoding is to get a particular file size -- like if you need a 700MB file to fit on a CD.

[Mini-Me]

That's actually pretty cool. It sounds like CRF suits my needs very well for my x264 encodes, even if the units used aren't the easiest to understand. I'll just have to play with it I guess, but I'm glad you guys cleared that up for me.

I'm dreading these encodes though, because my sources are interlaced, and I know the x264 devs haven't gotten around to adaptive MBAFF yet. Waiting months (years? heh) for adaptive MBAFF would get me better quality in smaller files, but I have to capture my videos now, and I don't have enough storage space to keep ~236 hours of footage in lossless Huffyuv all at once.