resize vs. crop/letterbox

[DeusExMachina]

A burning question I have had for some time is how to make the decision between cropping, letterboxing, or resizing. To get maximum image quality, how does one calculate at what point one should crop/letterbox vs. resize.
For example, I suspect that if I had a source file with a res of 724x482 that I wanted to encode to DVD, I would be far better off cropping 2 pixels from top and bottom and one off each side than I would be to resize the image to 720x480. Likewise, if I had a file that was, say, 710x470, I suspect maximum quality would be achieved by letterboxing 5 pixels all around rather than resizing. At some point, however, resizing becomes more appropriate.
How does one go about making the decision when to do either?
For example, a common size seems to be 512x384 (due to being half the pixel space.) Should this be resized or letter boxed?

[major]

If you letterbox from a small image size, your TV will show wide black areas and this will probably not look very good.

[DeusExMachina]

Yes, that is clear. What I was meaning was the appearance of the actual video content, rather than the complete overall image. There is going to be some video degradation due to passing through the resize algorithm, so at small discrepancies from the target size, letterboxing seems appropriate. At some point the letter boxing, as you say, gets to be too prevalent. I guess what I am asking is if it is just a simple addition of black bars to the video content, or does the aspect ratio play a part. In other words, if you have a video that is 512x384, like in my example, would there be a 512x384 video surrounded by 104 pixel bars top and bottom and 48 pixel bars side to side, or are these value chosen differently, and the video resized anyway? At this size, would it be better to add the bars (104x104x48x48 ?) or resize for the best displayed image?

The algorithm for letterboxing in general is kinda confusing. It does not seem to be a simple addition. I have a clip that was obviously incorrectly encoded; I can see from the file that it SHOULD have been 16:9 but is set to 4:3. This is easilty taken care of, but weirdly, when I use quicktime to letterbox in 16:9, instead of getting the aspect ratio correct, it simply drops the original file's video, still squashed, into a 16:9 container, large black pillar boxing right and left. using mplayer works fine.

[DeusExMachina]

Are there certain sizes, such as 512x384 that resize better? I tend to see certain sizes a lot.

[major]

You should generally take into account the fact that mpeg-4 videos are normally in square pixel aspect while DVDs meant to be played on a TV do not use square pixel (ie, 720/480 is not equal to 4/3). So one should only crop/pad the width or the height, and let the other side resize accordingly.

[DeusExMachina]

Ok, that I get. I suppose then that in general it is best to crop the smaller dimension, unless one happens to be at some ratio of the original that might result in better encoiding after resize. Is this correct?
Now I am just left wondering why there are particular sizes that seem to crop up (pun definitely NOT intended) again and again on RIPed DVD files (such as the 512x384 size above.) Is there something about these sizes that stem from optimal video after passing through video resizing?
Quicktime's choice to pillar box in the above example is also a bit odd.

[Case]

Now I am just left wondering why there are particular sizes that seem to crop up again and again.

My guess: To fit two episodes per cd-r, each file has to be 350 MB. For a 42-minute episode, with 23.976 fps, in XviD, that means a video bitrate of 1022 kbps. This bitrate is a bit much for 496x368, not enough for 528x400, and just about right for 512x384. As a bonus, 512x384 is exactly 4:3.

[DeusExMachina]

Ah, ok, that is what I am getting at. So since we are dealing with different aspect ratios in the source and target (square pixels and rectangular pels and all that) the important thing for maximum quality after a resize is to preserve aspect ratio. Correct?
So if I am going from an incorrectly encoded source (in 4:3 when it should have been 16:9) and since cropping takes place first in the conversion stream, to get maximum quality through the resize, I should first crop any bars I do not want if they exist, then figure out the closest 4:3 value to the cropped dimensions, then calclulate this ratio. Then find the size closest to the target (720x480) that corresponds to THIS ratio, then subtract to get letterbox values, then encode. Since the 16:9 issue is just a matter of setting a flag, this should take care of itself.
Is all this correct?

As a concrete example, if I start with a 512x384 mpeg with 24 letterboxed pixels top and bottom, and I wished to retain maximal quality through a resize, I would crop theoriginal bars to yield a 512x360 intermediate clip. This gives you a display pixel ratio of 1.422. using the lowered value as a starting point, this gives you a size of 682.67, or 683x480, for a final target of 720x480, with pillars of 19 and 18 pixels left and right. Is this correct?

Or should the process be this:
Calculate ratios for dimensions independantly. So 720/512=1.40625 and 480/384=1.25. Since we are cropping to 360, the final vertical size should be 360x1.25=450, so add back 15 pixels top and bottom.

Which of these is correct? As I think about it, the latter seems to make more sense. Perhaps I should just ask what algorithm is used by QT when the letter box field is checked?

[major]

There is no definitive rule, as one shall also admit the possibility that the original file has a wrong aspect. This said, the most general rule is to crop/pad the source in order to bring it to either 4:3 or 16:9 aspect (no other possibilities allowed), then resize to 720x480 (x576 if PAL), and add the DVD 16:9 flag if appropriate.

[DeusExMachina]

hmm. OK.
Three more test questions should resolve the issue for me then.

So… in the situation where 512x384 is already at 4:3, do not pad/crop. Just resize, correct?

In the situation where it already has black bars, say 12 pixels on each side, crop, pad to take the new size to 4:3, then encode (if I have a 512x360 clip, say, with the 12 pixel bars, crop the 12, to get a 512x336 clip, then pad with 24 pixels top bottom to get back to 512x384, then encode.) Correct?

In the weird situation above, where the file has the wrong AR flag, should one pad to the correct target ratio, then encode, or pad to the original ratio, then encode? In other words, if one has a 4:3 file that should have been 16:9, should one pad to 4:3 then encode to 16:9, or pad to 16:9, then encode.

Thanks a lot for these clarifications.

[major]

So… in the situation where 512x384 is already at 4:3, do not pad/crop. Just resize, correct?

Yes.

In the situation where it already has black bars, say 12 pixels on each side, crop, pad to take the new size to 4:3, then encode (if I have a 512x360 clip, say, with the 12 pixel bars, crop the 12, to get a 512x336 clip, then pad with 24 pixels top bottom to get back to 512x384, then encode.) Correct?

Yes.

In the weird situation above, where the file has the wrong AR flag, should one pad to the correct target ratio, then encode, or pad to the original ratio, then encode? In other words, if one has a 4:3 file that should have been 16:9, should one pad to 4:3 then encode to 16:9, or pad to 16:9, then encode.

You should bring it to 16:9, unless it is already precisely at 4:3, in that case just resize.

[DeusExMachina]

OK. WHEW!!!
Now I finally have a handle on that, I think. This is pretty much the procedure I have been using, but was never quite clear if this was correct. It has always been nagging me, since there are occasionally files that don't seem to follow the expected rules, like the one above where QT decided to pillarbox the file in 16:9. Weird

Anyway, thanks for all the clarifications.

[DeusExMachina]

OK, one last exemplar for clarification. If I have a file that is very close to the 4:3, say 672x496, which is 1.3548 (as opposed to 1.3333, i.e., 4:3) should I just resize, or should I pad to 672x504 before I encode to DVD 720x480?

Likewise, if I have the same file but it is obviously incorrectly set to 4:3 when it should be at 16:9, should I:

1) not bother to pad/crop, since it is so close to 4:3, and just set to 16:9

2) pad to 672x504 (4:3) and then resize to 720x480 and set aspect ration correctly

3) crop to 672x378 (16:9) and resize to 720x480 and set AR to 16:9

or lastly

4) pad to 882x496 (16:9) and resize to 720x480, setting AR 10 16:9.

The answer to those 5 issues should resolve the last bit of confusion.

Thanks so much.

[major]

OK, one last exemplar for clarification. If I have a file that is very close to the 4:3, say 672x496, which is 1.3548 (as opposed to 1.3333, i.e., 4:3) should I just resize, or should I pad to 672x504 before I encode to DVD 720x480?

Such small differences are due to rounding of width and height to multiple of 16, they are not generally visible unless you're an eagle.

Likewise, if I have the same file but it is obviously incorrectly set to 4:3 when it should be at 16:9, should I:

1) not bother to pad/crop, since it is so close to 4:3, and just set to 16:9

2) pad to 672x504 (4:3) and then resize to 720x480 and set aspect ration correctly

3) crop to 672x378 (16:9) and resize to 720x480 and set AR to 16:9

or lastly

4) pad to 882x496 (16:9) and resize to 720x480, setting AR 10 16:9.

The answer to those 5 issues should resolve the last bit of confusion.

Thanks so much.

If this clip is just anamorphically deformed to 4:3, then I would go for 1) for the same reason as in the previous point.

[DeusExMachina]

That is pretty much what I ended up doing. Now just to deal with the darkening issues!
Thanks a lot for the clarifications.