mpeg2 to mpeg4 video quality

[Starkian]

ok so here's my problem,I'm gonna try to explain the best as i can
I have some TV caps that i would like to keep but when I convert them from an mpeg2 to mpeg4 it always seems a step down from the original.no matter what setting i use!

Ok, here's some technical info:
1- the originals tv caps (SD channel) are in mpeg2-CBR12Mbps - i always use the highest quality possible for my cap
2- the footage is a live performance on a outdoor stage from sundown to the evening.
3- I use Ulead VideoStudio11 Plus for editing

Here's what I tried so far
1-H.264 Main Profile @ 640by480 @ 4Mbps using Ulead VS11Plus, which was a no go beacause
a-the color in Quicktime was not the same than those in the preview window of VS11Plus, they were way too much "fade" b-I get blockiness artifact
2- i also try xvid and divx (6.8.3) in 1 pass quality mode quantizer 2 , for xvid I also try using the mpeg quantization matrix
3- now i am with x264 and megui

the videos in mpeg4 are not that bad but seems a step down from the originals which at the base are good but not excellent, they seems less sharp and more fade compare to mpeg2.

I know that everyone embrace mpeg4 so i hope you wouldn't be mad at me.

P.S. I don't care about file size or encoding time , the only things that matters is end results.

[Soopafresh]

Post a small clip of the original mpeg2

[jagabo]

the videos in mpeg4 are not that bad but seems a step down from the originals...

Of course they are. Any time you compress with a lossy codec you get loss of quality. Some things you can do to minimize the losses:

Leave the frame size intact and use the MPEG 4 PAR/DAR flags to specify the display aspect ratio. Unfortunately, not all players will respect the AR flags.

Turn off B frames. Use only I and P frames. B frames are encoded with lower quality (higher quantizers) to reduce the file size.

You shouldn't see any "fade" in your MPEG4 encodes. You are probably not comparing them the same way. It may be that your DVD player software is compensating for the fact that computer monitors are very different from televisions. Video normally looks drab on a computer monitor compared to a TV.

Televisions are designed to receive fuzzy analog video and they use sharpening filters to make the picture sharper. Computer monitors expect to receive extremely sharp video from a graphics card and don't perform any artificial sharpening.

[Starkian]

thanks a lot of the answers!
@ Soopafresh : since the mpeg2 is in CBR12Mbps and a sample below 6MB would be 2-3 seconds , i took differents snapshots





@jagabo : thanks for the help. I know that Mpeg2 and Mpeg4 are lossy codec but I didn't think it would be really noticiable.For the comparaison , i simply watch both in VLC .
For the mpeg4 AR flag , I don't know if I set it in the right place , when i create my avisynth script,the video Input DAR is correctly set to ITU 4:3 NTSC and I don't use any cropping and resizing and after I'm done with the aviscript the video window shows up with AR control, since i don't use and crop/resize ,the video (720by480)have a AR of 1.5 which i switch to NTSC 4:3 , but when I watch it in VLC the AR is 1.5.Do i set it at the wrong place? Do I have to set the script to resize to 640 by 480 or is it a problem with VLC?

Turning off B frames really help.
One more question, so here's my encoding parameters

--crf 22.0 --ref 16 --mixed-refs --no-fast-pskip --filter -2,-1 --subme 6 --trellis 2 --partitions all  --8x8dct --qcomp 0.9 --me umh --threads auto --thread-input --progress --no-dct-decimate --no-psnr --no-ssim --output "output" "input"

For the filter (deblocking strength and threshold) , the values means what exactly ?
a value of -6 would lead to what compare to +6 ?

[jagabo]

the video Input DAR is correctly set to ITU 4:3 NTSC and I don't use any cropping and resizing and after I'm done with the aviscript the video window shows up with AR control, since i don't use and crop/resize ,the video (720by480) have a AR of 1.5 which i switch to NTSC 4:3, but when I watch it in VLC the AR is 1.5. Do i set it at the wrong place?

You need to specify the DAR in the output file. This is generally done in the output codec's configuration dialog. With Divx you set it in the "Format out" section on the Video tab in the Divx Codec Properties dialog (use NTSC 4:3, not square pixel). I don't know where the setting is in h.264 in VS11. I believe you specify the DAR indirectly in h.264 by setting the "sample aspect ratio" (shape of individual pixels). Try setting the sample aspect ratio to 10:11.

Do I have to set the script to resize to 640 by 480 or is it a problem with VLC?

You don't want to resize to 640x480 or you'll lose resolution. Just set the DAR in the output codec.

Turning off B frames really help.

Keep in mind that the lower quality of B frames is transient. You notice it when stepping through frames one by one, but it's not very noticeable at normal playback speed.

For the filter (deblocking strength and threshold) , the values means what exactly ?

I'm not sure about this but I believe those are hints for deblocking on playback. As you know, heavy DCT compression leads to blocky artifacts. h.264 decoders perform deblocking to smooth out the blocks (they interpolate the colors from corner to corner of the blocks). The codec can't always tell which blocky artifacts where caused by compression and which might be real parts of the picture. So you have settings to control the threshold and strength of the deblocking. Xvid and Divx have deblocking options in thier playback configuration dialogs. If you're not using heavy compression you won't have much DCT blocking. So you shouldn't see much difference.

With h.264 compression you might compare Constant Quantizer mode and Constant Rate Factor mode. CQP mode is constant quality in a mathematical sense. CRF is similar but it takes into account what the codec thinks you will be able to see.

[poisondeathray]

Just to clarify you are a bit confusing when you used the term "mpeg4" which could refer to both h.264/x.264 (mpeg4 part 10) and DivX/Xvid (mpeg-4 part 2) and about 10 other things

http://en.wikipedia.org/wiki/Mpeg-4

Just to add to what the others have already suggested, in MeGUI with x264, you have to signal DAR separately in the avisynth script. For example for a 4:3 DAR:

global MeGUI_darx = 4
global MeGUI_dary = 3

For 16:9 DAR it would be :
global MeGUI_darx = 16
global MeGUI_dary = 9

(you get the idea)

As for deblocking strength, the default (0,0) is usually used, negative values give you sharper pictures. Test a few settings on some sample clips (like (-1,-1) to see if it improves your situation). Here is the technical explanation I copied down from somewhere a long time ago (sorry I don't have the original link)

The first parameter is AlphaC0 (default: 0). This adjusts thresholds for
the H.264 in-loop deblocking filter. First, this parameter adjusts the
maximum amount of change that the filter is allowed to cause on any one
pixel. Secondly, this parameter affects the threshold for difference
across the edge being filtered. A positive value reduces blocking arti-
facts more, but will also smear details.
The second parameter is Beta (default: 0). This affects the detail
threshold. Very detailed blocks are not filtered, since the smoothing
caused by the filter would be more noticeable than the original blocking.
The default behavior of the filter almost always achieves optimal quality,
so it is best to either leave it alone, or make only small adjustments.
However, if your source material already has some blocking or noise which
you would like to remove, it may be a good idea to turn it up a little
bit.

In your sample settings, I noticed a CRF of 22. Try using a CRF of 18 or even lower, you will notice it will have a much higher bitrate, but it will preserve more details instead of the blurring "plastic doll" effect of lower bitrates

Make sure you use the latest x264 version (use the autoupdate feature) as it is updated frequently (almost weekly) and there have been quite a few enhancements to both speed and quality over the last few months)

The next thing to try might be filters, preprocessing and avisynth - there are dozens here to try

If you are still concerned after playing with the settings, post a sample clip to a free hosting site such as mediafire or megaupload and someone here might be able to give you more suggestions

Cheers