Is it harder to encode DV into MPEG2???

[jeanl]

People,
I'm sorry if this post is redundant. I haven't found an answer to my problem looking through the forum.
I'm encoding home videos shot with a DV camcorder, using CCE-SP, VBR, 3 passes, with an average of 7500kb/s (max 9000 min 0). Typical home video: people, scenery etc.

I'm puzzled by the fact that even with a high bitrate like that (7500 kb/s) there is a very substantial degradation between the original DV movie and the MPEG2 result no matter how I tweak video quality etc.
I compare the result with, say, an excerpt from a commercial DVD, and I'm baffled at how much better the DVD quality is compared to my encoding even though the bitrate is fairly comparable. The background is still very sharp, there's no blockiness, tessalation etc, whereas my encoded movies show very large amounts of these! Before you ask, I have made sure the field order is right.

In any case, I'm puzzled. Is this the results of largely superior encoders used by professional studios, or is it more the nature of the original video material?
Am I the only one to be puzzled by that?!

Any pointer/explanation would be greatly appreciated!

Jeanl
P.S. I've tried TMPGenc too, which yields similar quality (not quite as good as far as I can tell)

[junkmalle]

Shakey video from a handheld camcorder, low light indoor scenes with lots of static noise -- hard for MPEG to handle.

[jeanl]

mmm, OK, but movies can be shakey too! Action scenes etc. Yet when I look at the details on the frames, the quality seems very high compared to what I get. I'm wondering whether there isn't something in the DV source that makes it hard for encoders to do a good job...
jeanl

[junkmalle]

Interlace makes it more difficult too. Movies are usually progressive.

[thor300]

In general, CCE-SP is used in cracked versions by lots of people, and low quality is to be expected. In those cases, use the trial versions and see the difference. Am not saying that you are using a cracked version, but some does and they get the low quality they should expect. Easy to fix with $58 for CCE Basic. Still, only a few very expensive DV cams can give a quality that is remotely close to those used by the studios. Maybe you are expecting too much out of a piece of consumer hardware.

[jeanl]

The trial version is what I'm using... But I'm reasonably happy with my DV quality. What puzzles me is the very significant loss when going from DV to MPEG2...
jeanl

[Edmund Blackadder]

Jeanl, give Canopus ProCoder a try. I was also very dissatisfied with both Cinema Craft and TMPGEnc when encoding DV video. Interlaced DV doesn't compress very well into MPEG2, especially with encoders such as Cinema Craft and TMPGEnc no matter how you tweak them.

I've almost given up on encoding DV to MPEG2 until I tried ProCoder. The difference was night and day. Even at lower bitrates it produces a clean picture from DV, but at high bitrates there's hardly any difference, and definitely no blocks and almost no extra mosquito noise. You'll be surprised at how well your DV source will compress into MPEG2 with ProCoder.

[jeanl]

AH!? OK, given your experience, I'll give it a shot!
Thanks a lot for the advice.
jeanl

[Cornucopia]

Maybe you're watching interlaced video (i.e. from DV material) on a PC monitor (aka progressive), without any good deinterlacing. This will always look worse than Film material (inherently progressive). What's your output/display device?

Scott

[edDV]

mmm, OK, but movies can be shakey too! Action scenes etc. Yet when I look at the details on the frames, the quality seems very high compared to what I get. I'm wondering whether there isn't something in the DV source that makes it hard for encoders to do a good job...
jeanl

Comparison to a commercial DVD is more like wondering why your home cooking can't match Chef Puck. Its a combination of technique, technology, skill and $$$-time investment.

Some of the main differences in approximate order.

- Commercial work is usually shot on film and perfectly exposed with controlled lighting.
- Every shot is from a tripod or steadycam with a pro cameraman. Minimal pans and zooms are used. MPeg compression depends on identical pixels over frames. That means the camera must be stable.
- The source is much higher quality (resolution, noise, dynanic range) than the destination DVD.
- Speaking of lighting, every scene is color corrected to fit within the 219 levels (16-235) available on a 8 bit DVD. All saturated blacks and whites are removed and massaged by hand.
- Action scenes are encoded-optimised frame by frame and allocated a disproportionate bit budget. During this process, the DVD transfer has a substantial amount of detail removed from the action scene frames to improve encoding quality vs. bitrate. In other words, action scenes are hand crafted over days of work.
- The equipment keeps video quality optimal during all processing steps.
- Encoders are selected to match the material and each scene might use a different optimized encoder.
- etc. etc.

A more realistic comparison to your work may be the local independent UHF news team using DV equipment (not Betacam) for video acquisition. Why does their stuff look better?

- They generally use lighting for every take.
- They use a tripod whenever possible. Their cameraman is trained to hold the camera steady.
- Their cameras and lenses are a couple of steps up in quality ($3000-10,000 camcorders)
- The recording quality (DV), editing equiment (probably AVID) and encoders are probably no better in performance than yours.

Hope this helps explain the difference.

[Soopafresh]

You might get better results also by running the DV material through a few VirtualDub filters - Deshaker is good for handheld material.

http://www.sundancemediagroup.com/articles/deshaker_guide.htm

Too much jerky motion always looks bad on Mpeg encodes.

[jeanl]

Thanks a bunch, guys, for the answers/pointers. Yes, I was watching all this on my monitor, but I was watching frame by frame, so I'm not sure de-interlacing is an issue here?
@edDV, thanks for the details. I can see what you're getting at.
@Soopafresh, yes, I will try some of the filters to see if things improve a bit.

@All, I read in another thread (in this one) in a not in passing from the author of HC that DV is much more challenging to encode than pre-encoded commercial movies (and don't mean transcoding here), this might also explain a bit of what I'm seeing.

In any case, all this points to the fact that I should have reasonable expectations from the encoder!
Thanks again for all the info/pointers...
jeanl

[edDV]

I don't see any great difficulty in encoding DV. About 40-50% of what you see on non-primetime TV (> 60% for non-movie cable) is most likely DV format originated. Everything you see on cable or DTV is MPeg2. Everything except 480p, 720p DTV is interlaced.

I think what you may be concluding is that it is more difficult to get quaity results from your own original material than copying a release print of a multi-million $ production. That is true.

[jeanl]

.
I think what you may be concluding is that it is more difficult to get quaity results from your own original material than copying a release print of a multi-million $ production. That is true.

I understand what you're saying, but my point is much narrower than that. I'm not upset at the final quality of what I have in terms of lighting, color etc. I know I start with somewhat low quality, and I'll end up with worse.
However, I'm puzzled by the very significant loss of quality that I see between the original DV video and the output MPEG video, even at bitrates that should guarantee comparable quality.
So, I can understand how multi-million production would yield high-quality results in general terms. I don't quite understand yet why it's so difficult to obtain reasonable quality encoding on my DV videos, meaning encoding that does not degrade them much beyond their admittedly relatively poor original quality... I know I'll end up with something that won't look like a movie. All I would hope for would be for my MPEG movies to be essentially as good as the DV original.
At this point, they are not. Not by a long shot! That's what puzzles me!

jeanl

[troyvcd1]

You are doing something wrong. How did you set your field order. You should use avisynth and the last command should be doubleweave().selectodd(). You can not set the field order in CCE. You need to set the field order in CCE to top field and use avisynth to restore the field order. At those bitrates you should not see a single block. I use CCE with avisynth and my dvds come out just as good as the original. Also there are some new filters in avisynth like IIP that actually make you encoded m2v BETTER than the original. If anyone doesnt believe me then they need to try IIP first.

[jeanl]

troyvcd1, I'm not sure I'm following you. Are you saying that selecting top-field first in CCE controls which field is output first? (not which input field is first?). I haven't started using avisynth yet, but I can give it a shot. Wouldn't I notice if my input field order was wrong? Would I just see a degradation of quality, or would I also see erratic movements in the video (which I don't see)?

Also, when you say "my dvds come out just as good as the original", are you encoding DV material, or commercial DVDs?

jeanl

[edDV]

So, I can understand how multi-million production would yield high-quality results in general terms. I don't quite understand yet why it's so difficult to obtain reasonable quality encoding on my DV videos, meaning encoding that does not degrade them much beyond their admittedly relatively poor original quality... I know I'll end up with something that won't look like a movie. All I would hope for would be for my MPEG movies to be essentially as good as the DV original.
At this point, they are not. Not by a long shot! That's what puzzles me!

jeanl

I think you partially missed my point. Everything I listed affects how the MPeg2 encoder will perform and what "they" do to increase quality.

Bottom line:
1. use a tripod
2. light the scene to control contrast.
3. make sure you are capturing black to level 16
4. minimize pans and zooms

The typical jerky overzoomed home video will stress any encoder and force it to compromise quality to achieve compression. As they say, garbage in - garbage out.

troyvcd1 may be correct. You might have a larger error that we haven't figured out. Have you compared your results with others?

Added :
"All I would hope for would be for my MPEG movies to be essentially as good as the DV original.
At this point, they are not. Not by a long shot! "

DV format @25 Mb/s, is lightly compressed and uses minimal temporal compression (frame to frame). Highest usable MPeg2 rate for DVD is 6-8 Mb/s. The required compression is mostly achieved by frame to frame compression using similarities between frames. Unless you shoot the video with compression in mind, you will never get your MPeg2 to look nearly as good as the DV original.

[troyvcd1]

You may have the right field order. It is either right or wrong. DV video is bottom field first. Dont worry about what it means you just have to get it right. CCE is set up to encode top field first. No setting can change this in the earlier versions of CCE. I am not sure about the newest versions. The correct way to set up the field order is with avisynth. It is complicated to use though. Yes I am doing dv video from my camcorder.

[Edmund Blackadder]

Bottom line:
1. use a tripod
2. light the scene to control contrast.
3. make sure you are capturing black to level 16
4. minimize pans and zooms

This is all nice and correct, but unfortunately to the encoders such as CCE and TMPGEnc this is not enough. I was shooting my DV in both PAL and NTSC, using high quality 3CCD cameras such as Sony DCR-VX1000, DCR-TRV900, PD-150, DCR-VX2000 and Canon XL1.

I had video from different situations, including camera on tripod, manual exposure control, controlled lighting and guess what? - CCE and TMPGEnc MPEG2 results still were not nearly as good as DV source. And trying to avoid any kind of motion or pans, just to satisfy the MPEG2 encoder, doesn't make any sense - you might as well shoot still pictures!

My point is, CCE and TMPGEnc are really bad when it comes to interlaced video. It's not your or my DV video's fault, but their inferior encoding engines. For progressive they are alright, but for interlaced (and not only DV) avoid them like a plague. As I mentioned before, ProCoder is 300% better for interlaced video. It'll visually give you what you put in. It does have some smart filtering that is unnoticeable to the eye, but improves the output quality to the DV level. Hell, even MainConcept MPEG2 encoder gave me better results from DV than CCE and TMPGEnc.

Jeanl, don't let others convince you that your DV video is crap - it's most likely not, if you, yourself, like the way it looks. It's a shame that those encoders cannot live up to your quality DV video. Let me promise you that if you encode with ProCoder you'll be a lot happier

If you're able to, please try ProCoder and post back the results.

[jeanl]

Well, I'm glad you posted back! I tried finding a trial version of procoder, but couldn't! It looks like I have to buy it to try it! Is that right?
I am really not convinced that it can't be done! I'm beginning to believe that the DV format poses special challenges. For one it's interleaved, but that should be OK if I don't mess the field order. But it's also blocky by nature and it might be that that poses problems...

If I really can't try procoder, then do you think I could send you a 1mn snippet of video for you to convert? I would be willing to post the results here of course, on that particular video. Nothing really hard to encode, just a regular home-video shot. That would be cool, but of course, I'll understand if you don't hat the time!

Thanks for the help though! I really appreciate it!
jeanl
EDIT: Never mind! I found a demo version of procoder. I'm trying it this minute!

[edDV]

I've had good results with Mainconcept (Vegas and Premiere versions). Procoder has given excellent results when used. I once used Procoder on a Hi8 to DV to DVD restoration project and was very pleased with the encode.

My point on pro camera and encoding technique was to explain why commercial DVDs look significantly better than amatuer attempts. I wasn't suggesting not to pan or zoom. I was suggesting you will get greater compression and better quality if you do your camera motion from a stable support.

[Richard_G]

All the advice about maximising the quality of the video with lighting, tripods, etc is all well and good, but this isn't always possible. For my multi camcorder projects I use tripods on all three camcorders, but for normal family type stuff I don't. What I want to achieve, and I think this is what jeanl is after, is a DVD compliant mpeg file that looks, to the viewer, no worse than the DV original.

I do all my editing with the source as DV and then save as DV (Ulead Mediastudio Pro using smartrender so only transitions are re-encoded, everything else is untouched). I then give the DV .avi files to Ulead DVD Workshop to let that do the encoding and authoring. I don't know what encoder Ulead uses (I think I saw somewhere that it uses the Mainconcept encoder) but except for the odd bit of dodgy footage, the final output, when viewed on a TV, shows no noticable degradation as watching the original DV footage straight out of the camcorder.

[edDV]

All the advice about maximising the quality of the video with lighting, tripods, etc is all well and good, but this isn't always possible. For my multi camcorder projects I use tripods on all three camcorders, but for normal family type stuff I don't. What I want to achieve, and I think this is what jeanl is after, is a DVD compliant mpeg file that looks, to the viewer, no worse than the DV original.

I do all my editing with the source as DV and then save as DV (Ulead Mediastudio Pro using smartrender so only transitions are re-encoded, everything else is untouched). I then give the DV .avi files to Ulead DVD Workshop to let that do the encoding and authoring. I don't know what encoder Ulead uses (I think I saw somewhere that it uses the Mainconcept encoder) but except for the odd bit of dodgy footage, the final output, when viewed on a TV, shows no noticable degradation as watching the original DV footage straight out of the camcorder.

Having a DVD that looks as good the DV master is everyone's goal. The technical hurdle is getting the 25Mb/s DV video bitrate down to 6-8Mb/s DVD MPeg2. I haven't seen a MPeg2 encoder yet that can't do that without loss. The DVD always looks worse than the original especially on moving objects and edges. Now that I use a HDTV for a monitor, the differences are more apparent. The biggest surprise was not how bad the DVD looked, it looks good. The surprise was how much better a 3CCD Sony PD-150 looks on a HDTV at full 720x480 resolution!

MPeg2 encoders vary in performance. Most of them produce better results if presented with a higher quality source. Apparently, some do better with DV sources than others. My experiences with CCE and TMPGEnc have been with other than DV inputs so I'm neutral in opinion on them.

Ulead, like Sony Vegas, Adobe Premiere/Encore and others use the Mainconcept OEM MPeg development kit and each adds their own secret formula for optimizing their encoder presets. I'm happy with the Mainconcept family of encoders for DV to MPeg2 encoding. I'm also happy with Mainconcept for MPeg2 to MPeg2 re-encoding.

Canopus Procoder is very good as well and has been shown to be better in many situations. I don't use it all the time because it requires extra steps in my process.

[GeorgeW]

Ulead, like Sony Vegas, Adobe Premiere/Encore and others use the Mainconcept OEM MPeg development kit and each adds their own secret formula for optimizing their encoder presets. I'm happy with the Mainconcept family of encoders for DV to MPeg2 encoding. I'm also happy with Mainconcept for MPeg2 to MPeg2 re-encoding.

Out of curiosity, do you know what those "secret" formula optimization settings are?

For DV, are you using the mainconcept defaults for GOP sizes (I believe PAL=12 and NTSC=15)? Or are you tweaking some settings in the advanced section of the encoder?

[Steve Stepoway]

Seems like a crucial issue in encoding quality has to be the nature of the source: a commercially produced movie starts with 4:4:4 video to encode (i.e. YUV sample for every pixel). MiniDV is 4:1:1 -- a luminance sample for each pixel, but color info for groups of 4 pixels on a scan line. That gets encoded to mpeg2 which (AFAIK) uses a luma sample per pixel, but color info for 2x2 groups. Perhaps the amazing thing is that MiniDV encodes as well as it does. I wonder if anyone has done a test of encoding 4:4:4 into mpeg2, then subsampling it to 4:1:1, and encoding that to see what gets lost in the process. Don't think I've ever seen it.

[junkmalle]

From what I've seen Canopus Procoder runs the video through a low pass filter. This reduces high frequency noise making the video a little easier to compress. It leads to slightly less pronounced macroblocks but also a less crisp image.

[jeanl]

I've made my first tests with procoder (high quality mode, but not master) and I must say that it looks better than what I was getting with CCE and TMPGenc.
This being said, I think my expectations were not reasonable, in that I was hoping that when looked at on a frame-by-frame basis, the original DV and the output MPEG would be nearly indistinguishable. I undertstand that's asking a lot when going from 25Mb/s to 7Mb/s, the fact is that when you watch the video normally (i.e. when you play it), the difference is very hard to notice, even with CCE or TMPGenc. It's just when you look at individual frames, especially when there's movement.
What puzzled me originally was that in commercial movies, even when there's movement, the quality is still exceptional. But according to edDV every action scene is hand-tweaked, so I could see how this could yield superior results. Anyway, I'll experiment some more, including with avisynth filters and let you guys know if you're interested...
jeanl

[Richard_G]

The other thing that may make a difference is, as has already been pointed out, NTSC DV is an imperfect source, the luminance compression is 4:1:1. PAL DV however has a compression of 4:2:0, the same as DVD compliant mpeg. This perhaps goes some way to explaining why it is possible for me to get an authored DVD from DV footage that is virtually indistinguishable from the original DV. I'm starting off with a superior quality source.

While I respect that some of the people contributing to this thread may well work in the broadcast industry, I feel that some comments may be over simplification. Suggesting that an mpeg will be inherently poorer quality that a DV file because the size has reduced from 25MB/sec to 8MB/sec seems logical. But, it isn't as simple as that. A DV file contains every frame in it's entirety, an mpeg only contains an entire frame (I frame) every so often (every 12 frames for PAL and every 15 for NTSC), the others are simply instructions to tell the decoder what has changed since the last I frame. With a static scene, this can result in a huge reduction in file size with no degradation whatsoever.

[Soopafresh]

Since there is no trial of Procoder you might want to test BFF interlaced settings with the free (and amazingly good) Quenc , which has settings for TFF/BFF. You'll have to serve it an AVS file.

Another option is to post a clip of the DV file and those of us with Procoder can encode it for you. You can post a 30MB slice (zip it first) on http://rapidshare.de/

[lenti_75]

obviously, youre using a low quality encoder, don't matter the light very much
I used Ulead, and now I use Procoder, and I could encode at 5000 and still don't see any difference