An exercise in high-quality DVD conversion

[diprotic]

Normally I author DVDs when either source material or space constraints limit the possible quality of the output.

This time, I'm authoring just 70 minutes of what I can best surmise is 720p60 h.264 source to DVD5. I'm hoping to learn how to crank MPEG2 to the limit.

The result isn't disappointing, but I've hit an upper bound in quality while generating an output file of just 1.35GB. Something nags inside saying there's some ultra-slow, ultra-high-quality approach that will take advantage of the available bitrate.

At first, I wanted to use mencoder, since it supports N passes and soft telecine. But even with a lot of tinkering with -fps, -ifps and -ofps, I couldn't get the 59.94fps converted to 29.97 (or 24, for that matter).

Then, I switched to ffmpeg:

# Pass 1
ffmpeg -i source.mp4 -target ntsc-dvd -pass 1 -threads auto -mbd rd -trellis 2 -cmp 2 -subcmp 2 -g 18 -bf 2 -flags +mv0 -mpv_flags +qp_rd+skip_rd -intra_vlc 1 -sc_factor 6 -b_strategy 1 -ec guess_mvs -dia_size 1025 -mbcmp rd -pix_fmt yuv420p -an -f rawvideo -y /dev/null

# Pass 2
ffmpeg -i source.mp4 -i audio.ac3 -map 0:0 -map 1:0 -target ntsc-dvd -pass 2 -threads auto -mbd rd -trellis 2 -cmp 2 -subcmp 2 -g 18 -bf 2 -flags +mv0 -mpv_flags +qp_rd+skip_rd -intra_vlc 1 -ec guess_mvs -dia_size 1025 -mbcmp rd -sc_factor 6 -pix_fmt yuv420p -b:v 8250k -c:a copy -bufsize 3000000 -y final.mpg

I gave lavc 8250kbps but it ultimately used less than 3000 average. I've added every high-quality switch I can find.

I decided to try the infamous -sameq switch, which did use a lot more bitrate, but produced a lot of buffer underrun errors and seemed to actually degrade quality slightly.

Is there anything I can to to "crank it to 11?" Or have I reached the upper bound of lavc/ffmpeg for MPEG2?

Attached 10 second samples of source and output for good measure.

Thanks in advance!

[yoda313]

You do know that going from hd to sd will ALWAYS result in lower quality no matter what you do?

[diprotic]

You do know that going from hd to sd will ALWAYS result in lower quality no matter what you do?

As far as I know that any transcoding degrades quality slightly. I've just never pushed the upper bounds so I don't know when I've hit it versus I just don't know how to crank it up all the way.

I'm perfectly fine if I have done everything right and it is as good as it gets—I just don't know that, which is why I'm asking

[yoda313]

Ok just so you know the limitations.

Basically if you are just using a file than you can go as high as your encoder will allow. Though standalone players may refuse a high bitrate mpeg2 file - sd that is, if its hd mpeg2 can go really high according to bluray specs.

If you are making an actual dvd with a video_ts folder and all that you must conform to the specs.

https://www.videohelp.com/dvd

Follow the max settings for your region and that is as high as you can go.

[diprotic]

Ok just so you know the limitations.

Basically if you are just using a file than you can go as high as your encoder will allow. Though standalone players may refuse a high bitrate mpeg2 file - sd that is, if its hd mpeg2 can go really high according to bluray specs.

If you are making an actual dvd with a video_ts folder and all that you must conform to the specs.

https://www.videohelp.com/dvd

Follow the max settings for your region and that is as high as you can go.

I understand that DVD-Video has a maximum combined bitrate of 9800kbps. I'm also aware of the other constraints of NTSC DVD-Video.

What I don't fully understand is why I gave ffmpeg 8250kbps video bitrate and it only used ~2-3000, while still leaving some blocking and banding artifacts, using 2-pass encoding.

I was hoping there is some kind of ultra-high quality motion estimation scheme, or some other technique for those who want the best of the best. Maybe it's a pipe dream reserved for industry pros only using proprietary hardware and software. I've just never produced a DVD-Video stream that can compare to a commercial DVD in terms of bitrate and overall quality. If it's possible with free tools like lavc, then I want to do it. If it's not possible, then I'm happy I pushed those tools to the limit.

[yoda313]

I've just never produced a DVD-Video stream that can compare to a commercial DVD in terms of bitrate and overall quality. If it's possible with free tools like lavc, then I want to do it. If it's not possible, then I'm happy I pushed those tools to the limit.

Actually using a hd source and downconverting to dvd video will give you a very good dvd in the end. It just won't be hd which is what I was getting at.

I have made great looking dvds from blurays.

The trick is using just the movie with no bonuses and using the max bitrate for dvd. Probably sacrificing a bit of audio and going with 348kpbs ac3 instead of 448kpbs would let you eek out a touch more video bitrate as well.

But remember studios are working with FILM MASTERS and you are working with consumer sources.

Regardless if it was actually film negative or if it was recorded digitally a-la 4k stuff you won't be able to match a studio source in that sense.

I would live with the knowledge that maxing out the bitrate and using a hd source you will get the best possible dvd that a consumer can make without access to the film negative or digital source files.

[jagabo]

Encoded with HcEnc via HcGUI, DVD compatible settings. You can go far beyond this if you don't need DVD compatibility.

[diprotic]

Encoded with HcEnc via HcGUI, DVD compatible settings. You can go far beyond this if you don't need DVD compatibility.

Hmm. So jagabo, you think the limiting factor is lavc/ffmpeg?

[diprotic]

I've just never produced a DVD-Video stream that can compare to a commercial DVD in terms of bitrate and overall quality. If it's possible with free tools like lavc, then I want to do it. If it's not possible, then I'm happy I pushed those tools to the limit.

Actually using a hd source and downconverting to dvd video will give you a very good dvd in the end. It just won't be hd which is what I was getting at.

I have made great looking dvds from blurays.

The trick is using just the movie with no bonuses and using the max bitrate for dvd. Probably sacrificing a bit of audio and going with 348kpbs ac3 instead of 448kpbs would let you eek out a touch more video bitrate as well.

But remember studios are working with FILM MASTERS and you are working with consumer sources.

Regardless if it was actually film negative or if it was recorded digitally a-la 4k stuff you won't be able to match a studio source in that sense.

I would live with the knowledge that maxing out the bitrate and using a hd source you will get the best possible dvd that a consumer can make without access to the film negative or digital source files.

Good to know. What I'm getting at though is I can't seem to max out my bitrate. The encoder, with current settings, doesn't use nearly all of what I give it, and I'm not sure if that's a limitation of my knowledge of ffmpeg/lavc or an actual limitation of ffmpeg/lavc, as jagabo implies.

[jagabo]

Encoded with HcEnc via HcGUI, DVD compatible settings. You can go far beyond this if you don't need DVD compatibility.

Hmm. So jagabo, you think the limiting factor is lavc/ffmpeg?

I'm sure ffmpeg can deliver similar quality. It's just a matter of getting the settings right. I'm don't really use it so I can't tell you what you need to specify.

[yoda313]

I'm sure ffmpeg can deliver similar quality. It's just a matter of getting the settings right. I'm don't really use it so I can't tell you what you need to specify.

I think bat999 should be able to help. I know I've gotten help from bat999 on winff and I think that uses ffmpeg if I'm not mistaken.

Though I'm sure there are other more frequent users of ffmpeg here too. Somebody should be able to help tweak the settings with you - speaking to diprotic here.

[_Al_]

You have to have it encoded to CBR 8000kbs or so, mpeg2 and 3000 is not enough doesn't matter what.

Not using ffmpeg but anyway,..., Did you try to get rid of all ffmpeg presets and just to go all default just to get 8000kbps? Why there is 2pass? You just need CBR 8000, can ffmpeg do that? 70minutes of 8000 CBR and 224kbps audio is fine for DVD5.

[ricardouk]

Altough not an expert...i tried an old favourite of mine...Quenc..here's how it compared to yours:



_AL_ is right, if the video at that bitrate fits a dvd5 go with CBR

[manono]

I'm perfectly fine if I have done everything right and it is as good as it gets

You haven't, and it isn't. You encoded as progressive 29.97fps with every fifth frame a duplicate. That makes for jerky playback (or jerkier playback, anyway). You should have removed all dupe frames and encoded as jagabo did - progressive 23.976fps with 3:2 pulldown to output interlaced 29.97fps. Either you don't know what you're doing, you don't know how to tell the encoder to do it right, or the encoder can't do it right. I don't use it either, and wouldn't. Use either HCEnc, CCE, or something else. Use a better quantization matrix. And there are some other non-optimal settings as well. This can be greatly improved and the final size increased significantly. I'm not a fan of CBR encoding, so I'd do it in VBR. Others disagree.

I understand that DVD-Video has a maximum combined bitrate of 9800kbps.

It has a max video bitrate of 9800. The max combined is 10080.

https://www.videohelp.com/dvd

If you're trying to push it to the max, you should make sure the max bitrate is much higher than 8000. 9500, maybe.

[diprotic]

...you don't know how to tell the encoder to do it right, or the encoder can't do it right.

I would say either one of these is correct. I'm beginning to think that ffmpeg is more a versatile quick-n-dirty tool, but definitely not the best in most cases.

I'll give HCEnc a try, since it's free and gets a lot of good press. I've beat my head against the wall with mencoder/ffmpeg long enough.

[diprotic]

Encoded with HcEnc via HcGUI, DVD compatible settings. You can go far beyond this if you don't need DVD compatibility.

Can I see your AVS script that you fed into HC? I would research it myself but since you've done exactly what I want just now, maybe you have it on hand

[jagabo]

Can I see your AVS script that you fed into HC?

I just did a quick:

ffVideoSource("input.mp4") 
SelectEven()
TDecimate()
BicubicResize(720,480)
AssumeFPS(24000,1001)

The frame rate from ffVideoSource was slightly off so I used AssumeFPS to make it exact. You'll have to check a longer section of video to see if something else should be done.

[poisondeathray]

It's actually possible without avisynth, or on linux (or you can use wine or avxsynth for linux). The preprocessing part can be done in ffmpeg, because some avisynth filters have been ported - and they are actually semi-functional now in ffmpeg. So I suspect it could be done in mencoder as well, or you can pipe ffmpeg to mencoder

You can emulate avisynth's selecteven() and tdecimate() using -vf select, and -vf decimate. The problem is ffmpeg's select filters don't change the framerate . When you use selecteven() or selectodd() in avisynth, the framerate is automatically changed in half . It's not in ffmpeg. You end of with 47.952fps after decimating. So one workaround is piping to another ffmpeg instance and using -r as an input option , not an output option . (-r as an input option is like avisynth's assumefps() - same framecount, just the playback speed is changed ; and -r as an output option is like avisynth's changefps() - frames are dropped or duplicates added)

-vf select="mod(n-1\,2)",decimate

https://www.ffmpeg.org/ffmpeg-filters.html#select_002c-aselect
https://www.ffmpeg.org/ffmpeg-filters.html#decimate-1



jagabo forgot to do this, but colormatrix for proper hd=>sd colors is possible in ffmpeg with the colormatrix filter ported from avisynth

-vf colormatrix=bt709:bt601

https://www.ffmpeg.org/ffmpeg-filters.html#colormatrix

The main problems are 1) lack of soft pulldown, even with the "-target film-dvd" option 2) bitrate limitations imposed by the -target options.

1) Mencoder has softpulldown (but it's supposed to be buggy) and ffmpeg is supposed to as well but it's not working in ffmpeg
softpulldown
https://www.ffmpeg.org/ffmpeg-filters.html#toc-mp

A better option is dgpulldown , a linux version is here
http://renomath.org/video/linux/interlace/dgpulldown4unix.tgz
Standard windows version, original author
http://neuron2.net/dgpulldown/dgpulldown.html

2) The various -target DVD options also appear to hardlimit to 6Mb/s. But you can use manual DVD compliant entries for maxrate, bufsize and lower the min quantizer, the same way as before the -target options were introduced. The min quant of 2 is standard, if you remove -target and set min quant to 1 it will allow higher bitrates ; the other options like -maxrate and -bufsize are still there for dvd compliance

Here is example of the method ffmpeg pipe to ffmpeg workaround (If you wanted to pipe instead mencoder and -vf softpulldown, you probably could) . I purposely chose a maxrate of 9500kbps, but you could theoretically chose 9800kbps as per the DVD specs

ffmpeg -i input.mp4 -s 720x480 -vf select="mod(n-1\,2)",decimate -f yuv4mpegpipe -pix_fmt yuv420p - | ffmpeg -f yuv4mpegpipe -r 24000/1001 -i - -aspect 16:9 -vcodec mpeg2video -qmin 1 -q:v 1 -bf 2 -maxrate 9500000 -bufsize 1835008 -packetsize 2048 -g 12 -dc 10 -vf colormatrix=bt709:bt601 output.m2v

Then I used dgpulldown to add soft pulldown flags

[theewizard]

well try Avs2DVD old version, can use quenc encoder at 1 pass constant quailty and high bit rate

that might do the trick for you,

remember avg bit rate is going to depend on the file

frames with lots of movement compress differently and use a higher bit rate

background frames of talking heads, will have a lower avg bit rate, no matter what encoder and settings are use

there is not enough 'change' from frame to frame,

you want high bitrate, record something in MJPEG
its like film, it has many individual frames, not compressed data between key frames

[poisondeathray]

I'm sure ffmpeg can deliver similar quality. It's just a matter of getting the settings right. I'm don't really use it so I can't tell you what you need to specify.

I'm pretty sure it can't . Unless something has changed drastically in the last few years, HCEnc is noticably better, especially at lowish bitrates. ffmpeg / libav encoded MPEG2 just falls apart.

Sure, you can hit the bitrate targets and even do some of the preprocessing using some of the suggestions in the earlier post, but end quality is still lower when doing comparisons to HCEnc, CCE, Mainconcept or TMPGEnc. It's definitely on a lower tier quality wise

The only "weakness" IMO of HCEnc is gradients and banding, where some other encoders do slightly better

[diprotic]

Thank you very much poisondeathray.

I'm open to HC since it still appears to be better than ffmpeg, but since I have so much time and knowledge invested in it, I'd like to try what you suggest.

Here is my exact invocation:

ffmpeg -i input.mp4 -s 720x480 -vf select="mod(n-1\,2)",decimate -f yuv4mpegpipe -pix_fmt yuv420p - | ffmpeg -f yuv4mpegpipe -r 24000/1001 -i - -aspect 16:9 -vcodec mpeg2video -qmin 1 -q:v 1 -bf 2 -maxrate 9500000 -bufsize 1835008 -packetsize 2048 -g 12 -dc 10 -vf colormatrix=bt709:bt601 -y new-sample.m2v

Even after dgpulldown, the playback in mplayer and VLC is choppy and slow (it's also ~23MB whereas yours is ~10MB.)

I'm running ffmpeg version 1.0.1.

Thoughts?

[poisondeathray]

Even after dgpulldown, the playback in mplayer and VLC is choppy and slow (it's also ~23MB whereas yours is ~10MB.)

I'm running ffmpeg version 1.0.1.

Thoughts?

I didn't download that sample, but probably vbv rate control wasn't implemented properly in that ffmpeg version - it's not adhereing to the vbv maxrate settings, hence the larger filesize with q:v 1.

Use a newer binary 2.0 at least

If you want, you can still do 2pass encode instead of 1pass quantizer (vbv capped) encoding, just modify your example in the 1st post. The main reason you didn't reach the bitrate target is the bitrate cap imposed by the -target dvd presets, but also lowering the min quantizer will allow for non DVD legal bitrates as seen in your example above if not capped with vbv settings . In fact, 2pass should give better results in "normal" scenarios where you use slightly lower bitrates - the example above was just to show it's possible to reach higher bitates, and do preprocessing with ffmpeg filters

[diprotic]

Sure enough, I pulled the latest ffmpeg svn and it works.

So I tweaked and tinkered...But you know what? Comparing ffmpeg/mencoder/lavc side-by-side with HCenc, there's just no comparison. HC does everything. It's faster than lavc at producing a higher quality result, and it natively supports pulldown, so there's no extra step.

I think HC is my new MPEG2 encoder of choice. For some reason I clung to the misguided notion that a mature, open-source encoder would necessarily trump a free encoder made by one (?) guy...but I was wrong, at least in the MPEG2 department.

There is one last piece of the puzzle: noise.

I've never encoded at >8000kbps before, and it's quite faithful at reproducing grain/noise. I took a look at the RemoveGrain() avisynth plugin and tried a few iterations but couldn't get smoothness, particularly in the speaker's suit, which is OK in the source but flickers quite distractingly in the resulting m2v.

I've uploaded a new source sample and my output to show what I mean. I'm using this AviSynth script:

LoadCPlugin("C:\Program Files\AviSynth 2.5\plugins\ffms2.dll")
FFVideoSource("sample.mp4")
ColorMatrix(mode="Rec.601->Rec.709")
SelectEven()
TDecimate()
BicubicResize(720,480)
AssumeFPS(24000,1001)

Thanks so much!

[poisondeathray]

So I tweaked and tinkered...But you know what? Comparing ffmpeg/mencoder/lavc side-by-side with HCenc, there's just no comparison. HC does everything. It's faster than lavc at producing a higher quality result, and it natively supports pulldown, so there's no extra step.

And if you compare at "typical" bitrate ranges, maybe 4-7 kb/s for a DVD , there is even a larger difference at the lower end.

There is one last piece of the puzzle: noise.

I've never encoded at >8000kbps before, and it's quite faithful at reproducing grain/noise. I took a look at the RemoveGrain() avisynth plugin and tried a few iterations but couldn't get smoothness, particularly in the speaker's suit, which is OK in the source but flickers quite distractingly in the resulting m2v.

I've uploaded a new source sample and my output to show what I mean. I'm using this AviSynth script:

LoadCPlugin("C:\Program Files\AviSynth 2.5\plugins\ffms2.dll")
FFVideoSource("sample.mp4")
ColorMatrix(mode="Rec.601->Rec.709")
SelectEven()
TDecimate()
BicubicResize(720,480)
AssumeFPS(24000,1001)

Thanks so much!

A couple things:

1) 24p is the wrong thing to do here because this last sample is 59.94p , not film. 59.94p is not DVD compliant, but if you wanted to retain the smooth motion, you would need a "59.94i" DVD ie. make it interlaced (it's also called 29.97i, same thing just different naming conventions) , either that or just leave it with SelectEven() or SelectOdd() for a 29.97p DVD with 1/2 the motion samples

2) I would use Colormatrix clamp=0, to prevent clipping (it's mislabelled as "clamp", but it actually clips values beyond 16-235)
ColorMatrix(mode="Rec.601->Rec.709", clamp=0)

3) One of the "quircks" of ffms2 as a source filter, is it sometimes gets the fps a bit wrong - jagabo mentioned this. I would "fix" it right after FFVideoSource(), but adding AssumeFPS(60000,1001)


I'll have a look at the noise, but point #1 is very important, you will get stuttery playback

[diprotic]

So I tweaked and tinkered...But you know what? Comparing ffmpeg/mencoder/lavc side-by-side with HCenc, there's just no comparison. HC does everything. It's faster than lavc at producing a higher quality result, and it natively supports pulldown, so there's no extra step.

And if you compare at "typical" bitrate ranges, maybe 4-7 kb/s for a DVD , there is even a larger difference at the lower end.

There is one last piece of the puzzle: noise.

I've never encoded at >8000kbps before, and it's quite faithful at reproducing grain/noise. I took a look at the RemoveGrain() avisynth plugin and tried a few iterations but couldn't get smoothness, particularly in the speaker's suit, which is OK in the source but flickers quite distractingly in the resulting m2v.

I've uploaded a new source sample and my output to show what I mean. I'm using this AviSynth script:

LoadCPlugin("C:\Program Files\AviSynth 2.5\plugins\ffms2.dll")
FFVideoSource("sample.mp4")
ColorMatrix(mode="Rec.601->Rec.709")
SelectEven()
TDecimate()
BicubicResize(720,480)
AssumeFPS(24000,1001)

Thanks so much!

A couple things:

1) 24p is the wrong thing to do here because this last sample is 59.94p , not film. 59.94p is not DVD compliant, but if you wanted to retain the smooth motion, you would need a "59.94i" DVD ie. make it interlaced (it's also called 29.97i, same thing just different naming conventions) , either that or just leave it with SelectEven() or SelectOdd() for a 29.97p DVD with 1/2 the motion samples

2) I would use Colormatrix clamp=0, to prevent clipping (it's mislabelled as "clamp", but it actually clips values beyond 16-235)
ColorMatrix(mode="Rec.601->Rec.709", clamp=0)

3) One of the "quircks" of ffms2 as a source filter, is it sometimes gets the fps a bit wrong - jagabo mentioned this. I would "fix" it right after FFVideoSource(), but adding AssumeFPS(60000,1001)


I'll have a look at the noise, but point #1 is very important, you will get stuttery playback

Doing Things Right(R) is very important to me, so thanks for pointing that out.

1) Regarding output FPS: Is there a "right" option, given my source and destination, or is it a matter of preference? How can I produce a 59.94i / 29.97i end product? (you already explained how to get 29.97p, thank you!)

2) I will modify the ColorMatrix accordingly, thank you.

3) I will add AssumeFPS(60000,1001) right after my video source as you suggest.

[jagabo]

I've uploaded a new source sample

Try:

ffVideoSource("sample.mp4", fpsden=1001, fpsnum=60000) 
BilinearResize(720,480)
AssumeTFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()

Encode interlaced.

I don't see any need to remove grain. A small amount of grain is your friend -- it prevents posterization artifacts. The source is already over filtered and over sharpened.

[diprotic]

I've uploaded a new source sample

Try:

ffVideoSource("sample.mp4", fpsden=1001, fpsnum=60000) 
BilinearResize(720,480)
AssumeTFF()
SeparateFields()
SelectEvery(4,0,3)
Weave()

Encode interlaced.

Thanks, I'll try those settings.

I don't see any need to remove grain. A small amount of grain is your friend -- it prevents posterization artifacts. The source is already over filtered and over sharpened.

I'm really not concerned about the source, it's the result that is somewhat bothersome. The speaker's suit looks so noisy and "flickery." Maybe I'm being too demanding, I dunno. I trust your judgment though, I'm definitely not the expert here.

[jagabo]

I'm really not concerned about the source, it's the result that is somewhat bothersome. The speaker's suit looks so noisy and "flickery."

Something else is wrong with your encode. Here's what I got with HcEnc and the above script.

[diprotic]

I'm really not concerned about the source, it's the result that is somewhat bothersome. The speaker's suit looks so noisy and "flickery."

Something else is wrong with your encode. Here's what I got with HcEnc and the above script.

Hmm. That does look fine, much better than my sample above. What's your HC.ini look like for this?

[jagabo]

Hmm. That does look fine, much better than my sample above. What's your HC.ini look like for this?

Pretty plain:

*INFILE           sample.avs
*OUTFILE          unfiltered.m2v
*LOGFILE          hc_log.log
*BITRATE          9000
*MAXBITRATE       9800
*FRAMES           0 449
*PROFILE          best
*AUTOGOP          12
*DC_PREC          10
*INTERLACED
*TFF
*CHROMADOWNSAMPLE 1
*CLOSEDGOPS
*INTRAVLC         2
*MATRIX           mpeg
*LUMGAIN          2