Bunch of DVDs to transcode - Choosing the right settings

[abolibibelot]

1.011765 is over maximum, but as lossy audio can store peaks above maximum (0dB in Audacity) it's probably not clipped as such. When audacity imports audio, it imports it in a format (32 bit float) that can also have peaks above 0dB, so the audio wouldn't actually be clipped until it's exported to another format. The peaks probably haven't been flattened. Plus when lossy audio is decoded, the output mightn't be exactly the same as the input (encoded audio with peaks of 0dB might decode with peaks just over 0dB).
If you want to know the formula for converting percentage to dB, it's "20 log10 x", so for your peak it's 20 x log10 1.011765 = 0.102dB, or next to nothing above 0dB. It's not enough that I'd worry about it. Maybe for peaks of 1.100000 I'd consider reducing the volume. But that's still less than 1dB above 0dB.

Well, that's not an important issue here indeed, but it would seem that there is some slight clipping, as in the example below (the first and most pronunced instance of saturation) the curve is definitely flattened when I apply an amplification of -5dB around that point, and then if I apply the "ClipFix" filter a curve appears that extends quite far below the flattening point.

http://share.pho.to/9n7u5

Would you believe 0.7 is only -3dB? The conversion between % and dB isn't intuitive (given dB is logarithmic). 20 x log10 0.7 = -3.09dB
There's no real downside to having peaks at 1.00 or 0.95. Well..... because there's a potential for the peaks to be slightly different than the original when decoding lossy audio, it's often recommended to keep the peaks at a maximum of -3dB to allow for a little headroom. I can't say I've ever bothered myself. Most playback systems should have some headroom over 0dB without clipping the audio.

I've read that 3dB is in fact, in terms of perceived intensity of sound, a 100% factor, in other words a sound being increased by 3dB sounds twice as loud, so it's not insignificant. If I look for instance at side A, track 01, foobar gives a track gain value of 12.29dB and a peak at 0.3812. Besides, it alternates between talking sequences and music sequences, the former having an even lower level (barely goes beyond -24dB in Audacity's VU meter). It could have used some dynamic compression... Some other tracks hace a close to normal level, so I think I'll have to amplify them on a track by track basis after all.

http://www.cjoint.com/c/EJprjb7iMLy

The QAAC version itself doesn't matter too much (unless there's a bug), as it's just a "front end" for using the CoreAudioToolbox for encoding. The latest version seems to be CoreAudioToolbox 7.10.5.0.
MeGUI's version of QAAC is updated more regularly than the one in the foobar2000 encoder pack. Just replace it with MeGUI's version, if need be.

In this case the one in foobar is the most recent (2.49 vs. 2.47). I recently installed the "encoder pack" for foobar, while MeGUI's QAAC wasn't updated since february.

I almost never normalise, but I was just pointing out QAAC can. If you want to normalise when re-encoding with foobar2000, I think the only way to do it is to open the audio via an Avisynth script and include normalize(). Avisynth will normalise and foobar2000 can do the re-encoding. That's how MeGUI does it.

As a side note, if you copy the script below, save it with an avsi extension (call it whatever you like), and stick it in the installed Avisynth plugins folder, you should be able to downmix and normalise 5.1ch audio simply by adding Downmix() to a script. It normalizes to 0.98, but you can modify it if you like.

function Downmix(clip a, float "centergain", float "surroundgain", bool "surroundfx")
{
    a.ConvertAudioToFloat()

    ## channel layouts: http://avisynth.nl/index.php/GetChannel
    ## (this is WAV 5.1)
    fl = GetChannel(1)
    fr = GetChannel(2)
    fc = GetChannel(3)
    lf = GetChannel(4)
    sl = GetChannel(5)
    sr = GetChannel(6)

    ## add center
    gc = Default(centergain, 1.0) * 0.7071
    fl = MixAudio(fl, fc, 1.0, gc)
    fr = MixAudio(fr, fc, 1.0, gc)

    ## add surround
    gs = Default(surroundgain, 1.0) * 0.7071
    fl = MixAudio(fl, sl, 1.0, gs)
    fr = MixAudio(fr, sr, 1.0, gs)

    ## cross-mix surround delayed & out-of-phase
    ## to emulate back speaker location ??
    fx = Default(surroundfx, false)
    fl = fx ? MixAudio(fl, sr.DelayAudio(0.02), 1.0, -0.7071*gs) : fl
    fr = fx ? MixAudio(fr, sl.DelayAudio(0.02), 1.0, -0.7071*gs) : fr

    MergeChannels(fl, fr)

    Normalize(0.98)
    #ConvertAudioTo16bit()
}

Thanks for that. Where does it come from ? I thought Avisynth was strictly a video utility, it makes it even more versatile...

Could be related to a frame rate issue, they say, the video being considered as 30 FPS instead of 29.97 (yet all the MKVs I created have a proper 29.97 FPS rate).

Frame rate is the only explanation I could think of, but I don't know why it'd happen and I didn't want to speculate too much. You could try using MakeMKV to convert the DVD files to MKV, and if it gets the chapters right, extract them from the MKVs it creates. It'd have to be easier than having to adjust them all manually.[/QUOTE]

I've seen it mentioned quite a few times but haven's tried it yet.
With this method, is it still required to index the files with DGIndex in order to load them in an Avisynth script, or does it somehow simplify the workflow ?

I just tried it with one of these DVDs, it extracted two "titles", one corresponding to the whole video, and one corresponding to the "performances" only. For the second one I can see a small bit of the intertitles between those performances, meaning the original chapter timings aren't perfectly accurate to begin with. But indeed for the "title 1" MKV the chapter markers are very close to the ones I defined manually.

Five last chapters extracted by MeGUI's Chapter Creator module (same with ChapterGrabber / ChapterXtractor) :

CHAPTER23=01:16:15.000
CHAPTER23NAME=Chapter 23
CHAPTER24=01:21:30.500
CHAPTER24NAME=Chapter 24
CHAPTER25=01:32:32.500
CHAPTER25NAME=Chapter 25
CHAPTER26=01:35:18.500
CHAPTER26NAME=Chapter 26
CHAPTER27=01:41:01.500
CHAPTER27NAME=Chapter 27

Manually corrected :

CHAPTER23=01:16:19.708
CHAPTER23NAME=[Performance 8]
CHAPTER24=01:21:35.424
CHAPTER24NAME=Double stroke coordination
CHAPTER25=01:32:37.852
CHAPTER25NAME=Epilogue
CHAPTER26=01:35:23.885
CHAPTER26NAME=[Performance 9]
CHAPTER27=01:41:07.495
CHAPTER27NAME=Credits

Chapters defined by MakeMKV, from MediaInfo :

01:16:19.641                     : en:Chapitre 23
01:21:35.457                     : en:Chapitre 24
01:32:38.119                     : en:Chapitre 25
01:35:24.285                     : en:Chapitre 26
01:41:07.628                     : en:Chapitre 27

(It's quite hungry in memory though : about 540MB when a DVD folder is opened.)

Here's an example of QTGMC's ability to stabilise the picture (in progressive mode) as well as remove noise. It's a torture test sample I used when I first started experimenting with QTGMC for noise removal rather than only use it for de-interlacing. There's sample encodes comparing QTGMC to FastDegrain and TemporalDegrain.
QTGMC(InputType=1, EzDenoise=6) and TemporalDegrain() proved to be fairly similar with that sample, with TemporalDegrain removing a tiny bit more noise, but it was designed for pretty heavy duty noise removal. QTGMC had avantages of it's own. It repaired some of the "single frame" blemishes a little better and reduced the "bobbing" more.

I already stumbled upon that thread, but didn't take the time yet to actually compare the samples (I downloaded them though, so I'll do it one of these days... so many things to do, so little time !).

They should, but they set the boundary. You still need to adjust the saturation up or down to see a difference. In my case I set it to zero (no colour).

Indeed... I have yet to figure out how it works exactly, but it does work.

I "think" when you use levels, if the picture becomes darker, the colour seems to become more saturated, or something like that. That's probably what's meant by "changes the chroma". I'm not 100% sure how it works to be honest. There's a script called Ylevels which functions the same way as Levels but it "Processes only luma. This makes it faster and allows better preservation of color saturation"

I found that one too, but reading the very brief documentation I'm not sure how to use it efficiently.

I'm not sure about the "preferred way of correcting the contrast ". I'm running out of time again. I'll have to ponder that later.
I rarely use "cont" myself. I tend to use Levels or Ylevels and expand or reduce the range of levels, and adjust the gamma and saturation if need be. Colour-wise, the result of using Levels and YLevels is definitely different, so I go with the one I prefer at the time.
For example if a video looks a bit washed out, or the black levels aren't very dark, I might do something like this:
YLevels(8,1,245, 0,255)
So it's not a TV to PC levels conversion, but something in between that effectively increases the contrast.
Likewise for less contrast:
YLevels(0,1,255,8,245)
Or to increase the contrast without making black darker.
YLevels(0,1,240,0,255)
That sort of thing. I just do what I find works. Then I come back the next day and do it again.
I'll read the doom9 threads you linked to later and see if I can get my head around the contrast question. Maybe someone else will come along.....

The first one is actually pretty good for GH2002, I just tweaked it a bit, while ColorYUV "levels="TV->PC" seems to be about as good as it gets for GH2006 (it's too much for GH2002, 3rd picture) :
http://share.pho.to/9n8uU

I think I'll go with this, it might be possible to do better but it's already nicer looking than the original, and it's been time-consuming enough already -- and now I have 42 videos to encode with those parameters... (Those three remaining have 19+12+11 "VTS".)
In such a situation, is there a way to batch-process all the encodes without having to create an Avisynth script for each and loading each script in MeGUI ? That is, having one AVS script which would be used for all videos ? Found this :
https://forum.videohelp.com/threads/357502-Making-a-bat-file-for-both-Avisynth-and-x264
It might take longer to make this work than doing it the boring way, but if I can learn something in the process...

[hello_hello]

Well, that's not an important issue here indeed, but it would seem that there is some slight clipping, as in the example below (the first and most pronunced instance of saturation) the curve is definitely flattened when I apply an amplification of -5dB around that point, and then if I apply the "ClipFix" filter a curve appears that extends quite far below the flattening point.

http://share.pho.to/9n7u5

Interesting. I guess that means the original audio was clipped at some stage, maybe even before it was converted to AC3 (and it looks like only in one channel).
How are you importing the audio into Audacity? Is it already PCM or is it AC3 and you're converting it to a wave file first? I ask, because your screenshot shows it as being 16 bit PCM, which means peaks can't be above 0dB. If it's AC3 and you import it directly.... and it probably requires the ffmpeg files..... it should be imported as 32 bit float.



In which case the peaks mightn't be clipped and you can simply reduce the gain a bit. I'm not 100% sure as I don't use Audacity much, But I thought it'd be worth a mention.

I've read that 3dB is in fact, in terms of perceived intensity of sound, a 100% factor, in other words a sound being increased by 3dB sounds twice as loud, so it's not insignificant.

Open an audio file and increase it's level by 3dB, then compare it to the original. It might sound a little louder, but it won't be twice as loud.
My memory of all this is pretty vague, but have a look at the chart here. Either+3dB or +6dB can be double, but it depends whether you're measuring power ratio or amplitude ratio. One uses the 10 log10 x formula and the other uses 20 log10 x. So +3dB is double the power but +6dB is double the volume. I'm pretty sure that's how it works. The amplitude ratio is also the square root of the power ratio, so if +3dB is 2x the power ratio and the square root of 2 is 1.4, a 1.4x amplitude ratio increase is +3dB.
And to make it even more confusing when you're measuring sound pressure level it requires a 10dB increase in order to be twice as loud.
Anyway.... I'm going to read some more on dB myself later to refresh my memory, but I'm pretty sure when you see a dB scale in programs such as Audacity, it's referring to amplitude ratio.

In this case the one in foobar is the most recent (2.49 vs. 2.47). I recently installed the "encoder pack" for foobar, while MeGUI's QAAC wasn't updated since february.

If you've got MeGUI set to update itself from the development update server, it should be using QAAC 2.55. That's the latest version, but you could always download the latest QAAC yourself and replace them both. https://sites.google.com/site/qaacpage/cabinet

Thanks for that. Where does it come from ? I thought Avisynth was strictly a video utility, it makes it even more versatile...

Avisynth processes audio. Most of the scripts you've posted here (or the AvsPmod screenshots) have included audio. MeGUI uses various decoders and encoders, but Avisynth does the required processing. Next time you encode audio, have a look in MeGUI's log file. It'll include the script it used for encoding (which would include any downmixing and normalising etc). Sometimes if I want to do something audio-wise and can't quite get it to work, I'll get MeGUI to do it, then I'll raid the log file.

I've seen it mentioned quite a few times but haven's tried it yet.
With this method, is it still required to index the files with DGIndex in order to load them in an Avisynth script, or does it somehow simplify the workflow ?

Make MKV will create MKVs directly from the DVD structure, but if you want to re-encode them, they still need to be indexed. MeGUI will use either L-Smash or FFMSIndex. Generally though, DGIndex seems to be the best method (the others don't always handle interlaced or telecined video correctly), however DGIndex won't index MKVs. If I have an MKV containing mpeg2 video, I generally open it with TSMuxer, remux it as a TS file, then index the TS file with DGIndex. For DVDs I'd generally try to use DGIndex with the vob files, but maybe MakeMKV will do it's thing without messing up the chapters, and you can then extract them.

In such a situation, is there a way to batch-process all the encodes without having to create an Avisynth script for each and loading each script in MeGUI ? That is, having one AVS script which would be used for all videos ? Found this :
https://forum.videohelp.com/threads/357502-Making-a-bat-file-for-both-Avisynth-and-x264
It might take longer to make this work than doing it the boring way, but if I can learn something in the process...

OneClick does batch encoding and you can create a script template and save that as part of a OneClick preset. That should process all the video the same way. Maybe you'd have to disable OneClick's auto-cropping and resizing etc if you're doing it in the script. I'm not really sure as I haven't batch encoded that way, but I've no doubt OneClick will do what you want if you create an Avisynth template and save it as part of a a OneClick preset.

There's a couple of batch loaders for MeGUI I've been meaning to try, but I think you still need to create the scripts individually yourself, and they'll load them into the job queue for you so you don't have to add them one at a time, but in your case I think OneClick should do the job.

So did you sort out the luminance level issue, in respect to how VirtualDubMod and AvsPmod were displaying the scripts?

[hello_hello]

I found that one too, but reading the very brief documentation I'm not sure how to use it efficiently....

The first one is actually pretty good for GH2002, I just tweaked it a bit, while ColorYUV "levels="TV->PC" seems to be about as good as it gets for GH2006 (it's too much for GH2002, 3rd picture)

Just fiddle

Ylevels() and levels() work a bit differently, but if you understand how levels() works then the Ylevels() documentation probably just explains the differences. For both:

Ylevels(0,1,255,0,255)

The first number is the source black level, the second is gamma (1 is no change), the third is the source white level, and then you've got the destination levels. So for the above example, nothing would change. The equivalent of ColorYUV(Levels="PC->TV") should be

Ylevels(0,1,255,16,235)

Or the other way around, ColorYUV(Levels="TV->PC")

Ylevels(16,1,235,0,255)

If you use Levels() instead, the saturation might appear to change a bit. Sometimes that's a good thing.
Anyway, that's the way I remember the difference in relation to ColorYUV, but I tend to try both Ylevels and levels and use whichever looks best for a particular video.

Edit: Thinking about it, by default I think Levels() limits the range to TV levels, so even if you were expanding from TV levels to PC levels, nothing would be allowed to have a value below 16 or above 235. If need be, the extreme blacks and whites are "clipped" to keep them within the TV levels range. Either way, you'd probably not see a visual difference, but the coring parameter prevents the clamping of levels
levels(16,1,235,0,255,coring=false)
I don't think Ylevels has a coring option as it never clamps the levels.

[abolibibelot]

Interesting. I guess that means the original audio was clipped at some stage, maybe even before it was converted to AC3 (and it looks like only in one channel).
How are you importing the audio into Audacity? Is it already PCM or is it AC3 and you're converting it to a wave file first? I ask, because your screenshot shows it as being 16 bit PCM, which means peaks can't be above 0dB. If it's AC3 and you import it directly.... and it probably requires the ffmpeg files..... it should be imported as 32 bit float.

Indeed, only the left channel has red lines.
I directly dragged and dropped the AC3 file. I did it again with "Open..." menu, same result. I can change the sampling mode to "32 bit float" afterwards by clicking on the arrow on the left pannel, but I'm not sure if it has any effect at that point. I don't know why it's not imported that way right away. In the "Quality" settings it's set to "Default sampling format : 32-bit float".

Another thing I noticed in Audacity : there seems to be a slight delay between the original AC3 audio and the reencoded AAC (QAAC), about 2100 samples between two identical peaks. I tried with QAAC's "no-delay" option but had the exact same result. Is it to be expected ?

If you've got MeGUI set to update itself from the development update server, it should be using QAAC 2.55. That's the latest version, but you could always download the latest QAAC yourself and replace them both. https://sites.google.com/site/qaacpage/cabinet

Yet it's set to "Auto update / Use stable update server".

Avisynth processes audio. Most of the scripts you've posted here (or the AvsPmod screenshots) have included audio.

Yes, but they only import and mix it, with no actual processing. Actually that's the way I started out when learning how to make AVS scripts for a specific project earlier this year (where I had to export audio separately and make sure it would stay synchronized), I'm not sure if it's even necessary to use variables and import/mix the audio within the AVS script for something more straightforward like a DVD conversion. I read yet another tutorial last night and it seemed to imply that MeGUI automatically loads the DGIndex-generated AC3 when a script calling a .d2v index file is loaded by the user -- well, let's try : no, it doesn't. So are the audio tracks supposed to be loaded manually ?

Make MKV will create MKVs directly from the DVD structure, but if you want to re-encode them, they still need to be indexed. MeGUI will use either L-Smash or FFMSIndex. Generally though, DGIndex seems to be the best method (the others don't always handle interlaced or telecined video correctly), however DGIndex won't index MKVs. If I have an MKV containing mpeg2 video, I generally open it with TSMuxer, remux it as a TS file, then index the TS file with DGIndex. For DVDs I'd generally try to use DGIndex with the vob files, but maybe MakeMKV will do it's thing without messing up the chapters, and you can then extract them.

That's what I did for those 3 remaining DVDs (composed of many individual segments of 5-10min but some of them still have chapter subdivisions) : MakeMKV created as many MKVs as there are titles on each one, then I extracted all chapters with this command in a .bat file (when there's none it creates a 0 byte XML) :

for /R %%F in (*.mkv) do "C:\Program Files\MeGUI_2507_x86\tools\mkvmerge\mkvextract.exe" chapters %%F > %%~pF%%~nF.xml

OneClick does batch encoding and you can create a script template and save that as part of a OneClick preset. That should process all the video the same way. Maybe you'd have to disable OneClick's auto-cropping and resizing etc if you're doing it in the script. I'm not really sure as I haven't batch encoded that way, but I've no doubt OneClick will do what you want if you create an Avisynth template and save it as part of a a OneClick preset.

There's a couple of batch loaders for MeGUI I've been meaning to try, but I think you still need to create the scripts individually yourself, and they'll load them into the job queue for you so you don't have to add them one at a time, but in your case I think OneClick should do the job.

In this case I wanted to avoid the hassle of loading as many files through the GUI and setting up the various parameters with a high risk of error.
So if anybody's interested, now or later, I managed to create multiple Avisynth scripts with the same content, using this batch command (the one proposed by "jagabo" in the aforementioned thread would only treat one file at a time, dragged and dropped on the .bat file, this one processes all .d2v files located in the same folder as the .bat) :

for %%F in (*.d2v) do (
	echo LoadPlugin^("C:\Program Files\MeGUI_2507_x86\tools\dgindex\DGDecode.dll"^) > "%%~nF.avs"
	echo Mpeg2Source^("%%F"^) >> "%%~nF.avs"
	echo QTGMC^(Preset="Medium",EzDenoise=1^) >> "%%~nF.avs"
	echo LanczosResize^(640,480^).YLevels^(10,1.1,255,0,255^).Tweak^(sat=1.05^) >> "%%~nF.avs"
)
pause

It can be adapted for any kind of script, and any kind of source file. It's important to add the "^" before or after parentheses (escape character, otherwise it's interpreted as part of the command and messes it up).
Then another batch file can encode all these AVS scripts :

[CODE]for %%F in (*.avs) do "C:\Program Files\MeGUI_2507_x86\tools\x264\x264.exe" --preset=slow --bframes 5 --crf=24 --vbv-bufsize 17500 --vbv-maxrate 17500 --stitchable --output "%%~nF.264" "%%~nF.avs"

Now I'd like to mux all video streams with their corresponding audio stream(s) using a similar command (again, much quicker and less prone to error than doing it via MeGUI or MKVMerge when there are as many files), and the tricky part is that some of those videos have two audio streams (drums only or in musical context), and it's apparently not possible to use a wildcard character (* / ?) in this case.
I just tried MKVBatch but so far it doesn't seem to work very well (doesn't recognize raw H264 streams, doesn't seem to mux anything) and the interface is barely readable (some options are hidden or incompletely displayed).

That about sums it up :


(But I'm nowhere near as geeky as I'd like to be...)

So did you sort out the luminance level issue, in respect to how VirtualDubMod and AvsPmod were displaying the scripts?

Apparently it was just that "YUV->RGB" option in AvsPmod which was incorrectly set to "PC" (I most likely fiddled with it at some point and forgot about it).

[hello_hello]

I directly dragged and dropped the AC3 file. I did it again with "Open..." menu, same result. I can change the sampling mode to "32 bit float" afterwards by clicking on the arrow on the left pannel, but I'm not sure if it has any effect at that point. I don't know why it's not imported that way right away. In the "Quality" settings it's set to "Default sampling format : 32-bit float".

I don't know why that is, to be honest.

Another thing I noticed in Audacity : there seems to be a slight delay between the original AC3 audio and the reencoded AAC (QAAC), about 2100 samples between two identical peaks. I tried with QAAC's "no-delay" option but had the exact same result. Is it to be expected ?

I'm not sure. I tried a few test encodes and foobar2000 reported the same number of samples for the encoded versions (I tried Nero and QAAC) and when I loaded them into Audacity they lined up, however the QAAC version contained about 9ms of silence at the beginning. It was still lined up with the original but the 1st 9ms was just silence. Maybe that's something to do with the "no delay" option. I haven't tried it without it yet. The real world is calling.... I'll come back to that later.

I read yet another tutorial last night and it seemed to imply that MeGUI automatically loads the DGIndex-generated AC3 when a script calling a .d2v index file is loaded by the user -- well, let's try : no, it doesn't. So are the audio tracks supposed to be loaded manually ?

DGIndex will extract the audio as it indexes and it's usually automatically loaded into the audio section for encoding, but the video and the audio are processed separately. If you open a file type for which MeGUI can't extract the audio, it'll create a script after indexing and load it into the audio section instead. That's assuming you selected audio as part of the indexing job. I don't know what OneClick does, but I assume it works the same way, behind the scenes.

I'll have to return and finish replying later.

[hello_hello]

I played around a bit and below is what happens when I use QAAC's no delay option, but when I didn't use it, the QQC encode looked identical to the original. I assume the 'no delay" option has something to do with the very beginning of the audio being silence, but given I only use "no delay" when re-encoding "soundtrack" audio I don't really care.

While I was playing around I discovered the AFTEN AC3 encoder was the odd-one out. When I converted to other formats using other encoders, foobar2000 reported the same number of samples as the original every time, except for AFTEN, when it didn't. The AFTEN AC3 encodes were a bit longer.
After a bit of playing around I discovered adding "-pad 0" to the AFTEN command line fixed that. It appears to work in a similar way to QAAC's "no delay" option. I don't quite understand though, why the AFTEN encode without -pad 0 appears to have a different total number of samples to foobar2000. For most other formats, the padding info is saved with the audio and the player should know exactly how much is padding and to ignore it, but for AC3, maybe there's not a similar option.

Anyway, the first waveform is the original (I split the stereo audio deleted the right channel after importing each time).
Second is qaac.
Third is QAAC with "no delay".
Forth is AFTEN with -pad 0
Fifth is AFTEN.

They all line up except the last one (although it's only about 6ms out), and QAAC with "no delay" has about 9ms of silence at the beginning where there should be audio.



I'm not quite sure what to make of this yet, but keeping in mind foobar2000 reports the same duration for each of these (same number of samples) except the last one, I'm thinking any encoder padding at the beginning is being ignored when the audio is imported (as it should be), but it's not if there's padding at the end.