Correct deinterlace/IVTC settings for this PAL DVD ("Ikiru", Kurosawa 1952)

[abolibibelot]

Hi,

I'd like to transcode a movie on a DVD to an MKV file, and I'm quite confused as to what kind of frame pattern it has and what settings I should use. It's an old movie, "Ikiru" from Akira Kurosawa (1952), and it's a PAL DVD in 4:3 picture format. I ran an encode using QTGMC as deinterlacer, but then realized it was probably wrong for a movie DVD (resulted in 50FPS frame rate). I'm quite new to this (at least to doing this right without a mostly automated software which takes those kinds of decisions for the user) ; from what I read it should be originally 24 or 23.976 FPS footage converted in some way to PAL 25 FPS, but both MeGUI and DGIndex declare it as "interlaced". I found some interesting tips in that thread :
http://forum.doom9.org/showthread.php?t=159639
but I'm not sure how to apply them for that particular movie (the above discussion deals mostly with NTSC, and anime as opposed to film footage). Following an advice in a former discussion on this forum I tried to apply a bob deinterlacer in MeGUI's Avisynth Script Creator module and examine the video frame by frame in the preview, but I'm not sure of what I should look for, what are all the possible patterns in that case, and how I should deal with each of them.
Besides I'd like to be sure if the other settings I chose are correct :
- crop(10, 4, -14, -4) resulting in 696x568 resolution (satisfies mod8 condition, not mod16, I let it as is with no resizing or over-cropping),
- aspect ratio ITU 4:3 PAL (1.367521),
- x264 at CRF=20, preset=slower, Tune=Film, Profile=High@L3.1
Another question : the audio is in mono, is there a special setting to get the AAC stream in mono too, or is AAC required to have at least two channels ? (I chose "Keep original channels" in QAAC's configuration, but it still produces a stereo stream -- anyway the encoder seems to deal with it efficiently as the bitrate is quite low at 83kbps, despite the high quality setting of True VBR Q=91. I'm only surprised that the maximum bitrate is only 93kbps, thus very close to the average, despite there being almost silent scenes -- except a continuous background noise -- and scenes with dynamic music or street noises, I'd have thought it would vary much more with a codec aimed at maximum compression efficiency.)

Here's a short sample (scene with a lot of movement, should make it easier to analyse) :
http://www.mediafire.com/watch/y4x0d02mxc7gv1y/Kurosawa_Ikiru_extrait_47min40.demuxed.m2v

And the current script :
LoadPlugin("C:\Program Files\MeGUI_2507_x86\tools\dgindex\dgdecode.dll")
LoadPlugin("C:\Program Files\MeGUI_2507_x86\tools\avisynth_plugin\NicAudi o.dll")
global MeGUI_darx = 4 *
global MeGUI_dary = 3
Vid = MPEG2Source("Kurosawa Vivre VTS_01_1.d2v")
Aud = NicAC3Source("Kurosawa Vivre VTS_01_1 T80 1_0ch 256Kbps DELAY 0ms.ac3")
VidDeint = Vid.QTGMC(Preset="Medium", EzDenoise=1).crop(10, 4, -14, -4)
Mix = AudioDub(VidDeint, Aud)
Return(Mix)

* Do those values (which the Avisynth Script Creator module always puts there) serve any actual purpose ? It seems like they get superseded by the option chosen in the "Display Aspect Ratio" field in MeGUI's "One-click encoder" (at first I calculated myself the theoretical aspect ratio by taking into account the cropped black bars, obtained a corrected ratio of 15:11, so I put "darx = 15" and "dary = 11", but since the aspect ratio was set at 4:3 (1.33333) in One-click encoder, the test encode ended up with the same 1.33333 ratio as the one done before with "darx = 4" / "dary = 3".)


Thanks in advance.

[jagabo]

QTGMC(preset="medium") # or Yadif(mode=1)
SRestore()

But there are much better versions of that movie, why bother:
http://www.amazon.com/Criterion-Collection-Blu-ray-Takashi-Shimura/dp/B0141RBHTU/

[LMotlow]

I'd like to transcode a movie on a DVD to an MKV file

Why? Is there a particular reason for re-encoding this classic issue, or do you just like lower quality?

[jagabo]

Given the poor shape it's in it may look a little better after cleanup and re-encoding.

[LMotlow]

Sorry, I didn't notice anything in this thread having to do with "improvements". I did notice some unnecessary cropping that messes up the original image ratio. I'll try taking another look.





Nope. Nothing there. Maybe other readers can point out something I missed in the above posts.

The Criterion digital restoration (that's restoration, not "enhancement") will be the real improvement.

[abolibibelot]

@LMotlow : Well, as for the "why" : I'd like to share it with someone (and maybe others then) who probably won't know how to read a DVD folder on her computer (or won't bother and will forget it, that's how people are nowadays, and there are so many sources of entertainment out there... considering it's not a very funny movie to begin with it better be easily accessible from a technical standpoint -- right now she's watching Star Trek: The Next Generation with her husband (their wedding was two weeks ago) and often falling asleep in the middle of a 45min episode, so a 2h20min movie in japanese about a guy dying from cancer might be a stretch. Not that I think ST:TNG is low grade entertainment by the way : I only watched one episode ("The inner light") but it's one of the very best and most moving stories I've ever seen on a screen, regardless of the genre.) (Her brother -- for whom I'm converting the drum lessons DVDs from the other thread by the way -- also has a very low attention span : once I tried to have him discover "2001 A Space Oddyssey" and he could barely make it through "The Dawn Of Man", asking me to stop just when people on the Moon started talking. *sigh*)
And (correct me if I'm wrong) I don't think there would be a noticeable quality loss with the settings I indicated (CRF=20 is usually considered almost "transparent" to the source).



@ jagabo : Indeed I haven't looked very long if there were better versions "available" (wink-wink). It has to have french subtitles though, and I only found lousy AVI files with french subs -- but I could probably find those in SRT somewhere (or convert the DVD subtitles to SRT), then resynchronize and remux that with the Blu-Ray version...

"Given the poor shape it's in it may look a little better after cleanup and re-encoding. "

I suppose you were answering to LMotlow here ? What other cleanup filters would you think of in this case ? Apparently it's not worth the trouble, but it's still interesting, as such old movies must have very specific kinds of defects.
Anyway, if I can at least learn something from it : what is the frame pattern here, how can I reliably determine it for any video, and what does SRestore do to it ? As for QTGMC vs. Yadif, I did some comparisons when I discovered the former, the quality improvement is so obvious that it's almost painful to use a lesser deinterlacer afterwards (although Yadif is considered one of the best of the rest), even though it's much, much slower.

[hello_hello]

* Do those values (which the Avisynth Script Creator module always puts there) serve any actual purpose ? It seems like they get superseded by the option chosen in the "Display Aspect Ratio" field in MeGUI's "One-click encoder" (at first I calculated myself the theoretical aspect ratio by taking into account the cropped black bars, obtained a corrected ratio of 15:11, so I put "darx = 15" and "dary = 11", but since the aspect ratio was set at 4:3 (1.33333) in One-click encoder, the test encode ended up with the same 1.33333 ratio as the one done before with "darx = 4" / "dary = 3".)

I don't use the OneClick encoder but when you add a script to the job queue via the queue button in the Video section of the GUI, MeGUI uses those claues to calculate the correct pixel aspect ratio (when anamorphic encoding is enabled in the script creator). What OneClick does if you load the same script for encoding, I don't know.

Or check in MeGUI's options to see if the "use ITU aspect ratiio" option is checked. That should make the aspect ratio fairly close to 15:11 when MeGUI opens a 4:3 video.

Can't you over-ride the aspect ratio in the OneClick encoder? You'd probably need to create a custom aspect ratio if you want exactly 15:11 (ITU is very close, but not quite the same).

[hello_hello]

I'd like to transcode a movie on a DVD to an MKV file

Why? Is there a particular reason for re-encoding this classic issue, or do you just like lower quality?

Removing the frame blending in this case. I've never been a fan of black and while movies that aren't black and white. Browny-greeny white doesn't do it for me. Or black borders. Cropping them saves having to put the TV in 4:3 mode to enable overscanning to remove them. And for interlaced video QTGMC does a better job at de-interlacing than my TV. Can you trust a player to resize using an ITU aspect ratio these days? Smaller file size is a plus. The more problematic the source, the more likely I'm going to learn something new while re-encode it.

I'll confess I don't know why cropping should mess with the aspect ratio. Just how much of the picture is encoded

[hello_hello]

The attached sample was encoded with the following script. I always crop and resize to square pixel dimensions. For a video such as this, I'd crop so I can resize to exactly 4:3. If you want anamorphic 4:3, remove the resizing and/or cropping etc. For myself, I'd probably resize to 640x480. There's not enough detail to see any quality loss through downscaling a little, but resize however you prefer.

LoadPlugin("C:\Program Files\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\Kurosawa Ikiru extrait 47min40.demuxed.d2v")
GreyScale() # make it black and white
QTGMC() # you might be able to de-interlace with MeGUI's "Yadif with Bob" instead, but QTGMC will probably do a better job.
SRestore() # deblends the 50fps video back to 23.976fps. SRestore
crop(14, 4, -14, -6) # remove the crud
Spline36Resize(704,528) # resize to 4:3 worth of square pixels.
gradfun3() # helps prevent banding/blocking when encoding, requires the dither package.

If you've already got QTGMC working, then adding the required files for SRestore and the Dither Package shouldn't be hard. If the Dither Package contains plugins you already have for QTGMC, replace them with the Dither Package versions if they're not the same.
QTGMC is fine for de-interlacing video (referring to your original question). If it's truly interlaced, QTGMC will de-interlace 25fps PAL to 50fps progressive (as will MeGUI's "Yadif with Bob"). Each frame will be unique and motion will look smoother than de-interlacing to 25fps progressive. If you de-interlace with Yadif, the "Yadif with Bob" option should look better.

Regarding the audio. If it's mono in 2 channels (ie stereo but each channel is the same) it takes very little extra bitrate to encode than a single mono channel. There's a slim chance a surround sound system might put a single mono channel in the centre channel rather than the left and right speakers, so it's probably better not to downmix to mono. If it is mono, MeGUI might automatically convert it to dual channel mono (ie stereo) by default. I'm not sure. Have you tried selecting the "convert to mono" option in the audio encoder configuration? My guess is the audio's a long way from being high fidelity, which probably also contributes to a low bitrate when converting it.

[abolibibelot]

The attached sample was encoded with the following script.

Thanks for this thorough contribution. I kinda renounced re-encoding this particular movie myself (which would require letting my computer -- midrange 2009 config -- run a whole day to get a good result, with QTGMC and x264 preset "slow", considering it's 2h23min long) after others convinced me I could find better quality alternatives (and indeed I did), but it's still an opportunity to learn some new tricks.
The output looks cleaner and sharper than the source (plus the aspect ratio is more realistic -- but maybe VLC player has trouble displaying ITU standard correctly) :
http://share.pho.to/9lkTj

But comparing it with the higher definition file I downloaded yesterday (third picture), the quality gain doesn't seem so obvious : the woman's face is more defined, but the overall picture seems softer, less sharp, while the "colors", or rather the shades of grey seem duller. Arguably it's not a high quality conversion (just 800kbps for 960x720 resolution, 954MB for 2h23m), but it shouldn't affect things like colors / contrast, should it ? Or is the PAL DVD video artificially oversharpened with enhanced contrast or boosted white balance, giving me a wrong impression ?

LoadPlugin("C:\Program Files\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\Kurosawa Ikiru extrait 47min40.demuxed.d2v")
GreyScale() # make it black and white

Is it more of an aesthetic preference, or does it greatly improve the encoding efficiency / accuracy ?

QTGMC() # you might be able to de-interlace with MeGUI's "Yadif with Bob" instead, but QTGMC will probably do a better job.

Indeed I made some tests when I discovered QTGMC after reading several discussions where it was praised as the best available deinterlacer, and the improvement over Yadif was stunning.

gradfun3() # helps prevent banding/blocking when encoding, requires the dither package.

In general what kind of footage benefits from using that filter ?

If you've already got QTGMC working, then adding the required files for SRestore and the Dither Package shouldn't be hard. If the Dither Package contains plugins you already have for QTGMC, replace them with the Dither Package versions if they're not the same.
QTGMC is fine for de-interlacing video (referring to your original question). If it's truly interlaced, QTGMC will de-interlace 25fps PAL to 50fps progressive (as will MeGUI's "Yadif with Bob"). Each frame will be unique and motion will look smoother than de-interlacing to 25fps progressive. If you de-interlace with Yadif, the "Yadif with Bob" option should look better.

So if using Yadif default setting, or QTGMC with "SelectEven", SRestore doesn't work properly because of the lost information ?

Regarding the audio. If it's mono in 2 channels (ie stereo but each channel is the same) it takes very little extra bitrate to encode than a single mono channel. There's a slim chance a surround sound system might put a single mono channel in the centre channel rather than the left and right speakers, so it's probably better not to downmix to mono. If it is mono, MeGUI might automatically convert it to dual channel mono (ie stereo) by default. I'm not sure. Have you tried selecting the "convert to mono" option in the audio encoder configuration? My guess is the audio's a long way from being high fidelity, which probably also contributes to a low bitrate when converting it.

Indeed, it's quite crappy as expected for a movie of that period (the 720p BR rip file is barely better in that regard). That was just out of curiosity, and I get that dual channel mono audio requires very little extra bitrate if the encoder is doing it right, which it apparently does.

[manono]

LoadPlugin("C:\Program Files\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\Kurosawa Ikiru extrait 47min40.demuxed.d2v")
GreyScale() # make it black and white

Is it more of an aesthetic preference...

An aesthetic preference? It's not black and white. Since greyscaling is so basic, it's just further evidence of a great movie being butchered when released for home video by this fly-by-night company. Get the Criterion version. Or, if you prefer PAL, the BFI version which doesn't have that atrocious field-blending:

http://www.dvdbeaver.com/film/DVDCompare2/ikiru.htm

[jagabo]

It's your DVD video that has crappy levels, no true blacks, blown out brights, and contrast pumping from an autogain filter. It also has severe oversharpening halos (what you're interpreting as sharpness and looks terrible when blown up on a big screen), impure greys, has been over compressed, etc. It was obviously made from a poor NTSC to PAL converted video tape. Here's the same frame from a cleaner 960x720 version I found (~7 GB x264 mkv):



Even this I doubt is from the Criterion Blu-ray release. It's got lots of scratches, film bounce, etc.

So if using Yadif default setting, or QTGMC with "SelectEven", SRestore doesn't work properly because of the lost information ?

SRestore() works by preferentially removing blended frames. You need to convert every field to a frame before SRestore() so it can remove those blended frames. If you decimate before SRestore() you will be throwing out a lot of the good frames, not leaving enough for SRestore() to work with.

Indeed, it's quite crappy as expected for a movie of that period

35mm negatives have the potential of well over Blu-ray's 1920x1080 resolution. What determines the quality of a release is how the movie was shot, whether negatives or prints are available, how well they were stored, how well they were digitized and processed, and finally the medium they're delivered on. One doesn't expect a 1050's film to be crappy simply because of when it was made. It's just that the surviving copies may not be in good shape and the potential profits may not justify an intense cleanup.

[abolibibelot]

It's your DVD video that has crappy levels, no true blacks, blown out brights, and contrast pumping from an autogain filter. It also has severe oversharpening halos (what you're interpreting as sharpness and looks terrible when blown up on a big screen), impure greys, has been over compressed, etc. It was obviously made from a poor NTSC to PAL converted video tape. Here's the same frame from a cleaner 960x720 version I found (~7 GB x264 mkv):
[pic]
Even this I doubt is from the Criterion Blu-ray release. It's got lots of scratches, film bounce, etc.

Damn... Apparently the poor thing is as lousy as my ability to appreciate the quality of a moving picture...
Over compressed also ? The whole movie is 6GB (without the menu and presentation), so it doesn't exploit the full storage capacity of the medium (I'm not sure how much it is for a commercial DVD, at least 8GB I'd say) but I've seen worse.
And so whenever a movie exists in Criterion edition it's consistently the best there is ?

35mm negatives have the potential of well over Blu-ray's 1920x1080 resolution. What determines the quality of a release is how the movie was shot, whether negatives or prints are available, how well they were stored, how well they were digitized and processed, and finally the medium they're delivered on. One doesn't expect a 1050's film to be crappy simply because of when it was made. It's just that the surviving copies may not be in good shape and the potential profits may not justify an intense cleanup.

Here I was refering to the audio only (I doubt it's significantly better on the 7GB BRRip you mentioned, even if it's the native uncompressed track). I don't know if there's a technical reason for that, but, for movies made in the 1950s or before (I'll admit I haven't seen that many) even when the picture is beautifully preserved, the audio generally feels dull and dusty.
But yes, I know that quality film has a lot of potential, and it might be interesting to see in a few years that current movies shot in native 1080p will quickly look awfully dated whereas superbly crafted film-based movies such as "2001: A Space Odyssey", made decades ago, may look even more stunning than they do now in their next stage of magnification.

[jagabo]

And so whenever a movie exists in Criterion edition it's consistently the best there is ?

I wouldn't go that far. But it's usually much better than your DVD.

Here I was refering to the audio only

Yes, audio has improved a lot since the 1950's.

[hello_hello]

LoadPlugin("C:\Program Files\MeGUI\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source("D:\Kurosawa Ikiru extrait 47min40.demuxed.d2v")
GreyScale() # make it black and white

Is it more of an aesthetic preference, or does it greatly improve the encoding efficiency / accuracy ?

I'd imagine a b/w movie should be b/w. I've seen a few where they appear to be a very mild shade of brown and white, or something similar, which might make the picture look a bit warmer. Similar to adjusting the colour temperature on your TV/monitor, I guess. Whether that sort of thing is done deliberately, I don't really know. I doubt it'd hurt compressibility much unless there's a lot of chroma noise. I tend to use greyscale() out of habit when encoding b/w movies.

gradfun3() # helps prevent banding/blocking when encoding, requires the dither package.

In general what kind of footage benefits from using that filter ?

I generally use it whenever I use noise filtering, because noise filtering can cause colour banding. Noise tends to prevent banding. Gradfun3() adds dithering, which is a noise you can't see, to help prevent banding, or it can help reduce existing banding.

So if using Yadif default setting, or QTGMC with "SelectEven", SRestore doesn't work properly because of the lost information ?

Correct. SRestore does have other functions such as "double blend" removal, which removes the blending that results when film is converted to 29.970fps using pulldown, but de-interlaced to 29.970fps progressive instead of the pulldown being removed, and it works pretty well. I don't use SRestore much myself though so I'm not overly familiar with it.

Indeed, it's quite crappy as expected for a movie of that period (the 720p BR rip file is barely better in that regard). That was just out of curiosity, and I get that dual channel mono audio requires very little extra bitrate if the encoder is doing it right, which it apparently does.

It's something I used to understand but really need to read about it again to refresh my memory. It's to do with the way the encoders work. They can encode the left and right channels combined as a single channel, then the difference between the left and right channels as another channel (L+R and L-R). And I think (for MP3 at least) they switch between stereo and joint stereo modes as they encode, depending on the audio content. AAC probably works in a similar manner, but for stereo audio that's "dual mono" there'd be no L-R information to encode.

Edit: I found a link. http://wiki.hydrogenaud.io/index.php?title=Joint_stereo

[hello_hello]

The output looks cleaner and sharper than the source (plus the aspect ratio is more realistic -- but maybe VLC player has trouble displaying ITU standard correctly)

There's no way for a player to know if a DVD should be ITU or not. All software players I know of resize to exactly 4:3 or 16:9. I've often wondered why they don't use ITU for 4:3 DVDs, given I'm pretty sure 99% of 4:3 DVDs would be ITU. Maybe people would complain if a 4:3 video doesn't completely fill a 4:3 monitor. Using 16:9 for 16:9 DVDs makes more sense, given ITU 16:9 DVDs are probably in the minority.

I don't know about VLC but it's easy to resize using MPC-HC. A single tap on the "6" key stretches the width to something pretty close to ITU. I think it stretches by about 2% each time you tap the 6 key, and ITU is about 2.3% - 2.5% wider than non-ITU (depending on which pixel aspect ratio religion you subscribe to), so it's not far off. tapping the 5 key resets it so it's easy to adjust it manually.

[abolibibelot]

I generally use it whenever I use noise filtering, because noise filtering can cause colour banding. Noise tends to prevent banding. Gradfun3() adds dithering, which is a noise you can't see, to help prevent banding, or it can help reduce existing banding.

Alright, but then doesn't this "noise-you-can't-see" in turn impact the compressibility ? :^p

There's no way for a player to know if a DVD should be ITU or not. All software players I know of resize to exactly 4:3 or 16:9. I've often wondered why they don't use ITU for 4:3 DVDs, given I'm pretty sure 99% of 4:3 DVDs would be ITU. Maybe people would complain if a 4:3 video doesn't completely fill a 4:3 monitor. Using 16:9 for 16:9 DVDs makes more sense, given ITU 16:9 DVDs are probably in the minority.

Those drum DVD I have to encode which are in 4:3 (like the one from the other thread I created the other day), I'm pretty sure they're strictly 4:3 (no black bars on the sides, and 4:3 / 1.3333 feels "natural"). But for films, indeed, it seems to be the vast majority.

[hello_hello]

Alright, but then doesn't this "noise-you-can't-see" in turn impact the compressibility ? :^p

Probably a little, as I guess effectively it's forcing the encoder to encode more detail, at least when it comes to smooth gradients.
Still, if you apply it after using a noise filter, it'd impact compressibility far less than the original noise would have, and it looks better than unnatural banding.
If you're using low bitrates or high CRF values though, it probably won't survive the compression and you might still end up with banding anyway, in which case there's probably less point to using it. For really problematic sections of video, I often encode them using CRF15 or CRF16 while applying Gradfun3 if that's the only way to prevent banding (you could probably tweak x264 settings to make it encode them more accurately, but either way, you're still using a higher bitrate), while I encode the bulk at a higher CRF value such as CRF18. Then I append the encoded video together, but while I'm conscious of bitrate I do tend to favour quality.

Those drum DVD I have to encode which are in 4:3 (like the one from the other thread I created the other day), I'm pretty sure they're strictly 4:3 (no black bars on the sides, and 4:3 / 1.3333 feels "natural"). But for films, indeed, it seems to be the vast majority.

I'd vote they're ITU. There's ITU-like black down the sides in places (it only takes 8 pixels each side for the video width to be 704).
I don't trust my eyes completely, and I've been convinced something is ITU (or not) then come back another day and wondered what I was thinking, but surely somewhere in those DVDs there's a straight on shot of something round, given drums tend to be round in nature.

This is the closest I could find in your samples. It's not straight on, but close. Even the "draw a circle" test was fooling my brain, because I could draw a circle around the floor tom in both pics (a perfect circle each time) and when I switched between them it looked like the drawn circle was changing shape along with the picture, even though it can only change size. When I got the drawn circles pretty much the same though, the image starts to look ITU to me. And without drawn circles, the snare drum looks like it's round in the ITU screenshot, but that's just me.....

4:3


ITU:


Edit: I thought I'd add.... I'm fairly convinced DVDs aren't always the correct aspect ratio, whether they're officially ITU or not. I was encoding some old 16:9 DVDs of Stargate SG1 a while back. The aspect ratio of MGM's roaring lion at the beginning of each episode wasn't exactly the same each time, and in one episode there was a shot of a diagram of a stargate being displayed on a laptop that was almost round, it faded into another diagram of a stargate on a computer screen that was squashed like a football, and it faded into a screenshot of a stargate that was actually round.... if you didn't use ITU resizing. How stuff like that goes unnoticed, or how/why it ends up that way, I don't know... but I'm fairly sure the pilot episode was ITU while the rest were not, although I couldn't prove it to myself.

And recently I've encoded quite a few different 16:9 DVDs containing BBC video, and developed a theory all BBC 16:9 DVDs are ITU even if there's no black down the sides. I can't prove it either, but I'm fairly convinced.....