Question regarding NTSC Film DVD

[txporter]

Ok, let's say I have a DVD that is NTSC film material that is pulled-down (3:2). So 23.976fps progressive displayed at 29.97fps. I am trying to understand the proper handling of this with Avisynth.

If I process the video through DGIndex with honor pulldowns for field operations, then I need to use an Avisynth script that looks something like this:

loadplugin("PathTo\DGDecode.dll")
loadplugin("PathTo\TIVTC.dll")

MPEG2Source("video.d2v")
TFM()
TDecimate(hybrid=1) <-- not sure about using hybrid=1 (this seems to be default in XVID4PSP)

If I process the video through DGIndex with force film for field operations, what do I need to do then? Something like this?

loadplugin("PathTo\DGDecode.dll")
loadplugin("PathTo\Decomb.dll")

MPEG2Source("video.d2v")
fielddeinterlace()

I am trying to understand the benefits and negatives to one versus the other. I am mainly ripping episodic disks of North American TV shows. I have been trying the first method I listed (although without hybrid=1), and I see what I am calling strobing. About half of the time (maybe 1-2s good then 1-2s strobing), the video seems to be moving up and down very slightly and then the other half it is perfectly smooth. I think it has something to do with the way I am processing the video, I am just not sure what I am doing wrong.

Should I be forcing film through DGIndex rather than post processing with avisynth?

Thanks in advance.

Jason

update: I guess I should say that I am trying to end up with 23.976fps progressive material.

[jagabo]

If you use Forced Film in DgIndex you don't need to deinterlace. You already get 23.976 fps progressive flim frames. You should only use this if your source is 23.976 progressive with 3:2 pulldown flags. Hard telecined video (like when recording broadcast TV with an MPEG encoder) will not work with this setting.

Using Honor Pulldown and TFM().TDecimate() will work better if your source is hard telecined or has a mix of soft and and hard telecined segments.

[txporter]

If you use Forced Film in DgIndex you don't need to deinterlace. You already get 23.976 fps progressive flim frames.

Using Honor Pulldown and TFM().TDecimate() will work better if your source has mixed soft and and hard telecine.

Ok, thanks. So if I force it to film in DGIndex, then all I need to do is serve it up with MPEG2Source and do whatever else I wanted to do with my script?

If I get jerky video after doing forced film, then that means I probably have a mix of soft and hard telecine, so I should honor pulldowns and IVTC. That sound about right?

[manono]

So if I force it to film in DGIndex, then all I need to do is serve it up with MPEG2Source and do whatever else I wanted to do with my script?

Yes, if it's soft telecine. How do you know? Run the Preview and if it says Progressive/Film, you're OK. Or have a look at the bottom of the D2V and it'll tell you the film percentage. The usual advice is that if it's over 95% film it's safe to use Force Film. Me, I refuse to use it unless it's 100% film.

Then if it's been hard telecined or is a hard/soft telecine mix make the D2V using Honor Pulldown Flags and IVTC in the script. A better script for mixes is:

TFM(D2V="C:\Path\To\Movie.d2v")
TDecimate()

But those aren't the only 2 choices and for other kinds of sources (pure interlace, field blended junk, etc.) you make the D2V using Honor Pulldown Flags and do other things (or nothing) in the script.

[txporter]

Thanks for the response, manono. That is pretty much what I thought. I was having trouble with the strobing that I spoke of in the first post and I thought it was some way that I was handling the video in avisynth. It was NTSC film material (98.5-99.9% film according to DGIndex). I tried Honored Pulldowns with TFM().TDecimate() and TFM().TDecimate(hybrid=1) as well as DGIndex Forced film with no IVTC in avisynth. All of them were exhibiting strobing in the final encode.

Well, it turned out that it was HCenc causing the problem. I was allowing HCenc (both 0.23 and 0.24beta) to auto-detect interlaced/progressive. When I forced it to progressive encoding, the strobing effect went away. Also, this happened with whatever codec that WMP 10(?) was using, but not with MPC-HC. It showed up on my tivo just like with WMP, so was using that as the debug player.

Also, as far as forced film vs honor pulldowns/IVTC: I think I am going to try Forced film for now. I noticed a large change in encoding fps with forced film vs running through TITVC (~140fps vs ~70fps).

I had tried the method described by Donald Graft to determine what type of video it was. AssumeTFF().SeparateFields() and look at a high motion scene for aabbcc (progressive), abcdef (interlaced), aaabbcccdd (3:2 pulldown).

[poisondeathray]

I don't know how "picky" you are, but if it's not 100% there is going to be a cadence break somewhere, and using "forced film" will have at least a few frames with errors

The reason why "forced film" is faster, is that it assumes the exact same cadence (any breaks or errors will result in artifacts), while TIVTC is adaptive and uses more cpu

[manono]

I don't know how "picky" you are, but if it's not 100% there is going to be a cadence break somewhere, and using "forced film" will have at least a few frames with errors

Yeah, but if you're lucky those breaks will be in black frames at the very beginning or very end. Often though, as you say, they occur at chapter breaks or elsewhere during the actual video and then you're screwed. I've been burned too many times with stray interlacing left over after a Force Film when the percentage was very high but not 100%. So I use that script above. And if you turn off post-processing (PP=0), it's nearly as fast as a Force Film.

[txporter]

If I look through the .d2v files, will those cadence breaks when it changes from x0 x1 x2 x3 ... to something more like x2 x2 x2 x2 or something like that? Theoritically, I should be able to go to those scene and see what is happening, right?

Also, when you say there is a cadence break, is it just one frame that will be messed up (or group if there are multiple together)? Or does it propogate throughout further frames?

manono, what does turning off post-processing do and what is this for? TDecimate?

What about using multi-threading? Is that possible for TITVC?

Thanks.

[manono]

If I look through the .d2v files, will those cadence breaks when it changes from x0 x1 x2 x3 ... to something more like x2 x2 x2 x2 or something like that?

Yeah, you can go and look for those places. Kind of a pain in the ass, though, when they're in the middle of the film.

Also, when you say there is a cadence break, is it just one frame that will be messed up (or group if there are multiple together)

If chapter breaks, you'll get 2 interlaced frames. Criterion is notorious for this because they (or their DVD production house) force chapter points during the encode, rather than when authoring. And it breaks the cadence. Otherwise, it can be any number of frames, and seemingly for no reason at all. Here's one I saved out 3 years ago before Donald Graft (neuron2) began to update the original DVD2AVI to become DGIndex:



It's going along at x0 x1 x2 x3 x0 x1 x2 x3, when all of a sudden you get this long string of 2s. Right in the middle of the movie. And for no good reason. It's no problem if watching on an interlaced CRT television, but it can wreak havoc on a progressive scan player outputting to a progressive display, depending on how good the player is and how quickly it spots the change.

manono, what does turning off post-processing do and what is this for? TDecimate?

It turns off the post-field match check for any interlacing that may have escaped:

TFM(D2V="C:\Path\To\Movie.d2v",PP=0)

It might be risky, but I've found it's usually pretty safe if used when the percentage is real high. It's all explained in the TFM doc:

PP -

Sets the post-processing mode. This controls how TFM should handle (or not handle)
any combed frames that come out of the field matching process. Possible options
are:

0 - nothing (don't even look for combed frames)
1 - find/hint combed frames but don't deinterlace
2 - dumb blend deinterlacing
3 - dumb cubic interpolation deinterlacing
4 - dumb modified-ela deinterlacing
5 - motion-adaptive blend deinterlacing
6 - motion-adaptive cubic interpolation deinterlacing
7 - motion-adaptive modified-ela deinterlacing

* You can also use the "clip2" parameter to specify an externally deinterlaced clip
from which TFM will take frames instead of doing the deinterlacing itself. See
the clip2 parameter description for more info and to see how the PP settings effect
operation with clip2.

* You can manually control what frames should be marked as combed as well as the
PP option through an overrides file (see the ovr parameter description)

Default: 6 (int)

What about using multi-threading? Is that possible for TITVC?

There are multi-threaded versions of AviSynth.

[txporter]

Thanks again for the in-depth reply.

I am looking through the TFM info on avisynth.org. It sounds like setting PP=0 will basically just not do anything with any frames that are found to be combed from whatever mode is set, right? Isn't this basically like using forced film then?

The latest version of XVID4PSP that I was playing with was using 2.5.8 MT. I just haven't played with multi-threading and I didn't know if there was information about there about whether using with TITVC was beneficial or harmful or whatever. I haven't really looked into it a whole lot, but I thought I just needed to add MT(filter) to use.

2 interlaced frames doesn't sound too bad. I didn't watch any of the shows all the way through, I just watched a bit and skipped around and went to the end to make sure that the AV sync was good. Nothing stood out, but I didn't look at one of those transitions in particular.

[manono]

Isn't this basically like using forced film then?

No, not at all. ForceFilm, in the absence of the RFF/TFF flags, just randomly drops every 5th frame. You'll get a jerky-playing-full-of-interlacing mess. An IVTC, even without post-processing, tries to match up the field pairs before Decimate/TDecimate then drops a dupe frame.

It sounds like setting PP=0 will basically just not do anything with any frames that are found to be combed from whatever mode is set, right?

Found to be combed after TFM has already done its thing. That's why it's post-processing

TIVTC works well with any MT version of AviSynth.

I didn't watch any of the shows all the way through,

Then, if it's hard telecine, you leave on the post-processing. If it's a high film percentage mixed hard/soft telecine, you're usually OK in leaving off the PP.

[txporter]

Awesome stuff! Thanks!

[txporter]

Just a follow-up on this to answer some of my own questions and provide some data on some of the suggestions.

I tried running TFM using PP=0. It was ~10% faster than running at default (which is believe is PP=6). I think for the relatively small improvement in encoding speed, I am better off just letting it run with default settings when I need to IVTC (in case there are some weirdly encoded scenes).

I tried running TFM/TDecimate with multi-threading. From what I could read online and from what I tried, TIVTC doesn't seem to work with multi-threading.

For simplicity, I have decided to adopt what manono and pdr suggested and am encoding with 3 basic profiles:
1. 100% film as forced film in DGIndex and progressive encoding in HCenc
2. <100% film (but still mostly film) as honor pulldowns with TIVTC and progressive encoding in HCenc
3. pure interlaced as honor pulldowns and interlaced encoding in HCenc.

I haven't run across any of the mixed blended stuff yet, so nothing needed on that front yet.

Thanks again for all the help!

[manono]

I tried running TFM/TDecimate with multi-threading. From what I could read online and from what I tried, TIVTC doesn't seem to work with multi-threading.

Works fine here with:

SetMTMode(2)

at the top of any scripts with:

TFM().TDecimate()

or any TIVTC variations in them. I have a Core2Duo at the moment.

[txporter]

I tried running TFM/TDecimate with multi-threading. From what I could read online and from what I tried, TIVTC doesn't seem to work with multi-threading.

Works fine here with:

SetMTMode(2)

at the top of any scripts with:

TFM().TDecimate()

or any TIVTC variations in them. I have a Core2Duo at the moment.

Yeah, I didn't write exactly what I meant. I did some searches and most of the threads that I saw were pointing to some threads on Doom9 which I couldn't access over the weekend. The snippets in google mentioned that TFM wouldn't work with multi-threading. I did try SetMTmode(2) and it did not error or anything like that, but I saw no/little speed benefit (0-10% fps).

Based on how SetMTmode or MT works, they don't seem very compatible with TFM anyhow though, so maybe that is why I didn't notice a speed benefit. SetMTmode basically feeds every other frame to a different thread. That seems to defeat the purpose of a filter that is looking for duplicate frames. MT splits each frame in two (or more) and hands each to a different thread. Now it would seem you need to compare the results between the two threads to determine which frame to mark as a dup. Maybe I am looking at it wrong??

This is on a Q6600.