Will DVD2AVI Force Film do the same as AVISynth Decimate?

[LisaB]

The basic problem is this: I have a MPEG2 source - I guess the encoder made a mistake, because every 5th frame is a duplicate (when viewed in VDubMod).

I was going to make a project file using DVD2AVI and then use the Decimate() filter of the Decomb package......But, I notice that by just using the Forced Film option in DVD2AVI, I can get the same result!

In other words - I create a project file in DVD2AVI with Force Film enabled (despite the fact that preview in DVD2AVI yields pure NTSC 29.97), and then use MPEG2Source() in AviSynth, and then open the AVS file in VDub, and the duplicate frames are gone - I get 24 fps progressive frames.

I am confused because I didn't think that Forced Film was intelligent, as Decimate() is. I am also wondering whether the duplicate frames are just a result of RFF flags in the MPEG2 stream, but I don't know how to go about figuring this out. I have DoPulldown and ReStream, but am confused about how I should use them to analyze this problem.

Lisa B.

[au7usa]

I sure hope it correct cause that the way i've been doing it.

Sometimes DVD2AVI cant "Force-FILM" on DVD's that i know were originally 23.976 fps. They say it due to editing, something about throwing the count off. I've been using AviSynth to restore them to FILM.

According to everything i've read, telecining creates new frames by comparing the before and after frames, and then blending the two together.

I think its just two different means to the same end.

http://www.doom9.org/ivtc-tut.htm

[au7usa]

also (shorter)
http://www.doom9.org/mpg/d2a-mpeg2dec.htm

[adam]

It sounds like your source is stored at 29.97fps, in other words it is hard telecined. There are physically 29.97 frames every second. Forced film does not, and cannot be used on these sources.

Forced film takes advantage of sources which are stored internally at 23.976fps, meaning the decoder telecines them to 29.97fps during playback. The vast majority of NTSC DVDs are like this. All forced film does is bypass the pulldown flags imbedded in the stream. Essentially it decodes it without parsing the flags, thus without telecining it to 29.97fps so the encoder is instead fed with 23.976fps progressive frames...like it is stored on the disk. It has nothing to do with being intelligent, all it is doing is frameserving the source directly. If the output is usable or not depends entirely on how the source is stored. If you enable forced film on a true NTSC source (stored at 29.97fps) it will randomly decimate frames to get to 23.976fps. Needless to say, the result will be random and terrible.

Now the framerate that dvd2avi reports is absolutely meaningless. This is only reporting the output framerate that you will get at your current settings. So, enable forced film and it will report 23.976fps. Disable it and it will report 29.97fps.

The only thing you need to look at is whether it says NTSC, FILM, or some percentage FILM, ie: 95% FILM.

If it shows 95% or more FILM then forced film will work. If it shows anything else, forced film won't work. You have to disable forced film and treat it as 29.97fps interlaced, meaning you can either encode it at that framerate or you can run an inverse telecine filter on it during encoding. Personally I prefer to use decomb for doing inverse telecines.

[au7usa]

Is your source a DVD or MPEG-2 capture?

[SLK001]

The FORCE FILM option only strips out the 2:3 pulldown flags for FILM based (24fps) encoded streams and changes the fps setting. If you have a hard telecined video, then FORCE FILM will only change the fps setting in the header for you (a bad thing).

DECIMATE should be used (under most circumstances) in conjunction with TELECIDE to remove the hard telecining from a video (usually, some sort of capture), and is generally not needed for a DVD (I've only had one DVD that I had to use DECOMB to "fix").

[au7usa]

So is this corrct then? (i'm not the original poster by the way)

---------------------------------
LoadPlugin ("C:\Ut\Video\DVD2AVI.dg\MPEG2Dec3dg.dll")
#
Mpeg2Source ("D:\Out Of Time\Out Of Time.d2v")
#
# Source material reports as 29.97 100% interlaced
#
# Convert to 23.97 fps Progressive
Telecide (guide=1,gthresh=50,chroma=true,threshold=30)
Decimate (Cycle=5)
#
LanczosResize (400,168,2,58,718,359)
AddBorders (0,4,0,4)
---------------------------------

This was from "Out Of Time" w/ Denzel, and refused to take the Forced FILM setting (It was 23.976 fps progressive but looked like crap)

AviSynth produced perfect result (I couldn't find anything wrong with it), and no audio synch problem.

[au7usa]

Oh i think i see what your saying!

The DVD rips that would accept Forced FILM were actually 24fps and were tagged to playback at 30fps, and those phantom frames didn't really exist.

I'm guessing that they did really exist in the "Out Of Time"......Hard Telecined?

Correct or No?

[LisaB]

Thank you very much Adam and SLK001! You have both fully answered the question I was asking. Unfortunately, you have given me contradictory answers, and I would really like to know which one of you truly understands the DVD2AVI Force Film.

SLK001 says:

The FORCE FILM option only strips out the 2:3 pulldown flags for FILM based (24fps) encoded streams and changes the fps setting. If you have a hard telecined video, then FORCE FILM will only change the fps setting in the header for you (a bad thing).

Adam says:

If you enable forced film on a true NTSC source (stored at 29.97fps) it will randomly decimate frames to get to 23.976fps.

LisaB asks:

So which is it? How does Force Film treat a non-film source? You two agree that the output stream will have a film frame rate. But Adam claims DVD2AVI will reduce the number of physical frames, where SLK001 claims it won't. Obviously, if SLK001 is correct, then force-filming a non-film source will cause the output stream to have a longer duration, and terrible audio synch will ensue.

Lisa B.

[adam]

No contradiction. Well dvd2avi definitely does not patch the header because its just a frameserver. It can't change anything in the source and all it ever does is export the movie 1 frame at a time. But the point is that it treats it as if it were a 23.976fps source. So when it frameserves it only outputs 23.976 frames each second, thus any extra frames are randomly decimated. Its not that it is removing them per se, its just that it is skipping them when outputting the video to your editor or encoder, because it assumes they aren't there. Go ahead and try it. Regardless of how the source is encoded, if you enable forced film and then frameserve to something you will always have true 23.976fps output, it just will look like crap if it was stored as anything other than 23.976fps internally.

[LisaB]

Well Adam,

I did the test. Encoded a short pure NTSC clip using TMPGEnc. Created d2v using force film. Opened the result in VDub (via avisynth). The resulting stream had fewer total number of frames, but the same duration.

So, I'm going with you on this one. DVD2AVI definitely decimates.

But look back at my initial post - I found that DVD2AVI seemed to decimate smartly - or at least too smart to be random. In other words, the frames that were decimated were the "duplicates", so that the output stream had few dupe frames...i can't say there weren't some dupes, but much fewer than one would expect from dumb decimation.

Can you tell me appropriate settings in DoPulldown that would allow one to undo soft telecining? I'm guessing I should use the following:

Frame Rate = 23.976
TFF = Even
Drop Frame = Disabled
No Pulldown = true
No RFF = true
all other options = No Change or off

Do you think this would work? I don't think you would want to leave TFF at No Change, because then the TFF flags would alternating...of course that doesn't matter if the source is true progressive frames, but still I think its a good idea. I don't know about Drop Frame, either, but figured that to strip out all the flags, it should be disabled...

Lisa B.

[adam]

I don't know exactly how dvd2avi determines which frames get parsed or not when forced film is applied inappropriately. I doubt its using any sort of advanced method of analysis, its probably just a set pattern it uses, which relies on the fact that most footage is telecined with the same pattern. Another thing to consider is that in a typical hard telecined source, the amount of progressive source frames is greater than the interlaced frames (duplicate frame created from repeated fields), so its largely a matter of probablility as well. But in any case, obviously its not preferable to use forced film under these circumstances.

Stripping the pulldown flags from the stream and then processing it will yield the exact same results as using forced film. Yes that is the correct method to use. Actually drop frame will always be flagged for an NTSC source regardless of whether its stored as 23.976fps or 29.97fps. You can either enable it or set it to no change, but disable it. As for tff I don't really think it matters for a progressive source.

[SLK001]

But it DOESN'T decimate. It just doesn't apply the PULLDOWN flags, which would give you MORE frames by a 2:3 pulldown at 29.97fps, but the SAME DURATION.

If the video is encoded at 23.976fps, then there are only "X" number of frames. When the PULLDOWN flags are applied, then "X times 1.25" frames are displayed - both at the SAME DURATION.

DVD2AVI has no magic formula for telling what a video is, it just looks for the flags. That's why you can't always depend on its determination. If you apply FORCE FILM, it just STRIPS the out the flags, and gives you a resulting fps that is based on this formula:

(ORIGINAL FPS x 4) / 5

If you apply a FF to a 23.976fps video, you will get a resulting fps of 19.181fps. Try that experiment and you'll see.

[adam]

SLK001 dvd2avi does not apply pulldown flags and it never strips them out. The pulldown flags (RFF/TFF) are either in the mpeg stream or they aren't. DVD2avi is a decoder just like any software player. Depending on whether you enable forced film or not it either parses the pulldown flags or it doesn't. For a typical NTSC DVD stored at 23.976fps interal yes all it does is ignore the flags and output the source as it is stored on the disk.

But with pure NTSC footage (hard telecined) it is not a matter of simply ignoring the pulldown flags because obviously they don't exist. The video is physically stored at 29.97fps and dvd2avi is going to output 23.976fps. I see from your example that its not a matter of always outputting 23.976fps, but rather a simple equation to determine the output framerate. I remember this now from all those posts where people applied pulldown to PAL footage and got strange results.

But the point is still the same. With a pure NTSC source you have 29.97fps individual frames. If you enable forced film it is only going to export 23.976 of those frames each second. It is selectively neglecting to frameserve certain frames and it is not reversing the telecine process ala an IVTC or bypassing the telecine process via not processing the pulldown flags. Some of your frames really are magically disappearing. This is what I would call decimation. I suppose a more accurate term would be frame dropping.

[LisaB]

SLK, I already did the test. My test clip had no RFF flags, yet the number of frames as reported by VDub actually decreased by the appropriate factor after running it through DVD2AVI with force film. It most certainly decimates.

I'll agree with you (SLK) on one thing. Adam says it doesn't patch the header, but I say that it must "virtually" patch the header....the frameserver provides frames to the encoder "on demand", so I think it is confusing when Adam says the frameserver "outputs 23.976 frames each second". It doesn't, unless you happen to be encoding in exactly real-time. It doesn't take exactly an hour to encode a 1-hour clip, so no, the encoder can't "guess" the framerate by the rate at which the frameserver is serving! I'm guessing that DVD2AVI updates the timecodes of the frames to reflect the new fps...this is "virtually" patching the header. (this last sentence is wild speculation, but the encoder must have *some* way of determining the fps, and it surely has nothing to do with the speed of frameserving)

Lisa B.

[au7usa]

Sorry to interrupt, but i'm just trying to keep up. Is the DVD-Rip that i had to use AVISynth stored as 23.976 or 29.97? (above)

I originally tried to ForceFILM but it wouldn't work right. I know the original must have been FILM from the frame count, and it doesn't seem economical to store something at 29.97fps when you could save space with 23.976fps.

Does an MPEG-2 stream have to start with a particular sequence? I know that when you cut MPEG-2 w/TMPGEnc it takes a while to get started correctly during playback. Is it establishing the field pattern?

If the DVD-Rip video was edited after encoding and playback started from the wrong field could it cause it to report as 100% interlaced? (a failsafe flag setting maybe?)

[FOO]

At last , an interesting topic.

I always assumed DVD2AVI just ignored pulldown on Forced Film
as SLK did. Never did check it. I am surprised that it decimates.

Related question : what does VdubMod do with a soft telecined
source ? If you frameserve out of it , is there 24 or 30 frames /sec ?

I will check , this is just a rhetorical question

[adam]

I don't mean that dvd2avi outputs 23.976fps every second of encoding time, I mean that for every second of your source it would output 23.976 frames. It may end up sending those frames to your encoder at 100 frames per second, but when all is said and done the total amount of frames outputted divided by the runtime in seconds will equal 23.976. I'm talking about the interval of frames, not the actual encoding time.

I am 100% positive that dvd2avi does not patch or change anything about the source. It is a decoder, it simply decodes the source and either parses the pulldown flags or doesn't. You can test this yourself using a DVD backup. Since its unecrypted dvd2avi can read it directly off the disk, and it functions as normal. Obviously it can't change anything in the mpeg2 stream since its on a read only medium.

[adam]

au7usa if dvd2avi reported your source as NTSC then it is physically stored at 29.97fps. Forced film will not work. If the results seem acceptable its just luck, or perhaps your source is 29.97fps progressive. That would surely give you all progressive output and dvd2avi would still report it as pure NTSC because it only looks to the pulldown flags to determine whether its FILM or not, and obviously a source encoded at 29.97fps has no pulldown flags.

As for why something would be encoded at 29.97fps even though its less efficient, well many things are still shot at 29.97fps. This applies to many things which are intended to orginally broadcast on television. That's why you've got to check whether you have a true film source or not in dvd2avi before applying forced film.

You could telecine film with any pattern of progressive and interlaced fields that you wanted, so no there is no order required but most things use a 2:3 or 3:2 pattern. That's 3 progressive for every 2 interlaced (its really all interlaced in effect its just that the 2 are new frames created from duplicate fields).

Foo, Vdubmod just decodes it as is like any other decoder, which means that it would parse the pulldown flags and all NTSC DVDs would yield interlaced 29.97fps output. I dont think it has an option to bypass the pulldown flags ala forced film.

Regarding forced film, yup, if all it did was ignore the pulldown flags then an NTSC source (non-film) would come out the same regardless of whether you enabled forced film or not, because the pulldown flags are already disabled in the stream itself.

[au7usa]

Gotcha. Thanks!

[LisaB]

FOO, when you think about the "95% rule", it seems clear that it must decimate....otherwise they would have to make it the "100% rule".

If you forced film on something that wasn't pure film...even if it is, say, 97% film...and if DVD2AVI *didn't* decimate....then the duration of the clip would change and you'd have audio synch problems! (and probably not the simple type, either!)

[FOO]

LisaB , True
Why 95% though ? What random behaviour we don t know about is
happening to the other 5% ?

[au7usa]

There's an article at Doom9 that describes DVD2AVI function in detail.
http://www.doom9.org/ivtc-tut.htm

[adam]

We have the 95% rule because the opening logo is almost always pure NTSC and sometimes the ending credits are too. These rarely account for more than 5% of the total runtime, so if you get 95% film or more showing then it just means that the actual film is fit for forced film. I just set the start time after the opening logo and test that way and just look for 100% film.

During those NTSC parts there are frames being decimated but you will rarely notice. There's not alot of action and its so short. So generally its a non-issue.