Modifying NTSC MPEG-2 with Restream

[Anonymous344]

I have an NTSC file from which I have removed the 2:3 pulldown flags using dgpulldown. The result reads as 23.976 interlaced in Mediainfo and Avinaptic. From what I understand, 23.976 fps interlaced should not be possible, so this file must be encoded as interlaced when it is really progressive; it looks progressive to me in dgindex, as there is no sign of combing.

I loaded the file in Restream. The frametype progressive and topfield first boxes were checked. I checked Progressive sequence option under Sequence Extension and wrote the file. (I still can't find exactly what this option means.) The result reads progressive.

Are there any problems with this approach? Basically, I want the file to read as progressive, and need to know if this is a valid way of accomplishing this.

[SLK001]

You need to inform us as to what you plan on doing with your video. If the original MPEG2 file was from a DVD and the original source was film, then removing the 2:3 pulldown flag should give you a 23.976 progressive stream. IIRC, the original software that DGIndex was based on would NOT show combing because the video displayed was only showing "I" frames and NOT every frame. I don't know if Donald rewrote the Preview part of the code which would allow you to check for combing. VirtualDubMPEG will load the file and allow you to step through each frame.

Your frametype should read "progressive", if the original was "film". You should NOT change the sequence to progressive, since a DVD's MPEG stream doesn't allow low delay streams, and "B" frames are purposefully sent out-of-order. Basically, if you don't know what an option is for, you shouldn't be changing them willy-nilly. You should uncheck the progressive sequence in Restream and rewrite your file. The corrected MPEG should now be importable into any authoring program.

[Anonymous344]

If the original MPEG2 file was from a DVD and the original source was film, then removing the 2:3 pulldown flag should give you a 23.976 progressive stream.

Yes; it should, and I believe the stream is progressive, but it is apparently encoded interlaced. (It shows no combing in Virtualdub either). I was just wondering if I could use Resteam could change how it is read by various applications without re-encoding (as I am having to take it into Restream to remove the closed captions anyway). It appears Restream cannot change the file's properties in this respect.

Basically, if you don't know what an option is for, you shouldn't be changing them willy-nilly

You are of course correct, but I have been unable to find detailed explanation of Restream's settings, so I was only experimenting, whilst keeping a back-up of the original file. It is sometimes the only way you can see what happens for yourself. I was just curious.

The corrected MPEG should now be importable into any authoring program.

Yes; it is, as was the original. I want to do some resizing. Now, I know that one should not resize interlaced material. I also know that attempting to deinterlace material that is not interlaced is also a bad idea. I am unclear as to exactly how one can treat progressive material that is encoded interlaced. Is there, for example, a problem with resizing and cropping such material?

I have no idea why DVD material is not simply encoded as what it is. Encoding it as one thing when it is really another seems unfathomable to me.

[jagabo]

How do you remove pulldown flags with DgPulldown? I tried it by using the custom settings and setting both the input and output frame rates to 23.976. That produced a file with progressively encoded frames. But Gspot said it was interlaced (pics and frames per second were reported as 23.976, fields per second as 47.952, top field first). MediaInfo reported it as 23.976 fps, progressive, top field first. Upon decoding with DgDecode the output was YV12 with progressive chroma channels.

[Anonymous344]

Yes; I used the custom setting with 23.976 to 23.976 too. My file reads as interlaced in Gspot, AviNaptic and Virtualdub. MediaInfo doesn't seem to say. Can I treat it as progressive in respect of cropping and resizing?

I have never used Dgdecode direct, though I have it because other applications use it. What happens when you decode with Dgdecode? Does it generate an AVI?

Thanks!

[manono]

You shouldn't really need to do anything to the M2V in DGPulldown or in ReStream. One thing you don't want to do is add in the Progressive Sequence flag to an NTSC video.

Can I treat it as progressive in respect of cropping and resizing?

You said that ReStream showed it as already being progressive, so whatever apps you used that said it wasn't are wrong. Or you're not reading them right. Or something. As for cropping and resizing, the usual way to do this is by creating a D2V project file in DGIndex with the Field Operation set for 'Forced Film' (if it's really soft telecined) and then using that 23.976fps D2V in an AviSynth script in which you can crop and resize before sending it to the encoder you plan on using.

Or, if making an AVI, send either the IFO or the VOBs to AutoGK and let it handle it. It'll use the same procedure I just described.

What happens when you decode with Dgdecode? Does it generate an AVI?

Not by itself, it doesn't. But when used together with the D2V project file created by DGIndex in an AviSynth script and opened in an encoder, that encoder can then encode for AVI. This is all explained in great detail in the docs included in the DGMPGDec package (which also includes the DGDecode.dll and DGindex).

[jagabo]

In short, the frames of your m2v after dgpulldown are progressively encoded. Using Mpeg2Source() in an AviSynth script will get you progressive YV12 frames. But you should have used the Forced Film setting in DgIndex instead of dgpulldown.

[Anonymous344]

You said that ReStream showed it as already being progressive, so whatever apps you used that said it wasn't are wrong. Or you're not reading them right. Or something. As for cropping and resizing, the usual way to do this is by creating a D2V project file in DGIndex with the Field Operation set for 'Forced Film' (if it's really soft telecined) and then using that 23.976fps D2V in an AviSynth script in which you can crop and resize before sending it to the encoder you plan on using.

Thank you both. I am reading the apps right, so they must be wrong.

I understand what you said about dgindex. If I wanted to create lossless AVI with Virtualdubmod, would using dgpulldown to remove the pulldown flags still have been wrong? i.e. would using forced film in dgindex still have been better? I could presumably then just load the Avisynth script into Virtualdubmod.

(I'm using Virtualdubmod because I don't know another way to generate lossless AVI. Sorry, I should have been clearer, in my previous questions.)

[jagabo]

Using dgpulldown the way you did doesn't seem to hurt the video but it's an unnecessary step. If you had used DgIndex in Forced Film mode then Mpeg2Source() in VirtualDub you would have gotten progressive YV12 frames at 23.976 fps. Both VirtualDub and VirtualDub mode can deal with that properly. And both can produce lossless compressed (or not) YV12 AVI from that.

[Anonymous344]

Thank you. I think I know all I need to know about this now.

I'll just rip the DVD, remove the closed captions in Restream and proceed to DgIndex.

[manono]

VDubMod ignores the pulldown flags, so if it's really 100% soft telecined you won't see any interlacing anyway and it'll be treated as progressive 23.976fps. That way of doing it produces inferior results compared to doing it via an AviSynth script, though (partly because you'll be filtering in VDub and converting to RGB). And it's risky because unless it's 100% film it's a recipe for audio desynch and possible interlacing. You can check the bottom of the D2V you make for the film percentage.

[Anonymous344]

VDubMod ignores the pulldown flags, so if it's really 100% soft telecined you won't see any interlacing anyway and it'll be treated as progressive 23.976fps. That way of doing it produces inferior results compared to doing it via an AviSynth script, though (partly because you'll be filtering in VDub and converting to RGB). And it's risky because unless it's 100% film it's a recipe for audio desynch and possible interlacing. You can check the bottom of the D2V you make for the film percentage.

What exactly do you mean by that way of doing it? Do you mean judging whether the file is interlaced or not? If you mean converting to lossless AVI, I don't know another way of doing it. I take it even when you import YV12 and export YV12 there is still a conversion to RGB in the middle.

DGIndex says it is film 100% after I have used forced film, so I guess I am in luck.

[manono]

By 'that way' I mean that opening the M2V and then filtering in VDub (the crop/resize) forces you to use Full Processing which forces a colorspace conversion to RGB:

What will be the main advantages of processing in YV12?

• speed increase:

That depends entirely on the external plugins whether they will have YV12 support or not. Speed increases like 25-35 percent are expected. Of course there will only be a large speed increase if both your source and target are in YV12, for example in DVD to DivX/Xvid conversions.

• no color conversions:

The colour information doesn't get interpolated (so often) and thus stays more realistic.

http://avisynth.org/mediawiki/FAQ_YV12

When doing the filtering in the AviSynth scrip you can use Fast Recompress in VDub(Mod) and remain YV12 all the way. When doing a lot of slow filtering I often open the script in VDubMod, and then save out a Lagarith lossless AVI which gets put into another AviSynth script using AVISource, for encoding to MPEG-2 using CCE.

[Anonymous344]

Ah! Right! Gosh, this is complicated. So far my steps are.

1. Rip the DVD.
2. Use pgcdemux to pull out the video and audio files.
3. Use Restream to remove closed captions.
4. Take the video into DGIndex, set to forced film, and save as d2v.
5. Write an avisynth script with the d2v as input and crop/resize it.
6. Open the avs in VDub(Mod) and fast compress to lagarith lossless AVI. I hope I can add an audio file here. If not, I'll have to mux it after step 3.

Let's hope that's it.

[manono]

3. I don't understand why this is a necessary step since you say you're removing them (not saving or converting them). They certainly won't be there after making the lossless AVI.
6. Yes, you can add audio (I think). But since the Lagarith AVI is only an intermediate file, I'm not sure why you want the audio in it. Are you actually cutting stuff out and want to cut the audio at the same time?

[Anonymous344]

3. I don't understand why this is a necessary step since you say you're removing them (not saving or converting them). They certainly won't be there after making the lossless AVI.

I didn't realize they wouldn't be there. Perhaps it sounds stupid, but I'm new at this and didn't realize I could remove them that way.

6. Yes, you can add audio (I think). But since the Lagarith AVI is only an intermediate file, I'm not sure why you want the audio in it. Are you actually cutting stuff out and want to cut the audio at the same time?

Yes; that's right.

[txporter]

If you don't mind spending some $$ on a great program, you could consider purchasing VideoRedo.

Your steps would then be:
1. Rip DVD
2. Combine VOB files into single mpg with VideoRedo.
3. Edit mpg file in VideoRedo to whatever you want (will be straight copy except for a few frames near cut points)
4. DGindex (can use it to demux audio/video if you want)
5. Use avisynth to do any additional filtering/cropping/etc - you could possibly go to your final codec here, not sure where you are trying to end up
6. Mux

[manono]

Yeah, closed captions are found only in MPEG-2 streams.

And doing the cutting first, before anything else, is a very good method. You can also do it in the script by, first, adding in the audio, and second, making a series of trims in the script. I don't handle audio in my own scripts, so I'll leave it to someone more knowledgeable than I to give you the gory details. It might help if you told us what kind of audio it is so they can tell you which filter to load (AC3, WAV, MP3, what?) while showing you sample scripts. A simple script adding WAV audio to an AVI goes something like this:

# Load capture segments from patched AVICAP32 which puts
# video in multiple AVI segments and audio in a WAV file
video = AVISource("capture1.avi") + AVISource("capture2.avi")
audio = WavSource("capture.wav")
# combine them into a single clip
AudioDub(video, audio)

Since you're using MPEG2Source, the first line would be some variation of:

Video=MPEG2Source("...

WAVSource is built into AviSynth, but for something like AC3 audio you'll need the NicAudio.dll found here:

http://avisynth.org/warpenterprises/

The included text file explains how to use it. After loading the audio and video, you use Trim commands to split it up into the parts you want to keep:

http://avisynth.org/mediawiki/Trim

And I'm sure that all this is quite a bit to take in all at once, so you might want to do it your tried-and-true way for awhile before jumping 100% into the AviSynth pit. However, once you learn your way around it, and if you're a serious video editor and encoder, you'll wonder how you ever lived without it.

[Anonymous344]

@txporter Thanks for the advice, but I already own a program that does what VideoReDo does. The reason I am not using it this time is that I have two sources that I need to recrop to match each other before editing together; I cannot just edit them together and crop new borders - which was my first thought. I really appreciate the attempt to help though.

@manono Yes; that does seem a bit of a jump at the moment. Thank you for the copious links though. The audio will be 2.0 pcm wav (made from a 5.1 AC-3 original by h3adache). For now I think I shall just add it in Vdubmod; though I shall give your suggestions serious thought. Thanks!