[solved] Major MPEG2 Cutting issues. Looking for program that will work.

[ShadowBlink]

Hi,

I have been trying for a week now to try to find a tool that will cut a very complex MPEG2 file correctly.

The MPEG2 consists of 29.97 FPS with repeat frames, however, the Repeat frames are interlaced with each other. 3:2 Pull down, I believe is the correct term. The GOP structure looks like this: IRRBPBRRPBBRRBB. It is encoded at variable bit rate with an average of 7,500 kbps.

I have tried Cuttermaran, but it does not like 29.97 fps files at all, and I have tried the very latest beta as well, v1.69 Beat 14 at the time of this post. The video would always play too fast.

Some cutters I have tried work with this video, but they only cut on the I-Frame. However, I need to make some cuts that are in the middle of the GOP structure. I do not wish to change the bitrate of each frame, just meddle with it's GOP Structure, but not too much. Some scenes have I-Frames placed at specific points.

I also need the .ac3 file cut at the same points as the mpeg2 file. I need a cutter that will take both.

Is there a program that will allow me to cut in the middle of GOP?
Or, is there a program that will allow me to change a frame to an I-Frame? I thought VirtualDub had this feature, but it out puts in AVI only.
Is there a program that will allow me to scan the mpeg2 file to get the bitrate of each frame and feed that to an encoder? Or is there an advanced feature on CEE or HC or QuEnc that will split a file at a given frame and start writing to a new file? I could use the program to feed the bitrate to some encoder, so not much re-encoding is done, just GOP Fixing.

Last night I had the idea to re-encode the video file with the deinterlacing option of CCE, but the video was stretched out of shape. I had access to the CCE maching, so I figured I would give it a try. There is only one CCE machine here so access is very limited. If I could get away with not using it, that would be great, but I can't lose quality of the file.

I also installed AviSynth to try out the trim feature. It is frame accurate, but now what? Is there a way to load that into a GOP fixer to write a new MPEG2 file with the correct GOP? I also have the Decomb funtion added and that really does a great job at eliminating the interlaced frames, so I could really see precisely where I needed cut points. The trouble is that if I have it decombed, then it is still 29.97 fps, and if I were to use a program to add repeat frames in there, would it add even more frames, or convert existing frames to R-Frames?

My last question, is there a way I can take out all the repeat (R-Frames) frames, and then later after cutting add them back in where they were? I use VirtualDub MPEG2 edition to look at my MPEG2 files to check them. I found it is the only program to show accurate frame numbers and to show the Repeat and interlaced frames correctly. It would be brilliant if someone added features to it that allowed demultiplexed mpeg2/ac3 output and allowed the upgrading of frames to an I-Frame.

Thanks, I hope someone here can help me with my predigament.

-Jon

[mwkurt]

Have you tried any of the Womble products? I recommend MPEG Video Wizard. Seems to cut MPEG files wherever I tell it to.
Mark

[jman98]

You can try both MPEGVCR and VideoReDo before buying and I think both can do what you want. Both are frame accurate MPEG-1/2 editors and will cut AC3 at the same point as the video with no problems.

[ShadowBlink]

I have trieed the trial version of Womble DVD, and the cuts were almost accurate. There were some display issues reguarding the Repeat frames. Womble would not show the interlacing and it would show two different frames for the same frame number depending on if you were moving backwards or forwards with the frame pointer. I have emailed Womble about that issue. It's a repeat/interlacing issue.

I will try out VideoReDo and see if it does what I need. It's a complext video file, so I think I need more advanced ways of editing.

I do have the avs scripts, so is there some way I can turn them into separate MPEG2/AC3 streams? I think I need to send them through an encoder, but how do I preserve the bitrate?

Thanks for all those that responded.

EDIT: I just tried VideoReDo, and it said that editing of elementary streams is not yet supported.

[Cornucopia]

With THAT MANY repeat frames, you've got a very non-standard GOP, which wouldn't surprise me that you're having trouble with it (unless I'm misreading your GOP labelling).
To fix the GOP or edit in the middle of a GOP requires re-encoding, Re-encoding entails loss of quality.

Cuttermaran has never had ANY trouble with my true 29.97 or pulldown files, so there's obviously something amiss with your structure. I suggest you use an MPEG analyzer to read the streams and mark (by hand if need be) the timecode of the I-frames.
Then, IIWY, I'd go back to the original master, do the edit as needed there, and encode from scratch, this time correctly, but using the I-frame list you just created (would have to be modified if you DID do edits though). IOW, restart from scratch.

Don't even think about AVISynth or Vdub, as they'll both decode to uncompressed (which will require re-encoding, etc.) the way you're trying to edit.

Scott

[Cornucopia]

If you want to try VideoReDo again, just mux your elementary streams together first (TMPGEnc, Imago, etc).

Scott

[ShadowBlink]

I'm not the original encoder. I am more-or-less trying to "fix" the original encoders ... er ... let's say interesting encoding style. The orignals are no longer with us, so this is all we have to work with.

If I use a program to modify the header to trick programs into believing it's a 23.97 fps file, cutterman works with it just fine, but I lose what VirtualDub calls R-Frames. They are however, different frames than the before an after frame. They are just a specially interlaced frame.

I guess what I need to do is deinterlace this file. Convert those R-Frames back into something good. I have decomb in the avs file and it looks great when I preview it in VDub or some other viewer. I can really see the original frames that are where the interlaced frames use to be. There is a lot of information in this video file still. I just need to extract it right.

Like I said before, I tried CCE on it last night but the film was stretched vertically, so the faces were long. Is there a way to do a really good way to deinterlace this file? Is there a program that will list all the I-Frames? Could I use HC or QuEnc to modify to deinterlace the film in production quality?

EDIT: The AVS command I use to deinterlace preview it, is FieldDeinterlace(blend=false).

Thanks.

[manono]

Is this thing soft telecined? Is it encoded as progressive 23.976fps with pulldown applied? Or is it hard telecined? Is it encoded as interlaced 29.97fps already telecined? I don't quite understand, and don't think you're being very clear. And I've never heard of an R-Frame. Is this some VDub MPEG-2 term?

My last question, is there a way I can take out all the repeat (R-Frames) frames,

If it's soft telecined, they're not there, but added upon playback. If it's hard telecined, then that's what an IVTC does.

I also have the Decomb funtion added and that really does a great job at eliminating the interlaced frames, so I could really see precisely where I needed cut points. The trouble is that if I have it decombed, then it is still 29.97 fps, and if I were to use a program to add repeat frames in there, would it add even more frames, or convert existing frames to R-Frames?

You're not making any sense at all. Decomb's IVTC consists of 2 parts; Telecide, which field matches, and Decimate, which removes the dupe frames. The result is 23.976fps progressive. If it's soft telecined, then set up VDub MPEG-2 to ignore the repeat frames, as VDubMod does by default.

EDIT: The AVS command I use to deinterlace preview it, is FieldDeinterlace(blend=false).

Bad move. That's a deinterlacer (and a bad one), and not an IVTC. The result will be a dupe frame every 5th frame and 29.97fps. Use Decomb as it was meant to be used; to IVTC it back to 23.976fps.

If you'd like some more informed advice, how about uploading a small (10 second or so) untouched sample somewhere.

[ShadowBlink]

Okay, I analysed the file very thoughly. It is a progressive mpeg2 stream that had 3:2 Pulldown applied to it somehow. The problem is that it wasn't applied correctly. Most players see it as a standard film and thus they do not play all the frames, especially the ones that are Pulldown frames. The pull down frames are interlaced frames of the surrounding frames. It is progressive, but with Repeated frames applied to force it to 29.97 fps. Programs like Cuttermaran look at the flags on the header and apparently the flag saying that it is pulled down is missing. Cuttermaran sees 29.97 fps, but not the interlaced pulldowned frames, thus the video plays faster than the audio.

So, I either need to somehow patch the header to say that it has pulldown, or I need to remove the pull frames. I think this was hard pulldowned.

What is the highest quality way of detecting the pulldown frames and removing them? I need a good IVTC script, but I am really tired now and it's hard to process words. :P CCE has a built in 3:2 pulldown remover. Will this detect the right frames and delete them on a re-encode? I have never used CCE to remove pulldown before ... welll I never had to remove pulldown as bad as this before. :P

Thanks for all your help.

[manono]

If you'd like some more informed advice, how about uploading a small (10 second or so) untouched sample somewhere.

Since you're not too good at this, that's still my suggestion.

But failing that, you seem now to be saying it's really hard telecined, with already telecined video encoded as interlaced 29.97fps. If true, then 3:2 pulldown wasn't applied afterwards. You would reencode it with IVTC applied to return it to its original progressive 23.976fps.

A simple IVTC script uses a D2V made with DGIndex and Video->Field Operation->Honor Pulldown Flags, loads both the DGDecode.dll and the Decomb.dll, uses MPEG2Source to open the video, and :

Telecide(Guide=1).Decimate()

It's thoroughly explained in the QuickStart.html included in the DGMPEGDec package. I wouldn't trust CCE's IVTC.

Okay, I analysed the file very thoughly.

Bah. Just open a vob in DGindex and run the Preview. Does it say Film/Progressive, or does it say NTSC/Interlaced?

[ShadowBlink]

If you'd like some more informed advice, how about uploading a small (10 second or so) untouched sample somewhere.

Since you're not too good at this, that's still my suggestion.

Sorry I was tired and didn't see that part of the message. Unfortuanately I can't upload a part of the video. It would make things easy, I know, but I can't.

A simple IVTC script uses a D2V made with DGIndex and Video->Field Operation->Honor Pulldown Flags, loads both the DGDecode.dll and the Decomb.dll, uses MPEG2Source to open the video, and :

Telecide(Guide=1).Decimate()

It's thoroughly explained in the QuickStart.html included in the DGMPEGDec package. I wouldn't trust CCE's IVTC.

Okay, I analysed the file very thoughly.

Bah. Just open a vob in DGindex and run the Preview. Does it say Film/Progressive, or does it say NTSC/Interlaced?

It says that the file is 99% Film.

What I tried to say earlier was that the very original film source was 23.976 FPS, but when it was encoded, it was encoded at 29.97 FPS in progressive mode. It's only the repeated frames that are interlaced together. In my earlier post I gave the GOP structure and R meant Repeat Frame.
So, if we look at a part of it:
IRRB
The first R is a repeat of Frame I, the second R is a repeat of frame 4. The first R is then interlaced with the second R frame. However, the R frame is a very faded 4th frame.

I will try out your script to see if it works. Thanks for putting up with me. I know it can be difficult and I wish I could provide screenshots or film, but I can't. I can only report what I see.

[Abond]

It says that the file is 99% Film.

Then it is 3:2 pulldowned (soft). While in DGIndex select Video-Field Operations-Forced Film. Then File- Save project and demux video. As a result you will have 23.976 progressive mpeg2 elementary stream. Give it to Cuttermaran.

[manono]

It says that the file is 99% Film.

Yep, Abond has the answer there. There's so much gibberish in your posts that it's hard to understand what you have, especially if you are unable to upload a piece of the source.

What I tried to say earlier was that the very original film source was 23.976 FPS

That could be said about both soft and hard telecined sources. After all, isn't all film 24fps, the stuff shown in movie theaters (excepting, maybe, some PAL 25fps stuff)?

...when it was encoded, it was encoded at 29.97 FPS in progressive mode.

Not if DGIndex shows it as Film/Progressive, it wasn't. It was encoded as progressive 23.976fps with pulldown applied.

It's only the repeated frames that are interlaced together.

You're looking at the interlaced 29.97fps output. In every 5 frame sequence, 3 frames will appear to be progressive and 2 appear to be interlaced. And it's not frames that are being repeated, but fields. I told you that VDubMPEG-2 could be set up to ignore the RFF flags, just as VDubMod does.

I will try out your script to see if it works.

As Abond said, it was encoded as progressive 23.976fps. You won't need to IVTC it, only Force Film.

[ShadowBlink]

I did as you suggested. DGIndex said it was outputting 23.976 FPS. However, the video file still plays faster than the audio in Cuttermaran. On top of that, it is still detected as 29.970 FPS.

I was able to take some screenshots of a non-important object in a non-important scene. This will illustrate what I try to say.
http://img.photobucket.com/albums/v193/StifflerStealth/VDubFileInfo.jpg
http://img.photobucket.com/albums/v193/StifflerStealth/FiveFrames.gif

So, are these repeated/interlaced frames hardcoded into the film if DGIndex can't rid the file of these?

[manono]

On top of that, it is still detected as 29.970 FPS.

That's because it is. The flags are still in place to output interlaced 29.97fps.

I've never used Cuttermaran, although people speak well of it, so I can't help there.

So, are these repeated/interlaced frames hardcoded into the film if DGIndex can't rid the file of these?

Not the frames, but the flags are. They're software (sort of) that tells the player how to output the fields/frames. Did you make a full D2V (File->Save Project). Open it in Notepad and at the bottom it'll tell you the Film Percentage. If high, then no, it's not hardcoded. It's pure progressive 23.976fps with 2:3 Pulldown applied. The M2V that DGIndex outputs still has the flagging. I just tried something that might help. I took a progressive 23.976fps with pulldown video and removed the RFF flags. It might be OK for you. I don't have Cuttermaran, so I can't be sure, but it should work.

Open that M2V that DGIndex gave you in DGPulldown. Tick the "Custom" box, fill in 23.976->23.976fps and "Convert". Should work for you. I think. Don't forget to reapply standard 23.976->29.97fps pulldown (default in DGPulldown) when done.

[ShadowBlink]

DGPulldown crashes when I try to do that custom setting. The avs code you gave earlier eliminates the pulldown frames as far as I saw in VDub. The small sample looked good, so I sent that off to the CCE box where it's in the queue. I really hope this cleans up the film. While I was add it, I should have added filters to smother out the film, it's a grainy. I was hoping there would be a way to convert this to 23.976 fps and remove the pulldown frames without re-encoding the thing. If you can think of anything else I can try, please let me know. Thanks for all your patience and help. I really do appreciate this.

[manono]

DGPulldown crashes when I try to do that custom setting.

Strange, I tested it out myself. It even says 23.976fps is OK on the main screen there. I also unchecked the dropframe and timecode boxes. Don't know if that has anything to do with it, though. Using version 1.0.8b3. I just tested again, following your exact procedure (setting Force Film and demuxing in DGIndex, followed by stripping the flags from the M2V using DGPulldown), and it works fine.

The avs code you gave earlier eliminates the pulldown frames as far as I saw in VDub.

Yes, if you make your D2V using Honor Pulldown Flags, then an IVTC will return it to progressive 23.976fps. But it's not necessary to do that if the film percentage is very high like yours is. If you make the D2V using Force Film, then it's already 23.976fps, and when encoding you won't need to IVTC, which slows you down and can sometimes introduce artifacts.

[ShadowBlink]

I tried again, because I used the wrong source. I used the 29.97 fps version ... oops. I have way too many versions of this file around :P It worked. Cuttermaran is perfectly in sync with the audio. No more pulldown frames either. Thanks again for your help. Now to run the file through degrainers and to clean it up. They should work perfectly now as well.