Easy way to IVTC 29.97 1080i HDTV rips to 23.976p

[cjbrown80]

I'm archiving several seasons of a show I recorded with a Ceton. Commercials were removed with VideoReDo and exported to .m2ts files. Source is mpeg2 1080i 29.97 telecine. I would like to know what is the best way to accurately IVTC these to either 1080p or 720p 23.976 x264 (NOT variable frame rate) for archival. All the threads I've found so far involve AviSynth scripting, which is over my head, and I don't have time to learn it. I have been trying many combinations of settings with Handbrake/VidCoder, but the resulting files still have repeating and/or missing frames and result in the jerky motion effect (confirmed by going through output video and finding repeating frames and gaps). When I go through the source file, advancing frame by frame, it appears that exactly every fourth frame is repeated, which implies the standard 2:3 telecine (right?) So why is the Handbrake engine missing some of them? Is there a Custom setting I can use to remove those repeated frames which should result in a 23.976 output? Or a different app I should try (other than AviSynth) that will perform this task accurately?

[poisondeathray]

When I go through the source file, advancing frame by frame, it appears that exactly every fourth frame is repeated, which implies the standard 2:3 telecine (right?) So why is the Handbrake engine missing some of them? Is there a Custom setting I can use to remove those repeated frames which should result in a 23.976 output? Or a different app I should try (other than AviSynth) that will perform this task accurately?

2:3 would show a 2:3 pattern of 2 combed frames, 3 clean frames

every 4th frame repeated suggests that it was deinterlaced . Perhaps your method of examining it / viewing is deinterlacing it ? How are you advancing frame by frame (what software, what decoder) ?

There are some GUI's that use avisynth and try to make things easier for you eg. ripbot, megui, staxrip . They use avisynth in the background and some auto analysis algorithms that get it mostly right >90% of the time

Commercials were removed with VideoReDo and exported to .m2ts files. Source is mpeg2 1080i 29.97 telecine.

Removing commercials when pulldown hasn't been removed is dangerous in that depending on where make your cuts, you might get cadence breaks (disrupt the regular 2:3 or 3:2 pattern, resulting in some combed frames, combed transitions)

[cjbrown80]

Thanks for your lightning quick response.

Well, my method of examining is VideoReDo, which is, most likely, doing some deinterlacing. My basic knowledge of this process is from the Handbrake wiki, if that helps.

I'm going to give old RipBot a tinker since I already have it installed. I'm assuming MegUI is worth a shot also since most of the other threads I've read on this subject reference it.

[poisondeathray]

If you have time, I highly recommend learning the basics - at least enough for IVTCing . There are cases where the auto analysis tools fail

ripbot would be easier to use , megui is a bit more complicated - but both examine the file and attempt to apply the correct avisynth processing for you . Megui is a bit more advanced and will allow you to adjust the avs script more easily - it's good in that if you eventually want to learn more, you can get more involved

A non deinterlacing viewer would be vdub (with no filters applied) . If you can't open the files, you can use the vdub ffinputdiver or other vdub plugins like the directshow input plugin to open the files

[cjbrown80]

Okay, I guess I know what my afternoon will be spent doing.

I've always been curious about the AviSynth scripting, I just can't find any resources that simplify it enough for me. Could you recommend one?

[poisondeathray]

I find it odd that handbrake/vidcoder would make those mistakes . It might be your edits that is causing it to fail or confusing it , or your source files might be non standard . I think if you set it to "detelecine", with "CFR" , on standard 3:2 material it should work . If in doubt, or ripbot/megui still have issues - you can post a video sample with steady motion and someone will examine it and make suggestions

I've always been curious about the AviSynth scripting, I just can't find any resources that simplify it enough for me. Could you recommend one?

I would start with the mediawiki, and to learn about basic field patterns neuron2 has a mini-guide

http://avisynth.nl/index.php/Main_Page#New_to_AviSynth_-_start_here
http://neuron2.net/faq.html#analysis

[vhelp]

what is the series ? perhaps someone already went through the trouble with it and can assist you more efficiently because some shows are time expanded, so the cadense will be illregular since they change the behavior of it, to the point it may not be fully restorable and best to leave it alone in those cases.

edit: just realized you said you cut out commercials in those. if you still have the original separate pieces, i would suggest ivtc'ing each piece first, then join them into one, otherwise, you might have a harder time working the complete single piece since the way you cut out the commercials will undoubtly have missaligned the cadense pattern which may throw off ivtc method for each piece that you joined in those series.

[cjbrown80]

Okay, here are samples. One of the Variable frame rate runs looked okay, but I don't want VFR because of it's compatibility issues, and I'm picky.
I did four runs in VidCoder with the Detelecine filter enabled, alternating frame rate settings between 'Same as source' and '23.976', and CFR and VFR.

edit: The only reason I posted 'same as source' clips is the Handbrake wiki mentioned it.

[jman98]

I use MeGUI, but my experience is that it NEVER produces AviSynth that will run without crashing from errors. NEVER. Others may have different experience, but I'm betting the OP will be like me. I can, however, edit the scripts that it produces and change them to make it work and they are relatively close to being correct, but never quite right.

AviSynth basically works like this:
command to open the video file for processing
command(s) to do stuff to the file

AviSynth should generally be avoided for audio processing, in my opinion. Just worry about doing video work with it. The OP can always ask for help. There are a lot of AviSynth gurus here and I've gotten help before from them on things that weren't very clear to me.

[cjbrown80]

Okay, I just finished trying RipBot and it appears it worked for my sample clip. I used the settings of deinterlace TFF->29.97, decimate Restore->23.976. I'm going to do a whole episode now to be sure, before moving on to other shows/seasons. The Handbrake engine doesn't have decimate settings (that I know of), so I guess RipBot is the way to go. I'm not sure what the decimate filter is doing exactly, but I like it.

[jagabo]

The Handbrake engine doesn't have decimate settings (that I know of)

Use Detelecine and set the frame rate to 23.976.

[cjbrown80]

The Handbrake engine doesn't have decimate settings (that I know of)

Use Detelecine and set the frame rate to 23.976.

I tried that on one of my sample clips (located above if you wish to examine) and it didn't work. The result still had duplicate frames and possible gaps which led to inconsistent motion.

[jagabo]

I don't normally use Handbrake but I just tried the detelecine/23.976fps setting and it properly handled a short hard telecined MPG file. I'll take a look at your source...

[smrpix]

Your source clip is inconsistent. Viewed field by field there are different cadences and different blendings not just on a shot by shot basis, but even within shots.

This could be caused by anything from transmission errors, capture errors, motion speed-ups/slowdowns in editing (almost certainly done at 29.97i), even non-synced hmi lights on set.

This may be doable, but it won't be easy.

[jagabo]

Your source clip is inconsistent. Viewed field by field there are different cadences and different blendings not just on a shot by shot basis, but even within shots.

This could be caused by anything from transmission errors, capture errors, motion speed-ups/slowdowns in editing (almost certainly done at 29.97i), even non-synced hmi lights on set.

This may be doable, but it won't be easy.

Agreed. No simple GUI will be able to handle it. It would be easiest to leave it interlaced or smart bob and encode at 59.94 fps (and live with the blending artifacts).

You can force Handbrake to encode interlaced by entering tff in the Extra Options box on the Video tab. Be careful about filtering though. The filters may not handle interlace video properly.

[manono]

It looks time compressed to me and the 'effective' framerate is greater than 23.976fps. AviSynth is about the only way to handle this (and even it will have huge problems). Since neither Handbrake nor its front end VidCoder use AviSynth, if you're really picky, as you claim, they can't be used for this.

jagabo's suggestion is the easiest to get some sort of decent output. At 23.976fps it'll play noticeably jerky from time to time.

[cjbrown80]

Thanks for the input. I'm in a constant struggle to balance my pickiness with my laziness. I have no attachment to Handbrake/VidCoder, but they do work for most simple conversions. I've been tinkering with VirtualDub, going through most of these HDTV source files frame at a time. You're right about them being a mish mash of frame rates/types... the term I would use is Frankensteined. I don't even want to get into the full explanation as to why they are broadcast in this manner, though I am curious.

I have found another show that seems to have two fields interlaced then four fields not interlaced, repeating. What would that be?

These files were edited in VideoReDo to remove commercials. Could that software have caused issues with frame rate, or at least the way individual frames were encoded? I know it recreates certain frames around where the cuts were made.

[manono]

I don't even want to get into the full explanation as to why they are broadcast in this manner, though I am curious.

If it is time-compressed, and I'm fairly certain it is, it's to shorten the show a bit to have it fit the allotted time. Say the show is 42min 30 seconds long but the slot is exactly 42 minutes. To remove the extra thirty seconds they begin cutting out fields or frames (and in your case, also blend fields/frames together). Or the show is the right length but the network or local station wants to squeeze in another advertisement.

http://www.forbes.com/2002/03/06/0306timemachine.html

http://everything2.com/title/time+compression

I have found another show that seems to have two fields interlaced then four fields not interlaced, repeating. What would that be?

I think I understand what you're saying, but you're saying it wrong. Fields aren't interlaced, frames are. You have 4 progressive frames and one interlaced frame in every 5-frame cycle? A simple IVTC can usually fix that but, as usual, please provide a sample for a more informed response.

[smrpix]

These files were edited in VideoReDo to remove commercials. Could that software have caused issues with frame rate, or at least the way individual frames were encoded? I know it recreates certain frames around where the cuts were made.

Not likely, the cadence changes from shot to shot as well as within shots. VideoReDo is pretty solid.

[cjbrown80]

Not likely, the cadence changes from shot to shot as well as within shots. VideoReDo is pretty solid.

I gotcha. So the repeating pattern is interrupted between shots, hence you might get 7 progressive frames in a row over where a cut was.

[cjbrown80]

I think I understand what you're saying, but you're saying it wrong.

Yep, my bad.

You have 4 progressive frames and one interlaced frame in every 5-frame cycle? A simple IVTC can usually fix that but, as usual, please provide a sample for a more informed response.

That was the previous example, at least how it appeared in VideoReDo. The show I'm referring to here is Dracula on NBC. Only this time I opened it with VirtualDub as recommended by Poisondeathray above. I advanced frame at a time and I saw two frames with combing then four frames without. Not sure if that points to a different process used, other than Telecine, etc. I've just jumped to a different segment of the same file and it appears to have three frames with combing then three without. Wtf? I'm uploading this sample here.

[jagabo]

two frames with combing then four frames without.

That's usually indicates a PAL to NTSC conversion (3:2:3:2:2 pulldown).

[manono]

I advanced frame at a time and I saw two frames with combing then four frames without. Not sure if that points to a different process used, other than Telecine, etc. I've just jumped to a different segment of the same file and it appears to have three frames with combing then three without. Wtf?

Same thing. The important piece of information there is that it's a 6-frame cycle, always indicative of PAL to NTSC. When you have 5-frame cycles (2 interlaced and 3 progressive), it's a sign of progressive film to interlaced NTSC.

[cjbrown80]

OK, is there a way I can reverse that to get it back to its original 25 fps? I would just like to restore all these shows I have saved to remove interlacing and have nice, smooth motion. I'm lost at this point as to the best way to accomplish this, hopefully without having to learn Avi Synth. I know that must be frustrating to the folks who are experts, but the time I can afford to spend on this sort of thing, I don't believe is enough to get far enough for it to be worthwhile. Apologies.

[jagabo]

OK, is there a way I can reverse that to get it back to its original 25 fps?

Yes. TFM().TDecimate(cycle=6, cycleR=1) or some variation.

[manono]

...hopefully without having to learn Avi Synth.

Then you can fuhgeddaboutit. As far as I know, not only can AviSynth do it very easily (much more easily than fixing your time-compressed Blacklist show), but it's the only way to return it to 25fps


.

[cjbrown80]

Hmmmm... Remember what I said earlier about balancing my pickiness with laziness? I guess it's more like balancing my stubbornness with laziness...

I Have a bit of time tonight and tomorrow to tinker with this. Not sure if that's realistic for a total AviSynth noob to accomplish anything. But thank you guys so much for your time. I hope you'll continue to monitor this thread for when I will undoubtedly have scripting questions.

Cheers!