Require Avisynth Script Help.

[Eldric]

I'm attempting to correct a test vid capture I pulled off of cable. From everybody I've talked to about this clip - it appears to have been improperly commercially telecined. As a result, I got an orphaned field that pops up every 47 fields within the clip. This is what I'm getting:

2123232323232323232321232323232323232323212

As you can see - you have an orphaned field appearing every 47 fields. I've already had some help from several other coders in this area and have been given a script which will convert the film from it's 29.970 fps capture to one that is 25 fps. However, I still would like to experiment this with and have a theory (perhaps not a very well thought out one) that I would like to test. My idea is to perhaps run a script that can be used to "Selectevery" orphan field in the series (after running the seperatefields function), and then duplicate it twice. After this is accomplished, I want to attempt to weave the fields back together again, and then running a telecide/decimate script to revert the clip back to it's original speed, but without "jerks" appearing when the camera pans horizontally or the actors move.

So far, I've determined this:

1) I run a seperatefields function to get the clip to the format I want.
2) I run a selectevery function after I have manually determined the exact pattern that the orphans show up in the film.
3) I run a duplicateframe function in order to duplicate these orphans.
4) I run the Weave function, to once again weave the fields back together into frames.
5) After all this is done, I run the telecine/decimate script I have to correctly IVTC the clip.

Mind you, all this is just an experiment for me, but I DO require help if only in the two most important parts - #'s 2 and 3. I've never used the selectevery function, however I do know how it targets frames - in the example given, I would set it to select the 2nd frame (the first orphan) and then select every 47th frame afterward. The duplicateframe function is completely beyond me. I have no idea how it would work here, in order to create the two duplicates of each frame, or even if the duplicates would show up in the correct place in the video stream.

I know, it sounds odd, but unfortunately when you're dealing with video that's been incorrectly telecined, this sort of video stream is what you end up with. The main reason I'd like to see if this is possible is that while I DO have a current script that resolves this problem for me, I'd like to be able to have something that I can customize a bit more, especially since I might run into a clip that's been butchered even WORSE than this one. So if anybody has had some experience with the "SelectEvery" command and the "Duplicateframe" command, then I'd very much like to get some feedback from you.

Thanks in advance for your time and patience,
Yours,
Eldric

[SLK001]

How do you end up with an extra FIELD in your stream? Where is it coming from.

[Eldric]

It's not so much an extra field, as it is an Orphaned field. The pattern throughout the clip is primarily 3:2 pulldown - this means that you get 3 dupe fields followed by 2 dupe fields. Whoever telecined this material literally dropped one frame (2 fields) from the clip in order to make more room (probably for commercials). So what you end up with is a 1 field orphan where you should have 3 dupe fields. My idea is to duplicate THIS orphan in order to rebuild this 3dupe/2dupe pattern, and fix this kind of badly done hack job.

[FulciLives]

In situations like this you really are better off "giving up" and just encode at 29.970fps rather than try to IVTC it to 23.976fps

I understand the obession to IVTC when possible but sometimes it just isn't worth doing as you can do more harm than good.

So again assuming you are trying to create a DVD just do 29.970fps

- John "FulciLives" Coleman

[Eldric]

My main issue here though it attempting to come up with an end-product that will be able to be played off a computer to a TV or VCR in any of the region-codes NTSC/PAL/SECAM without any of the compatibility problems you would normally have - such as the frame-speed difference. My thinking is if I can roll the film back to it's original speed, then the player (in this case, the computer) compensates. You have NTSC film - which is 23.976 fps. My understanding is that this film speed is constant, and when played back on a DVD player that is NTSC format, the player automatically compensates for the speed change - displaying it at 29.970 fps.

Eldric

[thor300]

I dont know what you are doing there, but generally the answer is NO i think. NTSC Film cant not be played back on a NTSC DVD player, it need pulldown to do so.
NTSC/PAL/SECAM are not region codes, they are different systems.
You will not overcome what you call "compatibility problems you would normally have" anyway, so just encode your captured stuff at 29.97. I could misunderstand you here, its possible you ment that 23.976 is easier to convert to 25fps PAL, but there are just as good methods for converting from 29.97 to 25fps also. From what you said, "then the player (in this case, the computer) compensates" it looks like the misunderstanding is more on your side, so if im wrong please clarify what you want to do.

[FulciLives]

Not all NTSC that is 29.970fps can be restored to 23.976fps via IVTC.

With some sources it just is not possible.

So I would leave it NTSC 29.970fps

If you need to convert it to PAL format then there exist ways to convert NTSC 29.970fps to PAL 25fps

See the guide by Xesdeeni and look up my past posts on the subject.

- John "FulciLives" Coleman

[Eldric]

Okay, my thoughts here are as follows:
1) my apologies, I said region codes but I ment broadcast systems.
2) What I am attempting is to successfully IVTC a clip that can then be downloaded and viewed (using a TV-out card) in anyplace in the world, despite the broadcast system used. This means that if you digitize a family home movie recorded in NTSC format, and you have family in Europe, they'd be able to download it from your webpage and view it using their computer to play it and cables to their TV or VCR (either equipped to view only PAL), and have no compatibility issues.
3) Thirdly, my desire to IVTC a clip is to remove unnecessary frames and therefore make the video file smaller.

That pretty much is everything afaik. I'm still experimenting with IVTC but my main goal is to be able to take a piece of film, revert it back to it's original FPS (or if it's Hybrid, the most compatible FPS available), so that 1) the file size is smaller (if applicable) and more importantly 2) that the video stream will be playable via a TV-out card, with no problems whatsoever, despite if the broadcast-system used by the filmmaker and the computer user are different.

Eldric


Addendum: In this particular instance, my desire is to REPAIR a clip that's been damaged. For this experiment, I am using a clip that was badly telecined at the commercial level. In my first post in this thread, I explained how 1 frame was removed from the clip, in order to make room for advertisements, creating orphan fields in the process. My desire was to develop a script to rebuild the missing fields from the orphans - fixing the 3:2 pulldown pattern that is primarily what I'm seeing here.

[thor300]

Sorry eldric, i totally forgot about the part where screencards output to different formats. That will work i think, and i dont think it will be any difference if its 23.976 or 29.97. 23.976 is not more compatible with PAL than 29.97, i guess it will work anyway. I have watched 29.97 movies from laptop connected to PAL TV, no problems, i just cant be 100% sure if that TV was NTSC compatible, its 2-3 years ago, only remember it was a cheap Samsung.

[Eldric]

Thor,
I know some cards do but I'm not totally familiar with what's available out there on the market - in my case, I'm more familiar with my ATI card that allows you to select the broadcast type you wanna output. However, I'm not sure if this is the case for all cards, and the last thing I want is to fail to take that into account - meaning that because someone doesn't have that option with their video card, they're stuck viewing video on their computer because it's not compatible with their A/V equipment due to the FPS of the clip.
Eldric

[thor300]

Dont worry eldric, you can be pretty sure that all screencards sold in europe are PAL. Maybe many chose to watch it on PC anyway, not everyone keep the TV and PC so close, could require a very long cable.

[FulciLives]

NTSC 23.976fps is no more compatable than NTSC 29.970fps when it comes to PAL 25fps.

True it can be easier to convert NTSC 23.976fps to PAL 25fps than it is to convert NTSC 29.970fps to PAL 25fps but ...

What you are trying to do makes no sense.

There is no one format that works everywhere.

As for a computer ... if using the video card output then it should have no problem displaying on a TV no matter what format video file you play is in relation to the format of the TV.

Although it might look strange FPS wise it will probably be very subtle.

- John "FulciLives" Coleman

[Eldric]

What about correcting the orphaned field problem? Should I even still attempt that or is that prettymuch futile? Please remember, another of my reasons for performing IVTC is to create a smaller filesize by removing unnecessary frames. Clips such as the one I'm using here were damaged when they were improperly telecined. I know I'm not the only one who has had this problem, I've found several posts on this site, Neuron2.net and Doom9.org where people are having the same problem. Perhaps my idea sounds odd since I want to add several fields, prior to weaving them back together, only to delete frames when the video is IVTC'd. The end result is that I want to make sure that I got a smooth playing video that has unnecessary frames removed (making it smaller), without the sudden jerks that are the result of the video stream being damaged prior to my getting ahold of it.

[FulciLives]

Don't know what to tell you about that.

I've encountered some PAL DVD discs that were improperly telecined from NTSC sources and these can look really bad with jerky motion but I've never seen a NTSC DVD that looked "bad" in the same way when you simply leave it at 29.970fps ... at least not when viewing on a normal NTSC TV.

For computer playback I'd say leave it at 29.970fps but deinterlace it perhaps with some form of "blending" deinterlace.

- John "FulciLives" Coleman

[Eldric]

Thing about IVTC is that is gets rid of those interlaced frames when you perform the telecide/decimate functions. Of course, you have to know what you're doing (which is what I'm learning). As for leaving the playback at 29.970 fps - if I want to make the filesize smaller, one of the things I'm going to have to do is IVTC it (if possible), so removing those pseudo frames that were placed in there to shift the clip from it's original FPS to a higher one are going to have to go. :/

*shrugs* it's why I'd like to learn how to correctly use the duplicateframe and the selectevery functions in Avisynth, for this experiment so I can at least try to correct some of the problems I'm seeing.

Eldric

[FulciLives]

Well Eldric I do understand the concept of IVTC and I have done it myself many times.

All I am trying to say is that with some sources an IVTC just cannot be done properly.

It sounds like the source you have is one such source.

I'd try the AviSynth forum over at the doom9 website

Usually someone there is even willing to download and look at your video if you can cut out a short 5 second clip of it and upload it somewhere for someone to download it.

Those guys over there are pretty experienced so one of them might be able to figure it out if it is possible.

- John "FulciLives" Coleman

[adam]

You are essentially trying to do by hand, what most IVTC filters do automatically. Have you tried decomb.dll yet? It has a pattern guidance function that can account for "stray" fields or any other abnormal occurrences. Just read the docs for the syntax, its pretty simple to use.

[dphirschler]

eldric, take a look at my sript here:

http://pigseye.kennesaw.edu/~dhirschl/starwars/starwars.avs

I am doing a manual IVTC operation in this script, but I think the basic code would need very little modification in order to work for you.


Darryl

[LisaB]

Eldric,

If this is a strict pattern, then the solution isn't that difficult.

SelectEvery(48,1,2,2,2,3,4,5,6,7,...,45,46,48)

This would just say that "for every group of 48 fields, I'll select every field, but I'll select the field #2 three times"

(i'll note that in your initial example, your periodicity is 48 not 47)

You obviously don't need DuplicateFrame, since SelectEvery takes care of it.

The problem you will probably find is that the pattern is not absolutely strict throughout the entire video...there may be an extra field in there on occasion...For example, if field #2 is an orphan, then so should field #48,002. In other words, does your pattern stay exactly the same, even 48,000 frames later?

So, here is a possible approach:

duplicate EVERY field...you could use SelectEvery(2,1,1,2,2)

now your pattern will be 4,2,4,6,4,6,4,6,...

weave it back together, use Telecide, and then get rid of all those duplicate frames using Decimate or MultiDecimate....

[Eldric]

Well, it would appear that I really am well and truly up **** creek here. I thought the pattern was that the orphan would appear every 48 fields. I apparently was wrong. Deciding to investigate further, it seems that there's a variation as the 1 minute's worth of video progresses. From what I can tell, the number increases from every 48 fields to suddenly every 53 fields. Exactly how or why, I have no clue. I guess that my idea to trying to design a script that would allow the user to repair this type of problem has gone "Up in smoke" now. Thanks anyway for your help guys. I know you tried but unless the pattern remains constant (and this one didn't) it would take days or even weeks to get through just one 23 minute video clip.

Thanks anyway,
Eldric

[adam]

I know you tried but unless the pattern remains constant (and this one didn't) it would take days or even weeks to get through just one 23 minute video clip.

That's why you use decomb and its pattern guidance. It recognizes changes in the telecining pattern and adjusts accordingly. If you give it a try I bet you will be pleased with the result. It can inverse telecine even the most screwed up sources.

[Eldric]

The problem is I HAVE been using decomb to perform IVTC on clips.
This is the script I've been primarily using:

LoadPlugin("F:\decomb511.dll")
AVISource("F:\capture.avi")
Telecide(Order=1,Guide=1)
Decimate(cycle=5)

With this script I've had a 99% success rate with IVTCing test video that I've pulled off of my cable unit. However, the remaining 1% has been the clips I've pulled off of TV shows from the 60s and 70s where this orphan frame problem has occured.

Due to the help of a gentlemen named Scharfi's Brain, both on Neuron2.net and Doom9.org, it was identified that the problem was due to 1) the original clip being 25 fps, and 2) the clip being improperly telecined by whatever commercial agency there was out there. I was given the following two scripts to solve this problem.

import("F:\intellibob.avs")
LoadPlugin("F:\kerneldeint140.dll")
loadplugin("F:\avisynth_c.dll")
loadCplugin("F:\SmartDecimate.dll")
AVISource("F:\capture.avi")
assumetff()
x=kernelbob(6)
smartdecimate(1250,2997,bob=x)

AND the one I'm currently testing:

function kernelbob(clip a, int "th",bool "mask")
{ mask=default(mask,false)
th=default(th,5)
ord = getparity(a) ? 1 : 0
f=a.kerneldeint(order=ord, sharp=true, twoway=false, threshold=th,map=mask)
e=a.separatefields.trim(1,0).weave.kerneldeint(ord er=1-ord, sharp=true, twoway=false, threshold=th,map=mask)
interleave(f,e).assumeframebased
}

LoadPlugin("F:\kerneldeint140.dll")
loadplugin("F:\avisynth_c.dll")
loadCplugin("F:\SmartDecimate.dll")
AVISource("capture.avi")
assumetff()
x=kernelbob()
smartdecimate(1250,2997,bob=x)


Once again, the test clip I'm using is one that was taken off of digital cable and was originally SHOT in the US. The reason why it should be 25 fps is beyond me, since the standard is typically 23.976 fps. Anyway, I know the first script worked, but once again, my main concern is will it work with a piece of film that is in an even WORSE condition than the one I was able to get my hands on? Anyway, like I said, I've been using decomb for the majority of my IVTCing, but I'm still having problems.

Eldric