De-interlacing and MeGUI

[hello_hello]

The story.....
MeGUI has different de-interlacing "categories", and each lets you choose several different de-interlacing methods accordingly (or it automatically selects one after an analysis). After some recent experimenting and discussing it with one of the MeGUI developers, it appeared MeGUI was adding the wrong de-interlacing to the script for one of those "categories" (#8 below), which has been changed. However I'm still unsure as to whether MeGUI's de-interlacing is correct, but as the current MeGUI developer lives in PAL-Land as I do, neither of us are very expert when it comes to NTSC.

Currently the de-interlacing categories for MeGUI are as follows:

1. "Progressive"
2. "Interlaced"
3. "Film"
4. "M-in-5 Decimation"
5. "Hybrid film/interlaced. Mostly film": Mix of film & interlaced
6. "Hybrid film/interlaced. Mostly interlaced" Mix of film & interlaced
7. "Partially interlaced": Mix of interlaced & progressive
8. "Partially film": Mix of film & progressive

Most of the above is fairly self explanatory and the appropriate de-interlacing methods for each category are what you'd expect. My question involves the distinction between numbers 5 and 8... as to whether there should be one, and if so, what should it be? Currently MeGUI applies exactly the same de-interlacing for both categories, which to my way of thinking either makes one of them redundant, or #8 is possibly still wrong.

This is an example of what MeGUI currently adds to the script.

5. "Hybrid film/interlaced. Mostly film": Mix of film & interlaced:
tfm(order=-1).tdecimate(hybrid=1)

8. "Partially film": Mix of film & progressive:
tfm(order=-1).tdecimate(hybrid=1)

Obviously both of the above IVTC/de-interlace in exactly the same way (output is 23.976fps), which got me wondering, should #8 be doing something else?
I assume #5 applies to 29.97fps video which is a mixture of telecined 23.976 and 29.97 interlaced, whereas #8 might apply to a mixture of 23.976 and 29.97 progressive, although I'm not exactly sure.

Anyway, my question is..... logically what would the difference be between #5 and #8 in terms of the type of video it applies to, ie what's the difference between "Mix of film & interlaced" and "Mix of film & progressive"..... well the interlaced vs progressive part is obvious, but does "film" mean the same thing in both instances..... and is it logical they both should be de-interlaced in the same way?

Cheers.

[Selur]

To get things clear, can someone explain what 'film' in this content is ment to mean?
From what I know 'FILM' normally means that you have progressive content @24 (23.976) fps that is flagged to be displayed at 29.97fps during playback. So I assumed that 'FILM' would be kind of equivalent to soft-telecine content.

Looking at 5. and 8.:

1. both would require to handle the soft telecine
2. 5. would require deinterlacing on the interlaced parts
3. both might require an additional decimation call, in case the progressive or interlaced part didn't have a frame rate of 24 (23.976) fps

Assuming the tfm step can properly differentiate between interlaced and progressive frames, there would be no real reason do differentiate between the two options ( aside from being tired of explaining people that there was no need for a second option).
If this assumption is right, in the other hand a new question pops up: "Why at all differentiate between the options 5., 6. and 8. ?"
Also, if the selection between 5. and 8. additionally also allows the selection of different options to handle this type of content, the differentiation is needed. (shouldn't be joined, since only one of many options can handle both cases)

-> It might help if you could post which options of handling with content of type 5. to 8. are available depending on the type selection.

Sorry, this probably throws up more questions. (haven't used MeGui for ages)

Cu Selur

[hello_hello]

Thanks for the input.

The options for 3 and 8 are exactly the same, which is why I wondered.... why have both? The question arose after I discovered MeGUI was adding de-interlacing to the script in order to output a 23.976 frame rate after it analysed the video and selected "partially film" itself, but if I manually selected partially film it'd add de-interlacing in order to output a 29.97 frame rate.... which did my head in until I realised what was happening (the difference for tdecimate was basically just "hybrid=3" when it should have been "hybrid=1").
So that was fixed to make the de-interlacing the same whether MeGUI chose "partially film" or whether it was selected manually. Anyway, these are the various options available when selecting either 5 or 8. I've only ever seen MeGUI select the first one in each category after analysis. It changes a little if the field order is also selected, but these are basically the three choices. They're exactly the same for both 5 and 8.

LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\TIVTC.dll")
tfm(order=-1).tdecimate(hybrid=1)

LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\TIVTC.dll")
LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\EEDI2.dll")
LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\TDeint.dll")
edeintted = SeparateFields().SelectEven().EEDI2(field=-1)
tdeintted = TDeint(edeint=edeintted,order=-1)
tfm(order=-1,clip2=tdeintted).tdecimate(hybrid=1)

LoadPlugin("C:\Program Files\MeGUI\tools\avisynth_plugin\Decomb.dll")
Telecide(guide=1).Decimate(mode=3,threshold=2.0)

DGDecode still classes hard-telecined 29.97 film as "film' doesn't it, hence it's "force film" mode?

I've attached a little sample. Maybe you (or someone) could tell me what it actually is. As best as I can tell it's hard-telecined, but MediaInfo says it's 23.976 progressive with 2:3 pulldown, MPC-HC says it's 29.970 and DGIndex says it's 83.18% FILM.
When I apply IVTC, the frame count in MeGUI's preview window drops from 1796 to 1436 and it looks correct with IVTC applied.
Anyway, as a result of our discussion, the MeGUI developer recently added MeGUI's analysis result to the log file, so now I can at least see what it's doing, even if I don't fully understand it. For the sample I attached:

Progressive sections: 3
Interlaced sections: 1
Partially Static Sections: 112
Film Sections: 47
Source is declared hybrid film/progressive.
Source is hybrid bff and tff at 39 bff and 7 tff.

So I'm figuring "hybrid film/progressive" gets MeGUI to choose the "partially film" option (#8), which it did in this case, and would make sense given #5 is "hybrid film/interlaced, but I'm still back where I started trying to understand the difference between the two in terms of how they should be de-interlaced, or if they should just be de-interlaced the same way.

MeGUI added this to the script after the analysis:
tfm(order=-1).tdecimate(hybrid=1)

For the record, after it analysed the attached sample, AutoGK declared it was pure film, and added this to it's script:
Telecide(movie,guide=1,post=2).Decimate(mode=0)

NTSC does my head in.....

[Selur]

They're exactly the same for both 5 and 8.

then there really seems to be no reason to show the user two options
(aside from the nor wanting to answer the question why no differentiation was made )

Looking at the clip I'm not really surprised that an automated detection has problems. Nearly half of the clip has no or nearly no motion.
-> only way to really find out is probably to check out the interlaced pattern manually or believe in the flags.

[hello_hello]

The automated detection was actually getting it right (at least as far as adding IVTC correctly to the script goes), it's just the various de-interlacing "categories" I was interested in understanding a little better. I'd used the sample to show the MeGUI developer how manually selecting "partially film" caused MeGUI to add different de-interlacing to the script than it would if MeGUI selected "partially film" itself after the analysis. It was actually the manual selection which was wrong.

"Partially film" seems to mean "hybrid film/progressive" as opposed to "hybrid film/interlaced", but I'm still not sure I understand what "hybrid film/progressive" actually looks like. I guess it'd be a combination of 23.976 with hard-telecine and 29.970 progressive, or something..... but I suppose in the end they'd be converted to a constant 23.976 frame rate the same way. I think.....

[manono]

DGDecode still classes hard-telecined 29.97 film as "film' doesn't it, hence it's "force film" mode?

Do you mean DGIndex? And the answer is 'no'. Hard telecine shouldn't be Force-Filmed and is considered to be Video by DGIndex. Only soft telecine can be Force-Filmed successfully.

When I first read your questions, I immediately thought it was using DGIndex terms and that maybe Donald Graft was even involved in offering advice as to what MeGUI should do after its analysis.

I take the differences as being:

5. A mix of progressive, soft telecined film and true interlaced 29.97fps video
8. A mix of progressive, soft telecined film and progressive 29.97fps material (which anime has sometimes, as well as other sources I've seen).

But I could easily be wrong. I've never used MeGUI.

I've attached a little sample. Maybe you (or someone) could tell me what it actually is. As best as I can tell it's hard-telecined, but MediaInfo says it's 23.976 progressive with 2:3 pulldown, MPC-HC says it's 29.970 and DGIndex says it's 83.18% FILM.

MediaInfo just checks the first frame which is soft telecined with pulldown. Give it a sample beginning with that black part and it'll say differently. MPC-HC is correct as all NTSC DVDs output 29.97fps. DGIndex is correct as the black section in the middle is hard telecine and needs an IVTC, although because it's black a Force-Film will work as well. All the rest of it is soft telecine with pulldown.

[hello_hello]

Do you mean DGIndex? And the answer is 'no'. Hard telecine shouldn't be Force-Filmed and is considered to be Video by DGIndex. Only soft telecine can be Force-Filmed successfully.

Thanks.

5. A mix of progressive, soft telecined film and true interlaced 29.97fps video
8. A mix of progressive, soft telecined film and progressive 29.97fps material (which anime has sometimes, as well as other sources I've seen).

Assuming that's correct, would they both be de-interlaced in the same way?

DGIndex is correct as the black section in the middle is hard telecine and needs an IVTC, although because it's black a Force-Film will work as well. All the rest of it is soft telecine with pulldown.

So to help me understand this, could you please explain why this section appears to have "interlacing lines", if that's the correct term for it, which it probably isn't, but I guess you know what I mean. I thought the presence of those "interlacing lines" meant it's hard telecined, as they come and go in a 3:2 pattern, but I guess that's not the case. When I use separatefields() and step through them I can see the 3:2 pattern. Is that because it's hard telecined or is DGDecode applying 3:2 pulldown when it's decoding? If it's the latter then how do you tell the difference between hard telecine and soft telecine when looking at the video?

This is the way the video appears in MedGUI's preview before any IVTC is applied.



I basically understand the NTSC concept, at least in theory, but not having worked with much NTSC video I guess I don't really understand it in practice. Generally I just rely an the analysis getting it correct, which most of the time it seems to do. On the odd ocassion I think it hasn't, I ask AutoGK for a second opinion as it invariably seems to get it right. It's probably time to understand this a little better though, so I don't have to keep feeling too much dumber than software.

[manono]

Assuming that's correct, would they both be de-interlaced in the same way?

That's a big assumption as I'm not privy to what the developers intended by those distinctions. Anyway, if the aim is to reduce them to a common framerate (as opposed to a VFR encode), then that hybrid stuff is one way. I wouldn't use it, though, because I hate blending. I was always partial to SmartDecimate because it'll choose the most likely bobbed field and the result will be more pleasing looking (to me) for number 5. If I'm right about what number 8 is, and if you have to reduce it to a common framerate, it's either the choice of jerky playback or blended blurry playback that still plays a little bit jerky. I'd say either make a DVD of it, which can have both 23.976 'base' framerates (with hard and/or soft telecine) and true 29.97fps content, or do a VFR encode.

If I'm right about what number 8 is (pure progressive content), then technically there's no deinterlacing to be done as there wasn't any interlacing to begin with. All that'll be done with that hybrid script is to create 4 frames from what was originally 5 by blending two of them together.

So to help me understand this, could you please explain why this section appears to have "interlacing lines", if that's the correct term for it,

My guess is because they made the D2V project file using 'Honor Pulldown Flags' which gives you, in effect, hard telecine ready for the IVTC. If the picture was taken from a 29.97fps video, that's the answer. Even though the DVD itself may be mostly composed of soft telecine, if you use 'Honor Pulldown Flags', the result will be interlaced 29.97fps.

...how do you tell the difference between hard telecine and soft telecine when looking at the video?

You don't even have to look at the video. Just open it in DGIndex and run the preview. If it says 'Film', it's soft telecine. If it says 'Video', it's not. And that assumes the flags were set properly in the DVD. That's true most times, but not all the time.

[theewizard]

i usually stay out of interlace discussions, i just throw away the file

i would like like to know how IF possible

to remove crap like this, ( fix it) when all you have is the converted file, flv, avi, mp4, wbem etc..

how would you do a 'hard decimate' or what ever, avisynth and virtual dub ?

[jagabo]

how would you do a 'hard decimate' or what ever, avisynth and virtual dub ?

If you mean 30p with a duplicate every 5 frames (like the sample in post 3): TDecimate()

[manono]

If you mean 30p with a duplicate every 5 frames (like the sample in post 3): TDecimate()

But that sample is (mostly) soft telecine. You were playing it with a deinterlacing player, maybe? I think theewizard must be referring to something else, maybe something he downloaded that wasn't handled correctly. There's nothing wrong with the sample.

[theewizard]

the video is 25fps

https://forum.videohelp.com/attachment.php?attachmentid=16902&stc=1&d=1364190790

https://forum.videohelp.com/attachment.php?attachmentid=16903&stc=1&d=1364190790

https://forum.videohelp.com/attachment.php?attachmentid=16904&stc=1&d=1364191036

[manono]

So you weren't referring to the sample in post 3? I didn't think so. However, screenshots tell us nothing. Samples please.

[theewizard]

General
Complete name : C:\Users\Mr Wizard\Videos\Star Trek Of Gods and Men (Official Complete Film) (High).wbem
Format : WebM
Format version : Version 2
File size : 629 MiB
Duration : 1h 28mn
Overall bit rate mode : Variable
Overall bit rate : 994 Kbps
Writing application : google
Writing library : google

Video
ID : 1
Format : VP8
Codec ID : V_VP8
Duration : 1h 28mn
Bit rate : 818 Kbps
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 3:2
Frame rate mode : Constant
Frame rate : 25.000 fps
Standard : NTSC
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.095
Stream size : 518 MiB (82%)
Language : English
Default : Yes
Forced : No

Audio
ID : 2
Format : Vorbis
Format settings, Floor : 1
Codec ID : A_VORBIS
Duration : 1h 28mn
Bit rate mode : Variable
Bit rate : 128 Kbps
Channel(s) : 2 channels
Sampling rate : 44.1 KHz
Compression mode : Lossy
Stream size : 81.0 MiB (13%)
Default : Yes
Forced : No

the mp4 verson has the same problem, but is a lower resolution file

the segmented mp4, does not have this problem, it has the video drop out problem, but is the one i ended up using as it was more pleasing to watch

movie is 88 minutes

i will make a couple of cut samples and post them

i had to retrieve the video from the recycle bin just to make the snap shots

these are single frame shots, that is the way those frames look

thats why i just delete and start over with another source IF i can find it

[theewizard]

here are samples from the mp4
i could not edit the wbem to get samples

https://forum.videohelp.com/attachment.php?attachmentid=16906&stc=1&d=1364193810

https://forum.videohelp.com/attachment.php?attachmentid=16907&stc=1&d=1364193810

Media Info

General
Complete name : C:\Users\Mr Wizard\Videos\Star Trek Of Gods and Men (Official Complete Film) (SD).mp4
Format : MPEG-4
Format profile : Base Media / Version 2
Codec ID : mp42
File size : 296 MiB
Duration : 1h 28mn
Overall bit rate mode : Variable
Overall bit rate : 468 Kbps
Encoded date : UTC 2012-10-14 19:44:49
Tagged date : UTC 2012-10-14 19:44:49
gsst : 0
gstd : 5310866
gssd : B4A7DA664HH1363553202064253
gshh : r13---sn-a5m7znek.c.youtube.com

Video
ID : 1
Format : AVC
Format/Info : Advanced Video Codec
Format profile : Baseline@L2.1
Format settings, CABAC : No
Format settings, ReFrames : 1 frame
Format settings, GOP : M=1, N=60
Codec ID : avc1
Codec ID/Info : Advanced Video Coding
Duration : 1h 28mn
Bit rate : 370 Kbps
Maximum bit rate : 1 603 Kbps
Width : 540 pixels
Height : 360 pixels
Display aspect ratio : 3:2
Frame rate mode : Constant
Frame rate : 25.000 fps
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Progressive
Bits/(Pixel*Frame) : 0.076
Stream size : 234 MiB (79%)
Tagged date : UTC 2012-10-14 19:44:51

Audio
ID : 2
Format : AAC
Format/Info : Advanced Audio Codec
Format profile : LC
Codec ID : 40
Duration : 1h 28mn
Bit rate mode : Variable
Bit rate : 96.0 Kbps
Maximum bit rate : 104 Kbps
Channel(s) : 2 channels
Channel positions : Front: L R
Sampling rate : 44.1 KHz
Compression mode : Lossy
Stream size : 60.8 MiB (21%)
Title : IsoMedia File Produced by Google, 5-11-2011
Encoded date : UTC 2012-10-14 19:44:50
Tagged date : UTC 2012-10-14 19:44:51

[manono]

I'd say you were right to delete them. Among other things, some idiot resized them without taking the interlacing into account. I think they're ruined beyond all hope, but maybe someone else has some ideas.

[hello_hello]

manono,
Thanks for the info above. I'm starting to get my head around it a little more now.
Cheers.

[jagabo]

If you mean 30p with a duplicate every 5 frames (like the sample in post 3): TDecimate()

But that sample is (mostly) soft telecine. You were playing it with a deinterlacing player, maybe?

I see what happened. Normally, when I open MPEG 2 video in VirtualDub the MPEG 2 source filter performs soft pulldown and delivers interlaced frames in the normal 3 progressive 2 interlaced pattern. But since this video was in an M2TS container the DirectShow source filter was used instead. That was deinterlacing and delivering 30p with no comb artifacts, just the duplicate every 5th frame, all the way through the video. But I just used DgIndex to create a d2v file and open it via Mpeg2Source() and I see you are right. So something in the DirectShow pipe was deinterlacing.

[theewizard]

I'd say you were right to delete them. Among other things, some idiot resized them without taking the interlacing into account. I think they're ruined beyond all hope, but maybe someone else has some ideas.

I HAVE ONE IDEA, but no idea how to accomplish it

split every frame into feilds, drop the first feild

change it so the first bottom feild is now the new first top feild etc..
in other words shift ALL feilds one space left

re-combine,

not sure how much it would help, or what coding to use to try it

like you said with the sizing buggered, going to that extreme might not even help

[manono]

split every frame into feilds...

That's what I did when determining that it had been resized improperly. Even the fields are interlaced, something that should never happen. So, if you switch the fields, it'll still look nasty, just more so, if only for having been reencoded yet again.

jagabo has a method where he shrinks the resolution and then resizes it again, in the process blurring out the interlacing. But yours looks pretty bad to begin with, even if it didn't show the interlacing.

[JoeBolden]

.... you have to reduce it to a common framerate, it's either the choice of jerky playback or blended blurry playback that still plays a little bit jerky......or do a VFR encode.

I have a similar Asian (Taiwan) NTSC DVD. dgmpgdec158 DgIndex shows Interlaced and D2V Parse Output shows [GOP: open] xx[x]: xx,xx....2 all 2's, Visually looking at some motion frames are 3-2 some are 3-3 or even 2-4, 2-5 etc

Using Handbrake HandBrake -> Telecining -> Variable Frame Rate Detelecine
Would this be a good option converting to mkv?

[manono]

If you're asking me (and you seem to be), I wouldn't hazard a guess without a sample or two.

And I've never used Handbrake in my life or made a VFR encode. The movies I work with require neither. And I make DVDs for the most part. There are others more qualified to answer but most wouldn't give advice without samples either. From your description, you have a PAL2NTSC DVD which doesn't require a VFR encode, but unblending. Only a sample or two will prove (or disprove) that idle speculation.

[JoeBolden]

I wouldn't hazard a guess without a sample or two.

Thank you, I think you may be right Samples uploaded

[jagabo]

TestA1.avi and TestA2.avi used 3:2:3:2:2 pulldown to convert a progressive 25 fps PAL source to interlaced NTSC. Recombine the fields, then decimate to 25 fps. I have no idea if MeGUI has this ability. In AviSynth:

AviSource()
AssumeTFF()
TFM()
TDecimate(mode=2, rate=25)

[manono]

Handbrake doesn't use AviSynth, so I'm pretty sure it can't handle it. MeGUI probably can, but I've never used it either.

[JoeBolden]

Thank you jagabo & manono,
I am trying MeGUI (2325 beta (March 25, 2013) for the first time. (to get back on topic) MeGUI has an AVS Script creator and this is what processing as of this reply. Will report results with sample later. (actual movie name and local location changed for display here) Edited I forgot Assume TFF() will it matter? Ok to leave out AviSource() my prior sample was avi but using dv2?

-[Information] Log for job2 (video, movie.avs -> movie.mkv)
--[Information] [3/26/2013 3:41:09 PM] Started handling job
--[Information] [3/26/2013 3:41:09 PM] Preprocessing
--[Information] [3/26/2013 3:41:09 PM] Avisynth input script
# Set DAR in encoder to 175 : 96. The following line is for automatic signalling
global MeGUI_darx = 175
global MeGUI_dary = 96
LoadPlugin("C:\MeGUI_2308_x86\tools\dgindex\DGDecode.dll")
DGDecode_mpeg2source(X:\movie.d2v", info=3)
LoadPlugin("C:\MeGUI_2308_x86\tools\avisynth_plugin\ColorMatrix.dll")
ColorMatrix(hints=true, interlaced=true, threads=0)
LoadPlugin("C:\MeGUI_2308_x86\tools\avisynth_plugin\TIVTC.dll")
TFM()
TDecimate(mode=2, rate=25)
#crop
#resize
#denoise
--[Information] [3/26/2013 3:41:09 PM] Job commandline: "C:\MeGUI_2308_x86\tools\x264\avs4x264mod.exe" --level 4.1 --crf 18.0 --sar 175:144 --output "(X:\movie.mkv" "X:\movie.avs"

[jagabo]

Mpeg2Source(...,Info=3) should pass the field order along to TFM(). So you don't need the AssumeTFF().

[JoeBolden]

Thanks jagabo, Still processing BTW

...
5. "Hybrid film/interlaced. Mostly film": Mix of film & interlaced:
tfm(order=-1).tdecimate(hybrid=1)....

After analyzing I was presented with the same recommendation. When I went to edit avs script (MeGUI) and save got error message has to be mode 0,1 or 3. I tried "M-in-5 Decimation" set to 2 and then edited script and it saved ok.

UPDATE Going to have to redo this and might be busy for a couple days.