Best solution for deinterlacing hybrid content to VFR progressive?

[Pendell]

There are already several threads on the subject, but I've been perusing them for days and they all fizzle out with nobody coming to a real solution, or someone brings up Topaz and the discussion goes off the rails into a debate about AI upscalers.

My goal is simple. I have DS9 and VOY on NTSC DVD. These shows both combine 30i and 24i elements, creating a difficult problem for smooth deinterlacing. My goal is to create a variable framerate 480p video that moves between 24p and 30p where appropriate.

I generally use Vidcoder for my video encoding needs when dealing with HD sources that don't need any filtering or processing, but with SD sources it's extremely inadequate. Someone suggested turning on both its Detelecine filter and its EEDI2 deinterlacer with Interlace Detection, and that it should intelligently detect the changes and apply each filter appropriately. However, in practice, this barely works at all. Handbrake's underlying engine seems to do no pre-filter analysis of the footage of any kind, just guessing on the fly about the frame rate of each shot, and usually guessing wrong. In all my tests, 30p video will randomly switch to choppy 24p, and vice versa.

I've had some cursory experience with using QTGMC in AviSynth to deinterlace true video sources (DV tapes) from 60i to 60p, but this is obviously a far more complex task, and there's so many conflicting approaches. I don't want to upscale or anything, I just want the best option for taking these DVDs and converting them to a VFR 480p output.

If I can't find a good solution to this I guess I could fall back on using QTGMC to turn the whole thing into 60p. It would preserve the judder present in the source, but I'd ideally like to remove that, and make something genuinely nice to watch, if possible.

[jagabo]

You would have to detelecine the 24p sections and 30p sections separately. Then convert both to 120p (5 repeats for the 24p sections, 4 repeats for the 30p sections). Put the sections back together, then use DeDup() to remove duplicates and produce timecodes for the remaining frames.

But the only way to view the 24p sections without judder is to view it on a 120 Hz display. On a 60 Hz display you'll still see 3:2 repeat judder.

[Pendell]

You would have to detelecine the 24p sections and 30p sections separately. Then convert both to 120p (5 repeats for the 24p sections, 4 repeats for the 30p sections). Put the sections back together, then use DeDup() to remove duplicates and produce timecodes for the remaining frames.

But the only way to view the 24p sections without judder is to view it on a 120 Hz display. On a 60 Hz display you'll still see 3:2 repeat judder.

I... haven't heard this one before. What would be the purpose of upconverting everything to 120p just to bring it back down to the appropriate framerate? With regards to watching 24p content on a 60Hz display, I'd much rather the viewing device take care of that. Have the appropriate film sections run at 24p (or more accurately 23.976p), allowing 60Hz devices to handle the pulldown on its own and those with 120Hz devices to get the full experience.

How would I go about deinterlacing the 24p and 30p sections separately? If you're talking about poring over each episode manually, I simply don't have the time or energy to do that for 348 episodes of television.

[Bwaak]

I have DS9 and VOY on NTSC DVD. These shows both combine 30i and 24i elements, creating a difficult problem for smooth deinterlacing.

I suppose you mean 24p (or 24F), not 24i.

My goal is to create a variable framerate 480p video that moves between 24p and 30p where appropriate.

Why do you want to reduce 30i to 30p?

Recently I worked on a video that was mostly film (24p with pulldown), but transitions were made in video mode (30i). I did two conversions first: to 24p and to 60p. Then I brought both of them into an NLE onto 24p timeline, and replaced combed parts on 24p track with parts from 60p track. I turned off frame resampling and rendered as 24p. The transitions still look good in 24p, not too choppy, and they are short anyway.

I don't like VFR.

I simply don't have the time or energy to do that for 348 episodes of television.

Then I would deinterlace everything into 60p.

[Pendell]

I suppose you mean 24p (or 24F), not 24i.

Yes, although I'll note, at least for the first season, DS9 seems to switch on a whim between soft telecine (true 23.976 progressive just with flags for a DVD player to telecine) and hard telecine (59.94 interlaced fields per second)

Why do you want to reduce 30i to 30p?

I'm not certain that the CGI effects are running at 60 fields a second. I'm not certain how to test it, but they're not "soap opera" smooth, you know what I mean? They look to me like 30p native content that's just been rendered out interlaced. I wouldn't want to output unnecessary extra frames.

Recently I worked on a video that was mostly film (24p with pulldown), but transitions were made in video mode (30i). I did two conversions first: to 24p and to 60p. Then I brought both of them into an NLE onto 24p timeline, and replaced combed parts on 24p track with parts from 60p track. I turned off frame resampling and rendered as 24p. The transitions still look good in 24p, not too choppy, and they are short anyway.

I don't like VFR.

If I output the entire thing at 24p, the film content would be perfect, but pretty much any "space" scene would come out choppy and gross. Also for my use case, none of my software or hardware has any issues with VFR video, so I consider it an optimal solution to the problem.

I simply don't have the time or energy to do that for 348 episodes of television.

Then I would deinterlace everything into 60p.

I will if I have to, but I was just wondering if there was any possible solution developed in the AviSynth space for handling hybrid content reliably. I've heard of AnimeIVTC, which tackles a lot of issues that old SD anime seems to share with DS9 and VOY, but it also seems to include extra filters and settings optimized for anime that I don't know if I want to apply to a live action show.

Does QTGMC have any native method of handling hybrid content? I've heard lots of people mention timecodes files, would one of those help in creating the VFR output I'm looking for?

[jagabo]

What would be the purpose of upconverting everything to 120p just to bring it back down to the appropriate framerate?

To get precise time codes for all the appended 24p and 30p segments. DeDup() builds time codes of the transitions between different frames. 120 fps is lowest frame rate that's an integer multiple of both 24 fps and 30 fps. The 24p segments will get a frame change every 5 frames. The 30p segments will get a frame change every 4 frames. Every frame will have a precise time code and there will be no judder. But again, that's only true if you view the video at 120 fps.

But again, if you watch on a 60 fps display the judder will return for the 24p segments. That's why it was suggested you just stick with 60p. The DVDs are natively 60 fields per second. It's easy to convert that to 60 frames per second. Playing the resulting video will look the same (judder wise) as playing the original DVD on a 60 Hz TV. For smarter TVs which run their panels at higher frame rates (eg 120 Hz) the incoming 60p 3:2 frame repeats may be detected and convert to 5:5 repeats for the 120 Hz display. The 2:2 repeats will also be converted to 5:5 repeats. So you will get judder free playback for both segments.

[hello_hello]

There are already several threads on the subject, but I've been perusing them for days and they all fizzle out with nobody coming to a real solution, or someone brings up Topaz and the discussion goes off the rails into a debate about AI upscalers.

My goal is simple. I have DS9 and VOY on NTSC DVD. These shows both combine 30i and 24i elements, creating a difficult problem for smooth deinterlacing. My goal is to create a variable framerate 480p video that moves between 24p and 30p where appropriate.

See my post here (TIVTC is an Avisynth plugin).
https://forum.videohelp.com/threads/409300-3-2-pulldown-flags-and-29-97PsF-in-the-same...de#post2687420

[TomB-0]

Posting late as this is one of the top results and I've been trying to figure this out myself. I'm not trying to upscale but hopefully you can learn something from this. I have a flawless looking straight up encode.


Problems

There are two main issues with encoding these DVDs:

1. The content is VFR,

A. encoding to CFR involves dropping or duplicating frames. Dropping down to 23.970 looks terrible and stuttery because you lose frames. The motion looks bad if you drop 29.97 to 23.976 for one simple reason: If there is an object moving across the screen at 50 pixels per frame and you remove a frame then it moves 100 pixels in one step due to the missing frame. Where originally it moved 50px,50px,50px,50px, dropping a frame has it now moving 50px,100px,50px over the same period.This is noticeable and annoying. Dropping frames is not the solution.

B. Because it's VFR, trying to apply CFR at all (Even adding too many frames) looks bad because the frame times are wrong. Where in the middle of the footage some frames are displayed for 41ms (23.976fps) and some are displayed for 33ms (29.97fps). To increase frames in CFR you'd need to encode into a framerate that divides exactly into both 29.97 and 23.976, even then you'll likely end up with some merged frames (blurry!). If you apply CFR the frame times are stretched to a consistent number for all frames which is why it looks jerky

C. Either way you go, CFR looks bad.

Do not reencode CFR! Keep the variable framerate or it looks awful.

2. The content is telecined

I played around with all kind of inverse telecine filters, almost all the examples online ultimately rely on a `decimate` filter, which drops frames and causes motion issues. However you do it, it will mess with the frame times.

While there may be a way to do this with decimate, ultimately it's not necessary.


Goals

It's worth considering what we want to achieve here. If you do a straight up 1:1 VFR encode without any detelecine filters the motion looks perfect but the telecined frames are jarring.

1. Remove telecine artifacts (deinterlacing the correct frames)
2. We can't deinterlace the progressive frames as that causes new artifacts
3. Keep VFR for the best looking motion

Solution

The solution is the following:

- Detect interlaced frames
- Deinterlace only those frames
- Keep VFR

While I'd played around with VapourSynth a lot, I actually had the best luck with straight up ffmpeg as it doesn't decouple the timecodes from the video making the process a lot tidier.

The follow filter chain in ffmpeg can solve this:

fieldmatch=mode=pc_n_ub:combmatch=full:combpel=70,nnedi=weights=/usr/lib/vapoursynth/nnedi3_weights.bin:deint=interlaced

`fieldmatch` identifies any deinterlaced frames and tags them as iterlaced for the deinterlace filter, the settings above worked for all my samples, I haven't noticed any jarring interlaced frames in playback.

`nnedi` is the deinterlace filter I'm using, specify the path to the weights.bin file. You can use `yadif` or another alternative, nnedi looks better but since it's only on certain frames you're unlikely to tell the difference here. `:deint=interlaced` is important so that only the frames identified by `fieldmatch` are deinterlaced.

I have also verified that this approach doesn't insert any extra frames, extracting a frame before/after a deinterlaced frame so all the frame times are perfect, only the interlaced frames are fixed

(You can extract a frame using

ffmpeg -i "file.mkv" -vf "select=eq(n\,60184)" -vframes 1 60184.png

where 60184 is the number of the frame given by fieldmatch in the output, try comparing the same frame from the source video and the result)

tl;dr here's a script that encodes every file in a directory using those filters, I'm encoding to AV1/Opus but choose your codecs.

#!/bin/sh
for FILE in *; do
  INFILE=$FILE
  OUTFILE="${FILE//mkv/av1\.mkv}"

  if [ ! -f "$OUTFILE" ]; then
        if [[ ${INFILE} != *".av1.mkv"* ]]; then
        ffmpeg  -i "$INFILE" \
            -vf "fieldmatch=mode=pc_n_ub:combmatch=full:combpel=70,nnedi=weights=/usr/lib/vapoursynth/nnedi3_weights.bin:deint=interlaced" \
            -ac 6 \
            -map_metadata 0 \
            -pix_fmt yuv420p10le \
            -c:v libsvtav1 -crf 24 -preset 3 -c:a libopus -b:a 384k -c:s copy \
            -map "0:v"  -map "0:a" -map "0:s"  \
            "$OUTFILE"
        fi
    fi
done

[poisondeathray]

fieldmatch=mode=pc_n_ub:combmatch=full:combpel=70,nnedi=weights=/usr/lib/vapoursynth/nnedi3_weights.bin:deint=interlaced

This will have problems on many DVD's - Hard telecine sections, hard telecine DVD's . You can think of it as "23.976p in 29.97i" with encoded field repeats . It's not soft telecine which is encoded 23.976p, with repeat field flags. In the hard telecine case, there will jerky playback (every 4th , 5th frame are duplicates) because the duplicates remaining after field matching are not decimated

VFR means each section plays at their correct content frame rate, without duplicates or drops. ie. 23.976p content sections play at 23.976p , 29.97p content sections play at 29.97p , interlaced content sections get double rate deinterlaced to 59.94p sections (because they were 59.94 fields/s, with each field having unique motion) . There are other content rates too - e.g. anime pans can be 12fps

[manono]

I simply don't have the time or energy to do that for 348 episodes of television.

Then bob them all with the best AviSynth bobber you want to use. Simple, and they'll play fine at 59.94fps, just as your television would play it if you created the VFR encode.

[manono]

1. The content is VFR,

Nope, it's all 29.97fps as required for NTSC DVD.

2. The content is telecined

Some of it is, but not the CGI parts.

[TomB-0]

Nope, it's all 29.97fps as required for NTSC DVD.

If I force the framerate to 29.97 (which you suggest is the case anyway) then non-cgi sections are sped up so I'm not sure how that can be the case. Watching the framerate in mpv's info panel, the framerate does change between scenes.

I wonder if it's because I used MakeMKV which produces output as documented here: https://www.extremetech.com/extreme/324943-deep-space-nine-project-update-why-makemkv-...iles-dont-work

The reason why MakeMKV files don't work in this project is that variable frame rate MakeMKV files do not represent the video data contained within the VOB in a manner that AviSynth can work with. The progressive 1-2-3-4-4 and 1-2-3-4-5-5 pattern encoded into MakeMKV files plays back smoothly because the file stream contains a plethora of hints telling the player software how to play it and which fields/frames should be displayed. Turning that stream into something that other applications can process without breaking motion is not easy. AviSynth can convert the file to a constant frame rate and keep audio synchronized, but not without breaking motion in the 29.97 fps sections.

So yeah, it looks like it is VFR in this instance. If I play back the file straight from ripping with makemkv the motion is good but there are visible interlaced frames (most noticable in dark scenes). Converting it with ffmpeg also uses these same hints from the makemkv output and the encode is smooth.

If I deinterlace every frame I lose significant quality as frames which are not interlaced get deinterlaced (and the framerate is still VFR, it's either 47 or 60fps depending on the scene)

Without changing a thing in the file straight from MakeMKV and playing it back with deinterlacing off I'd expect jerky motion if that were the case but it's fine.

Some of it is, but not the CGI parts.

Which is why `fieldmatch` is useful as it detects the frames that need deinterlacing.

[manono]

If I force the framerate to 29.97 (which you suggest is the case anyway) then non-cgi sections are sped up so I'm not sure how that can be the case.

First, that's not what I suggested. I suggested bobbing it (doubling the framerate to 59.94fps). Second, the length stays the same, nothing is speeded up or slowed down.

Don't use MakeMKV. Decript it to VOB files and then use DGIndex to create a D2V project file to be opened in an AviSynth script using MPEG2Source as the source filter. It's all explained in the docs included in the DGMPGDec package.

No, it's still not VFR. There may be a mix of hard telecine, soft telecine and true interlaced sections, but they all play at 29.97fps (actually, 59.94 fields per second).

Bobbing it is far and away the simplest and easiest solution. They'll play on your television set just as the DVD does. You're the one that said he has 348 episodes to do. Just stay far away from MakeMKV.

[TomB-0]

Yes, you're right. Having looked into that further there are still some frames with interlacing artifacts. Playing around with the fieldmatch parameters helps but it never gets them all.

Since I already ripped all the episodes with makemkv and all the video data is still there you just need to load the video from a source that won't take fields from different frames. This VapourSynth script works as long as you use lsmas as a source.

from vapoursynth import core
import havsfunc
import os

clip = core.lsmas.LWLibavSource(os.getenv('FILENAME'), fpsnum=30000,fpsden=1001)
clip = havsfunc.QTGMC(Input=clip, Preset="Very Slow", TFF=True, InputType=0)
clip.set_output()

I've gone for QTGMC to do the deinterlacing but this works. There are no motion issues and no extra interlaced frames.

[TomB-0]

After playing back a few files using that approach, the motion is noticeably jerky in the live action parts while the 30fps CGI is fine. After further research, the issue is outlined here: http://avisynth.nl/index.php/VFR

It looks like I can either encode to 120fps or VFR. I went for the VFR approach.

For anyone interested, here's the scripts I ended up with:

for FILE in *; do
   INFILE=$FILE
   OUTFILE="${FILE//mkv/av1\.mkv}"

   if [ ! -f "$OUTFILE" ]; then
        if [[ ${INFILE} != *".av1.mkv"* ]];then
        if [[ ${INFILE} != *"mkv" ]];then
        FILENAME=`realpath "$INFILE"` vspipe -c y4m '/path/to/gettimecodes.vpy' - | \
        ffmpeg -i - -c:v copy -f null -

        FILENAME=`realpath "$INFILE"` vspipe -c y4m '/path/to/tfm.vpy' - | \
        ffmpeg -i "$INFILE"  -i - \
            -ac 6 \
            -map_metadata 0 \
            -pix_fmt yuv420p10le \
            -c:v libsvtav1 -crf 24 -preset 3 -c:a libopus -b:a 384k -c:s copy \
            -map "1:v"  -map "0:a" -map "-0:s:1"   \
            -vf "setdar=4/3" \
            "$OUTFILE"

             mkvmerge -o "final_${OUTFILE}" --timestamps 0:"${INFILE}timecodes.txt" "$OUTFILE"
        fi
    fi
done

Which uses these two vapoursynth scripts. It requires https://github.com/dubhater/vapoursynth-tivtc

gettimecodes.vpy

from vapoursynth import core
import os

clip = core.lsmas.LWLibavSource(os.getenv('FILENAME'), fpsnum=30000, fpsden=1001)
clip = core.tivtc.TFM(clip,mode=1,output=os.getenv('FILENAME')+"tfm.txt")
clip = core.tivtc.TDecimate(clip,mode=4,output=os.getenv('FILENAME')+"stats.txt")
clip.set_output()

tfm.py

from vapoursynth import core
import os

clip = core.lsmas.LWLibavSource(os.getenv('FILENAME'), fpsnum=30000, fpsden=1001)
clip = core.tivtc.TFM(clip, mode=1)
clip = core.tivtc.TDecimate(clip,mode=5,hybrid=2,dupThresh=1.0,input=os.getenv('FILENAME')+"stats.txt",tfmIn=os.getenv('FILENAME')+"tfm.txt",mkvOut=os.getenv('FILENAME')+"timecodes.txt")
clip.set_output()

requirements: ffmpeg, vapoursynth, vapoursynth-lsmash, vapoursynth-tivtc