Why is removing heavy dot crawl and rainbowing so difficult to achive?

[themon-master]

Hello!
As you've read from the title, I've been trying to remove heavy dot crawl and rainbowing, but I haven't gotten much luck with.

[Attachment 94272 - Click to enlarge]
As you can see, the dot crawl and rainbowing is very aggressive so I can see why it's difficult to remove.
The best results I've gotten is

clip = core.dotkill.DotKillT(clip, offset=3, dupthresh=45, tratio=1)

which doesn't remove too much, but it's a start.
I don't really have much too say, I just need help with the source.

[pandy]

Because they are inherent part of color encoding and eventual color decoding - part of spectrum for luminance signal is shared with chrominance - proper separation may be not possible.
You can try filtering in frequency domain - use some plugin or software allowing to perform signal processing in frequency domain - in avisynth you can use for example http://avisynth.nl/index.php/FFTQuiver - alternatively you can try to separate luminance and chrominance and process independently.

[Selur]

Problem is that the image was reencoded and resized multiple times and thus normal decrawler detection methods do not catch the crawlers.

applying other denoisers / grain removers on the separated fields before applying IVTC or similar might help.
i.e. TemporalDegrain2 with temporal radius 3

helps quite a bit
using TemporalDegrain and TemporalDegrain2 in succession seems to work even better.

[themon-master]

Problem is that the image was reencoded and resized multiple times and thus normal decrawler detection methods do not catch the crawlers.

I know the source was re-encoded a couple of times, I can tell because of the bad deinterlacing and the original SD scaled to HD (to fit TV resolution). That makes sense why any of these plugins wouldn't work.
Also I was able to remove the more of the analog artifacts a little more with this script.

clip = core.bifrost.Bifrost(clip, luma_thresh=6.2, variation=8, interlaced=True, blockx=14, blocky=12)
clip = havsfunc.LUTDeRainbow(clip, cthresh=13, ythresh=13, y=True, linkUV=True, mask=False)
clip = core.dotkill.DotKillT(clip, offset=3, dupthresh=45, tratio=7)
clip = core.vivtc.VFM(clip, 1)
clip = core.dotkill.DotKillS(clip, iterations=4)
clip = core.vivtc.VDecimate(clip)

[Selur]

hmm,...

clip = core.dotkill.DotKillT(clip, offset=3, dupthresh=45, tratio=7)

that does cause ghosting here.

Did you try:

clip = core.std.SeparateFields(clip, tff=False) # resolution 1440x540
clipEven = clip[::2]
clipOdd = clip[1::2]
clipOdd = core.std.SetFrameProps(clip=clipOdd, _FieldBased=vs.FIELD_PROGRESSIVE) # scan type: progressive
# removing grain using TemporalDegrain2
clipEven = degrain.TemporalDegrain2(clipEven, degrainTR=3, degrainPlane=4, postFFT=0, fftThreads=1)
# removing grain using TemporalDegrain2
clipOdd = degrain.TemporalDegrain2(clipOdd, degrainTR=3, degrainPlane=4, postFFT=0, fftThreads=1)
# converting half-height progressive to interlaced
clip = core.std.Interleave([clipEven, clipOdd])
clip = core.std.DoubleWeave(clip=clip, tff=False)  # resolution: 1440x1080
clip = core.std.SelectEvery(clip, 2, 0) # scan type: bottom field first

befofe the IVTC, like I suggested? That work better here than what you suggested.