NTSC to PAL source with Frame-blending and field order judder

[PRAGMA]

So this is a very strange DVD from South Africa. It is Mew Mew Power, the only English DVD.

Samples:
- intro: https://mega.nz/file/t9gjxTLY#ILclLeRtd5_NLoJZyqL1e5yjs_TpQMsO9Ym9_mAD0wc
- episode scenes: https://mega.nz/file/54ZGkI5S#TypW9D7wIt6XLIvtz4wGip2Jl54-fflH-Wmk4x_gv4k

This source has frame blending from NTSC to PAL conversion in an inconsistent pattern (from what I can tell), as well as some kind of outline overshoot on some fields. It also seems to have the field order incorrectly set to BFF when it's TFF.

Incorrect Field Order

If you Bob deinterlace (to look at each individual field) you will see it's sort of Jerky. Where it kind of goes back and forward a field (now "frame").

Jerky Showcase (at 1fps): https://mega.nz/file/VpgiSTJZ#dDqLpsDibPElEZv7EYn46Y9pGebzsYPiyVvFxxW17FI

When deinterlaced by MPV using real-time deinterlacing, it does not happen, even when seeking frame-by-frame.
When deinterlaced in VapourSynth by Bob/QTGMC/YADIF, it happens.

I've already found a fix for it, swap each field and then weave them back together (BFF to TFF by swapping the fields):

clip = core.std.SeparateFields(clip, tff=True)  # False or True doesn't matter, both work
clip = core.std.SelectEvery(clip, cycle=2, offsets=[1, 0])
clip = Weave(clip, tff=True)

or via SetFieldBased (BFF to TFF by editing the field flags):

clip = core.std.SetFieldBased(clip, value=2)  # where 1 is what it currently is

Fix Showcase (at 1fps): https://mega.nz/file/Mkw2zC5Y#N6wrjhNbfMAl9RFEkq1-B0dV4nBfCMCiJyUN9KjAg_k

I'm unsure why it happens in VapourSynth, yet it did not happen to MPV. Could this be a DGIndex error?

Also, both of these approaches to fixing the issue will also apply to unaffected frames or potentially progressive frames.
While this source is thankfully 100.0% interlaced, and every frame seems to have this issue, therefore that's not a huge concern.
But it would be ideal to have a more concise fix since if you discard every 2nd bobbed frame (every 2nd field) the effect does not happen, so it's clearly not
happening on EVERY field/frame per se.

Is there a better way to fix this if we assumed there could be a mix of progressive (software pulldown) frames mixed in?

---------

This is the result I've managed to get so far (Bob deinterlaced with QTGMC, every field kept):
https://mega.nz/file/kxp3xCbD#LEBIFLIXJQOZqP9N7pn5hCDmV3DWvLFzzirkvc5s538

Script: https://gist.github.com/rlaphoenix/6070c5efd04a987e5d3e2366ce6e59eb

Also any ideas on fixing the frame blending or the weird spouts of outlines (where no color and only bits of the outline are ghosting)?

[poisondeathray]

Could this be a DGIndex error?

Yes. The jerkiness when double rate deinterlacing looks like a DGIndex/MPEG2Source/D2V source filter issue . DGSource, LSMash work ok

[poisondeathray]

Also any ideas on fixing the frame blending or the weird spouts of outlines (where no color and only bits of the outline are ghosting)?

Some of them are tricky, without any nearby "clean frames" (not even to clone from or paint manually). It's a very crappy source. I'd take a serious look for another source

I don't see an "easy automatic" way for many of the contaminated frames, and some of them are "unique" frames with unique motion (basically critical that you fix it; unless you are willing to accept some duplicate and bad motion)

You can make use of some filters to speed up the work, such as high dpir filtering strength . But you have to roto/mask/layer/composite back the details from a weaker filter layer and/or nearby frames. It will take some manual work IMO if you want to clean it up properly . If the whole DVD is like this, it's a lot of work. Find some fans/forums and divide up the work

[Selur]

It also seems to have the field order incorrectly set to BFF when it's TFF.

Desktop\mmp_intro_sample.m2v is reported as TFF by MediaInfo.

General
Complete name                            : c:\Users\Selur\Desktop\mmp_intro_sample.m2v
Format                                   : MPEG Video
Format version                           : Version 2
File size                                : 51.3 MiB
Duration                                 : 2 s 640 ms
Overall bit rate mode                    : Constant
Overall bit rate                         : 163 Mb/s

Video
Format                                   : MPEG Video
Format version                           : Version 2
Format profile                           : Main@Main
Format settings                          : BVOP
Format settings, BVOP                    : Yes
Format settings, Matrix                  : Default
Format settings, GOP                     : Variable
Format settings, picture structure       : Field
Duration                                 : 2 s 640 ms
Bit rate mode                            : Constant
Bit rate                                 : 163 Mb/s
Maximum bit rate                         : 5 000 kb/s
Width                                    : 720 pixels
Height                                   : 576 pixels
Display aspect ratio                     : 4:3
Frame rate                               : 25.000 FPS
Standard                                 : PAL
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Interlaced
Scan order                               : Top Field First
Compression mode                         : Lossy
Bits/(Pixel*Frame)                       : 15.708
Time code of first frame                 : 01:01:26:21
GOP, Open/Closed                         : Open
Stream size                              : 51.3 MiB (100%)
Color primaries                          : BT.601 PAL
Transfer characteristics                 : BT.470 System B/G
Matrix coefficients                      : BT.470 System B/G

Looking at the file the last quarter (starting the 'MewMew Power'-logo) seems to be just broken.
For the fun of it, I threw DPIR, Despot and BasicVSR++at the first 1600 frames (script: https://pastebin.com/BuS0QG6e) while going down to 18fps and attached the output.
Also attached a more attentional approach using McTemporalDenoise (script: https://pastebin.com/Sr1muQxH).
=> with some scene-by-scene filtering, one could probably find a balance between throwing away stuff and keeping details, but I agree with poisondeathray you should better look for a better source. Especially for the last part of the intro, I see no way to rescue anything like:
when the frames next to it look similar.

Cu Selur

[rrats]

The only English DVD? Why not try finding a Japanese one. I know it's a stupid question but still. If it is the only dubbed version, then you can just replace the audio

[rrats]

4Kids Entertainment licensed the anime series for North American broadcast under the title Mew Mew Power, and produced an edited English-language localization. The first twenty-three episodes of the series would air on 4Kids TV in the United States, with

yeah it's just the Tokyo mew mew. I'm pretty sure that one at least has a NTSC dvd, if not a blu-ray release

[PRAGMA]

Could this be a DGIndex error?

Yes. The jerkiness when double rate deinterlacing looks like a DGIndex/MPEG2Source/D2V source filter issue . DGSource, LSMash work ok

Yep, this seems to be an error with d2vsource and not DGIndex, at least on the face of it. I've made a GitHub issue on the d2vsource repository.

[PRAGMA]

It's a very crappy source. I'd take a serious look for another source

That's hard because it's literally the only source for the 4Kids dubbed version for the West. The only other option would be to use the very good JP SD-on-BD release and spend ages re-creating the cuts for the Dub version. This would be very time-consuming to find which scenes from which episodes are part of the Dub intro, timing the cuts with the dubbed audio, and so on.

I don't see an "easy automatic" way for many of the contaminated frames, and some of them are "unique" frames with unique motion

The blending pattern seems very erratic. Sometimes there are blends separated by only 1 or 2 good frames, sometimes a good second apart. However, to me, they all seem to be duplicate frames from either the previous or the next frame. Meaning if you just discard that frame it's probably fine.

You can make use of some filters to speed up the work, such as high dpir filtering strength . But you have to roto/mask/layer/composite back the details from a weaker filter layer and/or nearby frames.

Yeah, I knew of DPIR for a while but never gave it a shot until now. It's done quite a good job! Sadly the blocking in the intro is still quite apparent but it's somewhat better. However, what do you mean about compositing back the details from a weaker filter layer/neighboring frames?

It will take some manual work IMO if you want to clean it up properly . If the whole DVD is like this, it's a lot of work. Find some fans/forums and divide up the work

Unfortunately, it is the whole series, let alone just this DVD.

[PRAGMA]

Desktop\mmp_intro_sample.m2v is reported as TFF by MediaInfo.

Yes, that's correct. In the end, it seems to be a d2vsource problem. Hence why my fix works. Thankfully it's not something worse like an erratic pattern to hard pulldown or something!

Looking at the file the last quarter (starting the 'MewMew Power'-logo) seems to be just broken.

I don't personally see this issue on my end when loading with D2VSource. I see you loaded with DGDecodeNV. If you try loading with D2VSource and apply my fix (SetFieldBased to 2), does it happen? I presume the image at the end of your post is what you are showcasing here?

For the fun of it, I threw DPIR, Despot and BasicVSR++at the first 1600 frames while going down to 18fps and attached the output.

I gave a look at DeSpot from lostfunc, and I don't personally like the results. It blends and manipulates random parts of frames in quite big chunks quite noticeably. It did work somewhat for some of the artifacts, but when it's more of bigger lines or curves it does not. Here's a before/after of what I mean about it blending and such noticeably: https://imgsli.com/MTY3OTYw (Notice the back of the neck, ear, and bits of the hair).

[PRAGMA]

4Kids Entertainment licensed the anime series for North American broadcast under the title Mew Mew Power, and produced an edited English-language localization. The first twenty-three episodes of the series would air on 4Kids TV in the United States, with

yeah it's just the Tokyo mew mew. I'm pretty sure that one at least has a NTSC dvd, if not a blu-ray release

While this is Tokyo Mew Mew, it's more specifically the Dub "Mew Mew Power" which only ever had one English DVD release, a 6-volume PAL release in South Africa. While there are two PAL Australian DVDs as well as some French DVDs, only the South African release is complete with the rest being only a handful of episodes.

Tokyo Mew Mew on the other hand has a spotless SD on BD release that is well worth your money.

[PRAGMA]

The only English DVD? Why not try finding a Japanese one. I know it's a stupid question but still. If it is the only dubbed version, then you can just replace the audio

There's a huge amount of edits, cuts, and such that would need to be made to the Video track for the Dub audio to sync up. But it's an idea I am tempted to try (at least for the intro). But I'm fairly confident the last scene of the Intro with the Logo was made by 4Kids so recovering that might not be easy.

[PRAGMA]

Interestingly, in Selur's encode there's a scene where what should be the next frame is blended and low opacity. Yet on my Script, I have that frame (but its heavily compressed).

See:

[Attachment 70257 - Click to enlarge]

The previous frame on both my script and his encode is correct, yet this frame I have it normal (but heavily compressed), he has it blended. Strange!

[Selur]

Result might look better depending on the position of sRestore and McTemporalDenoise , moving McTemporalDenoise before sRestore might help.

[poisondeathray]

I don't see an "easy automatic" way for many of the contaminated frames, and some of them are "unique" frames with unique motion

The blending pattern seems very erratic. Sometimes there are blends separated by only 1 or 2 good frames, sometimes a good second apart. However, to me, they all seem to be duplicate frames from either the previous or the next frame. Meaning if you just discard that frame it's probably fine.

You can make use of some filters to speed up the work, such as high dpir filtering strength . But you have to roto/mask/layer/composite back the details from a weaker filter layer and/or nearby frames.

Yeah, I knew of DPIR for a while but never gave it a shot until now. It's done quite a good job! Sadly the blocking in the intro is still quite apparent but it's somewhat better. However, what do you mean about compositing back the details from a weaker filter layer/neighboring frames?

I was referring to fixing the "outlines" problem on some frames . They look like motion artifacts from a bad encoder and insufficient bitrate.

eg. This frame is a "unique" frame, and the frame before and after are heavily blended

(This is an apng, it should animate in most browsers)


You can use high dpir filtering strength (or some other denoiser), enough to remove the "outlines", but high strength (not just dpir, any denoiser) will also damage wanted details. In this example - look at the BG latticework and 5 vertical line detail beneath the eyes (lower eyelashes)? , tongue is desaturated, face and hair shadowing, chin dimple. I don't know about you, but that detail loss bothers me.

You can use compositing techniques to add back the details; such as combining a lower strength dpir layer for the BG. Or add details from other fields and frames - if they were clean or you cleaned them in another step - basically clone them into the current frame. Many of the motions in this style of animation are simple translations - so you can go a few frames forward or back, and often motion tracking can help assist copying textures into the current frame. Usually that compositing part would be done in something like After Effects, Nuke, Natron - ie. compositing applications which have tools to make accurate masks and accessory tools. A procedural mask in vapoursynth/avisynth usually would not be accurate enough for work like that. And of course you can add other filters afterwards - sharpen, upscale etc.. many machine learning algorithms work well on this type of content

Not just the "outlines" either ; some of the blocky frames are "very blocky" , but a few frames away in a pan might have some clean(er) textures you can use

[poisondeathray]

Here is an example of cleaning up blocky frames using similar method of combining layers (or frames in this case, into a larger frame). No deblocking filter (not even machine learning ones) will clean up bad frames like that cleanly.

You spatially align 2 frames together near the beginning and end of the pan (cleaned with some filters - in this example using dpir, realesrgan) into a clean plate then animate it. If it was a larger scene, longer pans, you can patch more adjacent frames. You can make real 50p motion if that was your goal (the example below was compared to "50p QTGMC" version, and slowed for comparison purposes at 5fps) , or 24p, or time it to whatever you want. The duplicates and blends are just replaced with the animated motion frames. You can add motion blur (in modern animation productions, it would probably have motion blur @24p. But if you keep it at fake 50p, you can probably reduce the intensity of blur). Usually you want to keep the similar motion in terms of easing, acceleration, holds in the original animation. Because blends and unintended duplicates were a bit jerky, I chose to make it smoother, but keeping the strings of hold frames. You have full control because the animation for many scenes for this type of animation are just translation/zoom of static layers. The only different motion I could see is the pink girl's motion which is slightly more closed in the 2nd half


[Attachment 70283 - Click to enlarge]

[PRAGMA]

I don't see an "easy automatic" way for many of the contaminated frames, and some of them are "unique" frames with unique motion

The blending pattern seems very erratic. Sometimes there are blends separated by only 1 or 2 good frames, sometimes a good second apart. However, to me, they all seem to be duplicate frames from either the previous or the next frame. Meaning if you just discard that frame it's probably fine.

You can make use of some filters to speed up the work, such as high dpir filtering strength . But you have to roto/mask/layer/composite back the details from a weaker filter layer and/or nearby frames.

Yeah, I knew of DPIR for a while but never gave it a shot until now. It's done quite a good job! Sadly the blocking in the intro is still quite apparent but it's somewhat better. However, what do you mean about compositing back the details from a weaker filter layer/neighboring frames?

I was referring to fixing the "outlines" problem on some frames . They look like motion artifacts from a bad encoder and insufficient bitrate.

eg. This frame is a "unique" frame, and the frame before and after are heavily blended

(This is an apng, it should animate in most browsers)


You can use high dpir filtering strength (or some other denoiser), enough to remove the "outlines", but high strength (not just dpir, any denoiser) will also damage wanted details. In this example - look at the BG latticework and 5 vertical line detail beneath the eyes (lower eyelashes)? , tongue is desaturated, face and hair shadowing, chin dimple. I don't know about you, but that detail loss bothers me.

You can use compositing techniques to add back the details; such as combining a lower strength dpir layer for the BG. Or add details from other fields and frames - if they were clean or you cleaned them in another step - basically clone them into the current frame. Many of the motions in this style of animation are simple translations - so you can go a few frames forward or back, and often motion tracking can help assist copying textures into the current frame. Usually that compositing part would be done in something like After Effects, Nuke, Natron - ie. compositing applications which have tools to make accurate masks and accessory tools. A procedural mask in vapoursynth/avisynth usually would not be accurate enough for work like that. And of course you can add other filters afterwards - sharpen, upscale etc.. many machine learning algorithms work well on this type of content

Not just the "outlines" either ; some of the blocky frames are "very blocky" , but a few frames away in a pan might have some clean(er) textures you can use

This result is very impressive. You gave some nice information, but I'm curious about your exact process here. I'm not sure where to begin. You stated that VS/AVS isn't accurate enough, so I presume you did not use VS/AVS for this?

[PRAGMA]

Here is an example of cleaning up blocky frames using similar method of combining layers (or frames in this case, into a larger frame). No deblocking filter (not even machine learning ones) will clean up bad frames like that cleanly.

You spatially align 2 frames together near the beginning and end of the pan (cleaned with some filters - in this example using dpir, realesrgan) into a clean plate then animate it. If it was a larger scene, longer pans, you can patch more adjacent frames. You can make real 50p motion if that was your goal (the example below was compared to "50p QTGMC" version, and slowed for comparison purposes at 5fps) , or 24p, or time it to whatever you want. The duplicates and blends are just replaced with the animated motion frames. You can add motion blur (in modern animation productions, it would probably have motion blur @24p. But if you keep it at fake 50p, you can probably reduce the intensity of blur). Usually you want to keep the similar motion in terms of easing, acceleration, holds in the original animation. Because blends and unintended duplicates were a bit jerky, I chose to make it smoother, but keeping the strings of hold frames. You have full control because the animation for many scenes for this type of animation are just translation/zoom of static layers. The only different motion I could see is the pink girl's motion which is slightly more closed in the 2nd half


[Attachment 70283 - Click to enlarge]

When you talk about making a clean plate by aligning 2 frames together, what do you mean exactly? I'm interpreting it as you trying to re-create the original image that was panned left to right with, by merging the frames that are good together with each other, which would reduce the overall compression and blocking. Some blocky/compressed frames however would still be left over, right?

The result you have here is quite nice, but if you notice the Yellow girl has a missing finger now, and the other finger outline is all mashed up. Not sure how that has happened other than denoising+deblocking effectively masking the finger, which I could totally see happening.

[poisondeathray]

This result is very impressive. You gave some nice information, but I'm curious about your exact process here. I'm not sure where to begin. You stated that VS/AVS isn't accurate enough, so I presume you did not use VS/AVS for this?

VS/AVS used only for dpir. The masking/compositing in AE. View some tutorials on "rotoscoping", "layers and compositing" - there are many free ones youtube, and many courses you can take

It's slow because of manual input. You specify what areas you want to include/exclude. You can draw bezier masks. You can combine them with vapoursynth methods of masking and filters (you can frameserve into programs with avfs, or use a lossless intermediate).

On that example, there is no way to isolate the BG procedurally/automatically or with any accuracy in VS/AVS. You can't use luma mask, chroma mask, sat mask, motion mask, or some other technique. There are some "AI" BG removers (or FG isolation) that might work , but they are hit and miss and require lots of trial/error

When you talk about making a clean plate by aligning 2 frames together, what do you mean exactly? I'm interpreting it as you trying to re-create the original image that was panned left to right with, by merging the frames that are good together with each other, which would reduce the overall compression and blocking. Some blocky/compressed frames however would still be left over, right?

Not merging "good" frames as in mean/median/averaging to denoise (although you can use that technique to help generate the clean plate. )

The clean plate is like a big photo montage, a single big frame. It's like a jigsaw puzzle, and overlapping frames extending the size and dimensions of the clean plate. In this case , it's only 2 frames, because that's enough to cover the scene (the zoom and the pan) . The "seams" where each frame overlaps can be masked away by "top over bottom", or "bottom over top" using a feathered mask. There are other compositing techniques to "fix" various problems - they fall under the "compositing" category

In general, you stragetically choose "good" frames to include into the clean plate. You want to include as few as possible (but enough for complete coverage), and choose the cleanest frames (or after filtering become clean)

The result you have here is quite nice, but if you notice the Yellow girl has a missing finger now, and the other finger outline is all mashed up. Not sure how that has happened other than denoising+deblocking effectively masking the finger, which I could totally see happening.

Yellow girl finger - it's missing and mashed up in the QTGMC version, or a simple bob version, or a TFM version too . It looks to me like the last fingers are flexed into the palm. If you go frame by frame, check other frames, do you see any reference that has a "cleaner" finger reference? You can "fix" the finger if it was present in a source frame (it's not present that I can see); or you can photoshop it into the clean plate to make it look however you want.

[PRAGMA]

It's a very crappy source. I'd take a serious look for another source

It took a while, and it's still in the final polishing stages, but alas!

[Attachment 70317 - Click to enlarge]

This is using the Japanese SD on BD release as source clips. Done a ton of research and contacted people to figure stuff out:
https://gist.github.com/rlaphoenix/e2ef083a4beba29baf1731d5845199ca

I even managed to re-create the final scene with an explosion of colors and the Logo!
A community for this anime already knew what was used, and from there it was simple enough.

[Attachment 70323 - Click to enlarge]

Preview export: https://mega.nz/file/Y4Al2DiS#rYqP_9V4GZl-fD7AxtIynVQKxu4xU9W_KtiY6d7wogQ

Comparisons:


[Attachment 70319 - Click to enlarge]

[Attachment 70318 - Click to enlarge]

[Attachment 70320 - Click to enlarge]

[Attachment 70321 - Click to enlarge]

[Attachment 70322 - Click to enlarge]

[poisondeathray]

It's a very crappy source. I'd take a serious look for another source

It took a while, and it's still in the final polishing stages, but alas!

That's usually the "best" answer for these types of problems . It's not that difficult to splice/edit frames from other source(s) in a NLE. And for any remaining problems , you can use various compositing fixes. Sometimes you might have to reframe it slightly, but those are minor issues compared to starting with crap.

This is using the Japanese SD on BD release as source clips. Done a ton of research and contacted people to figure stuff out:

Yes - community / fan forums are usually the best resource, and 99% of the time, someone already has addressed the problem. At the very least, they provide important info on what is available, and a better starting point