removing black frames from interlaced footage without quality loss?

[DerBraune]

Hi!
I got a VHS rip of an old anime called Serial Experiment Lain. Unfortunately there are some black frames randomly in the video stream every few seconds to minutes. Weirdly enough after a black frame, the interlacing is very visible and goes away again after the next black frame.
https://www.youtube.com/watch?v=sK3y4lO61rc

The original uploader stated "The video is uncompressed, probably to retain every single grain of vhs tape goodness for the sake of authenticity." But it isn't. There are some minimal compression artifacts. Also it's only 4gb over 1.5 hours of footage with 720x480 at 29.97fps per file.
My question would be if it would be possible to remove those black frames possibly without reencoding the whole stream. Is there maybe a program that can remove black frames, replace those with the nearest nonblack and just reencode those bits from the last I-frame to the next and copy everything else from the original stream?

I have tried ffmpeg with the following command:
"ffmpeg -i in.mpg -vf blackframe=0,metadata=select:key=lavfi.blackframe. pblack:value=99:function=less -vsync cfr -c:a copy out.mpg"
It worked wonderfull! Unfortunately the quality loss is very extreme. I would like to keep it very high quality.

After that I tried to include "qtrle -pix_fmt rgb24" to get a lossless file and bother about compressing it later. The only problem, for some reason it looks like it uses chroma subsampling, or some other sort of lossy compression, even though I used rgb24, which should be lossless. This would mean, that it was already there before this process. Checking with MediaInfo revealed, that the original file also has 4:2:0 chroma subsampling. The interlacing would explain, why it is not easy to see. But the ffmpeg version also still has interlacing, but it now reports as "interleaved fields" and I'm not sure how to change that to upper first. As I understand it, that explains the different screenshots. But I also don't know enough to be sure.
https://imgur.com/a/8CaBGlV

Also the finished file was ~150gb and I'm not sure I can get it down again to the ~4gb of the original file without losing too much quality.

This is my first time having anything to do with VHS rips and interlaced video, so please correct me if I am wrong with anything and feel free to reply if you have any suggestions on this topic!



MediaInfo Data

-Videostream-
ID : 224 (0xE0)
Format : MPEG Video
Format version : Version 2
Format profile : Main@Main
Format settings : CustomMatrix / BVOP
Format settings, BVOP : Yes
Format settings, Matrix : Custom
Format settings, GOP : M=3, N=15
Format settings, picture structure : Frame
Duration : 1 h 38 min
Bit rate mode : Variable
Bit rate : 5 909 kb/s
Maximum bit rate : 8 000 kb/s
Width : 720 pixels
Height : 480 pixels
Display aspect ratio : 4:3
Frame rate : 29.970 (30000/1001) FPS
Standard : NTSC
Color space : YUV
Chroma subsampling : 4:2:0
Bit depth : 8 bits
Scan type : Interlaced
Scan order : Top Field First
Compression mode : Lossy
Bits/(Pixel*Frame) : 0.570
Time code of first frame : 00:00:00:00
Time code source : Group of pictures header
Stream size : 4.08 GiB (94%)
Color primaries : BT.601 NTSC
Transfer characteristics : BT.601
Matrix coefficients : BT.601

[Selur]

Do you have access to the uncompressed source or just the YouTube reencode?
If you got the uncompressed source compress part of it that shows the problem with a lossless video compression format and share a sample of that video.

[DerBraune]

Here you go! I should have done this in the first place haha.

It's about the same part as the YouTube video. The YouTube video was just to demonstrate the black frames.

Hope that helps!

[jagabo]

You cannot remove the black frames from the MPG file without reencoding. But if you're willing to reencode there are avisynth filters that can replace single black frames with a copy of the frame before, the frame after, or a motion interpolate frame.

[Selur]

okay, mpeg-2 not lossless by any standard.
Since you already found a command line that you can work with I would suggest to use a lossless compression format instead of mpeg, also your want to add '-flags +ildct+ilme' to make sure the output stays interlaced. Going for any other color space than yuv420p seems like a waste of space since your source is only yuv420p.
-> I would recommend to extend your command line to save to a lossless format which support interlacing. (for example lossless H.264)

---

Personally I would keep the 'original' and a Vapoursynth or Avisynth script to take care of the filtering and then use that script with the original as source for later reencoding.

Cu Selur

Ps.: It's probably cheaper to simply buy the Blu-ray of Serial Experiment Lain

[lordsmurf]

This was likely captured with a bad capture card.
The BM cards are especially known (infamously) for inserting black frames.

Just buy the DVD or BD if this same show exists in that format.

[DerBraune]

Since you already found a command line that you can work with I would suggest to use a lossless compression format instead of mpeg, also your want to add '-flags +ildct+ilme' to make sure the output stays interlaced. Going for any other color space than yuv420p seems like a waste of space since your source is only yuv420p.
-> I would recommend to extend your command line to save to a lossless format which support interlacing. (for example lossless H.264)

Thank you very much! Unfortunately MediaInfo still reports as "interleaved fields" instead of interlaced and it looks like on the image I linked in the original post again: https://imgur.com/a/8CaBGlV
I used the following command:
"ffmpeg -i in.mpg -vf blackframe=0,metadata=select:key=lavfi.blackframe. pblack:value=99:function=less -vsync cfr -c:v libx264 -crf 0 -preset medium -pix_fmt yuv420p -c:a copy -flags +ildct+ilme out.mp4"
Would you know how to get it to use interlaced?

You cannot remove the black frames from the MPG file without reencoding. But if you're willing to reencode there are avisynth filters that can replace single black frames with a copy of the frame before, the frame after, or a motion interpolate frame.

Personally I would keep the 'original' and a Vapoursynth or Avisynth script to take care of the filtering and then use that script with the original as source for later reencoding.

Never had to use those, so it's easier for now to just use ffmpeg and reencode it afterwards with the adobe media encoder. But thanks for the suggestion! If I don't get it to work, I'll take a look into it.

This was likely captured with a bad capture card.
The BM cards are especially known (infamously) for inserting black frames.

Interesting! Thanks for the info!

Ps.: It's probably cheaper to simply buy the Blu-ray of Serial Experiment Lain

Just buy the DVD or BD if this same show exists in that format.

I already have it, it's not about getting it for free, but it's just another experience to watch it with all the VHS artifacts and it fits soo well to the whole aesthetic and theme of the show. The bluray looks to clean haha. And the grading can also be very different sometimes.
Just a personal taste specific to this show.

[Selur]

Would you know how to get it to use interlaced?

when using x264 you also need to set the x264opts accordingly
see: https://ffmpeg.org/pipermail/ffmpeg-user/2016-April/031624.html

[johnmeyer]

Your capture has LOTS of additional problems beyond black frames, most notably the flagging at the top of the screen. A TBC used during capture can usually fix that.

The following script will replace all black frames with motion estimated versions. You can then save the result. I wouldn't worry about applying another level of compression because your source is so lousy that you'll never notice.

#Find And (Optionally) Fix Bad Frames
#John Meyer - December 13, 2016
#Rev. 2.0
#Thanks to Gavino and StainlessS for making the script more professional.

#This script detects single bad frames. 

#You can configure the script to write, to a file, the frame numbers of all frames which are detected as "bad".
#You can also configure it to automatically replace each bad frame with a new
#frame interpolated from its neighbors. 

#This script will fail if the bad frame happens immediately before or after a scene change.
#This script will also fail to find a bad frame if there is more than one bad frame in a row.

#It works very well for finding both blank frames and also "flash" frames (like those caused
#by a photographer's flash). It will also find single frames which have lots of 
#static or pixels. It can also find a frame with large x or y displacement from adjacent frames, like
#a film frame that wasn't properly registered in the film gate, or an analog
#video frame that lost vertical sync.

#When using VirtualDub, to create the text file containing the bad frame numbers,
#select "Run Video Analysis Pass" in the VirtualDub File menu. If you are simultaneously
#creating a fixed video file, you don't need to do this because the file will
#be created simultaneously as the fixed video file is created.

#The script uses ratios of the metrics for the current frame to the same metrics 
#on the two adjacent frames. Under normal circumstances, the metrics should be quite
#similar, and therefore the ratio should be very near to unity (i.e., 1.00). 

#Run through the video with the "METRICS" variable set to "True" and look at the metrics
#in order to determine an optimum threshold value. A larger threshhold will
#catch fewer bad frames, and a lower threshold will eventually create false positives. 
#The replacement code works well enough that if you end up replacing a few frames that are
#actually good, you probably won't notice it.


#-----------------------------
loadplugin("C:\Program Files\AviSynth 2.5\plugins\MVTools\mvtools2.dll")

#Control script operation by changing the following values :
#=====================================================================
VideoFile           = "E:\fs.avi"
global badthreshold = 9              # Set METRICS=TRUE to determine best value
METRICS             = FALSE          # TRUE will show Metrics ONLY (i.e., TRUE overrides all other selctions)
SHOWDOT             = FALSE          # TRUE will add "***" to each replacment frame (for troubleshooting)
REPLACE             = TRUE           # TRUE will replace each bad frame with a one that is interpolated from adjacent frames
FILEWRITE           = FALSE           # TRUE will create a file which contains the frame numbers of all bad frames
filename            = "E:\Bad.txt"   # Set to name and location where you want the frame numbers stored
#=====================================================================

source = AVISource(VideoFile).convertTOYV12().killaudio().assumeTFF()

script = """Subtitle("\nPrevious Ratio = " + String( YDifferenceFromPrevious(source) /
         \ Max(YDifferenceFromPrevious( selectevery(source, 1, -1)),0.00001) ) + 
         \ "\nNext Ratio        = " + String( YDifferenceToNext(source) /
         \ Max(YDifferenceToNext(selectevery(source, 1, 1)),0.00001)), lsp=0)"""
MetClip = Scriptclip(source, script)

FileFixed = (FILEWRITE)
    \ ? WriteFileIf(source, filename, "
    \ YDifferenceFromPrevious() / Max(YDifferenceFromPrevious( selectevery(1, -1)),0.00001) 
    \ > badthreshold && YDifferenceToNext() / Max(YDifferenceToNext(selectevery(1, 1)),0.00001)
    \ > badthreshold", "current_frame", append = false) : Source

output = (METRICS) ? MetClip : (REPLACE) ? ReplaceBadI(FileFixed,showdot) : FileFixed
return output

#------------------------------

function ReplaceBadI (clip c, bool SHOWDOT)
{
  even        = c.SeparateFields().SelectEven()
  super_even  = SHOWDOT ? even.subtitle("***").MSuper(pel=2) : even.MSuper(pel=2) 
  vfe         = manalyse(super_even,truemotion=true,isb=false,delta=2)
  vbe         = manalyse(super_even,truemotion=true,isb=true,delta=2)
  filldrops_e = mflowinter(even,super_even,vbe,vfe,time=50)

  odd         = c.SeparateFields().SelectOdd()
  super_odd   = SHOWDOT ? odd.subtitle("***").MSuper(pel=2) : odd.MSuper(pel=2) 
  vfo         = manalyse(super_odd,truemotion=true,isb=false,delta=2)
  vbo         = manalyse(super_odd,truemotion=true,isb=true,delta=2)
  filldrops_o = mflowinter(odd,super_odd,vbo,vfo,time=50)

  Interleave(filldrops_e,filldrops_o)
  Replacement = Weave()

  fixed       = ConditionalSelect(c, "
                Prev  = YDifferenceFromPrevious()
                Prev1 = Max(YDifferenceFromPrevious(SelectEvery(1,-1)),0.00001)
                Next  = YDifferenceToNext()
                Next1 = Max(YDifferenceToNext(SelectEvery(1,1)),0.00001)
                Prev/Prev1 < badthreshold && Next/Next1 < badthreshold ? 0 : 1", \
                c, selectevery(Replacement,1,-1))

  return fixed
}

[DerBraune]

Would you know how to get it to use interlaced?

when using x264 you also need to set the x264opts accordingly
see: https://ffmpeg.org/pipermail/ffmpeg-user/2016-April/031624.html

I'm very sorry, but I am unsure what exactly to add or change. Would you be able to help?

Your capture has LOTS of additional problems beyond black frames, most notably the flagging at the top of the screen. A TBC used during capture can usually fix that.
The following script will replace all black frames with motion estimated versions.

You are right, but no worries about the flagging. The black frames is the only thing I would like to have removed.

Thank you very much for the script! Unfortunately, since it's anime, the frames could change a lot from one to the other, so I think interpolation is not the best solution. Copying the previous or next frame would give better results I think.

[manono]

Would you know how to get it to use interlaced?

It's not interlaced. Some idiot already blend-deinterlaced it. It's aliased, yes, but not interlaced. And the blends can be removed to return it to its original framerate of 23.976fps:

SRestore(omode="pp3")
TDecimate(Mode=1)

And making it 23.976fps again should be done before the black frame removal.

Why anyone would want to work with this junk is beyond me, but to do it you'll need a working knowledge of AviSynth.

[johnmeyer]

Thank you very much for the script! Unfortunately, since it's anime, the frames could change a lot from one to the other, so I think interpolation is not the best solution. Copying the previous or next frame would give better results I think.

The replacement worked perfectly and, if you use manono's SRestore script first, the replacements should be even better.

However, if you simply want to replace the black frames with a duplicate of the previous frame (I understand your logic about anime), the following should work. I didn't test it, but all I did was replace the motion estimation with a call to use the previous field.

function ReplaceBadI (clip c, bool SHOWDOT)
{
  even        = trim(c,1,0).SeparateFields().SelectEven()

  odd         = trim(c,1,0).SeparateFields().SelectOdd()

  Interleave(even,odd)
  Replacement = Weave()

  fixed       = ConditionalSelect(c, "
                Prev  = YDifferenceFromPrevious()
                Prev1 = Max(YDifferenceFromPrevious(SelectEvery(1,-1)),0.00001)
                Next  = YDifferenceToNext()
                Next1 = Max(YDifferenceToNext(SelectEvery(1,1)),0.00001)
                Prev/Prev1 < badthreshold && Next/Next1 < badthreshold ? 0 : 1", \
                c, selectevery(Replacement,1,-1))

  return fixed
}

[jagabo]

That video has a further problem where the top and bottom fields are swapped in some frames.

[DerBraune]

It's not interlaced. Some idiot already blend-deinterlaced it. It's aliased, yes, but not interlaced. And the blends can be removed to return it to its original framerate of 23.976fps:

SRestore(omode="pp3")
TDecimate(Mode=1)

And making it 23.976fps again should be done before the black frame removal.

Why anyone would want to work with this junk is beyond me, but to do it you'll need a working knowledge of AviSynth.

Yea I mean I am weird sometimes.
Sometimes I just think of something and then I'll have to find a way to make it work. That's just me. Don't judge.
Unfortunately I have to say I have never used AviSynth before.
I'll have to take a look into it, but unfortunately I also do not really have any programming knowledge, that could guide me.

Why do you think it was 23.976fps and then blend deinterlaced?
MediaInfo tells me it is interlaced right now. Is it reading it wrong? Or was it maybe interlaced again at 29.97 fps?
Sorry, but I do not know enough about it to follow your thoughts, but I would like to learn and understand, if you would like to explain.

The replacement worked perfectly and, if you use manono's SRestore script first, the replacements should be even better.

However, if you simply want to replace the black frames with a duplicate of the previous frame (I understand your logic about anime), the following should work. I didn't test it, but all I did was replace the motion estimation with a call to use the previous field.

Thank you very much! I'll first have to take a look into AviSynth and hope I understand it enough to do this haha.

That video has a further problem where the top and bottom fields are swapped in some frames.

That sounds interesting. Would you have an example of that?

[jagabo]

After looking at the video some more... I don't think manono is right about field blending. It's basically a film source but some of the effects were added after the film was telecined* -- resulting in 30p and 60p effects. Then the video tape was played on a crappy VHS deck and captured by a device that caused the black frames and field swaps when it lost vertical sync.

As an example of the field swap problem here's a normal frame, #302:

[Attachment 55291 - Click to enlarge]

And two frames later (there's a black frame between them) a field swapped frame:

[Attachment 55292 - Click to enlarge]

Those artifacts aren't from motion, they are because the two fields have swapped postistion. Applying AviSynths SwapFields() filter fixes that frame producing a normal image like the first.

But you can't just use SwapFields() on the entire video. It fix the bad frames but all the originally good frames will now suffer from the problem. I worked out a way to detect which frames needed SwapFields() and fixes only those frames:

Mpeg2Source("black frames example no reencode.d2v", Info=3) 

v1 = TFM(pp=0)
v2 = SwapFields().TFM(pp=0)
testclip = Subtract(v1.mt_edge(thy1=30, thy2=30, mode="0 -1 0 0 2 0 0 -1 0", chroma="-128"), v2.mt_edge(thy1=30, thy2=30, mode="0 -1 0 0 2 0 0 -1 0",chroma="-128"))

ConditionalFilter(testclip, v1, v2, "AverageLuma(testclip)", "lessthan", "126", show=false)

Since TFM() was used with the pp=0 option there are still combed frames where effects were added at 60 fields per second. You can retain those by encoding as 30i or 60p (QTGMC) or lose some of the motion smoothness in the effects and encode at 24p. Attached is a 60p version created by adding QTGMC(preset="fast") at the end of the above script.

I didn't address the problems like cropping, levels/color adjustments, fixing the black frames, etc. But those need to be fixed too.

* The mix of 24p, 30p and 30i is confirmed by what I read about Blu-ray release where they went back to the original film and recreated the effects digitally.

[DerBraune]

Wow, very interesting! Yes, that sounds logical and the fixed result looks great! Thank you so much for looking into that so much. I will have to figure out avisynth and try this tomorrow.

[DerBraune]

Hi again! Okay so I've looked into it, but unfortunately I've come to a halt and I'm unable to proceed. It would be awesome if someone could help me.

I've used the following script with the central part from jagabos last post:

LoadPlugin("D:\...\dgmpgdec158\DGDecode.dll")
LoadPlugin("D:\...\TIVTC\TIVTC.dll")

video = MPEG2Source("H:\...\black frames example no reencode.d2v", Info=3)

v1 = TFM(pp=0)
v2 = SwapFields().TFM(pp=0)
testclip = Subtract(v1.mt_edge(thy1=30, thy2=30, mode="0 -1 0 0 2 0 0 -1 0", chroma="-128"), v2.mt_edge(thy1=30, thy2=30, mode="0 -1 0 0 2 0 0 -1 0",chroma="-128"))

ConditionalFilter(testclip, v1, v2, "AverageLuma(testclip)", "lessthan", "126", show=false)

return video

But all I get is "Invalid arguments to function TFM". The error occures on the line with "v1 = TFM(pp=0)". Does someone know what to do? Do I maybe need to load another dll first? Is more information needed to help?

I'm using avisynth 2.6 and AvsPmod to edit the script.

I'm sorry if this may be an incredibly beginner question.

[jagabo]

You have explicitly named the output of Mpeg2Source() "video". Then in the TFM lines you relying on the implicit name "last"

[DerBraune]

Awesome! Changing that fixed it. After that I saw the masktools2.dll was also missing, but now it works.

Thanks so much for the quick response!