fixing damaged frames (as if the video was captured from a bad tape)

[sashakhina]

Hi! I have this video with artifacts as if it was captured from a bad tape. But it was not shot on tape. Is there any way to fix this problem except for cutting out these frames manually and replacing them with copies of the following frames? I tried some repairing software, it didn't help.

[poisondeathray]

Where is the video from, what format and codec?

Did you rule out a decoding or playback error?

What is the distribution of the defect and affected frames ? eg. Does it occur randomly, or every "x"th frame in a pattern? . Also where on the frame spatially (same location, or different, such as top, bottom etc..)

Is the photo reference overlay ever affected? The defect seems to skip it in the screenshot

Posting an unprocessed video sample would help.

[sashakhina]

It's an mp4 file, H.264 codec.
No errors while playback. I didn't try to decode it to some other format.
This video was shot on Blackmagic camera 2 years ago, then edited, and exported to mp4. I don't have the original now, just this exported version. I also have ten 2-hour parts of this video course and only this part has the issue.
So these "lines" appear randomly and all over the frame, not in a specific location. They appear every other 1-3 seconds.
I attached the video sample.
Thanks

[babygdav]

If the original footage that you saved to disk was pristine and did not have such artifacts, then it likely reflects a hard drive or RAM error.
RAM errors can result in data loss during read/writes, so when you were writing that file to disc, the ram errors showed up as such in the video.
Disc errors occur as the drive goes bad. When you read information off the drive, you get artifacts in videos, even lost files.

But without knowing the state of the original file when written to disc, what you have is what you have.
You can use VirtualDub, Photoshop, After Effects and other similar tools to copy the good parts of the frame into the bad parts of another frame.
Sorry, but generally a frame by frame manual task.

[aedipuss]

If the original footage that you saved to disk was pristine and did not have such artifacts, then it likely reflects a hard drive or RAM error.
RAM errors can result in data loss during read/writes, so when you were writing that file to disc, the ram errors showed up as such in the video.
Disc errors occur as the drive goes bad. When you read information off the drive, you get artifacts in videos, even lost files.

But without knowing the state of the original file when written to disc, what you have is what you have.
You can use VirtualDub, Photoshop, After Effects and other similar tools to copy the good parts of the frame into the bad parts of another frame.
Sorry, but generally a frame by frame manual task.

don't now where this newb has come from but again it's out in left field.

[abolibibelot]

Sorry, but generally a frame by frame manual task.

Bad frames can be fixed practically flawlessly by interpolation of adjacent frames if there's not too much motion and if there aren't too many bad frames in a row with either FrameSurgeon or Morpheus. I recently wrote a comparison of these two Avisynth functions plus another, Morph, which is generally not as good and not as stable but can be preferred for some tricky frames.
The manual work can be greatly accelerated using Sawbones. (All three tools written by StainleSS from Doom9 if I'm not mistaken.)

If the original footage that you saved to disk was pristine and did not have such artifacts, then it likely reflects a hard drive or RAM error.
RAM errors can result in data loss during read/writes, so when you were writing that file to disc, the ram errors showed up as such in the video.
Disc errors occur as the drive goes bad. When you read information off the drive, you get artifacts in videos, even lost files.

I have seen things you people wouldn't believe... Well, at least I have seen way weirder things happening for no such obvious reason. There are many, many moving parts when editing / rendering video, and I've learned the hard way not to trust anything that goes in or comes out, everything must be scrutinized and triple-checked.

[KarMa]

Seems like the line errors keep happening in the same location throughout the short sample. Maybe a hardware or overheating issue. I've just never seen a source (existing video file) that is known to be clean, be damaged in such a localized manner after processing or decoding-->encoding. I have seen these line errors make their way to broadcast/distribution for other companies, and I always assumed this type of error happened at the camera end.

[poisondeathray]

In general, the problem is 1) detection 2) what to do about it 3) how much time and/or how good of a job you want to do

Every 4,5th frame are duplicates. ie. It looks to be 23.976, not 29.97. You can decimate it right off the bat no matter which method you chose

Definitely the "best" quality, most selective method would be manual cloning / content aware fill, but it becomes tedious quickly. The static areas are easily done. There are tips and tricks to speed things up and you can prefilter and alpha paint from different layers



The frames with defects frequently occur with something occurring in the left and right borders. One way is to autodetect, according to those. ie. Cut out the middle, and use the L+R black strip as the detection layer. Whenever Y>16.1, replace with last frame , or some filtered frame . You could do this with conditionalfilter() in avisynth. The problem is ~1/20 defective frames didn't have a defect there (only centrally), so you miss some . You might be able to combine it with logic "OR" by detecting the top right area of the board, if the artists' hand never goes there to catch the missing frames

IMO, replacing an entire frame is overkill, not very selective. Also causes stutter. e.g if the defect only occurs on the static periphery, you are actually causing more damage by using a duplicate, when you could have just replaced the periphery for a essentially perfect repair



As KarMa pointed out, the "lines" are always in the same line height location when they occur. They also only occur 1 frame duration. This makes it at least partially "treatable" with some types of filters . But if this sample was not representative, it might not work or might need to be modified e.g. if line defects occur elsewhere

This is sort of an "inbetween" solution. A brute force method where every frame is passed through a dirt removal filter with strong settings. Normally that causes artifacts and collateral damage - but it's limited by a line mask to limit the damage. It's not "perfect"; there are some frames that have residual artifacts, but it might be "good enough" and it's "automatic." At least it reduces the amount of manual touch up work you have to do. eg. You might take this as one of the clone layer sources

(I modified the blocksize of the removedirtmc function, there might be other settings where you can get better results)

output video attached below

LsmashVideoSource("video sample.mp4")
TDecimate()
orig=last

m=imagesource("linemask.png", fps=orig.framerate, start=0, end=orig.framecount-1).converttoyv12(matrix="pc.709")

r=orig.removedirtmc_new(100,false)

overlay(orig, r, mask=m)




function RemoveDirt(clip input, int limit, bool _grey)
{
  clensed=input.Clense(grey=_grey, cache=4)
  alt=input.RemoveGrain(2)
  return RestoreMotionBlocks(clensed,input,alternative=alt,pthreshold=4,cthreshold=6, gmthreshold=40,dist=1,dmode=2,debug=false,noise=limit,noisy=12,grey=_grey)
}

function RemoveDirtMC_NEW(clip,int limit, bool "_grey")
{
_grey=default(_grey, false)
limit = default(limit,6)

prefiltered = RemoveGrain(clip,2)
superfilt = MSuper(prefiltered, hpad=32, vpad=32,pel=2)
super=MSuper(clip, hpad=32, vpad=32,pel=2)

bvec = MAnalyse(superfilt,isb=true, blksize=8, overlap=2,delta=1, truemotion=true)
fvec = MAnalyse(superfilt,isb=false, blksize=8, overlap=2,delta=1, truemotion=true)

bvec_re = Mrecalculate(super,bvec,blksize=4, overlap=0,thSAD=100)
fvec_re = Mrecalculate(super,fvec,blksize=4, overlap=0,thSAD=100)

backw = MFlow(clip,super,bvec_re)
forw = MFlow(clip,super,fvec_re)

clp=interleave(backw,clip,forw)
clp=clp.RemoveDirt(limit,_grey)
clp=clp.SelectEvery(3,1)
return clp
}

[johnmeyer]

I've posted this one before. I wrote a function which detects frames which differ significantly from its neighbors and then fixes that frame by replacing it with one that is derived from its neighbors by using motion compensation. If your artifact only persists for one frame, and if the detection logic in my AVISynth script can detect it, you will be able to remove all the offending frames.

Here's the link:

Finding individual "bad" frames in video; save frame number; or repair

[sashakhina]

Thank you all for trying to help!
poisondeathray, your output result is great!
johnmeyer, I saw your other post on this subject. I want to try it too. The problem is that I work on Mac and AVIsynth is for Windows only. So I'll ask somebody to let me use their comp.
I'll post the result then.
Thanks!

[babygdav]

Very smart solutions posted!

Anyways, https://www.macworld.co.uk/feature/mac-software/best-virtual-machine-software-3671133/?amp

Virtualbox or boot camp can get windows on your Mac free.
https://www.microsoft.com/en-us/software-download/windows10

Windows will say it's a trial, but functionality never expires.

[sashakhina]

I've posted this one before. I wrote a function which detects frames which differ significantly from its neighbors and then fixes that frame by replacing it with one that is derived from its neighbors by using motion compensation. If your artifact only persists for one frame, and if the detection logic in my AVISynth script can detect it, you will be able to remove all the offending frames.

Here's the link:

Finding individual "bad" frames in video; save frame number; or repair

johnmeyer or poisondeathray, maybe you could fix my 2-hour mp4 file (3 GB) with your scripts for $compensation? It would be just so easier for me to pay somebody than to solve this problem myself. I'm really not good with codes.

[aedipuss]

I've posted this one before. I wrote a function which detects frames which differ significantly from its neighbors and then fixes that frame by replacing it with one that is derived from its neighbors by using motion compensation. If your artifact only persists for one frame, and if the detection logic in my AVISynth script can detect it, you will be able to remove all the offending frames.

Here's the link:

Finding individual "bad" frames in video; save frame number; or repair

just a question. what type of avi file is it expecting as input?

[sashakhina]

It is mp4 (h.264)

[aedipuss]

It is mp4 (h.264)

sorry. my question was for johnmeyer about his script. not your video.

[abolibibelot]

just a question. what type of avi file is it expecting as input?

If it's an Avisynth script, any common video file can be processed, with the right source filter. Or did you mean something else, like the colorspace ?

[johnmeyer]

just a question. what type of avi file is it expecting as input?

Here's another link to the script:

https://forum.doom9.org/showthread.php?p=1789584#post1789584

As you can see, it uses AVISource, which expects an AVI file that can be decoded by your computer. You can replace that line with whatever other file opening function you want.

As for volunteering to fix the OPs video, it will be a little bit complicated for two reasons. First of all, as already pointed out, it is 23.976 video, not 29.97 and therefore must be telecined. This is usually just two lines of code, but since the description of how this video was created doesn't quite match what we see, I have to assume that there may be sections that have other problems.

The second issue is detection. The detection algorithm in my script works really well for more extreme bad frames, like a photographer's flash, but may not be sensitive enough to detect just those narrow lines. Therefore, I'd have to invent some new detection logic, which is more than I really want to do.

Finally, poisondeathray's idea of only replacing part of the frame because the artifact does seem limited to only certain scan lines makes lots of sense. However, my script would have to be altered to merge only a section of the motion-estimated correction frame.

[abolibibelot]

The second issue is detection. The detection algorithm in my script works really well for more extreme bad frames, like a photographer's flash, but may not be sensitive enough to detect just those narrow lines. Therefore, I'd have to invent some new detection logic, which is more than I really want to do.

Wouldn't it work by using a mask as suggested by “poisondeathray”, but for the detection ? Or cropping the picture so as to only analyse those two bands of pixels, before running the detection algorithm.

[johnmeyer]

The second issue is detection. The detection algorithm in my script works really well for more extreme bad frames, like a photographer's flash, but may not be sensitive enough to detect just those narrow lines. Therefore, I'd have to invent some new detection logic, which is more than I really want to do.

Wouldn't it work by using a mask as suggested by “poisondeathray”, but for the detection ? Or cropping the picture so as to only analyse those two bands of pixels, before running the detection algorithm.

Maybe, but if you look at how the YDifference functions work, you need a LOT of pixels to be different from their mates in the two adjacent frames in order for the metrics to spike, signalling a problem. In this video, the defect is only one scan line high and not the entire width of the frame, so even if you only look at 1/4 or 1/5 of the frame, via masking, most of the pixels are going to be identical and therefore the YDifference comparison won't be much different than it is when comparing two good frames.

I suspect that the solution to automatic detection would require a different approach, probably something which compares entire scan lines, rather than doing the YDifference on individual pixels.

[abolibibelot]

Maybe, but if you look at how the YDifference functions work, you need a LOT of pixels to be different from their mates in the two adjacent frames in order for the metrics to spike, signalling a problem. In this video, the defect is only one scan line high and not the entire width of the frame, so even if you only look at 1/4 or 1/5 of the frame, via masking, most of the pixels are going to be identical and therefore the YDifference comparison won't be much different than it is when comparing two good frames.

But if it's been established that only those two very small strips of pixels have problems on some frames, only those strips could be analysed, not 1/4 or 1/5 of the frame. Or do you think that it wouldn't work for some reason ?

[abolibibelot]

I tested the bad frame detection script with the tweak below, I have little experience with it, and not much with Avisynth in general (at least at this high level which is sufficiently advanced to be indistinguishable from magic !) ; so far it doesn't work so well, as the algorithm fails to specifically detect the frames with the defects (even though they become very conspicuous with this treatment), but it might give some ideas to someone somewhere... I couldn't get Avisynth to crop the video to a 2 pixels height, so as to isolate the defective strips, apparently the smallest crop possible is 6 pixels in height (with a height of 4 I get "Error loading Avisynth!"). Then I used StackVertical to copy those thin strips until it added up to the original 1080 height. There might be a better way of doing this.
With the first "source" line and badthreshold = 20, frames 72, 162, 212 are correctly identified as bad, but 66 too, although it is not affected. So far I don't quite understand what "badthreshold" refers to exactly, and how to adjust it based on the results of the "metrics" (but I played with it for about 10 minutes).
Also, there's the problem of duplicated frames (like 170-171) which by design (as I understand it) could not be properly identified with this method.

video = LSmashVideoSource("E:\video sample.mp4")

crop1 = video.crop(0,250,0,-824)
crop2 = video.crop(0,386,0,-688)
stack1 = StackVertical(crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1, crop1)
stack2 = StackVertical(crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2, crop2) # 6 x 18 x 10 = 1080 -- there may be a more elegant way of achieving this...
source = StackVertical(stack1, stack1, stack1, stack1, stack1, stack1, stack1, stack1, stack1, stack1)
#source = StackVertical(stack2, stack2, stack2, stack2, stack2, stack2, stack2, stack2, stack2, stack2) # enable one "source" line, to target the upper or lower defective strip

global badthreshold = 2              # Set METRICS=TRUE to determine best value
METRICS             = TRUE          # TRUE will show Metrics ONLY (i.e., TRUE overrides all other selctions)
SHOWDOT             = FALSE          # TRUE will add "***" to each replacment frame (for troubleshooting)
REPLACE             = FALSE           # TRUE will replace each bad frame with a one that is interpolated from adjacent frames
FILEWRITE           = TRUE           # TRUE will create a file which contains the frame numbers of all bad frames
filename            = "E:\video sample - bad.txt"   # Set to name and location where you want the frame numbers stored

script = """Subtitle("\nPrevious Ratio = " + String( YDifferenceFromPrevious(source) /
         \ Max(YDifferenceFromPrevious( selectevery(source, 1, -1)),0.00001) ) + 
         \ "\nNext Ratio        = " + String( YDifferenceToNext(source) /
         \ Max(YDifferenceToNext(selectevery(source, 1, 1)),0.00001)), lsp=0)"""
MetClip = Scriptclip(source, script)

FileFixed = (FILEWRITE)
    \ ? WriteFileIf(source, filename, "
    \ YDifferenceFromPrevious() / Max(YDifferenceFromPrevious( selectevery(1, -1)),0.00001) 
    \ > badthreshold && YDifferenceToNext() / Max(YDifferenceToNext(selectevery(1, 1)),0.00001)
    \ > badthreshold", "current_frame", append = false) : Source

output = (METRICS) ? MetClip : (REPLACE) ? ReplaceBadI(FileFixed,showdot) : FileFixed
return output

#------------------------------

function ReplaceBadI (clip c, bool SHOWDOT)
{
  even        = c.SeparateFields().SelectEven()
  super_even  = SHOWDOT ? even.subtitle("***").MSuper(pel=2) : even.MSuper(pel=2) 
  vfe         = manalyse(super_even,truemotion=true,isb=false,delta=2)
  vbe         = manalyse(super_even,truemotion=true,isb=true,delta=2)
  filldrops_e = mflowinter(even,super_even,vbe,vfe,time=50)

  odd         = c.SeparateFields().SelectOdd()
  super_odd   = SHOWDOT ? odd.subtitle("***").MSuper(pel=2) : odd.MSuper(pel=2) 
  vfo         = manalyse(super_odd,truemotion=true,isb=false,delta=2)
  vbo         = manalyse(super_odd,truemotion=true,isb=true,delta=2)
  filldrops_o = mflowinter(odd,super_odd,vbo,vfo,time=50)

  Interleave(filldrops_e,filldrops_o)
  Replacement = Weave()

  fixed       = ConditionalSelect(c, "
                Prev  = YDifferenceFromPrevious()
                Prev1 = Max(YDifferenceFromPrevious(SelectEvery(1,-1)),0.00001)
                Next  = YDifferenceToNext()
                Next1 = Max(YDifferenceToNext(SelectEvery(1,1)),0.00001)
                Prev/Prev1 < badthreshold && Next/Next1 < badthreshold ? 0 : 1", \
                c, selectevery(Replacement,1,-1))

  return fixed
}