Repair missing pixels in video?

[TravelRock]

Hi,

On a trip to the South Pacific in 2002 I had a DV-camcorder that had a problem with really high humidity and because of this a lot of my videos from the trip have missing pixels (see screenshoots). Any chance theres is any software that would be able to repair this? Perhaps with help of AI?


[Attachment 56933 - Click to enlarge]

[Attachment 56934 - Click to enlarge]

The video is filmed in PAL 576i.

Would be really great to finally be able to repair these. Thanks for any suggestion and help!

[DB83]

Upload a short sample of the original (not a youtube re-encode) There are some avisynth geniuses here (not me I'm afraid)


(You will never recover what was behind the blocks but you could be able to replace them with what is around them)

[johnmeyer]

It is possible that the tape is still mostly OK, but has stretched because of the heat/humidity on your trip. I have seen this sort of thing clear up when you do these two things:

1. Use a DV cleaning tape. Sony makes them, so I'm pretty sure they are safe. Just make sure not to exceed the recommended cleaning time (about 10 seconds, I think).

2. Fast forward the tape to the end and then rewind.

After doing these two things, try capturing the problem sections again.

As for repairing the pixels, the problem is not the repair (which can be done), but the identification. These bad pixels will come and go, and won't appear in the same place. Therefore, you can't just create a mask and apply a repair, as I did here:

https://www.youtube.com/watch?v=Z12TutFSg8c

[Cornucopia]

One thing that might also help: these kinds of errors are often random and exist in different spots from one frame to the next, so it should be possible to use temporal denoisers that borrow from succeeding/preceding frames in those same spots to fill in the errors (and little else). Use of pixel differencing thresholds can narrow the region of interest.

Scott

[johnmeyer]

One thing that might also help: these kinds of errors are often random and exist in different spots from one frame to the next, so it should be possible to use temporal denoisers that borrow from succeeding/preceding frames in those same spots to fill in the errors (and little else). Use of pixel differencing thresholds can narrow the region of interest.

Scott

The OP didn't mention the persistence from one frame to the next, but when I've seen this on DV tapes, the pixelization often last for many seconds, and is not just a single frame.

For single frame problems, you can replace the entire frame with a motion interpolated frame, as I did in this fix:

https://www.youtube.com/watch?v=uzMFodrGHDs

[Cornucopia]

I've seen both. And, why replace an entire frame?


Scott

[TravelRock]

Hi,

Thanks all for your suggestions!

@DB83 I've attached a short clip here to show what it looks like, and if someone wants to try out something.

@johnmeyer I've tried similar actions numerous times with forward/rewind and cleaning and I've not been able to get better results unfortunately.

@johnmeyer & @Cornucopia You're right about the lost pixels come and go and in various places of the frame. See the attached.

I was kind of hoping some kind of smart AI software could look a few frames back and forward and based on that calculate/guess what should be in the missing pixels. But I do understand it's quite a challenging task and perhaps there's no software doing this.

[johnmeyer]

I've seen both. And, why replace an entire frame?


Scott

Because it's easy and, as you can see in that video I linked to, the result can be perfect. I do this all the time for things like a photographer's flash. You can make it disappear completely by using the adjacent frames to interpolate a result.

[TravelRock]

@Cornucopia - How would I go about trying out the suggestion of "temporal denoisers that borrow from succeeding/preceding frames" and "Use of pixel differencing thresholds can narrow the region of interest"? And do you still think it would be able to fix the problem if looking at the vid clip I attached above?
I'm on Mac and got FFMPEG running as well as Adobe software. But perhaps some other software would be needed?

Thanks!

[Cornucopia]

What I was thinking was more AVISynth, aka script-based. ffmpeg doesn't have that level of manipulation (neither does Adobe stuff). Not sure how much option you've got on a Mac.

Noise is noise, and there are a whole palette of tools to work with denoising, depending on the character of the the noise. That particular noise has some definite characteristics: Certain horizontal macroblocks appear to be delayed in time compared to other macroblocks, within those macroblocks there are other blocks that are further delayed, there is very little of the image that is WITHOUT signal (it is there, just at the wrong place/time), and it is (obviously) blocky. These characteristics should guide you toward your choice of method.

johnmeyer may have a simpler solution, as doing WHOLE frame replacement is much easier than doing segments of frames. It all depends on whether you end up getting a smoother, more natural (restored?) version as a result, and depends on how much effort you are willing to put into getting there.
Also, if the issue is persistent enough, you may not have enough surrounding "clean" frames with which to borrow from, making it almost mandatory to do more region/segment-based manipulation.

Scott

[johns0]

Thread moved to the mac forum where you can get more help.

[TravelRock]

@johnmeyer How would I go about trying out your suggestion of "replace the entire frame with a motion interpolated frame"? What software or what kind of actions/process would I need to use?

Thanks!

[johnmeyer]

I didn't realize the OP was using a Mac. The software I use is AVISynth, but that is PC only.

[TravelRock]

@johnmeyer I didn't state I was primarily on Mac since I might be able to get a hold of and run the "repair" on a PC.
Is AVISynth script based as FFMPEG?
And if so, could you perhaps suggest something?

Thanks!

[johnmeyer]

Here is a script I developed several years ago that does exactly what I described: it automatically detects a single bad frame and then automatically replaces it with a frame synthesized from the adjacent frames. For some clips, it can work perfectly.

The discussion of the script begins here:

Finding individual "bad" frames in video; save frame number; or repair

and here is a link directly to the script:

https://forum.doom9.org/showthread.php?p=1789584#post1789584

If you have multiple frames in a row which are bad, this script will not work automatically.