Is it possible to remove fly from video?

[Bencuri]

I shot a video outdoors, unfortunately they flies were not sleeping, there were a few that found the field of view of my camera. Fortunately most flew in a path that are possible to cover creating masks. But there was one that flew in front of me while I was in motion. Would it be possible to remove this somehow from the video, too? Maybe with an AI tool or something? I have Sony Vegas, but I can obtain other things, too.

The problematic video segment is attached.

There is one condition that is a neccessary aspect to care about: the fames in the video should remain as is. The frame order and the number of frames should not be changed, because this is a tutorial video, and the audio and video should be in the original, perfect sync.

[poisondeathray]

You can use dirt removal and/or interpolation between clean frames with animated masks to limit the repair +/- minor clean up with photoshop/after effects if required. Another way is to motion track textures/areas from clean frames/areas into target frame(s). This falls under the umbrella term of "compositing" . There is also a bird in the BG sky near the tree below the guitar between frames 32-38 , not sure if you wanted to remove

[Bencuri]

You can use dirt removal and/or interpolation between clean frames with animated masks to limit the repair +/- minor clean up with photoshop/after effects if required. Another way is to motion track textures/areas from clean frames/areas into target frame(s). This falls under the umbrella term of "compositing" . There is also a bird in the BG sky near the tree below the guitar between frames 32-38 , not sure if you wanted to remove

What you write about is not familiar to me, it is too PRO, so to say, I won't be able to do that. I figured out a way myself however: with pan crop in Vegas, go frame by frame (the fly is on 14 frames or so), select the fly with the mask pen tool on each frame, and shift the previous fame each time to that mask area. This way the masked area will be clean (the fly is never at the same place on adjacent frames), and the motions in that small area are not too different this way.

However mine also takes time, your method seems to be much more efficient. May I ask you to take this attachment below, it is similarly cut, but in 24mbps, and redo the correction on it? This way I will have a good quality version that I can insert into the video once I have finished editing. Please render it in 24mbps too. I don't need the bird to be removed, only the fly.

[poisondeathray]

What you write about is not familiar to me, it is too PRO, so to say, I won't be able to do that. I figured out a way myself however: with pan crop in Vegas, go frame by frame (the fly is on 14 frames or so), select the fly with the mask pen tool on each frame, and shift the previous fame each time to that mask area. This way the masked area will be clean (the fly is never at the same place on adjacent frames), and the motions in that small area are not too different this way.

However mine also takes time, your method seems to be much more efficient.

Less "efficient" in terms of time, but usually higher quality in terms of matching the scene or filling the mask area . It's basically the same thing as you are doing in terms of masking part, but not just using a simple frame offset

Here are the 16 frames at qp1 I-frame (near lossless), I preserved the overbrights this time (you have Y values >235 in the source for this outdoor scene)

[Bencuri]

Thank you!

[Bencuri]

You can use dirt removal and/or interpolation between clean frames with animated masks to limit the repair +/- minor clean up with photoshop/after effects if required. Another way is to motion track textures/areas from clean frames/areas into target frame(s). This falls under the umbrella term of "compositing" . There is also a bird in the BG sky near the tree below the guitar between frames 32-38 , not sure if you wanted to remove

I need to do a similar fly removal again, can you describe where you ran your cleanup process? Did you do it in avisynth exclusively, or used some other tool as well?

[poisondeathray]

You can use dirt removal and/or interpolation between clean frames with animated masks to limit the repair +/- minor clean up with photoshop/after effects if required. Another way is to motion track textures/areas from clean frames/areas into target frame(s). This falls under the umbrella term of "compositing" . There is also a bird in the BG sky near the tree below the guitar between frames 32-38 , not sure if you wanted to remove

I need to do a similar fly removal again, can you describe where you ran your cleanup process? Did you do it in avisynth exclusively, or used some other tool as well?

Other compositing tools like after effects - because avisynth is the wrong tool for the rotoscoping / animated masks part

[Bencuri]

Can you link a Youtube tutorial that shows the basics how to use these animated masks in After Effects, and somehow relates to my problem? I have never used that software yet.

[poisondeathray]

Can you link a Youtube tutorial that shows the basics how to use these animated masks in After Effects, and somehow relates to my problem? I have never used that software yet.

No tutorials come to mind immediately, but if you search there should be tutorials to do similar things for various software. The general category this would fall under is "object removal" , and the general techniques fall under "compositing"

The masking part should be similar in any editor - that's the most important step for any type of decent quality "object removal" workflow. Other keyword to search for would be "rotoscoping" tutorials.

The "fix" part could also be done with Content Aware Fill in AE, or even Propainter which has a vapoursynth port. You use the same mask as input and they remove/fill the area. InpaintDelogo in avisynth might be able to do it too


(These days, there are various machine learning ("AI") algorithms that can automask objects once you mark it in 1 frame - they fall under "segmentation" algorithms. This potentially saves lots of time and manual work. It would be similar to "rotobrush" in AE, but these days, they are much smarter and more advanced. However, a motion blurred fly "blob" probably wouldn't be detected automatically by most algorithms, (unless you train one yourself) because the borders are too indistinct, the object too unclear and object shape changes)

[Bencuri]

Thanks!

You mentioned back then, when this thread was opened, that my method is not time efficient, but can be very accurate, compared to the automated methods. If I consider bug removals through a bunch of frames only, does that statement prevail even today? Or since then, the fixes you mentioned have been developed further and it's no use wasting time with my (let's call it) manual method even in small scale corrections? I mentioned it earlier, above, that I tracked the fly with masks on each frame in a duplicated video segment, and then shifted it +1 frame in time, so that the mask would show a pre-fly state of the mask instead of the fly.

[poisondeathray]

Thanks!

You mentioned back then, when this thread was opened, that my method is not time efficient, but can be very accurate, compared to the automated methods. If I consider bug removals through a bunch of frames only, does that statement prevail even today? Or since then, the fixes you mentioned have been developed further and it's no use wasting time with my (let's call it) manual method even in small scale corrections? I mentioned it earlier, above, that I tracked the fly with masks on each frame in a duplicated video segment, and then shifted it +1 frame in time, so that the mask would show a pre-fly state of the mask instead of the fly.

It's up to you, and how much time you're willing to spend (not just fixing, but learning) vs. how well you want it fixed . If you think it's good enough with vegas and the offset fix, than use that

The repaired demarcation area (ie. the mask itself) can be accurate, because you're masking with the pen tool - that's the basic way masking is done in most editors.

To fix these types of problems properly, that mask accuracy is critical. Not much has changed in terms of a motion blurred fly (it's too blurry, object shape changes). If it was a car, a person, some well delineated object, then the mask could likely be mostly automatic with "AI" segmentation methods these days . That saves time vs. the pen tool

But the offset frames fix in vegas can be problematic and is not that accurate in the sense that fixed area can be very different. That clip has relatively small motion, but what happens when the fly goes over the hands ? That's an important part of this video - If you shift/offset frames, it might not look great. In general, think of larger motions, cars driving by, people walking etc... - distortions over fast moving areas will be more noticable

[Bencuri]

"But the offset frames fix in vegas can be problematic and is not that accurate in the sense that fixed area can be very different. That clip has relatively small motion, but what happens when the fly goes over the hands ? That's an important part of this video - If you shift/offset frames, it might not look great. In general, think of larger motions, cars driving by, people walking etc... - distortions over fast moving areas will be more noticable"

Yes, I am aware of that. I am only asking about situations like the one before, when there is a proper frame segment available to replace the fly. Luckily it is possible to do the same workflow in the recent video, but this time I want to look into the methods that you referred to earlier as well, to see how the workflow goes in that case.