Multi-frame logical reference - a new video enhancement concept

[forensic]

Thank you for al lthe great feedback. Message thread is now complete and I understand the difficulties.

[Soopafresh]

With the low cost of Hi Def security cams, why not go the hardware route? Sounds like you're trying to turn a Volkswagen into a Porsche. Unfortunately, there's only so much information present in the source file with standard def media. You can try a 30 day of Video Enhancer, which uses Super Resolution upscaling technology, but I suspect you're only going to see bigger looking blobs: http://www.thedeemon.com/VideoEnhancer/

[forensic]

.

[Soopafresh]

I would e-mail Deemon, who wrote the software for VideoEnhancer. He's a nice guy.

[forensic]

Thanks

[guns1inger]

While the idea *might* be interesting as an intellectual challenge, or even for some video restoration work, it has little value for restoration for use as evidence as you are introducing too much information that simple was not present at the time. Especially with the idea of a library of known objects. A wheel is not a wheel, other than both are round and share general characteristics. If you need to identify a car in a video however, then you need to know what type of wheel it actually has, not what type of wheel was drawn from a library of wheels simply because it was round. If the source doesn't have enough detail for a trained viewer to make the call on a detail, then it doesn't have enough for a filter to make an intelligent object replacement.

I believe the first filter also has a few problems, not the least of which is the need for a quality still image taken from the same position as the original video. Also, how does it address moving cameras, instead of simple fixed placement cameras ? It also has to be able to address the fact that the world is a 3 dimensional place, and in many locations the moving objects will be moving behind objects closer to the camera. And while it might allow you to get a clearer background image and perhaps even slightly sharper edges around your moving image (which in most security footage would be the actual data in which you would be most interested), it wont be able to reconstruct the major details of, say, a face, or logo.

The real question that the technique raises in my mind is this : if, to make it work, I have to source a clean, clear version of the background to the footage, which is going to require using a good camera from the same physical location as the source footage, why not simple use a better camera there in the first place.

Bottom line - the data you really need no longer exists. It cannot be retrieved from the remaining digital image data you have. Substituting it with something else might allow you to get a better image, but it does not restore the source image or give you the original image data. In some cases that might be enough. In court it would not.

[AlanHK]

I am looking for comments about the viability of my enhancement theory.

I suggest asking this on Doom9

[filmboss80]

Feasible, perhaps; practical, no. Adequate lenses and ample video data storage are getting cheaper all the time. The end result will be the same for less cost. Enhancement filters can make videos look sharper, but cannot add true detail that was not recorded in the first place. Soopafresh made an effort to give you a very wise answer in his first post. Reject it if you want, but wisdom is still wisdom.

[forensic]

Thanks

[JohnnyMalaria]

In principle, the concept makes sense. However, there are a number of practical issues that would make it very difficult to implement. As someone who has written image analysis software for a number of unusual reasons, I would list these as the main issues:

1. Noise. By far the biggest menace. It exists in a number of forms - the inherent noise of the security camera's imaging which will be much higher than from your still reference image. Noise from the compression scheme used. The problem here is that as the content of the frame changes due to a moving object, regions across the whole of the frame will change - within and without the region of interest. Given the basic premise that you want to extract detail from temporally and spatially separated information from what should be the same part of a moving object, noise will render that pretty much impossible. Mathematically, it is very common to end up with an infinite number of possible solutions to noisy information when the noiseless case has a single, definitive solution.

2. Lighting. Compared to your low noise, high resolution reference objects, your object in your video will likely move across different areas of light and shade as well as casting shadows on the still background. This would challenge the edge processing.

3. Point-of-view. You would need a large number of reference images (such as the tires) to be able to accommodate the positioning of the object - e.g., square on, side on, rotating etc.

Considerable research has been done into the identification of noisy, poor resolution images. Intel have developed an open source computer vision (OpenCV) library with all kinds of useful functions including the detection of faces in a photograph, including the identification of a specific person against a reference. Unfortunately, such schemes require a reference image of the person of interest.

I recall watching a program within the last couple of years of a system in the UK developed by one of the local police forces that would compare poor quality images from security videos against a nationwide database of mug shots of known criminals. Given the reference images, probabilities of a match can be made. There is also a live surveillance system that monitors everyone and flags whether a suspect for a crime is present. The information is not stored except for the flagged individual(s). In this latter case, high quality video is used.

[forensic]

Thanks

[guns1inger]

While you wait for someone to write you a filter, you might want to have a look at this : http://www.giveawayoftheday.com/topaz-moment/

And more specifically, this feature : http://www.topazlabs.com/topazlabs/03products/topaz_moment/gallery/40acquire_night_vision/index.html

Perhaps this might help you in the meantime. They also have super resolution with noise reduction and detail enhancement video filters.