How do I do an image-search within a video file?

[eranos]

How do I do an image-search within a video file? I want to find the exact moment in the video where a specific frame appears, which I have as a jpeg.
Ideally would be a way to do this within Adobe Premiere but I couldn't find any mention of this ability online.

[DB83]

Not possible.

Bit for bit the jpeg is different to a video. And both will also vary according to compression.

[eranos]

Hi DB83, thanks for the reply.
This is unfortunate. I understand that the frame in the video and the image in the jpeg are different in their encoding, but they look practically identical.
I would imagine that some sort of image-search could work if they're visually similar enough, but I guess no one created such a tool yet.
I would think that it would work somewhat like Premiere's Synchronize option, which can synchronize similar audios even if they were recorded with different mics or use different types of audio formats.

[smrpix]

Its a numbers game. That's why audio has always preceded video when it comes to technical tricks.

To match a piece of audio you are comparing a pattern of 48,000 bits of information per second. In typical 1080 video it's over 63,000,000 per second, almost 4,000,000,000 per minute or nearly 224,000,000,000 per hour.

Someone will figure out how to do it, but it's not there yet.

[eranos]

Interesting. Thanks!
I guess I'll have to keep doing it manually for now.

[smrpix]

Also note that in audio matching you're searching for certain patterns of peaks and valleys over time. Matching a still to a video is like trying to pick out a specific cymbal crash from a recording of an entire symphony. Yep. Manually is still the best option for now.

[jagabo]

You can do it with an AviSynth script. Load the video, the image, then compare frames with a runtime function and print out the frame number that matches (within a threshold). It's pretty simple if the jpeg image is the full frame (so you can compare the full video frame with the full image frame). It's much harder (and can't reasonable be done with AviSynth) if the jpeg image is just a small unknown crop of the video frame.

Example script:

function AbsDiff(clip v1, clip v2)
{
    # compute abs(v1-v2)
    Subtract(v1, v2)
    Overlay(last.ColorYUV(off_y=-126), last.Invert().ColorYUV(off_y=-130), mode="add")
}

v = LSmashVideoSource("video.mp4")
i = ImageSource("image.jpg", fps=v.framerate).ConvertToYV12()
diff = AbsDiff(v, i)

WriteFileIf(diff, "match.txt", "AverageLuma<3.0", "current_frame", append = false)

That subtracts the image from each frame of the video. If the video frame matches the image the result is a black frame (aside from a little noise because the jpeg compression is imperfect). On other frames the result is not black. If the average luma of the frame is less than 3.0 it prints the frame number to a text file, match.txt.

The frame I was searching for:

[Attachment 48908 - Click to enlarge]

A diff example of a non-matching frame:

[Attachment 48909 - Click to enlarge]

Diff of the matching frame is all black. I'm sure you can imagine what it looks like.

The contents of match.txt

367

Frame 367 of the video matched the image file. If you get to many matches you need to lower the threshold. If you don't get a match you need to raise the threshold (or maybe the image isn't in the video).

As noted earlier, this won't work if the image is only a portion of the frame. For example, this small image was cropped from the same frame:


[Attachment 48910 - Click to enlarge]

There is no way to know beforehand what portion of the frame to compare too (subtract from). It gets even more difficult if the cropped image is further distorted (brightness, contrast, rotated, scaled, etc.):


[Attachment 48911 - Click to enlarge]

It would be very difficult to match that image to the video frame. It would probably require some A.I. algorithm.

[eranos]

Thanks jagabo!
Your solution is interesting.
Unfortunately, since I don't know anything about Avisynth, let alone scripting in it, and since I'll need to calculate the timecode from the frame number one by one, I doubt it'll be faster for me than looking up the frame manually.
Hopefully, your solution will be helpful to someone looking into a similar problem in the future.
It might also help me if I run into a frame which I have no idea where in the movie it was taken from.
Thanks!

[pandy]

You will find some answers here (and in similar places):

https://www.learnopencv.com/image-recognition-and-object-detection-part1/

[eranos]

[Attachment 48915 - Click to enlarge]

[jagabo]

since I'll need to calculate the timecode from the frame number one by one, I doubt it'll be faster for me than looking up the frame manually.

You can modify the script to get the time rather than frame number. current_frame / framerate = time (seconds.decimal). I believe it's also possible to show in time format (hh:mm:ss.ms) or SMPTE format (hh:mm:ss:ff).