How to take a snapshot pic when motion is detected on FFMPEG?

[rajhlinux]

Hello,

I am using FFMPEG to transcode a live USB webcam, how can I make it detect motion and take snapshots?

This command is able to show the motion vector paths when objects are in motion, this only works for pre-recorded videos, not for live streaming.:

ffmpeg -i output.mp4 -vf "mestimate=epzs:mb_size=16:search_param=7, codecview=mv=pf+bf+bb" -c:v libx265 output.mp4 -y

Not sure how to use "mestimate" filter on "ffmpeg" to make use of the motion vector data so that I can set some threshold value on detecting motion in which it can take snapshots.

39.157 "mestimate":
https://ffmpeg.org/ffmpeg-all.html#mestimate

The webcam stream is MJPG and being transcoded to HEVC video. Using GPU acceleration.

Thanks for any advice.

[jagabo]

mabye scdet with a low threshold and sc_pass:
https://ffmpeg.org/ffmpeg-filters.html#scdet-1

[rajhlinux]

mabye scdet with a low threshold and sc_pass:
https://ffmpeg.org/ffmpeg-filters.html#scdet-1

Hello, thanks for your reply.

Seems like I can use this, but I have no idea how to write the syntax for it.

There are no examples of "scdet"

Thanks.

[jagabo]

Using scdet:

md pic
ffmpeg -i input.mp4 -vf scdet=threshold=20:sc_pass=1 -vsync vfr pics\pic%%05d.jpg

The output images will go in a sub folder named pic. Adjust the threshold value as necessary. Bigger values will give fewer frames.

You can also use the select filter:

md pic
ffmpeg -i input.mp4 -f image2 -vf "select=gt(scene\,.5)" -vsync vfr pics\pic%%05d.jpg

Adjust the scene value (.5) as necessary. Bigger values will give fewer frames.

[rajhlinux]

Hello jagabo, thanks for your reply.

Super Helpful and will try your example.

I also did some reading on FFMPEG filters and saw that they have "minterpolate" filter which is based on motion interpolation:
https://ffmpeg.org/ffmpeg-filters.html#minterpolate

Whats interesting about "minterpolate" is that it has the option of "scd" (Scene change detection), which uses motion vectors for it's detection algorithm.

How can I use minterpolate's "scd" parameter to test a condition in which it has detected a great threshold motion from "scd"?

Thank You for your help.

[jagabo]

I don't think the scene change detection of minteroplate will help you. It's there to prevent the filter from trying to interpolate motion across scene changes. It appears to use a similar algorithm as scdet and select. And doesn't seem to have any way of exporting the detection.

[october262]

Hello,

I am using FFMPEG to transcode a live USB webcam, how can I make it detect motion and take snapshots?

This command is able to show the motion vector paths when objects are in motion:

ffplay output.mp4 -vf mestimate=epzs:mb_size=16:search_param=7,codecview=mv=pf+bf+bb

But I am not able to use it on "ffmpeg" to make use of the motion vector data from "mestimate" filter so that I can set some threshold value on detecting motion in which it can take snapshots. Also it only works with "ffplay".

The webcam stream is MJPG and being transcoded to HEVC video. Using GPU acceleration.

Thanks for any advice.

try this here - https://www.cloudacm.com/?p=3423

[rajhlinux]

try this here - https://www.cloudacm.com/?p=3423

Hello, thanks for the reply.

I have tried it, not great because it is either too sensitive and false triggering snapshots or not detecting anything at all when playing with the threshold values.

Its like hit or miss.

I did some research on this and proper motion detection uses "Motion Vectors / Motion Estimation" algorithm, it is used by OpenCV as well. FFMPEG has it (mestimate) but no documentation on it at all other than just the option and parameter syntax with no examples.

[rajhlinux]

I don't think the scene change detection of minteroplate will help you. It's there to prevent the filter from trying to interpolate motion across scene changes. It appears to use a similar algorithm as scdet and select. And doesn't seem to have any way of exporting the detection.

I guess you're right. I'm not fluent with FFMPEG, I do know they have conditions and Expression Evaluation usages to test filters or what not:
https://ffmpeg.org/ffmpeg-utils.html#Expression-Evaluation

But little documentation on it for examples. I assume it is not compatible with "minteroplate" as you suggest.

I strongly believe that the "select" filter does not use motion vector algorithm, since the algorithm is really basic and can not distinguish movement of objects, it is more based on video parameters, like change of pixel color or what not.

Thanks for your help with "scdet". I will do further testing on this.

[jagabo]

Yes, of course, true motion detection requires the calculation of motion vectors. Not knowing the nature of your video, and not knowing how to extract motion vectors in ffmpeg, I was hoping simple scene detection (pixel-by-pixel subtraction) was sufficient for you.

I have some idea how to do this in AviSynth. The mvtools filter has the ability to overlay motion vectors onto the video. You can then subtract the original from the motion vector image to get only the motion vectors:


[Attachment 66269 - Click to enlarge]

That's th eoriginal image on the left, the motion vectors on the right. The butterfly is moving across the frame. You could then output only the frames where the average brightness of the motion vector frame exceeds some threshold.

I suspect you can do something like this in ffmpeg but I don't know how. I did find a way to show motion vectors from an encoded video:

https://www.youtube.com/watch?v=HZF8JX5UYD8

ffmpeg -v error -flags2 +export_mvs -i input.mov -vf codecview=mv=pf+bf+bb -y mv.mp4

But I don't think that won't work with all i-frame codecs like mjpeg.

[rajhlinux]

Hello jagabo, thanks for your reply. I will look into "AviSynth" they seem to have it for linux.