VideoHelp Forum
+ Reply to Thread
Results 1 to 14 of 14
Thread
  1. Member
    Join Date
    Oct 2021
    Location
    Wheeling WV USA
    Search Comp PM
    i try to regularly convert .mp4 videos i get into .webm format as the latter is substantially smaller. i have scripted this to manage a queue of videos to convert using a pool of 3 instances of ffmpeg. sometimes i want stop everything. i can't get ffmpeg to stop and resume from a new process. this is often due to a need to reboot. if i kill ffmpeg my scripts just put the video back in the queue after setting the global stop flag. this in Xubuntu Linux 18.04.6 LTS.

    i'm looking for a way to force ffmpeg to stop with information stored in its output video that would allow it to gracefully resume. has anyone heard of such a thing, like maybe an add-on or source patch? one option is to run it in a virtual machine which can save process state and resume without ffmpeg ever knowing. unfortunately, for complex reasons, a VM is not an option for me.
    Quote Quote  
  2. Member Cornucopia's Avatar
    Join Date
    Oct 2001
    Location
    Deep in the Heart of Texas
    Search PM
    Probably best to change your scripts so they log each file as it is completed, then check for interrupt/break, then continue to next file. That way, you aren't directly interrupting ffmpeg, but the script command processor, which expects occasional interruptions.
    And have it start by looking at list and the log, and seeing which file was the last one completed, and pick up immediately after. If none are complete, it starts at the beginning of the list. If all are complete, it aborts.


    Scott
    Quote Quote  
  3. Originally Posted by Skaperen View Post
    i try to regularly convert .mp4 videos i get into .webm format as the latter is substantially smaller. i have scripted this to manage a queue of videos to convert using a pool of 3 instances of ffmpeg. sometimes i want stop everything. i can't get ffmpeg to stop and resume from a new process. this is often due to a need to reboot. if i kill ffmpeg my scripts just put the video back in the queue after setting the global stop flag. this in Xubuntu Linux 18.04.6 LTS.

    i'm looking for a way to force ffmpeg to stop with information stored in its output video that would allow it to gracefully resume. has anyone heard of such a thing, like maybe an add-on or source patch? one option is to run it in a virtual machine which can save process state and resume without ffmpeg ever knowing. unfortunately, for complex reasons, a VM is not an option for me.
    Ctrl+Z pauses it.

    Or you could get the PID with pgrep ffmpeg then use kill -s SIGSTOP <PID> to suspend.

    Then resume with fg command or kill -s SIGCONT <PID>
    Last edited by codehound; 19th Oct 2021 at 14:51.
    Quote Quote  
  4. @codehound: How would you restart those after a reboot? That just sounds wrong.
    users currently on my ignore list: deadrats, Stears555
    Quote Quote  
  5. Originally Posted by Selur View Post
    @codehound: How would you restart those after a reboot? That just sounds wrong.
    omg, i have overseen that requirement too ^^

    The direct solution would be something as they describe here https://unix.stackexchange.com/questions/43854/save-entire-process-for-continuation-after-reboot
    But it is an obscure way, sounds pretty intrusive and unsecure.


    It would be really a lot of work to kill and "resume" the process by script, especially if we produce a single output file...
    What i can imagine that could work would be something like first encode to segments, e.g. 100 frame segments and only when all frames are encoded, remux using concat filter and codec copy.
    In that case, before a reboot, one would need to send ctrl+c to the running ffmpeg instances and after the reboot one would need do check if there are unfinished encodes by comparing the duration of the source files and the sum of the duration of encoded segments.
    I would then delete the last, unfinished segment and start my "resuming" ffmpeg process with a corresponding seek (-ss).

    All this would require the input files to be frame accurately seekable by ffmpeg.
    Last edited by emcodem; 23rd Oct 2021 at 07:41.
    Quote Quote  
  6. Member
    Join Date
    Oct 2021
    Location
    Wheeling WV USA
    Search Comp PM
    Originally Posted by Cornucopia View Post
    Probably best to change your scripts so they log each file as it is completed, then check for interrupt/break, then continue to next file. That way, you aren't directly interrupting ffmpeg, but the script command processor, which expects occasional interruptions.
    And have it start by looking at list and the log, and seeing which file was the last one completed, and pick up immediately after. If none are complete, it starts at the beginning of the list. If all are complete, it aborts.


    Scott
    the script already does that. it's right before the code that goes to get another file to convert. some of the longer files (videos of 6 hours, plus or minus) take over 24 hours to convert. since they play OK, i assume all the time is the vp9 video encoding. given the much smaller size of the .webm files, i consider the time spent to be worth as i have over 25% of my laptop drive and nearly as much of my backup drive occupied by videos. now that's shrinking as i manually check that the .webm files play OK and look decent.

    the script just does one file at a time. the queue is in a designated directory tree. it moves the file it will work on and verifies that it actually moved. the move gives it its process ID number. that way i can run 3 of these scripts in parallel in different terminal windows or batch slots. i run 3 of them on a 4 core processor.

    it's in the middle of a video i want to stop and resume. i have read that the best way to do this is to run ffmpeg inside VirtualBox and just put my whole script (i run 3 instances in parallel so i would stop all 3) in suspended stop state followed by saving an image of the VM (a VirtualBox operation). then i can reboot the real hardware as needed and restart every with a reload of the VM image. thus ffmpeg would only see a glitch in wallclock time. but, i have not been able to get VirtualBox running.
    Quote Quote  
  7. Member
    Join Date
    Oct 2021
    Location
    Wheeling WV USA
    Search Comp PM
    i, too, was thinking about some kind of segmentation scheme. if the input can be segmented in one pass, such as conversion to a raw uncompressed format with some finite number of bytes per frame (i can then chop that up into segments), maybe that would help. concatenating the segments in .webm format (audio an video) would be my big worry. ffmpeg would need to be able to do that (with a filter?).

    the first big step, producing the raw frames, might take a while. but that would be OK to restart back at the beginning.

    i would re-design the script to look at how many segments have been converted to webm and the status flags last saved by the previous instance of the same script. i would spend the time thinking about that if i can see and test the things i need ffmpeg to do (1: convert to raw fixed frames) (2. convert each segment into a bunch of webm files) (3. concatenate the webm segments files at copying rate to a single webm file). if ffmpeg can segment the mp4 file into a bunch of mp4 segments then that might be a better alternative (no giant raw frame temp space).
    Quote Quote  
  8. Member
    Join Date
    Oct 2021
    Location
    Wheeling WV USA
    Search Comp PM
    it would be nice to know if ffmpeg could respond to signals with a usable or specified action. i can make the script send it signals at a regular interval with zero accumulated error (there would be a little bit of jitter that could be more than a frame or a few).

    how this whole scheme could work with interlace video is unknown. but it does need to splice the audio right with no clicks at segment cut points.
    Quote Quote  
  9. First, reading all you say i understand that you are very passionate about what you do and you don't shy away from trying new stuff, thats cool.

    Originally Posted by Skaperen View Post
    but, i have not been able to get VirtualBox running.
    Hmpf, i have never met any situation like this but i am sure you invested lots of research about it - with the conclusion that the best way to do what you want is to invest a lot of manpower in enhancing your encoding scripts (compared to e.g. buying a vbox compatible laptop for 150$) ... just joking here, but as you said this, lots of people will now be wondering what was the problem with that and be sure they could solve it. I'll leave that aside and concentrate on the ffmpeg specific part

    Originally Posted by Skaperen View Post
    it would be nice to know if ffmpeg could respond to signals with a usable or specified action.
    This is not currently anyhow the intention of ffmpeg itself. The only commands it supports "while running" at this moment are:
    • ? show this help
    • + increase verbosity
    • - decrease verbosity
    • c Send command to first matching filter supporting it
    • C Send/Queue command to all matching filter
    • D cycle through available debug modes
    • h dump packets/hex press to cycle through the 3 states
    • q quit
    • s Show QP histogram

    "Send command to first matching filter supporting it": Well, i Don't know which filter really supports it but it is guaranteed not even close to what you are looking for because in your case we'd need writers or readers instead of filters to react to the sent command.

    So we are far away from your usecase. If you ask the ffmpeg community they'd directly point you in direction of implementing your own program that replaces ffmpeg binary, using the libav* codec, filter, format and utility library.

    Originally Posted by Skaperen View Post
    Me, too, was thinking about some kind of segmentation scheme. If the input can be segmented in one pass, such as conversion to a raw uncompressed format with some finite number of bytes per frame (i can then chop that up into segments), maybe that would help. concatenating the segments in .webm format (audio an video) would be my big worry. ffmpeg would need to be able to do that (with a filter?).
    Sorry but i really cannot follow your thoughts here. There is no reason to first "dump the raw uncompressed" frames to disk and segment it. If at all, maybe you want to keep the audio in PCM format but the video doesnt make sense and eats too much disk space. All you need to do is to instruct ffmpeg to encode in segments, as described here:
    https://askubuntu.com/questions/948304/how-to-automatically-segment-video-using-ffmpeg...ut-re-encoding

    After encoding all the segments, you can easily stitch them together using the concat input protocol and set whatever output format you like...
    Last edited by emcodem; 3rd Nov 2021 at 03:07.
    Quote Quote  
  10. Originally Posted by Skaperen View Post
    i, too, was thinking about some kind of segmentation scheme. if the input can be segmented in one pass, such as conversion to a raw uncompressed format with some finite number of bytes per frame (i can then chop that up into segments), maybe that would help. concatenating the segments in .webm format (audio an video) would be my big worry. ffmpeg would need to be able to do that (with a filter?).

    the first big step, producing the raw frames, might take a while. but that would be OK to restart back at the beginning.

    i would re-design the script to look at how many segments have been converted to webm and the status flags last saved by the previous instance of the same script. i would spend the time thinking about that if i can see and test the things i need ffmpeg to do (1: convert to raw fixed frames) (2. convert each segment into a bunch of webm files) (3. concatenate the webm segments files at copying rate to a single webm file). if ffmpeg can segment the mp4 file into a bunch of mp4 segments then that might be a better alternative (no giant raw frame temp space).
    ffmpeg -i in.mp4 \
    -c:v hevc -crf 33 -strict -2 \
    -c:a libopus -b:a 64k \
    -vf "select='\
    between(t, 20, 50) \
    +between(t, 2*60 +10, 3*60 +5) \
    +between(t, 5*60, 5*60 +30) \
    +between(t, 19*60 +40, 20*60 +10)', \
    +between(t, 1*3600 +15*60, 1*3600+ 15*60 +47) \
    +between(t, 1*3600 +36*60 +45, 1*3600 +37*60 +5)', \
    setpts=N/FRAME_RATE/TB" \
    -af "aselect='\
    between(t, 20, 50) \
    +between(t, 2*60 +10, 3*60 +5) \
    +between(t, 5*60, 5*60 +30) \
    +between(t, 19*60 +40, 20*60 +10)', \
    +between(t, 1*3600 +15*60, 1*3600+ 15*60 +47) \
    +between(t, 1*3600 +36*60 +45, 1*3600 +37*60 +5)', \
    asetpts=N/SR/TB" output.mp4 \

    one command to do multiple segments and re-encode them into one video
    put in your time codes once for the -vf video filter then copy them for the -af audio filter
    notice first between no + and last +between for -vf (video) and (-af) audio have ',
    have a \ at the very end lets you look at the command before pressing enter

    20 to 50 seconds
    2:10 to 3:05
    5:00 to 5:30
    19:40 to 20:10
    1:15:00 to 1:15:47
    1:36:45 to 1:37:05
    Quote Quote  
  11. Member
    Join Date
    Oct 2021
    Location
    Wheeling WV USA
    Search Comp PM
    Originally Posted by emcodem View Post
    After encoding all the segments, you can easily stitch them together using the concat input protocol and set whatever output format you like...
    my per-frame idea was just the other extreme end of segmenting. but if i can get ffmpeg to output lots of tiny segments that have the result of the big work to encode VP9 and these segments can be concatenated together to make a single clean webm file, then we have half the solution.

    the other half is to be able start it back up where it left off. if telling it how many segments to skip encoding and it can get to that point real fast (not just skipping the writing) then that should work. if it need keep some active stare data file across the reboot to read when restarting, that would be OK.

    can i get it to segment starting at every I-frame and include all the b/p-frames that modify it, all into one properly contained file that can be stitched with others or played to see a quick burst?

    i have 2 or 3 use cases for segmenting:

    1. making the restart work for the conversions small when i reboot. this would involve stitching in webm format or at least stitching uncontained vp9 and making webm from it.

    2. breaking up a live stream so i can spy on it w/o starting another stream. my BW is not infinite. these come in mp4 format. this the same as needing to stop the feed and still have something to see.

    3. playing around and having fun like randomly shuffling segments.
    Quote Quote  
  12. Originally Posted by Skaperen View Post

    can i get it to segment starting at every I-frame and include all the b/p-frames that modify it, all into one properly contained file that can be stitched with others or played to see a quick burst?
    yeah thats kind of default for the segment format (unless u specify break_non_keyframes option). Just give it a whirl, e.g.
    ffmpeg -i INPUT %YOURCODECSTUFF% -f segment -segment_time 1 c:\temp\segment\output_%10d.mkv

    It will split at each I-frame after at least 1 second duration.
    Last edited by emcodem; 7th Nov 2021 at 14:03.
    Quote Quote  
  13. Member
    Join Date
    Oct 2021
    Location
    Wheeling WV USA
    Search Comp PM
    it certainly makes sense to begin the next segment at an I-frame, otherwise the next segment depends on this segment. but 1 second could be a few frame groups (is that the correct term for an I-frame and all other frames until just before the next I-frame?). my thought would be to specify the time of one frame or less (but maybe not zero) when trying to get segments of one frame group.
    Quote Quote  
  14. Exactly. Well i'd just have a play with time values, e.g. -segment_time 0.2.
    It would be even better if you know the GOP (group of pictures) size up in front, you could force creating fixed GOP length while encoding with the -g option.
    Honestly i thought that the GOP's for VP9 would be usually more than 1s because you want to save as much bitrate as possible...
    Quote Quote  



Similar Threads

Visit our sponsor! Try DVDFab and backup Blu-rays!