cutting off ffmpeg downloads of very long videos

[Skaperen]

i periodically download some live videos that run for too long. this is using ffmpeg. it saves a file named what/ever.mp4.part. if i let it run until it ends, it creates what/ever.mp4 which usually works fine. but if i cut off ffmpeg it just leaves me with that what/ever.mp4.part file which is unplayable. if i try to use ffmpeg to convert it to .mp4 it tells me there is a "moov" item missing and it cannot determine the duration of the video. i have noticed it has some "m3u8" stuff in it.

the reason i want to cut it off is that i want to use some of the beginning stuff long before the live feed ends. these are videos of severe weather and they can be very short or very long.

[lomero]

this is "Video Streaming Downloading" section ...
if you have some request to edit your video, we have right section on "Video Conversion" or "Editing" section

[codehound]

i periodically download some live videos that run for too long. this is using ffmpeg. it saves a file named what/ever.mp4.part. if i let it run until it ends, it creates what/ever.mp4 which usually works fine. but if i cut off ffmpeg it just leaves me with that what/ever.mp4.part file which is unplayable. if i try to use ffmpeg to convert it to .mp4 it tells me there is a "moov" item missing and it cannot determine the duration of the video. i have noticed it has some "m3u8" stuff in it.

the reason i want to cut it off is that i want to use some of the beginning stuff long before the live feed ends. these are videos of severe weather and they can be very short or very long.

Use .ts as download format and you wont get this issue. Then convert to .mp4 post download.

ffmpeg -i "videoURL" -c copy "anyname.ts"

Post download:

ffmpeg -i "anyname.ts" -c copy "output.mp4"

Or alternatively use yt-dlp

yt-dlp --no-part "videoURL"

[LZAA]

How do you stop downloading?

[codehound]

Ctrl+c

(Hold down ctrl key on keyboard, simultaneously hit 'c'.)

[Skaperen]

Or alternatively use yt-dlp

yt-dlp --no-part "videoURL"

that looks like something that will help.

[Skaperen]

How do you stop downloading?

whatever way i can. Ctrl+C, q, reboot

[LZAA]

Url?

[Skaperen]

Use .ts as download format and you wont get this issue. Then convert to .mp4 post download

so at any time during the download the .ts file is technically complete to that point in the video as received? most live videos i download are using m3u8 according to messages from ffmpeg. this can be saved as .ts without heavy conversion work?

[codehound]

Use .ts as download format and you wont get this issue. Then convert to .mp4 post download

so at any time during the download the .ts file is technically complete to that point in the video as received?

Yes

[BosseB]

if i let it run until it ends, it creates what/ever.mp4 which usually works fine.
but if i cut off ffmpeg it just leaves me with that what/ever.mp4.part file which is unplayable.

If you hit the q key in the window where ffmpeg is running you will get a complete file because ffmpeg will finalize the output before exiting.
Ctrl-C just kills it immediately and the moov atom is not properly inserted.

If you know beforehand the length of the recording you want to make you can just set the -t argument to the number of seconds you want to record and ffmpeg will stop there and finish the file properly.

[Skaperen]

If you know beforehand the length of the recording you want to make you can just set the -t argument to the number of seconds you want to record and ffmpeg will stop there and finish the file properly.

in some cases i want to record it all but just look at the recording i have so far.

in other cases i do not want to continue recording but as i recall typing q did not work. i will need to try it again to try to see why. i will need to wait until another event to try again (a river flood gauge monitoring camera). if anyone knows a public video source i can test on (m3u8) that could help.

[mister_ nex]

i want to cut it off is that i want to use some of the beginning stuff long before the live feed ends. these are videos of severe weather and they can be very short or very long.

can try this code from https://ostechnix.com/download-a-portion-of-youtube-video-with-youtube-dl-and-ffmpeg/

If you don't want to download the entire video but only a part of it, you can combine ffmpeg and youtube-dl to download a specific portion of the youtube video like below

$ youtube-dl --external-downloader ffmpeg --external-downloader-args "-ss 00:01:00.00 -to 00:02:00.00" -f best "https://www.youtube.com/watch?v=your_link"

[Skaperen]

i usually want to let the ongoing download continue. in other cases i don't want to continue. but the latter case should be as simple as doing the former case followed by killing the download. these are not YouTube Live videos. i don't know how these protocols work but i could design my own protocol based on UDP, TCP, or SCTP. TCP would make for an ugly design. but there is no point making my own design as so many others have been done with a mix of software to support many, none of which supports everything, or needs to.

the determination of when to view the video needs to be able to be done at a time after the recording starts. it is not the case that at start recording time that i know when i want to look at the video. and i may want to view the beginning or the current point (live) or any point in between. the examples in the ostechnix link seem to assume you know ahead of time.

[Skaperen]

Use .ts as download format and you wont get this issue. Then convert to .mp4 post download

so at any time during the download the .ts file is technically complete to that point in the video as received?

Yes

and it will have what is needed to include that moov object in the mp4 to say how long that video is (when i grabbed a copy while letting the origin recording just keep on going), or ffmpeg knows how to time it and create it when converting from ts to mp4?

would it be practical to just leave it in .ts format? can it be played, at least by something? mplayer? ffplay? can it be converted to .webm format (ts to webm)? is ts format substantially larger than webm format?

suppose i make my videos available in webm format? how would that impact people that expect or want mp4 format? i am converting my video collection to webm format. Firefox plays this one fine so i think i can just leave them that way.

[abolibibelot]

and it will have what is needed to include that moov object in the mp4 to say how long that video is (when i grabbed a copy while letting the origin recording just keep on going), or ffmpeg knows how to time it and create it when converting from ts to mp4?

TS (Transport Stream) is designed for the specific purpose of streaming video content reliably even in case of transmission issues ; contrary to regular video files which have a single header (sometimes footer) metadata structure, it is made of multiple tiny segments, each having its own metadata (which incidentally tends to fool data recovery algorithms), so indeed a recording can be stopped at any point and the resulting file should play fine. Then normally the conversion to a playback container like MP4 or MKV, or WEBM since you mention it, is straightforward, quick and lossless.

would it be practical to just leave it in .ts format? can it be played, at least by something? mplayer? ffplay? can it be converted to .webm format (ts to webm)? is ts format substantially larger than webm format?

Because of that particular “modular” structure, the so-called “container overhead” is significantly larger with TS, so converting to MP4 or MKV, or WEBM since you mention it, shrinks the file appreciably (can be a few dozen megabytes for a single video file, can be a few gigabytes over a whole collection). Also the playback is smoother (with the TS format there's a slight delay when requesting a specific timecode by clicking on the progress bar, and the picture can look garbled for a split second).

suppose i make my videos available in webm format? how would that impact people that expect or want mp4 format? i am converting my video collection to webm format. Firefox plays this one fine so i think i can just leave them that way.

I don't know much about the WEBM container, it seems to be a “flavor” of Matroska video (the headers are very similar). Nowadays it should play fine on any computer, but older standalone devices may choke on it.

I took the habit of keeping DIFF files made with xdelta (as suggested in this thread) when converting a native file to another container — TS to MP4 for instance ; that way I can go back to the original if I ever need to. For instance : I had an original TS recording with a size of 1508381528 (1438MB), converted it to a MP4 file with a size of 1449442452 (1382MB), then created a DIFF file from the TS with the MP4 as source / reference, which has a size of 38235448 (36MB). If I ever need to, I can re-create the TS, 100% identical to the original, from the MP4 and that small DIFF file. (I have had inconsistent results regarding the size of the DIFF files, specifically when converting from TS to MP4, sometimes setting a “source window size” parameter equal to the size of the source / reference file yields a larger DIFF file, although it is supposed to maximize the efficiency ; also I did extensive tests with the last two available versions, 3.0.11 and 3.1.0, there's no consistency regarding which one performs the best, i.e. produces the smallest file from a given pair of input files — most of the time version 3.0.11 yields a slightly higher compression, as the author remarked when releasing those last two updates more than 5 years ago, but sometimes it's the opposite, and the difference can be a factor 2 or 3. I couldn't get any useful insight on those matters. To keep it simple, use version 3.0.11 and set the -B parameter to the size of the source / reference input file, that's the most efficient combination on average — the maximum value for -B in v. 3.0.11 being 2147483648, v. 3.1.0 allows to set it higher but in my limited tests it didn't work as expected, in a very specific case... explaining further would be really outside of the scope of this thread.)

If you kept the unplayable interrupted downloads, you may be able to fix them with either of the 2 tools mentioned in this thread (Untrunc, MP4 Repair) :
https://forum.videohelp.com/threads/352660-Fixing-corrupt-MP4-video-recovered-from-SD-card

[Skaperen]

i am trying ts format but it still saves in mp4 format. the cause may be because i am using youtube-dl. the command line a script runs looks like this code snippet:

format = 'ts'
cmd = ['youtube-dl','--verbose','--merge-output-format',format,url]
subprocess.call(cmd)

the script is in Python3 and is more elaborate than the snippet shown. but this shows the exact command line i am running. i still get a mk4.part file. i now must conclude that youtube-dl is my trouble maker.

[abolibibelot]

By now youtube-dl is pretty much considered a deprecated utility (hasn't been updated since June, while things evolve very fast on that front, which means that many features are bound to stop working properly over time), so you'd better start using yt-dlp instead.
Then TS is not among the formats available with the --merge-output-format option (quote from embedded guide obtained with “yt-dlp -h” : “One of mkv, mp4, ogg, webm, flv” — it was exactly the same in youtube-dl). Here I was suggested to use --hls-use-mpegts, but it may not work in that case. Above “codehound” suggested that you use --no-part, although I'm not sure how it affects the output.

[Skaperen]

vBulletin kept telling me i could not process my submission because i had logged in (i didn't so maybe i need to change my password). it said to reload the window, which i did 3 times (it erased my post the first time). but that didn't help. i tried "Go Advanced" 2 times and that didn't help, either. some logic in the board code seems to be broken (a bad algorithm?). fortunately, i saved my post in a file. so i totally exited the browser and started a new one and came back here. i have an icon that does all that (a custom .desktop file). so, here we go, my original post ...

the man page for youtube-dl didn't list it, either. but i always try things just in case the man page is for an older version (happens a lot). now i am wondering if i can use ffmpeg directly on the URLs i have (one at a time). maybe no as these URLs go to an HTML page where the video URL is on a particular button. but i assume yt-dlp knows how to find it as youtube-dl did. so yt-dlp needs to let me ask for ts format and/or let me add or override ffmpeg options. for youtube-dl i just created an empty directory and changed into there before running it. then my script just looked what was in there and moved to where i was collecting them. i'll write some new scripts after some manual testing of yt-dlp options.

one thing i tried with youtube-dl that failed was to make my own copy of ffmpeg in an earlier directory listed in $PATH. but that didn't work so it must have invoked /usr/bin/ffmpeg by full path, bypassing the scan of $PATH. i do that in my scripts, too. i had a script at ~/cmd/ffmpeg which writes args to a file and invokes /usr/bin/ffmpeg. but it was never run. i would have been able to change the args if it did.

anyway, i got yt-dlp from github. i'll try to get Ubuntu to add it to their repositories if it works for me. i'll also use it for occasional downloads from YT itself (my own or family or free videos/music).

[Skaperen]

i installed yt-dlp using "pip3 install --no-deps yt-dlp". in an empty directory i tried "yt-dlp --format ts ${URL}" and got the message "Requested format is not available" and exits with status 1. changing "ts" to "mp4" and doing the command, again, and it begins downloading. i also tried "webm" and that fails the same as ts did.

edit:

it should set up ffmpeg to convert as the video comes in, if you ask for a different format.