How to start viewing a downloaded video before it has finished?

[BosseB]

I have a scripted system on Linux for downloading real time streaming video for later viewing, which uses ffmpeg to retrieve the videos through their m3u8 urls into mp4 files.
Some of these videos are 2 or even 3 hours long and I would like to be able to start viewing them while they are still being downloaded.
But when I point my video player to the mp4 file being written it fails to display anything.

So I am wondering if there is a way to do the downloading such that the file being written at the end can be simultaneously viewed from the beginning including performing timed playback jumps?

My ffmpeg command for the download looks like this:

CMD="ffmpeg -hide_banner -referer \"${VIDEOURL}\" -i \"${M3U8URL}\" -r 30 -vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -x264-params keyint=30:scenecut=0:open-gop=0 -c:a aac -t ${CAPTURETIME} ${TARGETFILE}"

The reason for the complex filter is that I want the keyframes start on even seconds to enable simpler and faster editing.

[larley]

I've noticed that you're re-encoding the video file. Why is that? You could also just copy the codecs over with

-c copy

If you want to pre-play the video, then use

-f mpegts

to set the mpegts protocol.

[BosseB]

I've noticed that you're re-encoding the video file. Why is that? You could also just copy the codecs over with

-c copy

If you want to pre-play the video, then use

-f mpegts

to set the mpegts protocol.

Well, I used copy earlier but then it was near impossible to edit the files because there were always artifacts of surrounding scenes at the cut points in the final video. And re-encoding after the video was downloaded took about 10 minutes per hour of video so was not useful either.
With the on-the-go re-encoding I got control of the frame borders and now applying an edit command with 10 cut points in a 1-hour video takes something like 10 s to process.

Before I did this though, I could still not play the video unless it had been completely downloaded.
It looks like something necessary for playback is added to the video at the end of recording and if that is missing the video is unplayable. Some closing data needed for playback is located at the end of the file, maybe?

This always (also in the present solution) happens if there is an interruption of the download just before it would have ended anyway...

I have noted on my PVR box that it is possible on that to start viewing a recording that is still not finished.
But that is on a system recording off-air via an antenna so it is not streaming video...

[Obo]

Haven't tried myself, but maybe adding "-movflags +faststart" to the ffmpeg arguments may help as well. Or encoding as mkv.

[BosseB]

Haven't tried myself, but maybe adding "-movflags +faststart" to the ffmpeg arguments may help as well. Or encoding as mkv.

I will try the suggested flags.
Regarding encoding I need mp4 for the compression and my tools (self developed)....

[larley]

I have noted on my PVR box that it is possible on that to start viewing a recording that is still not finished.
But that is on a system recording off-air via an antenna so it is not streaming video...

They're recording the videos as .ts files, which support this kind of operation. So setting '-f mpegts' should work, as I've tested it

EDIT: Obo's solution works too

[A_n_g_e_l_a]

N_m3u8DL-RE --save-name test "http://xxxx,xxxx,xxxxx.mpd" --append-url-params --mp4-real-time-decryption --use-shaka-packager  --key xxxxxxxxxxxxxxxxxxxxxxx:xxxxxxxxxxxxxxxxxxxxxxx   --live-pipe-mux

produces a ts file that just grows. and vlc test.ts plays it. I was using this format to watch 4k Olympics output. If it isn't encrypted then only --live-pipe-mux is likely needed in the command. I found, by using vlc on my PC, I could cast to my TV; look under 'playback' and choose renderer.

[BosseB]

EDIT: Obo's solution works too

Not for me...
I tried to modify the ffmpeg command as follows (added -movflags +faststart after -r 30):

CMD="ffmpeg -hide_banner -referer \"${VIDEOURL}\" -i \"${M3U8URL}\" -r 30 -movflags +faststart  -vf scale=w=-4:h=480 -c:v libx264 -preset fast -crf 26 -x264-params keyint=30:scenecut=0:open-gop=0 -c:a aac -t ${CAPTURETIME} ${TARGETFILE}"

Then started a 3 min recording and after a minute or so tried to open the mp4 file in a video player, but it did not show up...

When it was all done it played fine so there is something blocking it from being viewed until fully recorded.

[LZAA]

'-movflags empty_moov'

[BosseB]

'-movflags empty_moov'

OK thanks!
So this did allow starting viewing of the mp4 file while it was downloading.

But the length of the video inside the video player (VLC) seems to be read at the moment of opening the file and is not updated while the video is downloaded further.
So the player stops at the location the recording was when it started to play the file even though the file is now longer.

I don't know what this setting might mean for the video itself, I will have to check it further to see if I find any problems.

What exactly is this moov thing?

[larley]

'moov' is an MPEG-4 atom (kinda like a container), that contains metadata about the video

[BosseB]

'moov' is an MPEG-4 atom (kinda like a container), that contains metadata about the video

Coming back to this thread a year+ later in order to ask if there has been any movement on ffmpeg in Ubuntu...
I have noted that there is a new flag 'hybrid_fragmented' that will update a working moov while downloading the video and when it is done update it such that the video is as finished the same as when not using the flag.

Using this seems to make it possible to play back an mp4 video from the start while it is being written in the end.
But then one needs to have the flag enabled in the ffmpeg of course.

I have now updated my Ubuntu server, which is doing my downloads, to 24.04.3 LTS but the flag is not available as far as I can see...
My apt ffmpeg version:

$ apt policy ffmpeg
ffmpeg:
  Installed: 7:6.1.1-3ubuntu5+esm4
  Candidate: 7:6.1.1-3ubuntu5+esm4
  Version table:
 *** 7:6.1.1-3ubuntu5+esm4 510
        510 https://esm.ubuntu.com/apps/ubuntu noble-apps-security/main amd64 Packages
        100 /var/lib/dpkg/status
     7:6.1.1-3ubuntu5 500
        500 http://archive.ubuntu.com/ubuntu noble/universe amd64 Packages

And starting ffmpeg shows this at the beginning:

ffmpeg version 6.1.1-3ubuntu5+esm4

Does this version contain the new flag handling?
How can I check?

Or else I don't understand how it can be enabled/used.

Similar but different:
I noted that yt-dlp was regularly becoming stale on Ubuntu and then I found that I could get it not via apt but directly from the source in a form that could be easily installed on Ubuntu and then when I symlinked that to my bin dir all yt-dlp problems disappeared.
See here

Is there some similar way to download ffmpeg into the user dir and get the very latest release?

[LZAA]

Version 7.1.

[BosseB]

OK so the most recent release is 7.1...

Question:
How can I get that onto my Ubuntu Server 24.04.3 LTS?

[Obo]

There's no official package for 24.04 IIUC. It's packaged for 25.04 though. If you really need a 7.1 (or newer) version, install a static build, e.g. from here: https://github.com/BtbN/FFmpeg-Builds/releases

[BosseB]

There's no official package for 24.04 IIUC. It's packaged for 25.04 though. If you really need a 7.1 (or newer) version, install a static build, e.g. from here: https://github.com/BtbN/FFmpeg-Builds/releases

Thanks for the link! Much appreciated!

It looks like the one I have used successfully for yt-dlp...

How can I see which version of ffmpeg it offers? Maybe version 8???

It does not say on the webpage how to use the download....

Does the tar.xz contain a single ffmpeg binary including everything or is there a set of files with libraries etc?

There are several linux64 versions that I believe can match my Intel core-i7 device:

ffmpeg-master-latest-linux64-gpl-shared.tar.xz 59.9 MB
ffmpeg-master-latest-linux64-gpl.tar.xz 124 MB
ffmpeg-master-latest-linux64-lgpl-shared.tar.xz 55 MB
ffmpeg-master-latest-linux64-lgpl.tar.xz 112MB

What does the "shared" and "gpl/lgpl" mean and which one should I download?

[Obo]

https://github.com/BtbN/FFmpeg-Builds

[BosseB]

I did a test as follows:
$ wget https://github.com/BtbN/FFmpeg-Builds/releases/download/latest/ffmpeg-master-latest-li...x64-gpl.tar.xz

This gave me a 550 MB big tar file (after decompressing the xz file)
Inside I found a bin dir containing ffmpeg, ffplay and ffprobe all sized 176 to 178 MB each.

I tested running ffmpeg and got this response:

$ ./ffmpeg
ffmpeg version N-121199-g5cb6d2221a-20250924 Copyright (c) 2000-2025 the FFmpeg developers
  built with gcc 15.2.0 (crosstool-NG 1.28.0.1_403899e)

So no real release version embedded...

If I symlink this ffmpeg into my private bin dir will that mean that it could be used like the one installed via apt?

[BosseB]

I did a test as follows:
If I symlink this ffmpeg into my private bin dir will that mean that it could be used like the one installed via apt?

Tested and it works!
So now I have installed it on one test machine by copying it into a subdir at $HOME and symlinking into $HOME/bin.
Will check the performance during a session now.

The difference in binary size is considerable between the ffmpeg installed via apt and the one from github. Like 500 times...

[Obo]

Yes, that's expected: it's a statically linked binary.

(Use

ldd <path-where-you-put-ffmpeg>

vs.

ldd /usr/bin/ffmpeg

to see what statically linked vs. linked against a whole lot of shared libraries means. The statically linked binaries are really meant for environments where you can't easily install the latest version via package manager. On my Mac "ffmpeg8" is linked against 68 shared libraries, whereas the statically linked binary has no dependency at all.)

[BosseB]

OK, I have now retrieved the very latest ffmpeg so I want to test it using movflag hybrid_fragmented, but I don't know where and how to put it on the ffmpeg command line..
I want to modify a working script that gets me mp4 formatted video which is not playable until closed such that it produces an mp4 file where the moov atom is adjusted so it is playable while being written (like a ts file is):

My command to download into an old style mp4 file looks like this and works fine since a long time to get mp4 output:

CMD="ffmpeg -hide_banner ${MODE} -i \"${M3U8URL}\" -r 30 -vf scale=w=-4:h=${VIDEORES} -c:v libx264 -preset fast -crf 26 -x264-params keyint=30:scenecut=0:open-gop=0 -c:a aac -t ${CAPTURETIME} ${TARGETFILE}"

where the names in all caps are values according to the names.

I tried inserting this into the command above:

  -movflags \"hybrid_fragmented\"

right after the {MODE} identifier before -i but when I ran the script it aborted with this message:

Option movflags not found.
Error opening input file https://*******/hls/msnbc_live/index.m3u8.
Error opening input files: Option not found

Obviously I have not used it correctly and I do not understand from the documentation how it should be formulated....
Any ideas what is wrong here?
How should a movflag be entered?

From the documentation:

‘hybrid_fragmented’

    For recoverability - write the output file as a fragmented file. This allows the intermediate file to be read while being written (in particular, if the writing process is aborted uncleanly). When writing is finished, the file is converted to a regular, non-fragmented file, which is more compatible and allows easier and quicker seeking.

    If writing is aborted, the intermediate file can manually be remuxed to get a regular, non-fragmented file of what had been written into the unfinished file.

What to do?

[LZAA]

ffmpeg -referer https://player.hotmart.com/ -i "https://vod-akm.play.hotmart.com/video/3RGb5QxbZ4/hls/master-pkg-t-1752932995000.m3u8?hdnts=st%3D1758752955%7Eexp%3D1 758753455%7Ehmac%3Def5e952c43237e32d8a15fc4ba78998 2f4cc8dae48318bf1835f0c06e29859a3&app=aa2d356b-e2f0-45e8-9725-e0efc7b5d29c" -movflags hybrid_fragmented -c copy video.mp4 -hide_banner
mime type is not rfc8216 compliant
[hls @ 000002a8b9c814c0] Skip ('#EXT-X-VERSION:4')
[hls @ 000002a8b9c814c0] Skip ('## Created with Unified Streaming Platform (version=1.14.4-30793)')
[hls @ 000002a8b9c814c0] Skip ('#EXT-X-SESSION-KEY:METHOD=AES-128,URI="https://contentplayer.hotmart.com/video/3RGb5QxbZ4/mp4/key/3RGb5QxbZ4-1752932995000.key"')
[hls @ 000002a8b9c814c0] Skip ('# AUDIO groups')
[hls @ 000002a8b9c814c0] Skip ('# variants')
[hls @ 000002a8b9c814c0] Skip ('# keyframes')
...
[hls @ 000002a8b9c814c0] Opening 'crypto+https://vod-akm.play.hotmart.com/video/3RGb5QxbZ4/hls/hdntl=exp=1758839844~acl=%2f*~da...5-e0efc7b5d29c' for reading
...

[BosseB]

ffmpeg -referer https://player.hotmart.com/ -i "https://vod-akm.play.hotmart.com/video/3RGb5QxbZ4/hls/master-pkg-t-1752932995000.m3u8?hdnts=st%3D1758752955%7Eexp%3D1 758753455%7Ehmac%3Def5e952c43237e32d8a15fc4ba78998 2f4cc8dae48318bf1835f0c06e29859a3&app=aa2d356b-e2f0-45e8-9725-e0efc7b5d29c" -movflags hybrid_fragmented -c copy video.mp4 -hide_banner

What does this mean (removed bulk of ffmpeg output)???

I see this at the end of your command:

-movflags hybrid_fragmented -c copy video.mp4 -hide_banner

Do you mean that the -movflags are misplaced in my command example?
Or what is your point?
Please elaborate what I need to do to convert my working script that produces a ts type file so it instead makes an mp4 that can be played before the full download is complete and the file closed...

[BosseB]

So I asked ChatGPT and got the following working command for the flags:

-movflags +frag_keyframe+empty_moov+default_base_moof -frag_duration 2000000

And this should be placed at the end just before the -t ${CAPTURETIME} argument.

This works to download the mp4 file on a test server, now I need to verify on my main server where I can actually view the videos.
But then I have to activate the latest ffmpeg first also there...

LATER
Yes! The files are now viewable while they download so no need to wait.