N_m3u8DL-R Issues with Concurrently Downloading Multiple Live Streams

[doganjr]

Hello friends,

I believe the problem I’m facing is related to running multiple scripts simultaneously that use the N_m3u8DL-RE library. However, this is just an assumption. I will explain the process in detail.

I’m working on a project that involves downloading and saving multiple streams concurrently.

Here is the script I’m using for each stream:

def download_m3u8(m3u8_url, key_id, gamename, formatdate):
    n_m3u8dl_path = r"C:/Users/username/Desktop/N_m3u8DL-RE_Beta_win-x64/N_m3u8DL-RE.exe"
    
    command = [n_m3u8dl_path, m3u8_url, "--save-dir",
     "2425", "--save-name", formatdate+"_"+gamename,
     "--append-url-params",
     "--mp4-real-time-decryption", 
     "--live-real-time-merge",
      "--key", key_id,
     "-sv", 'res="1280*":for=best',
     "-sa", "lang=en:for=best",
     "-M", "mp4",
     "--live-record-limit", "03:15:00"
     ]
    subprocess.run(command)

Some of these streams start at the same time, while others do not. Each stream lasts approximately 3 hours and 15 minutes.

I run the scripts when the streams start broadcasting live. For the first three streams in the queue, the downloading and saving process works perfectly. However, when the 4th, 5th, and 6th streams begin, the streams overlap, and issues arise in downloading and saving them(I believe its more about merging the sound and video files). If there are fewer than three simultaneous streams, there is no issue with downloading the remaining ones.

Here are the terminal outputs of scripts that completed successfully and those that failed:

Successful:

[Attachment 83921 - Click to enlarge]

Failed(Here it freezes, I am waiting for almost 6 hours to see it finished):

[Attachment 83923 - Click to enlarge]

To better illustrate the timing issue, I am sharing the creation timestamps of both successful and failed stream files:


[Attachment 83924 - Click to enlarge]

[Attachment 83925 - Click to enlarge]

Interestingly, the timestamps of the failed stream files show creation dates from 10 years ago, which surprised me. I wonder if this could be a clue to the issue. Additionally, the MP4 files that yielded by problematic download scripts cannot be played.

New Approach:

I modified my download script as follows, which successfully produced a .ts file for the stream(EVENTHOUGH I TRIED TO FORCE IT TO BE MP4 FILE), which is surprising. However, I haven’t had the chance to test this approach on multiple simultaneous live streams yet.

def download_m3u8(m3u8_url, key_id, gamename, formatdate):
    n_m3u8dl_path = r"C:/Users/username/Desktop/N_m3u8DL-RE_Beta_win-x64/N_m3u8DL-RE.exe"

    command = [n_m3u8dl_path, m3u8_url, "--save-dir",
               "2425", "--save-name", formatdate + "_" + gamename,
               "--append-url-params",
               "--mp4-real-time-decryption",
               "--live-pipe-mux",
               "--key", key_id,
               "-sv", 'res="1280*":for=best',
               "-sa", "lang=en:for=best",
               "-M", "mp4",
               "--live-record-limit", "03:15:00"
               ]
    subprocess.run(command)

[Attachment 83922 - Click to enlarge]

If anyone has encountered this issue before, has suggestions for a solution, or can recommend an alternative approach, I would greatly appreciate your insights.

Thank you in advance for your help!

Best regards!

[2nHxWW6GkN1l916N3ayz8HQoi]

Try setting a randomized tmp dir for each n_m3u8dl-re download command.

Edit: and also a different output folder for each command

[doganjr]

Try setting a randomized tmp dir for each n_m3u8dl-re download command.

Edit: and also a different output folder for each command

Hmm, actually, every essential document for each stream is stored separated folders.

Let me share some snaps from the stream folders for successful and problematic cases:

Successful:


[Attachment 83952 - Click to enlarge]


[Attachment 83953 - Click to enlarge]

Problematic:


[Attachment 83954 - Click to enlarge]


[Attachment 83955 - Click to enlarge]

Things getting more interesting in here in my opinion. "_dec" id's seems more reliable for audio segments(folder name: 0__avc1.64001F_3044804_)

However, .ms4 file id's at video segments folder(1_mp4a.40.5-104400-48000-2___en) are not consistent. Some of them are not decoded(I assume _dec is the abbreviation for "decoded"). Also, it seems like some of the files are decoded twice???(files that end with "_dec_dec")


[Attachment 83956 - Click to enlarge]


[Attachment 83957 - Click to enlarge]

[2nHxWW6GkN1l916N3ayz8HQoi]

I'll just copy paste what I said again. Try setting a randomized tmp dir for each n_m3u8dl-re download command.

n_m3u8dl-re --help
.
.
.
Options:
--tmp-dir <tmp-dir> Set temporary file directory
.
.
.

I don't see tmp-dir being used in your subprocess command.

  command = [n_m3u8dl_path, m3u8_url, "--save-dir",
               "2425", "--save-name", formatdate + "_" + gamename,
               "--append-url-params",
               "--mp4-real-time-decryption",
               "--live-pipe-mux",
               "--key", key_id,
               "-sv", 'res="1280*":for=best',
               "-sa", "lang=en:for=best",
               "-M", "mp4",
               "--live-record-limit", "03:15:00"
               ]

I had this issue once and it only worked with massive parallel downloading only after I put n_m3u8dl-re to handle each download in a separate temp folder.

[doganjr]

I'll just copy paste what I said again. Try setting a randomized tmp dir for each n_m3u8dl-re download command.

n_m3u8dl-re --help
.
.
.
Options:
--tmp-dir <tmp-dir> Set temporary file directory
.
.
.

I don't see tmp-dir being used in your subprocess command.

  command = [n_m3u8dl_path, m3u8_url, "--save-dir",
               "2425", "--save-name", formatdate + "_" + gamename,
               "--append-url-params",
               "--mp4-real-time-decryption",
               "--live-pipe-mux",
               "--key", key_id,
               "-sv", 'res="1280*":for=best',
               "-sa", "lang=en:for=best",
               "-M", "mp4",
               "--live-record-limit", "03:15:00"
               ]

I had this issue once and it only worked with massive parallel downloading only after I put n_m3u8dl-re to handle each download in a separate temp folder.

To be precise, here is the modified version of the function:

def download_m3u8(m3u8_url, key_id, gamename, formatdate):
    n_m3u8dl_path = r"C:/Users/username/Desktop/N_m3u8DL-RE_Beta_win-x64/N_m3u8DL-RE.exe"

    if not os.path.exists(r"2425temp/"+gamename):
        os.makedirs(r"2425temp/"+gamename)

    command = [n_m3u8dl_path, m3u8_url,
               "--tmp-dir", "2425temp/"+gamename,
               "--save-dir", "2425",
               "--save-name", formatdate + "_" + gamename,
               "--append-url-params",
               "--mp4-real-time-decryption",
               "--live-pipe-mux",
               "--key", key_id,
               "-sv", 'res="1280*":for=best',
               "-sa", "lang=en:for=best",
               "-M", "mp4",
               "--live-record-limit", "03:15:00"
               ]
    subprocess.run(command)

If I am still missing something please inform me.

[2nHxWW6GkN1l916N3ayz8HQoi]

I guess. The main idea is you need to add a unique tmp dir parameter for each n_m3u8 command. See if it works now.

[doganjr]

I guess. The main idea is you need to add a unique tmp dir parameter for each n_m3u8 command. See if it works now.

Nothing changed. Problem occurs as the same as I mentioned above. First three streams are good then extra overlapping ones are problematic. They all have unique temp files.


[Attachment 83961 - Click to enlarge]

Actually, this time, first three were defective, then remaining ones good.

[Attachment 83962 - Click to enlarge]

[2nHxWW6GkN1l916N3ayz8HQoi]

Hmm. That's weird. Do you get the same issue with multiple dec? _dec_dec . Or is it another issue

[doganjr]

Hmm. That's weird. Do you get the same issue with multiple dec? _dec_dec . Or is it another issue

It is the same issue, in my opinion, cannot talk precisely.

This is the files in a problematic stream folder.

[Attachment 83963 - Click to enlarge]

dec_dec issue in audio folder:

[Attachment 83964 - Click to enlarge]

video folder does not include any dec_decs and this time the video can be played(audio is good, too). not fully obviously, just the first 30 mins.

[Attachment 83965 - Click to enlarge]

Here is a weird thing. All the audio files seems like created after 42 mins the stream downloading process???(This is not the case for the successful downloads)

[Attachment 83966 - Click to enlarge]

I think temp folder idea was a good shot, maybe but just maybe, adding one more subfolder to the path(/2425temp/blabla) collapses the process somehow?

I am also just wondering that maybe the issue is related that these folder located in D:, I move the process to somewhere in C:. I know it may sounds ridiculous but I will give it a try with that way. Any more suggestions? Logs you may want to see to get some clue?

[2nHxWW6GkN1l916N3ayz8HQoi]

Without full script and full m3u8 input to test locally, I can't do much besides blind guessing. Setting a unique tmp was definitely part of the problem since you have overwriting issues if you do multiple parallel downloads at once with N_m3u8. But why is it not working still, no idea.

[doganjr]

Without full script and full m3u8 input to test locally, I can't do much besides blind guessing. Setting a unique tmp was definitely part of the problem since you have overwriting issues if you do multiple parallel downloads at once with N_m3u8. But why is it not working still, no idea.

To make the problem more clear or shrink the uncertainty circle, I am going to try downloading streams without real time encoding this night. In the mean time, if we assume that the problem is something like "decrypted files are being overwrited", I want to ask that is it possible to decrypt the live stream files by using another script that simultaneously works with the command that works on downloading the raw(not decrypted) stream files?