[rtmpdump] Download rolls back, stores the same part several times.

[etiam]

Hi,

I'm trying to download this video
https://www.ipam.ucla.edu/wowzavideo.aspx?vfn=10660.mp4&vfd=gss2012

Using rtmpsrv gave an rtmpdump command, but the download jumps back to lower completion percentages, and the resulting file repeats the content.
I'm on Linux Mint 13, using the latest git repository version of rtmpdump (compiled statically).
Sample from terminal:

~/rtmpdump $ ./rtmpdump -r "rtmp://128.97.46.13:1935/gss2012/" -a "gss2012/" -f "LNX 11,0,1,152" -W "https://www.ipam.ucla.edu/videoplayer/jwplayer.flash.swf" -p "https://www.ipam.ucla.edu/wowzavideo.aspx?vfn=10660.mp4&vfd=gss2012" -y "mp4:10660.mp4" -o mp4_10660.flv
RTMPDump v2.4
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Connected...
Starting download at: 0.000 kB
INFO: Metadata:
INFO: trackinfo:
INFO:   timescale             30000.00
INFO:   length                98732634.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            avc1
INFO:   timescale             48000.00
INFO:   length                157974528.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            mp4a
INFO:   audiochannels         2.00
INFO:   audiosamplerate       48000.00
INFO:   videoframerate        29.97
INFO:   aacaot                2.00
INFO:   avclevel              40.00
INFO:   avcprofile            100.00
INFO:   audiocodecid          mp4a
INFO:   videocodecid          avc1
INFO:   width                 1920.00
INFO:   height                1080.00
INFO:   frameWidth            1920.00
INFO:   frameHeight           1080.00
INFO:   displayWidth          1920.00
INFO:   displayHeight         1080.00
INFO:   framerate             29.97
INFO:   moovposition          32.00
INFO:   duration              3291.14
11077.478 kB / 25.96 sec (0.7%)
INFO: Metadata:
INFO: trackinfo:
INFO:   timescale             30000.00
INFO:   length                98732634.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            avc1
INFO:   timescale             48000.00
INFO:   length                157974528.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            mp4a
INFO:   audiochannels         2.00
INFO:   audiosamplerate       48000.00
INFO:   videoframerate        29.97
INFO:   aacaot                2.00
INFO:   avclevel              40.00
INFO:   avcprofile            100.00
INFO:   audiocodecid          mp4a
INFO:   videocodecid          avc1
INFO:   width                 1920.00
INFO:   height                1080.00
INFO:   frameWidth            1920.00
INFO:   frameHeight           1080.00
INFO:   displayWidth          1920.00
INFO:   displayHeight         1080.00
INFO:   framerate             29.97
INFO:   moovposition          32.00
INFO:   duration              3291.14
19733.950 kB / 21.86 sec (0.6%)

[aborted by user]
Then it might go up to 0.9% before going down to 0.8% again and so on, stuttering every other promille or so.
And here is the file that resulted from that run:
https://mega.co.nz/#!Vd1TRZoI!XEzGC11hOU1tu98k_x988UmptUAOwYBHX4-l4LOFXVE

The video stops after the first few seconds in the flash player, so it's not even feasible to watch it that way and I would really like to have a local copy anyway as I expect to rewatch this. rtmpdump does seem to be able to access all the material though. I left it overnight and it got up to ~25% before I stopped it (that percentage is plausible looking at the content at the end, but the sound is badly out of sync by then and all the repeats give an everpresent and unpleasant feeling of déjà vu, so that file is pretty much unusable)

Could you tell me what the problem is and point me to towards a way to fix it?

[bat999]

Could you tell me what the problem is and point me to towards a way to fix it?

Hi
Some streams will only download in 'real-time'.
Try adding --live to your command.

[etiam]

Try adding --live to your command.

Thank you bat999! That's considerably better.
No more starting over with --live (or --realtime, for that matter) added to the command.


That unmasked another problem though: The first minute or so works fine, but then the audio stays fine and the picture starts skipping frames. It looks a bit like a quick slide show of a photo sequence.
Here's a video sample:
https://mega.co.nz/#!FJli3TbJ!GsUe9gPYrSawNWC7SsiGT3b8TGM5XPW2t0rTQkv9q7s
(The slideshow-esque part starts around 1:10.)
I'm not sure about the reason here either, but it seems that dropping frames is a natural response if the bandwidth can't keep up with the transfer, and I seem to be having a surprisingly slow connection to this, so there's a strong candidate for why if you ask me.
Same question again then, in this inner layer of the onion of live stream downloading issues. Do you know what might be the cause, and is there something I can do to compensate for it?

(And should I have started a new thread for this followup?)

[medico]

Use -v dude!

[etiam]

Thanks, -v would be a more compact way to write --live , then

That works, but now that bat999 had pointed me in the right direction some of rtmpdump's manpage entries make a bit more sense:

--realtime -R
Download approximately in realtime, without attempting to speed up via Pause/Unpause commands
("the BUFX hack"). Useful for servers that jump backwards in time at the Unpause command.
Resuming and seeking in realtime streams is still possible.

That last line probably means -R is an even better choice in many cases. Whether I use -v or -R though, it solves the 'starting over-problem' and unmasks the 'slideshow problem'. I've been struggling with that new issue for a couple of days, but I'm not getting anywhere with it, so I'm starting a new thread here.

EDIT: Appearantly, no I'm not. According to the wishes of the MEGA Super Moderator I'm continuing right here in this thread.

So, to recap:

The issue I had with the download restarting could be resolved by adding either the --live or --realtime flag to the command, and I can now get a file that includes the full length of the video. The resulting file seems to work fine during approximately the first 70 seconds (consistent over many retries). After that, the audio keeps working well, but the picture basically turns into a sequence of still images, updated about once per second. It looks a bit like a slide show of a photo sequence.

So, again, I'm trying to download this video:
https://www.ipam.ucla.edu/wowzavideo...p4&vfd=gss2012

...and here's a short video sample of the result:
https://mega.co.nz/#!FJli3TbJ!GsUe9g...PW2t0rTQkv9q7s
(The problem starts around 1:10.)

I'm on Linux Mint 13, using the git repository version of rtmpdump from the 10th of April 2013 (compiled statically).
The first minute of a run would look something like this:

 ~/rtmpdump/rtmpdump --realtime -r "rtmp://128.97.46.13:1935/gss2012/" -a "gss2012/" -f "LNX 11,0,1,152" -W "https://www.ipam.ucla.edu/videoplayer/jwplayer.flash.swf" -p "https://www.ipam.ucla.edu/wowzavideo.aspx?vfn=10660.mp4&vfd=gss2012" -y "mp4:10660.mp4" -o mp4_10660.flv 
RTMPDump v2.4
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Connected...
Starting download at: 0.000 kB
  in approximately realtime (disabled BUFX speedup hack)
INFO: Metadata:
INFO: trackinfo:
INFO:   timescale             30000.00
INFO:   length                98732634.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            avc1
INFO:   timescale             48000.00
INFO:   length                157974528.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            mp4a
INFO:   audiochannels         2.00
INFO:   audiosamplerate       48000.00
INFO:   videoframerate        29.97
INFO:   aacaot                2.00
INFO:   avclevel              40.00
INFO:   avcprofile            100.00
INFO:   audiocodecid          mp4a
INFO:   videocodecid          avc1
INFO:   width                 1920.00
INFO:   height                1080.00
INFO:   frameWidth            1920.00
INFO:   frameHeight           1080.00
INFO:   displayWidth          1920.00
INFO:   displayHeight         1080.00
INFO:   framerate             29.97
INFO:   moovposition          32.00
INFO:   duration              3291.14
3138.303 kB / 12.31 sec (0.3%)

A typical data transfer rate I get seems to be in the range of 30-60 kB/s. My ISP is providing a symmetric 10 Mbps connection and I would assume that UCLA has plenty of capacity at their end. I'm located in Europe.
Some search engine work seems to suggest that the server could have a more or less fixed rate for outputting the video and this low transfer rate isn't coping with that, so frames get dropped to stay in sync. That's just my best guess so far though, and I don't have much experience with these issues.

Could you tell me the what's causing this problem and if there's something I can do to fix it?

[CristianoA]

Thanks, -v would be a more compact way to write --live , then

That works, but now that bat999 had pointed me in the right direction some of rtmpdump's manpage entries make a bit more sense:

--realtime -R
Download approximately in realtime, without attempting to speed up via Pause/Unpause commands
("the BUFX hack"). Useful for servers that jump backwards in time at the Unpause command.
Resuming and seeking in realtime streams is still possible.

That last line probably means -R is an even better choice in many cases. Whether I use -v or -R though, it solves the 'starting over-problem' and unmasks the 'slideshow problem'. I've been struggling with that new issue for a couple of days, but I'm not getting anywhere with it, so I'm starting a new thread here.

EDIT: Appearantly, no I'm not. According to the wishes of the MEGA Super Moderator I'm continuing right here in this thread.

So, to recap:

The issue I had with the download restarting could be resolved by adding either the --live or --realtime flag to the command, and I can now get a file that includes the full length of the video. The resulting file seems to work fine during approximately the first 70 seconds (consistent over many retries). After that, the audio keeps working well, but the picture basically turns into a sequence of still images, updated about once per second. It looks a bit like a slide show of a photo sequence.

So, again, I'm trying to download this video:
https://www.ipam.ucla.edu/wowzavideo...p4&vfd=gss2012

...and here's a short video sample of the result:
https://mega.co.nz/#!FJli3TbJ!GsUe9g...PW2t0rTQkv9q7s
(The problem starts around 1:10.)

I'm on Linux Mint 13, using the git repository version of rtmpdump from the 10th of April 2013 (compiled statically).
The first minute of a run would look something like this:

 ~/rtmpdump/rtmpdump --realtime -r "rtmp://128.97.46.13:1935/gss2012/" -a "gss2012/" -f "LNX 11,0,1,152" -W "https://www.ipam.ucla.edu/videoplayer/jwplayer.flash.swf" -p "https://www.ipam.ucla.edu/wowzavideo.aspx?vfn=10660.mp4&vfd=gss2012" -y "mp4:10660.mp4" -o mp4_10660.flv 
RTMPDump v2.4
(c) 2010 Andrej Stepanchuk, Howard Chu, The Flvstreamer Team; license: GPL
Connecting ...
INFO: Connected...
Starting download at: 0.000 kB
  in approximately realtime (disabled BUFX speedup hack)
INFO: Metadata:
INFO: trackinfo:
INFO:   timescale             30000.00
INFO:   length                98732634.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            avc1
INFO:   timescale             48000.00
INFO:   length                157974528.00
INFO:   language              eng
INFO: sampledescription:
INFO:   sampletype            mp4a
INFO:   audiochannels         2.00
INFO:   audiosamplerate       48000.00
INFO:   videoframerate        29.97
INFO:   aacaot                2.00
INFO:   avclevel              40.00
INFO:   avcprofile            100.00
INFO:   audiocodecid          mp4a
INFO:   videocodecid          avc1
INFO:   width                 1920.00
INFO:   height                1080.00
INFO:   frameWidth            1920.00
INFO:   frameHeight           1080.00
INFO:   displayWidth          1920.00
INFO:   displayHeight         1080.00
INFO:   framerate             29.97
INFO:   moovposition          32.00
INFO:   duration              3291.14
3138.303 kB / 12.31 sec (0.3%)

A typical data transfer rate I get seems to be in the range of 30-60 kB/s. My ISP is providing a symmetric 10 Mbps connection and I would assume that UCLA has plenty of capacity at their end. I'm located in Europe.
Some search engine work seems to suggest that the server could have a more or less fixed rate for outputting the video and this low transfer rate isn't coping with that, so frames get dropped to stay in sync. That's just my best guess so far though, and I don't have much experience with these issues.

Could you tell me the what's causing this problem and if there's something I can do to fix it?

i can download it using tubedigger download speed 400-500 KB/s

[etiam]

i can download it using tubedigger download speed 400-500 KB/s

So using rtmpdump is not required for this case. That's good to know.
I dusted off an old Windows VM and tried tubedigger as well. I get lower transfer rates ( less than 160 kB/s, and with lots of variation ), but it does seem to do the trick.
With that, my practical problem is solved. I can get the video downloaded.
If someone knows what's going on with rtmpdump or a workaround for the problem, I would encourage them to leave a note about it though. I'll report back myself if I get it sorted out.
Thank you for your help!

[filippo.rosa]

is there any news?
I have the exact same issue.
for now I use rtmpdump 2.1, but not without problems.

[etiam]

Hi filippo.

Unfortunately, as far as I know, this is still an unresolved issue.
It turned out that on closer inspection the Tubedigger downloads suffered from the slideshow issue as well.
I have tried with just about every common open source, freeware, shareware and trialware tool I could, and at least under my use they all fall short, most in similar ways. And I've had the problem verified by a friend in another location, so this seems to go beyond both rtmpdump and the specific qualities of my connection.

The least damaged download results so far I got by using cURL (which I believe handles rtmp streams by using librtmp).
That may be due to cURL handling the stream a bit less deficiently than the other tools, or it may be due to chance.
In either case, those results aren't good enough either.
If you do find a solution I would be very grateful to hear it.

[CristianoA]

try use Flash Stream Hunter

[etiam]

try use Flash Stream Hunter

Thanks for the suggestion. Unfortunately recordings of these videos using Flash Stream Hunter "finish" after only a few seconds worth of video for me.

[filippo.rosa]

try this:

/rtmpdump-2.1 $ ./rtmpdump -m 20 -r "rtmp://128.97.46.13:1935/gss2012/" -y "mp4:10660.mp4" -q -o - | avconv -i - -qscale 0 -copyts -async 1 /home/user/Videos/graduate2012.mp4

it isn't perfect but is the best I have found out.
works only with rtmpdump v 2.0/2.1.
pay attention to the file size!

let me know if you find something better!

[megaics]

Is there a way to fix an old .mp4 file downloaded without the --live flag? I tried with ffmpeg -i old_file.mp4 new_file.mp4 and the video doesn't jump back but the audio DOES jump back.

[filippo.rosa]

Have you tried with the avconv command:
avconv -i old_file.mp4 -qscale 0 -copyts -async 1 new_file.mp4 ?


I don't know if this will be helpful, but this version works for me (doesn't store the same part several times), but sometimes it crashes so I recommend run by:

#!/bin/bash

until ./rtmpdump -r "..." -y "..." -o filename -e
do
sleep 10
done

[aazerty]

Hi megaics ,

Generally for knowning which streams are inclued in the video : ffmpeg -i video.!!!

ffmpeg -i ZZZ_3rd_graduate2012.ts

ffmpeg version N-58869-gae33007 Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec  7 2013 22:01:45 with gcc 4.8.2 (GCC)

  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnu
tls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --ena
ble-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger -
-enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-li
bvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 56.100 / 52. 56.100
  libavcodec     55. 45.100 / 55. 45.100
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     3. 92.100 /  3. 92.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, mpegts, from 'ZZZ_3rd_graduate2012.ts':
  Duration: 00:01:03.66, start: 1.456978, bitrate: 597 kb/s
  Program 1
    Metadata:
      service_name    : Service01
      service_provider: FFmpeg
    Stream #0:0[0x100]: Video: h264 (High) ([27][0][0][0] / 0x001B), yuv420p, 960x556 [SAR 139:135 DAR 16:9], 25 fps, 25 tbr, 90k tbn, 50 tbc
    Stream #0:1[0x101]: Audio: mp3 ([3][0][0][0] / 0x0003), 48000 Hz, stereo, s16p, 128 kb/s

You can watch :
_ Stream #0:0
_ Stream #0:1

in other cases :
_Stream #0:'any number'

EXAMPLE You'll have to select :
first the video and write -map 0:0 ( 0:9 )
second the audio and write -map 0:1 ( 1:10 )

in the commands-line add to indicate your need : -map 0:0 -map 0:1
=====================

Now with your specific link : I have done
_ vcodec libx264
_ size 960x556 ( the original is 1920x1080 )
_ r 25 ( unstead 29,97 fps )
_ audio mp3

3rd try
=======

rtmpdump -m 20 -r "rtmp://128.97.46.13:1935/gss2012/" -y "mp4:10660.mp4" -q -o -   | ffmpeg -i - -vcodec libx264 -acodec libmp3lame
 -r 25 -async 1 -vf scale=960:556 -aspect 16:9 -f mpegts "ZZZ_3rd_graduate2012.ts"

ffmpeg version N-58869-gae33007 Copyright (c) 2000-2013 the FFmpeg developers
  built on Dec  7 2013 22:01:45 with gcc 4.8.2 (GCC)

  configuration: --enable-gpl --enable-version3 --disable-w32threads --enable-avisynth --enable-bzlib --enable-fontconfig --enable-frei0r --enable-gnu
tls --enable-iconv --enable-libass --enable-libbluray --enable-libcaca --enable-libfreetype --enable-libgsm --enable-libilbc --enable-libmodplug --ena
ble-libmp3lame --enable-libopencore-amrnb --enable-libopencore-amrwb --enable-libopenjpeg --enable-libopus --enable-librtmp --enable-libschroedinger -
-enable-libsoxr --enable-libspeex --enable-libtheora --enable-libtwolame --enable-libvidstab --enable-libvo-aacenc --enable-libvo-amrwbenc --enable-li
bvorbis --enable-libvpx --enable-libwavpack --enable-libx264 --enable-libxavs --enable-libxvid --enable-zlib
  libavutil      52. 56.100 / 52. 56.100
  libavcodec     55. 45.100 / 55. 45.100
  libavformat    55. 22.100 / 55. 22.100
  libavdevice    55.  5.102 / 55.  5.102
  libavfilter     3. 92.100 /  3. 92.100
  libswscale      2.  5.101 /  2.  5.101
  libswresample   0. 17.104 /  0. 17.104
  libpostproc    52.  3.100 / 52.  3.100
Input #0, flv, from 'pipe:':
  Metadata:
    audiochannels   : 2
    videoframerate  : 30
    aacaot          : 2
    avclevel        : 40
    avcprofile      : 100
    frameWidth      : 1920
    frameHeight     : 1080
    displayWidth    : 1920
    displayHeight   : 1080
    moovposition    : 32
  Duration: 00:54:51.14, start: 0.000000, bitrate: N/A
    Stream #0:0: Video: h264 (High), yuv420p(tv, bt709), 1920x1080 [SAR 1:1 DAR 16:9], 29.97 tbr, 1k tbn, 59.94 tbc
    Stream #0:1: Audio: aac, 48000 Hz, stereo, fltp
-async is forwarded to lavfi similarly to -af aresample=async=1:min_hard_comp=0.100000:first_pts=0.
[libx264 @ 031d3220] using SAR=139/135
[libx264 @ 031d3220] using cpu capabilities: MMX2 SSE2Fast SSSE3 SSE4.2 LZCNT BMI1
[libx264 @ 031d3220] profile High, level 3.1
Output #0, mpegts, to 'ZZZ_3rd_graduate2012.ts':
  Metadata:
    audiochannels   : 2
    videoframerate  : 30
    aacaot          : 2
    avclevel        : 40
    avcprofile      : 100
    frameWidth      : 1920
    frameHeight     : 1080
    displayWidth    : 1920
    displayHeight   : 1080
    moovposition    : 32
    encoder         : Lavf55.22.100
    Stream #0:0: Video: h264 (libx264), yuv420p, 960x556 [SAR 139:135 DAR 16:9], q=-1--1, 90k tbn, 25 tbc
    Stream #0:1: Audio: mp3 (libmp3lame), 48000 Hz, stereo, fltp
Stream mapping:
  Stream #0:0 -> #0:0 (h264 -> libx264)
  Stream #0:1 -> #0:1 (aac -> libmp3lame)
frame= 1545 fps=2.3 q=28.0 size=    4297kB time=00:01:01.63 bitrate= 571.2kbits/s dup=1 drop=2279
...
Not finished !!!

Cheers .

[etiam]

let me know if you find something better!

Now I have. Completely undamaged, as far as I can tell.
The crucial step was arranging for a better connection. I paid for some running time on a server close to the source and used a plain curl command from there.

Not exactly an elegant software solution, but it has the considerable merit of being the only thing that has really worked so far.

[etiam]

P.S. Despite my workaround, if you find a solution to the original problem, especially with rtmpdump, please consider sharing it for the benefit of the community.

[ponens]

P.S. Despite my workaround, if you find a solution to the original problem, especially with rtmpdump, please consider sharing it for the benefit of the community.

efiam, as alternative to rtmp you may use hls or hds.
Examples

livestreamer "hlsvariant://http://128.97.46.13:1935/CP2015/12507.mp4/playlist.m3u8" best -o 12507.ts

adobehds.php --manifest "http://128.97.46.13:1935/CP2015/12507.mp4/manifest.f4m"

[filippo.rosa]

P.S. Despite my workaround, if you find a solution to the original problem, especially with rtmpdump, please consider sharing it for the benefit of the community.

it will be done.
at the moment, I'm using the PauseBreak version of rtmpdump (see my previous post) and it's all good.