Need help downloading this vimeo with youtube-dl

[jagabo]

I've been wanting to learn a little about mp4 file structure for a while. This seemed like a good excuse to start. I wrote a program to walk through the MP4 header chunks, find the stsd chunk, display the Profile@Level, and clear the constraints flags.

I've seen one MOV file that had a different size stsd chunk. All the MP4 files and all the other MOV files had the same size chunk (unless they weren't AVC). I've seen a few where the two "profile-level-id" flags differ from each other. And some where the constraint byte has different bits set. I'm not sure if it's appropriate to clear the entire contraint byte or just the 0x08 bit. If you want to try the program out I'll upload a copy for you guys to test. It does an in-place change (and hence is very fast) so be sure to back up your video before proceeding.

[ponens]

Sounds useful. Yes, upload.

[jimdagys]

Jagabo,
Can you post the download link for that software. I might need directions on how to use it. I want to try it on this video (discussed in this thread) and see if it plays in my media player.

[jagabo]

The program is pretty crude for now but it seems to work. It takes a filename on the command line:

ClearConstraints filename.mp4

You can run the program from the command line or drag/drop files onto the EXE. The file must be an mp4 or mov file to be parsed correctly. If it finds an stsd chunk with an avc1 header it asks if you want to clear the entire constraints byte. If you answer y it will set the entire byte to 0. If you answer n it will clear only the 08 bit (other bits will remain unchanged -- if only the 08 bit is set these two operations are identical). If you answer anything else it will not change the constraint byte. This is done for both constraint bytes. When done you have to press any key to exit the program.

Sample log:

C:\>ClearConstraints filename.mp4

Found stsd chunk at offset 0x00768755 (7767893).
Found avc1 header.

first profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : y
Clearing entire constraint byte (0x08 --> 0x00).

second profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : n
Clearing constraint bit 08 (0x08 --> 0x00).



Found stsd chunk at offset 0x0076ac57 (7777367).
Not avc1 header, skipping.

Press any key to exit...

Let me know if you find files for which this doesn't work. Once thek kinks have been worked out I'll clean it up a bit and make a final version available.

<edit>
Outdated file attachment removed.
</edit>

[ponens]

Thanks.

[jimdagys]

I'm using Windows 7 netbook. Software doesn't seem to work.

C:\youtube-dl>ClearConstraints original.mp4
This version of C:\youtube-dl\ClearConstraints.exe is not compatible with the ve
rsion of Windows you're running. Check your computer's system information to see
whether you need a x86 (32-bit) or x64 (64-bit) version of the program, and the
n contact the software publisher.

[jagabo]

The program requires 64 bit Windows. I'll see if I can build a 32 bit version.

[jimdagys]

The program requires 64 bit Windows. I'll see if I can build a 32 bit version.

Thank you.

[jagabo]

Thanks.

I saw your earlier version of this post. Do you no longer think it's useful to allow custom changes to the constraint byte? And the Level byte? I was originally thinking about allowing one to change the Profile, constraint, and Level bytes but it's a bit of a mess figuring out which constraint bits are valid/needed for each Profile@level.

The program requires 64 bit Windows. I'll see if I can build a 32 bit version.

Thank you.

32 bit version attached. It should work with any size file.

I'm aware of a few problems at this time:

1) Make sure you keep an unmodified copy of your file since I'm working somewhat blindly here. It's always possible the program will make a change that makes the video unplayable.

1) The avc1 header varies in size. I have at least one file which has a header too short for the second constraint bit. Do not change the second (or first, for that matter) constraint bit if the Profile shows "????" or the Level has an unusual value. I plan to add a check for this so it will be automatic.

2) I don't know exactly what combinations of Profile, Constraints, and Level are valid. Clearing the entire Constraints byte, or even just the 08 bit, may make a file unplayable under some circumstances. Buyer beware!

<edit>
Outdated file attachment removed.
</edit>

[jimdagys]

Ok, your software (for my Windows 7 netbook) to play the video on the media player works perfectly. However, the video (downloaded with Tubedigger) must first be run through the Tubedigger convert tool, mp4 to mp4, (or this also works):
ffmpeg -i original.mp4 -c copy original1.mp4
I am a little confused about answering y/n on the second question. I tried it both ways (yes and no) and both played in the media player.
The strange thing, whether you answer yes or no to the second question, the command output is exactly the same. See below:

C:\youtube-dl>Mp4AvcConstraintChanger32 original[2].mp4

Found stsd chunk at offset 0x000001b9, length=0x009d.
Found avc1 header.

first profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : y
Clearing entire constraint byte (0x08 --> 0x00).

second profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : y
Clearing entire constraint byte (0x08 --> 0x00).

Found stsd chunk at offset 0x000b9193, length=0x0067.
Not avc1 header, skipping.
Press any key to exit...

C:\youtube-dl>Mp4AvcConstraintChanger32 original[2].mp4

Found stsd chunk at offset 0x000001b9, length=0x009d.
Found avc1 header.

first profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : y
Clearing entire constraint byte (0x08 --> 0x00).

second profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : n
Clearing constraint bit 08 (0x08 --> 0x00).

Found stsd chunk at offset 0x000b9193, length=0x0067.
Not avc1 header, skipping.
Press any key to exit...

[jagabo]

There are two constraint bytes in most headers (I don't know why, and sometimes they're different). The first question is for the first byte, the second for the second byte.

A byte is 8 bits. A bit can have the values 0 and 1 (sometimes called clear and set, off and on, etc.). The value of a byte can be represented in several way including decimal (0 to 255), binary, octal, hex, ascii text, etc. (depending on what's convenient in different contexts).

https://www.rapidtables.com/code/text/ascii-table.html

As noted before, if only the 08 bit is set answering y and n give the same result. You would see a different result if more than just the 08 bit is set. For example, if the constraint byte was 48 hex (72 decimal, 01001000 binary, the letter H as ascii text) answering y would result in 00 (hex), answering n would result in 40 (hex). If the constraint bit had the value 11111111 (FF hex, 255 decimal) clearing only the 00001000 bit (08 hex and decimal) would result in 11110111 (F7 hex, 247 decimal).

Why does the Tubedigger downloaded file need to be remuxed? Does it play as-is in a media player on the PC? Is it really an MP4 file (what does MediaInfo say about it)? Online videos are often stored as TS files (transport stream). So maybe the downloaded video is really a TS file (just misnamed with an MP4 extension).

[jimdagys]

Why does the Tubedigger downloaded file need to be remuxed? Does it play as-is in a media player on the PC? Is it really an MP4 file (what does MediaInfo say about it)?

I mentioned this in above post 5:

The 720p version of the video that I got from Tubedigger (probably the same as the video from youtube-dl)
won't play with Media Player Classic [Home Cinema]1.6.3.5818 (2320902). The video will play with VLC. If I run the video through this ffmpeg:
Code:
ffmpeg -i video.mp4 -c copy video1.mp4
or use the Tubedigger convert tool,
then the video will play with Media Player Classic.

Also, Ponens mentioned something about this in above post 29:

Assuming that "original.mp4" is the native mp4 (1546591581.mp4) from the "http-720p" source on the vimeo server, and not a "mp4" crafted by tubedigger or youtube-dl from the dash or hls sources.

And in above post 7:

If the dash version was downloaded, ffmpeg will "fix the headers" via codec copy.
There is a mp4 version that should not have this problem.

youtube-dl -f http-720p https://player.vimeo.com/video/372238757
The url for the mp4 can also be found by viewing the source of the vimeo player page.

Also, the original video from Tubedigger appears on the netbook desktop as a file, not a video, but after going through ffmpeg (or Tubedigger convert tool) then it appears as a video thumbnail. (see below screenshot).

[ponens]

I hadn't yet checked tubedigger for this particular video.

With the version of tubedigger that I just tried (v6.5.5), the tubedigger download list does include the http progressive mp4 version. This version is a single file, non-segmented mp4. It is the same as the file that exists on the vimeo server as 1546591581.mp4.

Tubedigger downloads the http progressive mp4 version exactly the same as if the url to the video were downloaded within the browser, or with curl, or with IDM. The md5 is ee703ecf0b3e78954f41163a48c7de78.

And Mp4AvcConstraintChanger32 works directly on the http progressive mp4 version, without any pre-processing of the file required.

It might be a different story for the hls versions. Or if newer versions of tubedgger also list the dash versions.

[jagabo]

Using View Page Source in FireFox I got this link and downloaded the file:

https://gcs-vimeo.akamaized.net/exp=1576736075~acl=%2A%2F1546591581.mp4%2A~hmac=5c989e...1546591581.mp4

You can right click on that link and select "Save Link As..." to download it. I got a 568 MB 720p mp4 file. Is that the file you are talking about? Does Tubedigger download that exact file without changing it? From my program I get:

Found stsd chunk at offset 0x000001ec, length=0x00b0.
Found avc1 header.

first profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : n
Clearing constraint bit 08 (0x08 --> 0x00).

second profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : n
Clearing constraint bit 08 (0x08 --> 0x00).

Found stsd chunk at offset 0x00099042, length=0x0067.
Not avc1 header, skipping.

Press any key to exit...

Do you get the same thing? And after doing that -- the file still doesn't play in your standalone player?

[ponens]

Does Tubedigger download that exact file without changing it?

Yes, if properly selected from the tubedigger list, the exact same file is downloaded.

And after running Mp4AvcConstraintChanger32 on such file, the patched file will play directly on a "tv box player" using the MStar MSD7802 cpu (the same cpu as the Ematic AT103B).

Separate and apart from all of the above, for whatever reason, it seems that jimdagys' windows 7 installation, and program installation of "Media Player Classic" (with however file associations are configured) does not recognize the vimeo server file 1546591581.mp4 as directly playable with Media Player Classic (as configured). Perhaps something about the mp4 is not recognized by Media Player Classic.

But the exact vimeo server file (1546591581.mp4) is directly patchable with Mp4AvcConstraintChanger32, and the patched file is directly playable on a MStar MSD7802 based tv box player (like the Ematic AT103B).

[jimdagys]

With the version of tubedigger that I just tried (v6.5.5), the tubedigger download list does include the http progressive mp4 version. This version is a single file, non-segmented mp4.

Now I see the problem. I am using Tubedigger v6.6.1 (not the source of the problem). The issue is that I normally choose "Chromium 3 (Android") in the Tubedigger browser settings. The downloads list I get (see below screenshot) are very different from your screenshot (thank you for posting the screenshot). So in this thread starting from post#1, I downloaded entry #9 (dash-720mp4). If I switch to IE in the Tubedigger browser settings, then the downloads list is very similar to your screenshot. In terms of downloading, I never really understood terms hls, http, dash, etc, so I didn't realize it might be better to download the http entry. However, since I didn't use the IE browser setting, the http version wasn't even listed. One other thing worth noting: When using Tubedigger (either discussed browser setting), when I just hover the mouse over the video, all the download entries appear. Clicking the "play" arrow in the video and playing the video (in the Tubedigger browser), does not add any additional download entries. There might be some advantage for me in this because my internet is so bad, that sometimes I have difficulties getting the stream video to play.

[jagabo]

Now I see the problem. I am using Tubedigger v6.6.1 (not the source of the problem). The issue is that I normally choose "Chromium 3 (Android") in the Tubedigger browser settings. The downloads list I get (see below screenshot) are very different from your screenshot (thank you for posting the screenshot). So in this thread starting from post#1, I downloaded entry #9 (dash-720mp4).

I'm still curios about why that video didn't play. I don't have TubeDigger so I don't know what file it's downloading. Can you make the file available somewhere. Maybe I can adjust my program to handle the other problem too.

[jimdagys]

The original video doesn't play in my version (1.6.3.5818) of Media Player Classic Home Cinema (MPCHC). I get error message. See below screenshot. The video does play in vlc. I can't upload a 20 second segment of the video using ffmpeg:

ffmpeg -i largefile.mp4 -t 00:00:20 -c copy smallfile1.mp4 -ss 00:00:20 -c copy smallfile2.mp4

because that segment (after going through ffmpeg) will play in MPCHC. You can download the latest version of Tubedigger and use for free (downloads a complete 50% of the file). Make sure you choose "Chromium 3 (Android)" in Tubedigger browser settings , and choose the 720p dash version to try to duplicate what I did. It is also possible that the Ematic media player won't play the "patched video" with your software for the same reason that MPCHC won't play the original video (bad headers?) which ffmpeg will fix. So the patched video not playing in the media player may have nothing to do with your software.

[ponens]

I'm still curios about why that video didn't play. I don't have TubeDigger so I don't know what file it's downloading. Can you make the file available somewhere. Maybe I can adjust my program to handle the other problem too.

The dash version of the video does play for jimdagys with vlc. See post #5:

The video will play with VLC

You can download the dash version of the video with youtube-dl.

Whether with tubedigger or youtube-dl, the full dash video is assembled by the mere sequential binary concatenation of the fragmented segments.

Something about the dash segmentation process introduces additional data into each individual dash segment that remains if all of the segments are merely concatenated.

Apparently, vlc is able to fully recognize mp4 files that were crafted though the mere sequential binary concatenation of fragmented dash segments (the dash init file + one or more of the segments).

Some player applications will have trouble with these files, either by a delay in playback or inability to playback.

For example, I have an older version of SMplayer [Version: 0.8.5 (32-bit)]. This player will have a delay before stating to playback a mp4 that was produced through the mere sequential binary concatenation of fragmented dash segments.

youtube-dl.exe "https://intheamericas.org/works/810-colombias-guardians-of-the-homelands/" -f dash-akfire_interconnect_quic-video-1546591586

The above downloads the 426x240 dash video version (74.5MB). Compare the downloaded video and a copy of the video processed with ffmpeg's -vcodec copy. This will give some idea of the additional data that is present in the merely concatenated file. Note however that the same issue applies to dash audio.

[ponens]

So in this thread starting from post#1, I downloaded entry #9 (dash-720mp4).

As you noted, if you download the dash version, either with tubedigger or youtube-dl, you will have to run the downloaded mp4 through a ffmpeg step in order to "fix the headers". My understanding is that "-vcodec copy -acodec copy" (rather than "-c:v copy -c:a copy" or "-c copy") is the preferred process, but apparently "-c copy" is working for you.

If you download the hls version, "-vcodec copy -acodec copy" plus the aac-bitstream filter "-bsf:a aac_adtstoasc" should be used if copying from .ts to a mp4 container, unless tubedigger does this automatically.

Your initial question in this thread was about how to download the 720p version with youtube-dl. Some of the initial replies suggested using "-f" parameters for youtube-dl that are for the hls or dash sources.

But in post #7, I suggested using "http-720p" as the "-f" parameter with youtube-dl in order to download the the http progressive mp4 version. This does work.

But if you're able to use tubedigger in IE mode, and select the http progressive mp4 source, this works just as well.

[jimdagys]

Ok, your patch is working directly on the dash stream played in the Ematic video player. Dash stream that Ponens gave:

youtube-dl.exe "https://intheamericas.org/works/810-colombias-guardians-of-the-homelands/" -f dash-akfire_interconnect_quic-video-1546591586

However, one must run the ffmpeg either before or after your patch is applied. This is because the dash video won't play in the Ematic unless it goes through ffmpeg at some point.
If the above youtube-dl video download is called ponens, then these are the results of playing or not in the Ematic.
ponens - no
ponens, ffmpeg -no
ponens, ffmpeg, patch -yes
ponens, patch -no
ponens, patch, ffmpeg -yes

[jagabo]

youtube-dl.exe "https://intheamericas.org/works/810-colombias-guardians-of-the-homelands/" -f dash-akfire_interconnect_quic-video-1546591586

The above downloads the 426x240 dash video version (74.5MB). Compare the downloaded video and a copy of the video processed with ffmpeg's -vcodec copy. This will give some idea of the additional data that is present in the merely concatenated file. Note however that the same issue applies to dash audio.

Looking at the chunk structure:

downloaded:

ftyp offset==0000 size=0018
moov offset==0018 size=04f5
    mvhd offset==0020 size=006c
    trak offset==008c size=0208
        tkhd offset==0094 size=005c
        edts offset==00f0 size=0024
            elst offset==00f8 size=001c
        mdia offset==0114 size=0180
            mdhd offset==011c size=0020
            hdlr offset==013c size=002d
            minf offset==0169 size=012b
                vmhd offset==0171 size=0014
                dinf offset==0185 size=0024
                    dref offset==018d size=001c
                stbl offset==01a9 size=00eb
                    stsd offset==01b1 size=009f
                    stts offset==0250 size=0010
                    stsc offset==0260 size=0010
                    stsz offset==0270 size=0014
                    stco offset==0284 size=0010
    trak offset==0294 size=01cf
        tkhd offset==029c size=005c
        edts offset==02f8 size=0024
            elst offset==0300 size=001c
        mdia offset==031c size=0147
            mdhd offset==0324 size=0020
            hdlr offset==0344 size=002d
            minf offset==0371 size=00f2
                smhd offset==0379 size=0010
                dinf offset==0389 size=0024
                    dref offset==0391 size=001c
                stbl offset==03ad size=00b6
                    stsd offset==03b5 size=006a
                    stts offset==041f size=0010
                    stsc offset==042f size=0010
                    stsz offset==043f size=0014
                    stco offset==0453 size=0010
    mvex offset==0463 size=0048
    udta offset==04ab size=0062
sidx offset==050d size=0034
sidx offset==0541 size=0034
moof offset==0575 size=0b54
mdat offset==10c9 size=4612e
sidx offset==471f7 size=0034
sidx offset==4722b size=0034
moof offset==4725f size=0b54
mdat offset==47db3 size=4a4ab
sidx offset==9225e size=0034
sidx offset==92292 size=0034
moof offset==922c6 size=0b50
mdat offset==92e16 size=47749
sidx offset==da55f size=0034
sidx offset==da593 size=0034
moof offset==da5c7 size=0b54
mdat offset==db11b size=461c5

(and many more sidx,sidx,moof,mdat chunks...)

remuxed:

ftyp offset==0000 size=0020
free offset==0020 size=0008
mdat offset==0028 size=49c6f38
moov offset==49c6f60 size=133683
    mvhd offset==49c6f68 size=006c
    trak offset==49c6fd4 size=e9b2c
        tkhd offset==49c6fdc size=005c
        edts offset==49c7038 size=0030
            elst offset==49c7040 size=0028
        mdia offset==49c7068 size=e9a98
            mdhd offset==49c7070 size=0020
            hdlr offset==49c7090 size=002d
            minf offset==49c70bd size=e9a43
                vmhd offset==49c70c5 size=0014
                dinf offset==49c70d9 size=0024
                    dref offset==49c70e1 size=001c
                stbl offset==49c70fd size=e9a03
                    stsd offset==49c7105 size=009f
                    stts offset==49c71a4 size=0018
                    stss offset==49c71bc size=0a54
                    ctts offset==49c7c10 size=57b00
                    stsc offset==4a1f710 size=3d798
                    stsz offset==4a5cea8 size=2f024
                    stco offset==4a8becc size=24c34
    trak offset==4ab0b00 size=49a81
        tkhd offset==4ab0b08 size=005c
        edts offset==4ab0b64 size=0024
            elst offset==4ab0b6c size=001c
        mdia offset==4ab0b88 size=499f9
            mdhd offset==4ab0b90 size=0020
            hdlr offset==4ab0bb0 size=002d
            minf offset==4ab0bdd size=499a4
                smhd offset==4ab0be5 size=0010
                dinf offset==4ab0bf5 size=0024
                    dref offset==4ab0bfd size=001c
                stbl offset==4ab0c19 size=49968
                    stsd offset==4ab0c21 size=006a
                    stts offset==4ab0c8b size=0028
                    stsc offset==4ab0cb3 size=0028
                    stsz offset==4ab0cdb size=24c3c
                    stco offset==4ad5917 size=24c34
                    sgpd offset==4afa54b size=001a
                    sbgp offset==4afa565 size=001c
    udta offset==4afa581 size=0062

The major difference is the mdat chunk (all the audio and video data) appears as many small segments in the download file, one single segment after remuxing with ffmpeg. I don't know if this is related to the dash segmentation or just the way the original file was produced.

And what you can't see here is the ftyp subtype of the former is "iso5" and the latter "isom". I tried changing the 5 to an m -- the video still played in all the players I tried. It's right at the beginning of the file:


[Attachment 51161 - Click to enlarge]

Could you try changing that and see if it makes any difference with the set-top box? You may still have to change the Constraint byte with my program. If that works I can modify my program to change it along with the constraint bit.

[ponens]

I don't know if this is related to the dash segmentation or just the way the original file was produced.

See the bento4 family of tools.

mp4fragment: creates a fragmented MP4 file from a non-fragmented one
mp4split: splits a fragmented MP4 file into discrete files
mp4dash: creates an MPEG DASH output from one or more MP4 files

For dash streaming, at minimum a fragmented (but still a single) file is required. Splitting the single fragmented file into multiple discrete files is optional but common.

For more on fMP4, See also; 1, 2, 3, 4

And what you can't see here is the ftyp subtype of the former is "iso5" and the latter "isom". I tried changing the 5 to an m -- the video still played in all the players I tried.

Could you try changing that and see if it makes any difference ...

Changing that byte resulted in SMPlayer (version that I mentioned previously) having a 3-5 second delay before starting playback, to not being able to play the file at all.

[jagabo]

But changing that byte resulted in SMPlayer (version that I mentioned previously) from have a 3-5 second delay before starting playback, to not being able to play the file at all. Vlc played both the non-edited and edited versions just fine.

I had no problems with SMPlayer (2017 version). MPCBE and MPCHC also had no problems. VLC played the patched video but did not seek correctly -- but it had the same problem with the original download. Some other players had problems with both the original and patched file. I guess remuxing with ffmpeg is going to be required, so I won't change iso5 to isom in my program. I'll upload a cleaner version (more checks for error conditions) tomorrow.

[ponens]

ffmpeg isn't required at all for this series of videos if the http progressive mp4 version is downloaded instead of the dash or hls version.

Of course other videos not mentioned in this thread may have only dash and/or hls available.

But for all of these "In The Americas with David Yetman" videos on vimeo, just download the http progressive mp4 version. Then use Mp4AvcConstraintChanger32 if need be. Done and done. It doesn't have to be any more complicated than that.

As I was checking some of these videos, I happened to notice the recent episode (809) about the colorado river which is quite well done.

[jagabo]

Of course other videos not mentioned in this thread may have only dash and/or hls available.

Yes, that it why i was considering whether it was feasible or not to just patch the subtype. It looks like those videos will have to be remuxed anyway (many players seem to have problems with them) so there's no point in patching the subtype.

[jagabo]

Here's a version with a little more internal error handling (messages displayed when obvious errors are detected, eg. not an mp4/mov file). Let me know if you thing the UI should be different. Or if you find any files where it doesn't work.