I've been wanting to learn a little about mp4 file structure for a while. This seemed like a good excuse to start. I wrote a program to walk through the MP4 header chunks, find the stsd chunk, display the Profile@Level, and clear the constraints flags.
I've seen one MOV file that had a different size stsd chunk. All the MP4 files and all the other MOV files had the same size chunk (unless they weren't AVC). I've seen a few where the two "profile-level-id" flags differ from each other. And some where the constraint byte has different bits set. I'm not sure if it's appropriate to clear the entire contraint byte or just the 0x08 bit. If you want to try the program out I'll upload a copy for you guys to test. It does an in-place change (and hence is very fast) so be sure to back up your video before proceeding.
Try StreamFab Downloader and download from Netflix, Amazon, Youtube! Or Try DVDFab and copy Blu-rays! or rip iTunes movies!
Try StreamFab Downloader and download streaming video from Youtube, Netflix, Amazon! Download free trial.
+ Reply to Thread
Results 31 to 57 of 57
Thread
-
-
Sounds useful. Yes, upload.
Last edited by ponens; 17th Dec 2019 at 18:20.
-
Jagabo,
Can you post the download link for that software. I might need directions on how to use it. I want to try it on this video (discussed in this thread) and see if it plays in my media player. -
The program is pretty crude for now but it seems to work. It takes a filename on the command line:
Code:ClearConstraints filename.mp4
Sample log:
Code:C:\>ClearConstraints filename.mp4 Found stsd chunk at offset 0x00768755 (7767893). Found avc1 header. first profile (64 08 20) High@3.2 Clear entire constraint byte? (y/n) : y Clearing entire constraint byte (0x08 --> 0x00). second profile (64 08 20) High@3.2 Clear entire constraint byte? (y/n) : n Clearing constraint bit 08 (0x08 --> 0x00). Found stsd chunk at offset 0x0076ac57 (7777367). Not avc1 header, skipping. Press any key to exit...
<edit>
Outdated file attachment removed.
</edit>Last edited by jagabo; 20th Dec 2019 at 17:52.
-
I'm using Windows 7 netbook. Software doesn't seem to work.
C:\youtube-dl>ClearConstraints original.mp4
This version of C:\youtube-dl\ClearConstraints.exe is not compatible with the ve
rsion of Windows you're running. Check your computer's system information to see
whether you need a x86 (32-bit) or x64 (64-bit) version of the program, and the
n contact the software publisher. -
The program requires 64 bit Windows. I'll see if I can build a 32 bit version.
-
The program requires 64 bit Windows. I'll see if I can build a 32 bit version.
-
I saw your earlier version of this post. Do you no longer think it's useful to allow custom changes to the constraint byte? And the Level byte? I was originally thinking about allowing one to change the Profile, constraint, and Level bytes but it's a bit of a mess figuring out which constraint bits are valid/needed for each Profile@level.
32 bit version attached. It should work with any size file.
I'm aware of a few problems at this time:
1) Make sure you keep an unmodified copy of your file since I'm working somewhat blindly here. It's always possible the program will make a change that makes the video unplayable.
1) The avc1 header varies in size. I have at least one file which has a header too short for the second constraint bit. Do not change the second (or first, for that matter) constraint bit if the Profile shows "????" or the Level has an unusual value. I plan to add a check for this so it will be automatic.
2) I don't know exactly what combinations of Profile, Constraints, and Level are valid. Clearing the entire Constraints byte, or even just the 08 bit, may make a file unplayable under some circumstances. Buyer beware!
<edit>
Outdated file attachment removed.
</edit>Last edited by jagabo; 20th Dec 2019 at 17:53.
-
Ok, your software (for my Windows 7 netbook) to play the video on the media player works perfectly. However, the video (downloaded with Tubedigger) must first be run through the Tubedigger convert tool, mp4 to mp4, (or this also works):
ffmpeg -i original.mp4 -c copy original1.mp4
I am a little confused about answering y/n on the second question. I tried it both ways (yes and no) and both played in the media player.
The strange thing, whether you answer yes or no to the second question, the command output is exactly the same. See below:
C:\youtube-dl>Mp4AvcConstraintChanger32 original[2].mp4
Found stsd chunk at offset 0x000001b9, length=0x009d.
Found avc1 header.
first profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : y
Clearing entire constraint byte (0x08 --> 0x00).
second profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : y
Clearing entire constraint byte (0x08 --> 0x00).
Found stsd chunk at offset 0x000b9193, length=0x0067.
Not avc1 header, skipping.
Press any key to exit...C:\youtube-dl>Mp4AvcConstraintChanger32 original[2].mp4
Found stsd chunk at offset 0x000001b9, length=0x009d.
Found avc1 header.
first profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : y
Clearing entire constraint byte (0x08 --> 0x00).
second profile (64 08 20) High@3.2
Clear entire constraint byte? (y/n) : n
Clearing constraint bit 08 (0x08 --> 0x00).
Found stsd chunk at offset 0x000b9193, length=0x0067.
Not avc1 header, skipping.
Press any key to exit... -
There are two constraint bytes in most headers (I don't know why, and sometimes they're different). The first question is for the first byte, the second for the second byte.
A byte is 8 bits. A bit can have the values 0 and 1 (sometimes called clear and set, off and on, etc.). The value of a byte can be represented in several way including decimal (0 to 255), binary, octal, hex, ascii text, etc. (depending on what's convenient in different contexts).
https://www.rapidtables.com/code/text/ascii-table.html
As noted before, if only the 08 bit is set answering y and n give the same result. You would see a different result if more than just the 08 bit is set. For example, if the constraint byte was 48 hex (72 decimal, 01001000 binary, the letter H as ascii text) answering y would result in 00 (hex), answering n would result in 40 (hex). If the constraint bit had the value 11111111 (FF hex, 255 decimal) clearing only the 00001000 bit (08 hex and decimal) would result in 11110111 (F7 hex, 247 decimal).
Why does the Tubedigger downloaded file need to be remuxed? Does it play as-is in a media player on the PC? Is it really an MP4 file (what does MediaInfo say about it)? Online videos are often stored as TS files (transport stream). So maybe the downloaded video is really a TS file (just misnamed with an MP4 extension).Last edited by jagabo; 18th Dec 2019 at 19:28.
-
Why does the Tubedigger downloaded file need to be remuxed? Does it play as-is in a media player on the PC? Is it really an MP4 file (what does MediaInfo say about it)?
The 720p version of the video that I got from Tubedigger (probably the same as the video from youtube-dl)
won't play with Media Player Classic [Home Cinema]1.6.3.5818 (2320902). The video will play with VLC. If I run the video through this ffmpeg:
Code:
ffmpeg -i video.mp4 -c copy video1.mp4
or use the Tubedigger convert tool,
then the video will play with Media Player Classic.
Assuming that "original.mp4" is the native mp4 (1546591581.mp4) from the "http-720p" source on the vimeo server, and not a "mp4" crafted by tubedigger or youtube-dl from the dash or hls sources.
If the dash version was downloaded, ffmpeg will "fix the headers" via codec copy.
There is a mp4 version that should not have this problem.youtube-dl -f http-720p https://player.vimeo.com/video/372238757
The url for the mp4 can also be found by viewing the source of the vimeo player page.Last edited by jimdagys; 18th Dec 2019 at 22:10.
-
I hadn't yet checked tubedigger for this particular video.
With the version of tubedigger that I just tried (v6.5.5), the tubedigger download list does include the http progressive mp4 version. This version is a single file, non-segmented mp4. It is the same as the file that exists on the vimeo server as 1546591581.mp4.
Tubedigger downloads the http progressive mp4 version exactly the same as if the url to the video were downloaded within the browser, or with curl, or with IDM. The md5 is ee703ecf0b3e78954f41163a48c7de78.
And Mp4AvcConstraintChanger32 works directly on the http progressive mp4 version, without any pre-processing of the file required.
It might be a different story for the hls versions. Or if newer versions of tubedgger also list the dash versions. -
Using View Page Source in FireFox I got this link and downloaded the file:
https://gcs-vimeo.akamaized.net/exp=1576736075~acl=%2A%2F1546591581.mp4%2A~hmac=5c989e...1546591581.mp4
You can right click on that link and select "Save Link As..." to download it. I got a 568 MB 720p mp4 file. Is that the file you are talking about? Does Tubedigger download that exact file without changing it? From my program I get:
Code:Found stsd chunk at offset 0x000001ec, length=0x00b0. Found avc1 header. first profile (64 08 20) High@3.2 Clear entire constraint byte? (y/n) : n Clearing constraint bit 08 (0x08 --> 0x00). second profile (64 08 20) High@3.2 Clear entire constraint byte? (y/n) : n Clearing constraint bit 08 (0x08 --> 0x00). Found stsd chunk at offset 0x00099042, length=0x0067. Not avc1 header, skipping. Press any key to exit...
-
Yes, if properly selected from the tubedigger list, the exact same file is downloaded.
And after running Mp4AvcConstraintChanger32 on such file, the patched file will play directly on a "tv box player" using the MStar MSD7802 cpu (the same cpu as the Ematic AT103B).
Separate and apart from all of the above, for whatever reason, it seems that jimdagys' windows 7 installation, and program installation of "Media Player Classic" (with however file associations are configured) does not recognize the vimeo server file 1546591581.mp4 as directly playable with Media Player Classic (as configured). Perhaps something about the mp4 is not recognized by Media Player Classic.
But the exact vimeo server file (1546591581.mp4) is directly patchable with Mp4AvcConstraintChanger32, and the patched file is directly playable on a MStar MSD7802 based tv box player (like the Ematic AT103B). -
With the version of tubedigger that I just tried (v6.5.5), the tubedigger download list does include the http progressive mp4 version. This version is a single file, non-segmented mp4.
Last edited by jimdagys; 19th Dec 2019 at 07:03.
-
-
The original video doesn't play in my version (1.6.3.5818) of Media Player Classic Home Cinema (MPCHC). I get error message. See below screenshot. The video does play in vlc. I can't upload a 20 second segment of the video using ffmpeg:
Code:ffmpeg -i largefile.mp4 -t 00:00:20 -c copy smallfile1.mp4 -ss 00:00:20 -c copy smallfile2.mp4
Last edited by jimdagys; 19th Dec 2019 at 14:30.
-
The dash version of the video does play for jimdagys with vlc. See post #5:
You can download the dash version of the video with youtube-dl.
Whether with tubedigger or youtube-dl, the full dash video is assembled by the mere sequential binary concatenation of the fragmented segments.
Something about the dash segmentation process introduces additional data into each individual dash segment that remains if all of the segments are merely concatenated.
Apparently, vlc is able to fully recognize mp4 files that were crafted though the mere sequential binary concatenation of fragmented dash segments (the dash init file + one or more of the segments).
Some player applications will have trouble with these files, either by a delay in playback or inability to playback.
For example, I have an older version of SMplayer [Version: 0.8.5 (32-bit)]. This player will have a delay before stating to playback a mp4 that was produced through the mere sequential binary concatenation of fragmented dash segments.
Code:youtube-dl.exe "https://intheamericas.org/works/810-colombias-guardians-of-the-homelands/" -f dash-akfire_interconnect_quic-video-1546591586
-
As you noted, if you download the dash version, either with tubedigger or youtube-dl, you will have to run the downloaded mp4 through a ffmpeg step in order to "fix the headers". My understanding is that "-vcodec copy -acodec copy" (rather than "-c:v copy -c:a copy" or "-c copy") is the preferred process, but apparently "-c copy" is working for you.
If you download the hls version, "-vcodec copy -acodec copy" plus the aac-bitstream filter "-bsf:a aac_adtstoasc" should be used if copying from .ts to a mp4 container, unless tubedigger does this automatically.
Your initial question in this thread was about how to download the 720p version with youtube-dl. Some of the initial replies suggested using "-f" parameters for youtube-dl that are for the hls or dash sources.
But in post #7, I suggested using "http-720p" as the "-f" parameter with youtube-dl in order to download the the http progressive mp4 version. This does work.
But if you're able to use tubedigger in IE mode, and select the http progressive mp4 source, this works just as well. -
Ok, your patch is working directly on the dash stream played in the Ematic video player. Dash stream that Ponens gave:
Code:youtube-dl.exe "https://intheamericas.org/works/810-colombias-guardians-of-the-homelands/" -f dash-akfire_interconnect_quic-video-1546591586
If the above youtube-dl video download is called ponens, then these are the results of playing or not in the Ematic.
ponens - no
ponens, ffmpeg -no
ponens, ffmpeg, patch -yes
ponens, patch -no
ponens, patch, ffmpeg -yes -
Looking at the chunk structure:
downloaded:
Code:ftyp offset==0000 size=0018 moov offset==0018 size=04f5 mvhd offset==0020 size=006c trak offset==008c size=0208 tkhd offset==0094 size=005c edts offset==00f0 size=0024 elst offset==00f8 size=001c mdia offset==0114 size=0180 mdhd offset==011c size=0020 hdlr offset==013c size=002d minf offset==0169 size=012b vmhd offset==0171 size=0014 dinf offset==0185 size=0024 dref offset==018d size=001c stbl offset==01a9 size=00eb stsd offset==01b1 size=009f stts offset==0250 size=0010 stsc offset==0260 size=0010 stsz offset==0270 size=0014 stco offset==0284 size=0010 trak offset==0294 size=01cf tkhd offset==029c size=005c edts offset==02f8 size=0024 elst offset==0300 size=001c mdia offset==031c size=0147 mdhd offset==0324 size=0020 hdlr offset==0344 size=002d minf offset==0371 size=00f2 smhd offset==0379 size=0010 dinf offset==0389 size=0024 dref offset==0391 size=001c stbl offset==03ad size=00b6 stsd offset==03b5 size=006a stts offset==041f size=0010 stsc offset==042f size=0010 stsz offset==043f size=0014 stco offset==0453 size=0010 mvex offset==0463 size=0048 udta offset==04ab size=0062 sidx offset==050d size=0034 sidx offset==0541 size=0034 moof offset==0575 size=0b54 mdat offset==10c9 size=4612e sidx offset==471f7 size=0034 sidx offset==4722b size=0034 moof offset==4725f size=0b54 mdat offset==47db3 size=4a4ab sidx offset==9225e size=0034 sidx offset==92292 size=0034 moof offset==922c6 size=0b50 mdat offset==92e16 size=47749 sidx offset==da55f size=0034 sidx offset==da593 size=0034 moof offset==da5c7 size=0b54 mdat offset==db11b size=461c5 (and many more sidx,sidx,moof,mdat chunks...)
Code:ftyp offset==0000 size=0020 free offset==0020 size=0008 mdat offset==0028 size=49c6f38 moov offset==49c6f60 size=133683 mvhd offset==49c6f68 size=006c trak offset==49c6fd4 size=e9b2c tkhd offset==49c6fdc size=005c edts offset==49c7038 size=0030 elst offset==49c7040 size=0028 mdia offset==49c7068 size=e9a98 mdhd offset==49c7070 size=0020 hdlr offset==49c7090 size=002d minf offset==49c70bd size=e9a43 vmhd offset==49c70c5 size=0014 dinf offset==49c70d9 size=0024 dref offset==49c70e1 size=001c stbl offset==49c70fd size=e9a03 stsd offset==49c7105 size=009f stts offset==49c71a4 size=0018 stss offset==49c71bc size=0a54 ctts offset==49c7c10 size=57b00 stsc offset==4a1f710 size=3d798 stsz offset==4a5cea8 size=2f024 stco offset==4a8becc size=24c34 trak offset==4ab0b00 size=49a81 tkhd offset==4ab0b08 size=005c edts offset==4ab0b64 size=0024 elst offset==4ab0b6c size=001c mdia offset==4ab0b88 size=499f9 mdhd offset==4ab0b90 size=0020 hdlr offset==4ab0bb0 size=002d minf offset==4ab0bdd size=499a4 smhd offset==4ab0be5 size=0010 dinf offset==4ab0bf5 size=0024 dref offset==4ab0bfd size=001c stbl offset==4ab0c19 size=49968 stsd offset==4ab0c21 size=006a stts offset==4ab0c8b size=0028 stsc offset==4ab0cb3 size=0028 stsz offset==4ab0cdb size=24c3c stco offset==4ad5917 size=24c34 sgpd offset==4afa54b size=001a sbgp offset==4afa565 size=001c udta offset==4afa581 size=0062
And what you can't see here is the ftyp subtype of the former is "iso5" and the latter "isom". I tried changing the 5 to an m -- the video still played in all the players I tried. It's right at the beginning of the file:
[Attachment 51161 - Click to enlarge]
Could you try changing that and see if it makes any difference with the set-top box? You may still have to change the Constraint byte with my program. If that works I can modify my program to change it along with the constraint bit.Last edited by jagabo; 19th Dec 2019 at 18:23.
-
See the bento4 family of tools.
mp4fragment: creates a fragmented MP4 file from a non-fragmented one
mp4split: splits a fragmented MP4 file into discrete files
mp4dash: creates an MPEG DASH output from one or more MP4 files
For dash streaming, at minimum a fragmented (but still a single) file is required. Splitting the single fragmented file into multiple discrete files is optional but common.
For more on fMP4, See also; 1, 2, 3, 4
Changing that byte resulted in SMPlayer (version that I mentioned previously) having a 3-5 second delay before starting playback, to not being able to play the file at all.Last edited by ponens; 19th Dec 2019 at 22:24.
-
I had no problems with SMPlayer (2017 version). MPCBE and MPCHC also had no problems. VLC played the patched video but did not seek correctly -- but it had the same problem with the original download. Some other players had problems with both the original and patched file. I guess remuxing with ffmpeg is going to be required, so I won't change iso5 to isom in my program. I'll upload a cleaner version (more checks for error conditions) tomorrow.
-
ffmpeg isn't required at all for this series of videos if the http progressive mp4 version is downloaded instead of the dash or hls version.
Of course other videos not mentioned in this thread may have only dash and/or hls available.
But for all of these "In The Americas with David Yetman" videos on vimeo, just download the http progressive mp4 version. Then use Mp4AvcConstraintChanger32 if need be. Done and done. It doesn't have to be any more complicated than that.
As I was checking some of these videos, I happened to notice the recent episode (809) about the colorado river which is quite well done. -
-
Here's a version with a little more internal error handling (messages displayed when obvious errors are detected, eg. not an mp4/mov file). Let me know if you thing the UI should be different. Or if you find any files where it doesn't work.
Similar Threads
-
Downloading Private/Password Video on Livestream/Vimeo
By AZSloth in forum Video Streaming DownloadingReplies: 9Last Post: 18th Aug 2019, 08:57 -
Downloading Interview - Youtube
By devilcoelhodog in forum Video Streaming DownloadingReplies: 4Last Post: 11th Aug 2019, 11:42 -
Downloading Just Audio From Youtube
By cornemuse in forum AudioReplies: 6Last Post: 11th Aug 2019, 11:39 -
Everything to know about Downloading Vimeo Videos
By glober in forum Video Streaming DownloadingReplies: 2Last Post: 22nd Dec 2018, 08:02 -
difficulty downloading streaming embedded vimeo?
By estidog in forum Video Streaming DownloadingReplies: 6Last Post: 27th Oct 2017, 10:18