VideoHelp Forum

Try DVDFab and download streaming video, copy, convert or make Blu-rays,DVDs! Download free trial !
+ Reply to Thread
Results 1 to 12 of 12
Thread
  1. Member
    Join Date
    Oct 2008
    Location
    United States
    Search Comp PM
    I have searched high and low and have been unable to locate a definition for the mpl files on HD AVCHD format camcorders. Does anyone have a definition or file format spec for these playlist files which are located at BDMV\Playlist\*.mpl in the AVCHD structure. I have opened it in hex and can ID some patterns, but a spec would help me with parsing the files to determine which mts files should be treated as a single video.

    Thanks,

    Mike
    Mike Strange
    Chief Solutions Designer
    Strange Solutions, LLC
    www.gotstrange.com
    Quote Quote  
  2. Member
    Join Date
    Nov 2008
    Location
    United States
    Search Comp PM
    I've been searching fruitlessly as well. I did find this useful link:
    http://www.blu-raydisc.com/Assets/Downloadablefile/BD-RE_Part3_V2.1_WhitePaper_080406-15271.pdf
    ... which provides some good conceptual background. But as yet haven't found details
    on the file's format/structure.

    It looks to me something like:
    - A Header of 45 bytes (content not yet understood)
    - Clip Name Section, 1 entry for each clip, 82 bytes each
    - Unknown section with 1 entry per clip, 14 bytes each, (In and Out point info?)
    - A 274 byte section with Date at beginning. This may also include per clip segments,
    but my camera fills these with all 0's, so I can deduce nothing.
    - Date section, 1 entry per clip, 66 bytes each
    - Trailer, 82 bytes (content not yet understood)

    If I learn anything else I'll post it here.
    Tom
    Quote Quote  
  3. Member
    Join Date
    Nov 2008
    Location
    United States
    Search Comp PM
    Oops. That was a header size of 69 bytes ( 0x45 (hex)).
    Quote Quote  
  4. Member
    Join Date
    Jan 2009
    Location
    Sweden
    Search Comp PM
    While not a spec, this might be enough for your needs: http://eiman.tv/misc/flashdump.txt

    It's some hacky Python code I wrote that extracts the relevant data from a playlist. It probably only works properly on playlists generated by my Canon HF100, but adjusting for other cameras should be easy enough.

    Sample output:

    edith:~/ mikaeleiman$ ./flashdump.py f2/00000.MPL
    Parsing f2/00000.MPL
    File list with 2 files
    File: 00000M2TS (00000, spanning: False) ['01', '00', '00', '00', '97', '2c', '03', '87', 'ff', '0c']
    File: 00001M2TS (00001, spanning: True) ['06', '00', '03', '87', 'ff', '0c', '06', '17', '8c', 'ec']
    unknown1 has 1 items
    PLEX has 1 items, date 2009-01-11 16:02:29
    CA index 0 date 2009-01-11 15:24:38 (files: ['00000', '00001'])

    edith:~/ mikaeleiman$ ./flashdump.py /Volumes/CANON/PRIVATE/AVCHD/BDMV/PLAYLIST/00000.MPL
    Parsing /Volumes/CANON/PRIVATE/AVCHD/BDMV/PLAYLIST/00000.MPL
    File list with 3 files
    File: 00000M2TS (00000, spanning: False) ['01', '00', '00', '00', '97', '2c', '02', 'c3', '8f', '8c']
    File: 00001M2TS (00001, spanning: True) ['06', '00', '02', 'c3', '8f', '8c', '05', '84', '8d', 'ac']
    File: 00002M2TS (00002, spanning: True) ['06', '00', '05', '84', '8d', 'ac', '07', '10', 'b8', '6c']
    unknown1 has 1 items
    PLEX has 1 items, date 2009-01-13 15:16:31
    CA index 0 date 2009-01-13 14:32:37 (files: ['00000', '00001', '00002'])
    Quote Quote  
  5. Member
    Join Date
    Nov 2008
    Location
    United States
    Search Comp PM
    m_eiman:
    Where did you learn what things like "PLEX", "CA", where to find data rate, that you have to split bytes, and so on?
    t
    Quote Quote  
  6. Member
    Join Date
    Jan 2009
    Location
    Austria
    Search Comp PM
    well this is not exactly the topic, but since i got some ideas for my script from here, i believe it's my duty to share it with you.

    My Camcorder: Canon HG-20

    The Story: since i wasn't very happy with this filie linked (xxxxx.mts <-> xxxxx.cpi -> xxxxx.mpl) solution to retrieve the recording date/time i was searching for another way to manage the task of finding this bit of information. After playing around a little bit with the software that came with my cam (pixela imagemixer 3 se) i noticed that it was able to retrieve the recording date/time directly from the mts files (without even the presence of a mpl file). So it was perfectly clear to me that this info has to be somewhere hidden in the stream. Firing up my hexeditor and after doing some probing it took not more than 5 minutes to see where. The result was the script i wrote (attached). I know it's not perfect, and not the most speedy one, but it's just meant as proof of concept. I tried to enable as many options as possible to allow it to be adapted for other cams too. I verified it's function against the mts files of my canon hg-20 and also against the files of a canon hf-100 and hf-10.

    sincerely martin

    canon_analyze_mts.zip
    Quote Quote  
  7. hi

    I cannot d/l from the link from above, but would be very interested...
    "You don't have permission to access /images/guides/p1933128/canon_analyze_mts.pl.zip on this server."

    Could you plz publish your tool again?

    Thanks
    Joey
    Quote Quote  
  8. Member
    Join Date
    Nov 2008
    Location
    United States
    Search Comp PM
    Yeah! I get the same error! If not too large, could you just post the code in a reply? Thanks!!!
    In the meantime, it feels like an apache configuration issue. I'm gonna click "Report this post" on the contribution and see if the webmaster can help.
    t
    Quote Quote  
  9. I'm a MEGA Super Moderator Baldrick's Avatar
    Join Date
    Aug 2000
    Location
    Sweden
    Search Comp PM
    Testing uploading it again

    canon_analyze_mts.zip

    edit: It should work now. The filename .pl caused some problems.
    Quote Quote  
  10. Thanks I could d/l now.

    Regards
    Joey
    Quote Quote  
  11. I am writing this as a summary of what i have learned in the last few weeks.

    As a start the brief story is that a family member has a Canon videocamera that had its memory card reformatted with lots of video still on it - as their local IT support person i was able to recover most of the video (mts files) and the Clip info (cpi) however the mpl file was damaged and not recoverable. Also they use the Canon supplied software, VideoBrowser, which is a real POS, but that is another topic. If you are using the VideoBrowser software you will find that it checks that all the default folders on a Canon formated memory card are there (ie PRIVATE\AVCHD\...). Without these folders it will tell you there is nothing to import. Next you need your mps files in the PRIVATE\AVCHD\BDMV\STREAM folder, and the cpi in the PRIVATE\AVCHD\BDMV\CLIPINF folder. Also as a side note it checks that the folder (ie PRIVATE\AVCHD\...) exists on a removable drive, originally i was trying to put the files on a USB hard drive which doesn't appear as removable and VideoBrowser wouldn't detect it, so if you are trying to recover into the software use a Flash memory device.
    If you are using another video management software (which i would recommend) you probably really only need the mts files, if you have the cpi I would keep them as well since they are very small. However if you are to get them into VideoBrowser they need to be imported. I found no way for it to detect mts/cpi files that are just copied into your hard drive! My original plan was just to copy the mts/cpi into the 'My Videos\Canon\20121121\' and let it syncronize with VideoBrowser - no luck.
    So the only way to get the videos into VideoBrowser is to import them, and the only way to import them is to have a mpl file at PRIVATE\AVCHD\BDMV\PLAYLIST usually a name like 00000.mpl. Now this file is a proprietary or at least unpublished format, so there is no documentation officially available on how to re-create this file. After finding a few resources on the internet, this is what i found out;

    -First get yourself a Hex Editor, something like the Free Hex Editor XVI32 (http://www.chmaas.handshake.de/), this will allow you to look at the structure and make modifications. If you don't have a mpl file to start with, i suggest you get one, even if it has nothing to do with your videos. I think its too involved and beyond the scope of things here to recreate one from scrach. I took 3 small videos on the same videocamera and used that mpl as a basis. It would be great to post some examples of the mpl files to demostrate the changes but it would take to much space. So here's the non-visual attempt; As stated before the structure is roughly;
    - [Header] - A Header of 69 bytes - Contains information about the number of clips and pointer addresses to other Sections
    - [Section A] - Clip Name Section, 1 entry for each clip, 82 bytes each
    - [Section B] - Unknown section with 1 entry per clip, 14 bytes each, (In and Out point info?), I believe this contains clip sequence information
    - [Section C] - A 274 byte section with Date at beginning. This may also include per clip segments, but my camera fills these with all 0's, so I can deduce nothing.
    - [Section D] - Date section, 1 entry per clip, 66 bytes each
    - [Section E] - Trailer, 82 bytes (content not yet understood)
    My objective was to take my existing 3 clip mpl file and expand it to import the 135 recovered video files. I started this by copying 1 clip information section (82b) and pasting it 132 times (135-3 clips). In my mpl the 82b clip looks like (in ASCII): 0P00000M2TS...1und, so the 3 clips would look like; 0P00000M2TS...1und0P00001M2TS...1und0P00002M2TS... 1und... - Also note that these are the file names so actually my files started off 00074.mts, so in this case i can either rename my file 00000.mts or in the mpl change the first entry from 00000M2TS to 00073M2TS.

    The next step is to do the same for Section B, the clip (in hex) looks like: 00 00 00 00 00 P# 00 C# 00 00 XX XX FF FF, so the 3 clips look like 00 00 00 00 00 01 00 01 00 00 XX XX FF FF 00 00 00 00 00 01 00 02 00 00 XX XX FF FF 00 00 00 00 00 01 00 03 00 00 XX XX FF FF, the XX are values that i don't know what they mean, but they don't seem to impact the import process. The P# I think is the playlist number, so for one playlist , ie all clips will just be 01, while the C# increments 1 for each clip, so (and remember you are working in hex for a lot of this file), 01, 02, 03,..., 09, 0A, 0B, 0C, 0D, 0E, 0F, 10, 11,... So i started like the clip section by selecting the 14 bytes and copying it 132 times. We will come back to renaming the CC values.
    The last section to do copying is Section D - Again if you look at the mpl file in the hex editor, towards the end of the file you should see the Date which looks like: [FF FF 2A YY YY M D H M S 91 ...] which is 66b long, the YY YY will be the Year, M - Month, D - Day, H - Hour M - Minute, S - Second, so you should have something like: [FF FF 2A 20 12 11 29 23 27 01 91 ...] and this is the timestamp of the video.

    Because normally the timestamp of the video's are recorded in the mpl file, if you lose that then the question arises how do you find out the timestamp of your videos? The information is not recorded in the cpi files but it is imbedded in the mts files. Go and get ExifToolGUI which is the GUI frontend for ExifTool which can read the mts and display the timestamp, this information can then be entered in your recreated mpl file.
    So you select the first FF of the Date section for one clip and tell the Hex Editor to block the next 66b, copy that section, then paste another 132 times.
    So after all this you have an mpl file which has all the clip sections but it still needs to have the correct information. In Section A as mentioned your actual mts file names need to match the cpi names and the name in the mpl file, ie if your mts file starts off 00188.mts, then the cpi needs to be 00188.cpi and in the mpl the first clip needs to be 0P00188M2TS. So go through Section A and make sure all your clip filenames, cpi filenames match what you have in the mpl file.
    Next in Section B you need to make sure that all the P# are equal to 01, this should be the case, next you will need to make sure that the first C# is 00 and then the next is 01 so that they augment by 1, remember as shown before these augment in hex values. The first C# actually might be a 01 but you should see the pattern in your mpl file and ajust as necessary.
    In Section D is probably the longest section, you must enter the proper time in the mpl file. the first time clip corresponds to the first filename, and hense the first mts video as you would expect. So i needed to use ExifToolGui to find the timestamp of each video and enter it in the mpl file, depending on how many files you have its easy to make a typo.
    ok - at this point you should have all the correct information in your mpl file. If you try to import the files you will probably find that VideoBrowser throws out an error saying it cannot find any files to import or that it was created by another camera, that it is non compliant, etc. Next is a little bit the structure of the file.

    At hex address E and F (in the Header) is the Pointer to Section B, so in your file you should find the hex address of the last part of section A, ie 0P00002M2TS...1und[This is the address you want], so lets say u is addess 1427h, then n is 1428, d is 1429, and Section B starts ar 142A(hex) so in address E you would put 14 and F would be 2A. Next at address 12 and 13 is the Pointer to Section C, look at 5th address after the last FF FF in Section B, ie: XX XX FF FF 00 00 00 00 [This is the address you want]. At this point it often helps to look at an orginal working copy of a mpl file to see these addesses and pointers.
    Next you will need to modify address 41, again in the header to reflect the number of clips you have, this number needs to be in hex. so if all you have is 3 then the value is 03, however if you have 20 then it will be 14h (20d=14h).
    The number of clips also needs to be entered in 2 other addresses. The first is in Section B and will be the 5th address after the pointer you entered in address E & F, so if you entered 142A, the 5 addresses after would be 142f where you would enter the number of clips in hex (for our example of 20 clips it would be 14h). Finally look at the beginning of Section D just before the first date entry for a pattern: C# 10 11 32 03 00 00 00 00 FF FF 2A YY, the C# address is again where you should enter the number of clips in hex.
    I believe after that you should be able to import you video files properly into VideoBrowser. Again i recommend you test on a small number of small video files to test the procedure before tackling a large number of big files. It certainly helped me to have a working mpl/mts/cpi structure that i could convert to test the changes. Also please note that this is not a guarenteed method, this is just what worked for me and might not work for you. In addition I am writing this from my scribbled notes from a few weeks ago, because now that i have been able to load the files i hope never to have to touch this again, and i would like to point out this oversight of basing so much importance on one file that whatever group and or companies that decided they were going to keep this information private that you see the failings in this post!

    Finally I hope that this helps someone so after searching Google will find a bit more into then I found and might build upon it. It is more than likely someone can create a small program to automate this to rebuild a mpl file based on the mts files.
    As a last note I just want to relate that after actually finishing the mpl and trying to import the video files i had, that VideoBrowser would only inport like 2 files and then stop, since with my testing i know that it could import at least 3-5 files with the testing i did before. So after some more testing i tracked it down to the cpi files that when i replaced the recovered cpi files with the test files then it started to import more but again it would stop after a certain number. i finally opened the problem cpi files in the hex editor to find that some of the cpi files had been damaged as well. I don't know how useful or what they add to Videobrowser (if anything at all) but i replaced the damaged ones with copies of good ones that i had recovered and renamed them to the name of the corresponding mts file. After replacing a considerable number of cpi files the import completed.
    Quote Quote  
  12. [Section B] - chapter info section with 1 entry per chapter, 14 bytes each chapter
    example
    unsigned AnsiChar data[14] = {
    0x00, 0x00, 0x00, 0x00, 0x00, 0x01, 0x00, 0x02, 0xXX, 0xXX, 0xXX, 0xXX, 0xFF, 0xFF
    };
    0xXX, 0xXX, 0xXX, 0xXX = YYYYYY (in dec) = chapter's first GOP minPTS32 value
    Quote Quote  



Similar Threads