SOLVED mp4 Bigendian smpte st 337 frames detected 1536 kbps cbr, Unplayable

[compscistudent]

MediaInfo detects "SMPTE ST 337" far into the file(skipping the header) and even detects the running time and frames, but no codecs work. I tried ffdshow, quicktime, tsdoctor, vlc, mpc-hc, and a few others. Strangely, vlc starts to play counting the time with no video or audio. there is data through the entire file. I attached about 10mb around 100mb into the file and the header with almost the first 5mb. (Source of file was an mp4 file downloaded 2015-04-10).

More information:
From Potplayer(regardless of header in file):

Media Type 0:
--------------------------
Unknown

[ AM_MEDIA_TYPE ]
+ majortype: MEDIATYPE_Stream {E436EB83-524F-11CE-9F53-0020AF0BA770}
+ subtype: Unknown GUID Name {08E22ADA-B715-45ED-9D20-7B87750301D4}
+ formattype: TIME_FORMAT_NONE {00000000-0000-0000-0000-000000000000}
+ bFixedSizeSamples: 1
+ bTemporalCompression: 0
+ lSampleSize: 1
+ cbFormat: 0

First 8 bytes: 0000001c 66747970
Readable text: 00s between the text: ftypmp42 mp42isomavc1 free mdat x264 - core 120 - H.264/MPEG-4 AVC codec - Copyleft 2003-2011 - http://www.videolan.org/x264.html - options: cabac=1 ref=3 deblock=1:0:0 analyse=0x3:0x113 me=hex subme=7 psy=1 psy_rd=1.00:0.00 mixed_ref=1 me_range=16 chroma_me=1 trellis=1 8x8dct=1 cqm=0 deadzone=21,11 fast_pskip=1 chroma_qp_offset=-2 threads=6 sliced_threads=0 nr=0 decimate=1 interlaced=0 bluray_compat=0 constrained_intra=0 bframes=3 b_pyramid=2 b_adapt=2 b_bias=0 direct=1 weightb=1 open_gop=0 weightp=2 keyint=300 keyint_min=30 scenecut=40 intra_refresh=0 rc_lookahead=50 rc=crf mbtree=1 crf=20.0 qcomp=0.60 qpmin=3 qpmax=69 qpstep=4 ip_ratio=1.40 aq=1:1.00

This may have enough information to decode the file, but I don't have the expertise. Seems like an interesting mystery, or maybe it's really simple. Please help.

EDIT:
See last post for full recover method.

[Baldrick]

Do you have a working sample from same source/camera(same codecs)?

[ndjamena]

https://forum.videohelp.com/threads/367607-Damaged-MP4-%28h264%29-File-Need-expert-help

Did you try just cutting off the file "header" with a hex editor? I'd try it myself but I don't remember the exact position the h.264 stream begins and I have nothing left on my PC to compare it to.

[ndjamena]

Try [MP4Box -diso] on it.

It looks as though MP4Box at least recognises the header in the truncated sample.

[compscistudent]

I have no other file like it that I know of, but I could search through my files to check. I tried mp4box -diso on the full file but it had the same error of missing bytes but the number was different. The full file: missing 62 033 847 bytes, but just the first 5mb I uploaded: 1 043 517 860. Is it possible my file is short 62mb and important information is at the end? I can search for similar files, but without a match would it be impossible? I think it might be possible to use a substitute header or footer and get distorted video out of it, have you heard of this?

[ndjamena]

MP4 can keep part of its headers at the end of the file. If headers are missing then you're kind of screwed.

You'd have to determine what exactly is missing.

Do you have any idea what's supposed to be in there?

You could try extracting the streams using MKVMerge or MP4Box but since nothing seems capable of playing the file it may be that something important is missing.

[compscistudent]

So I found a very similar file that has the same error when I truncate it and was trying to take the first few MB and fix it having the full working video to do something similar with this other one.. Has not worked yet but there must be a way. The basic layout is identical, which means the moov is at the end of the file and missing. I'll upload the header and footer of the working video which has the same layout and is core 130 vs the other one which is core 120. It was encoded with HandBrake 0.9.9, so my other file would have been through that or a similar program because the basic layout is the same.

Other header [readable] information at the end that's cut off(in the similar video file): moov lmvhd iods trak tkhd mdia mdhd hdlr vide minf vmhd dinf dref url fstbl stsd avc1 JVT/AVC Coding 8avcC colrnclc pasp Xstts ... udta nameStereo gmet hdlr mdir ilst

footer section: similar-footer.mp4
header section: similar-h-test.mp4

[ndjamena]

The end of another file won't be much good. Each file is unique, there could be an index or any other element that's supposed to fit at the end of the file. Even if you replaced the end of the file with a chunk the same size it's unlikely the pointers at the beginning of the file will point where they're supposed to at the end of the file, or that the replacement info will make sense to the old file.

Handbrake doesn't plan what goes into an mp4 before it starts writing it, so it's headers are pretty much whatever it needs where ever it can fit it.

MP4s aren't supposed to be played before being finalised, luckily you obviously have codec private data. I think the best thing you could try is Video Repair Tool, hopefully it can find all the frames and create a new file or something.

If that fails:
https://www.videohelp.com/software/recover-mp4-to-h264

[compscistudent]

I tested that recover tool on the beginning from the working clip and it got nothing, also didn't work on the broken file. Maybe I'll lookup how to redirect the size of the file to point to the footer at the end to repair it manually tomorrow.

[compscistudent]

So cutting the header at the beginning and end of the similar layout working clip together in a hex editor and changing the size in hex of the mdat atom worked perfectly once I looked up the easy format. I edited the end of a video i generated with identical options except for height and ar of my target video onto a copy and it doesn't play in anything but the header information is read fine by mediainfo. Handbrake won't let me change the ar or move the height above 404 which the target video has, this might be the only reason it is failing to play. I am using the exact right version of handbrake(0.9.8) that has core 120 and all the defaults of high profile match except for keyint and keyint_min, so it is probably the one that was used.

I would like to manually perform the 8x8 high profile reverse transfrom on the macroblock bitstream in the NAL and experiment with different picture parameter sets(only 7 bytes long in the similar file).

Here is a pdf explaining it: http://www.w6rz.net/H264HighProfile.pdf
and here is a slideshow also explaining it: http://www.slideshare.net/vcodex/the-h264-integer-transform

I don't have any experience doing matrix operations on bitstreams and it may take me some time to figure out how to find the macroblock streams. Anyone have experience with this? (Of course testing on working video first)

[compscistudent]

SOLVED: (full recovery of missing moov atom possible)

The video was fixed manually in a hex editor frame by frame for about 14 frames and more can be added if I want to make a program to do it.

Step 1: Use the same software to produce the identical options and header layout (Handbrake 0.9.8, high profile, large file, options from header).
Step 2: Use hex editor to insert the moov atom from that video to the end.
Step 3: Manually re-index chunks, slices/samples in stco/co64, stsc, stsz for video track for a few frames. (Find lengths by hex inspection)
Step 4: Adjust the width and height in exponential golomb coding in sds. (found in avcc code 67 after 0 nibble. sizes are multiples of 16. check 9.3 of iso for golomb examples)
Step 5: Adjust previous steps and add more frames manually or program it(I might).

Here is the same file with the above done for 6 chunks, 24 samples.
(I didn't try re-indexing the audio, so it may or may not be working if more indexing is added to that track)

VLC does not display it correctly, but Windows player, Potplayer, and probably others play it fine for the first few frames.

Edit:
The error codes I put in the first post are just the player complaining about chunk offsets(stco/co64) and sample sizes(stsz) that are not indexed.