Salvage FLV video, several downloads, missing chunks (source disappeared)

[abolibibelot]

Years ago I tried to download a video on C-Span's website (a speech by George Carlin), using StreamTransport and Replay Media Catcher. I tried repeatedly with both softwares but each downloaded file had glitches caused by missing chunks at different spots. I kept 5 FLV files, each one having a different size (from 275635KB to 289770KB). When I went back to the same page months later, the video had been replaced by one of inferior quality, missing a part at the begining and/or end.

In order to complete the video as much as possible, by completing the missing chunks in one file which are not missing in another, I tried WinHex in “synchronize and compare” mode, but it proved excruciatingly tedious, as there are many differences, and it can't be automated, while WinHex's compare feature is not practical at all (for each search a menu has to be opened, a “report” file has to be created, and there's often a random error message saying that it can't write that damn report file, making it even more tedious). WinMerge can compare two files in an automated way and successfully match equal chunks located at different spots / offsets, but only in “text” mode it would seem, not in “binary” mode, so it's not usable either in a case like this. HexWorkshop should be able to compare two files in their entirety in “resynchronizing compare” mode, but it's extremely slow if the “resync window” is set higher than a few hundred bytes (it seems to be freezing without ever completing the analysis).

Is there a better method to perform such a task ?

[october262]

youtube - https://www.youtube.com/results?search_query=George+Carlin+speech

[abolibibelot]

Well, thanks but that's a bit frustrating... I was trying to regenerate the video with the source(s) I have, which are probably in better quality than a video downloaded from YouTube which has suffered at least one generation loss (often more than that), and has been typically recompressed at a very low bitrate.
And a secondary purpose of a forum discussion is to expose possible solutions to a problem so that anyone having a similar problem in the future could use it as a reference and solve said problem without having to ask again. In this case, with that kind of answer, it could only generate the same kind of frustration in the future.

[Budman1]

FLV are usually not too difficult to repair, at least by cutting bad chunks since they start with Hex 12 with header information and 09, video and 08 audio. Replacing sections would be more difficult and comparing each other is definitely not going to work since minute changes in stream, hesitation points, etc. would possibly make similar sections not the same hex wise.
My best method, if I were doing it, would be to play 2 (or more) in multiple players with a script to show time to the decimal seonds and frame numbers and when area of glitch arises, find that area in another player and cut frame accurate and merge with the first. I do this a lot with MPC-BE since it is free, portable and allows skipping to key frame, frame by frame, and small, medium, large jumps. It also allows multiple processes to run at the same time.

I use FFMpeg and re-encode segment from B or P frame up to the next Key Frame (I) E.G.

Frame 0-49 IBBBPBBBP- GLITCH - PBBBPBBBPIBBBPBBBPBBBPBBBPI <--Bad Copy
Frame 0-49 IBBBPBBBPBBBPBBBPBBBPBBBPBBBPIBBBPBBBPBBBPBBBPI <--Good Copy
Frame 0-49 IBBBPBBBP<-Re-encode------------->IBBBPBBBPBBBPBBBPI < Cut Sections 1, 2, 3
Frame 0-49 I-CopiedBP|I-yyyyyyyyyyyyyyyyyyy-|I-CopiedBPBBBPBBBP-I < merged segments.

I know this is time consuming but it works and I do it a lot since I am not in a hurry most times and can watch an hour movie with 2 shuttles to move frames back and forth with multiple players.


[Attachment 49819 - Click to enlarge]

I use script for most everything. (Also am in middle of Creating A utility for this.) The below example, a GOP 250 (I-Frame every 250 frames) shows 'KEEP' checked for frames 0-58 (copied), 'TRIM check for Frames 59-249 (Re-encoded), All segments after that are consecutive 'KEEP' checks they they will all be copied as 1 segment. Then the 3 segments will be merged.

I hope this example is useful, but bear in mind Most editors will not cut at frame accurate so there may still be glitches or replays of a quick segment. This can be eliminated Easily with AVC and FFMpeg but FLV use a slightly different algorithm to cut precisely with FFMpeg.


[Attachment 49820 - Click to enlarge]

[sophisticles]

[QUOTE=abolibibelot;2557695]WinMerge can compare two files in an automated way and successfully match equal chunks located at different spots / offsets, but only in “text” mode it would seem, not in “binary” mode, so it's not usable either in a case like this.

If WinMerge can compare and merge the files in "text" mode then you're in luck assuming what is being saved in the text file are hex values of the bytes. Once you have the text file you can convert it to binary, with Unix and Linux you can do it right from the command line, with Windows there's a bunch of programs that are capable of converting binary to text and back again.

Google "reversible hex dump".

[abolibibelot]

@ Budman1

FLV are usually not too difficult to repair, at least by cutting bad chunks since they start with Hex 12 with header information and 09, video and 08 audio. Replacing sections would be more difficult and comparing each other is definitely not going to work since minute changes in stream, hesitation points, etc. would possibly make similar sections not the same hex wise.

Thanks for this in-depth reply. What do you mean by “minute changes in stream, hesitation points, etc.” ?
From what I could see in WinHex, the sections with no missing chunks seem to be identical, but the missing chunks are not aligned at sector boundaries.
In WinMerge it looks... more complicated (see below).

I use FFMpeg and re-encode segment from B or P frame up to the next Key Frame (I) E.G.

With which parameters ? And how can it re-encode a segment with missing data ?

I know this is time consuming but it works and I do it a lot since I am not in a hurry most times and can watch an hour movie with 2 shuttles to move frames back and forth with multiple players.

Sorry, “2 shuttles” ?

I hope this example is useful, but bear in mind Most editors will not cut at frame accurate so there may still be glitches or replays of a quick segment. This can be eliminated Easily with AVC and FFMpeg but FLV use a slightly different algorithm to cut precisely with FFMpeg.

In-depth indeed, I'd have to do quite a lot of research to fully grasp what you wrote !
But have you been in a similar situation, where you had several files with corrupted / missing “packets” of data, which could be at least partially combined from one another ?
And doesn't it cause audio glitches to simply merge two good sections before and after a bad one ? I remember being warned about that kind of issue years ago when I exposed a situation where I had to export / render a large movie in three parts.
Quotes from Cornucopia :
“I would suggest that you remember to render a separate pass of audio-only at the full length (and then AudioDub() it in AVISynth). This avoids clicks/glitches at the join points. Have done this successfully many times.”
“Regarding the audio, it's your call, but if it were me, yes I'd do it. Glitches in silence will be very low, but will still be glitches unless you are extremely lucky to be exactly at digital 0 for a good length of time (on both ends). Those clicks are of unusual character so are quite noticeable even at low levels.”

@ sophisticles

If WinMerge can compare and merge the files in "text" mode then you're in luck assuming what is being saved in the text file are hex values of the bytes. Once you have the text file you can convert it to binary, with Unix and Linux you can do it right from the command line, with Windows there's a bunch of programs that are capable of converting binary to text and back again.

I'm not sure if / how it can be done that way, because (other than being quite resource-intensive – with this recent computer based on an Intel i7 6700K with 16GB of RAM WinMerge can go through with the analysis, by maxing out one CPU core and allocating 2-3GB of RAM, but with my former machine based on a Dual Core E5200 with 4GB of total RAM it failed miserably, hence why this task was stalled for so long) the output looks seriously messy : there are identical sections, missing sections, but also many sections which are slightly or vastly different – this may correspond to what Budman1 described as “minute changes in stream, hesitation points”. And there are a lot of different sections, as evidenced by the amount of orange in the left column. Wouldn't it cause inconsistencies, resulting in playback issues, if I simply copied everything that is either different or missing from one file to another ? And there are five of them, all with differences at different spots... And in accordance with Murphy's Law, there are most likely many such chunks which are found in neither of them.

Here are three screencaps :

[Budman1]

Thanks for this in-depth reply. What do you mean by “minute changes in stream, hesitation points, etc.” ?
From what I could see in WinHex, the sections with no missing chunks seem to be identical, but the missing chunks are not aligned at sector boundaries.
In WinMerge it looks... more complicated (see below).

If you use the same codec and the stream you download is identical in all you download, the data will indeed probably be identical since it is the same data, encoded by the same codec with the same settings (bit rate, FPS, etc.) . The length of each encoded segment may NOT be the same due to buffering, stream errors, and others. This means you may have data in one video tag bytes 0-1000 before the next segment and the other video may be Tag, 0-800, then a tag 801-1010 . The data will be the same but in between may not be. If the the compare program is smart enough to realized they are the same then it has to show the headers or tags in between as extra and leave a space in the other to line up or it will not match. I use EXAm Diff for this type of thing.

[Attachment 49841 - Click to enlarge]

BTW... FLV always start the same way FLV.........0x12 for Script data, 0x09 for video and 0x08 for audio. Youcan remove bytes 0x12 up x09 or 09 to 09 and video will still play although you lose data toSee Below

[Attachment 49832 - Click to enlarge]

With which parameters ? And how can it re-encode a segment with missing data ?

Okay the only way to be sure to get all the video is to have a video with, say Carlin face from frame 0->70, zoom out Frame 71-199, Front face again 200-500.
Now assuming each scene change starts with a Key Frame 'I' type. If you were missing frames 71-199, then you would just find a copy of the video with a good frame sequence 71-199 and copy it to the first video.

The more likely scenario is that your streaming glitched between Frames 58 and 75. That means you need to have good frames 58-75 in another copy of the video. And since Frame 58 is not a Key frame it will have to be encoded up to Frame 70 and then copy and add frames 71-199. Now you would have key frames at 0 (Original start) and 58 (because of the encode) and 71 (original Scene change Key frame and 200 (Original 3rd segment Key frame) This is where the 'Shuttle' comes in.

[Attachment 49833 - Click to enlarge]

It is a device to basically step forward and back or whatever you program it for for many programs. I have the 2 because they have left, right, Depress, Depress+left, Depress+right, and Long Depress. There are others with more function but with only 6 functions, one shuttle eats up Short jump, medium jump, and long jump both forward and back for 6 functions all by itself. I bought 2, Mainly because if I bought another with more functions, the original one would be wasted and my wife wouldn't go for that. (I did make her believe I needed 2 though, LOL)

Order of steps:
1. Obtain a list of all key frames and all frames and type with FFprobe

setlocal enabledelayedexpansion
@echo off
::for %%a in (*.mp4,*.mpg,*.flv) Do (
for %%a in ("%~nx1") Do (
set /a count=0
cd %%~dpa
echo frame,media_type,stream_index,key_frame,pkt_pts,pkt_pts_time,pkt_dts,pkt_dts_time,best_effort_timestamp,^
best_effort_timestamp_time,pkt_duration,pkt_duration_time,pkt_pos,pkt_size,^
Width,Height,pix_fmt,sample_aspect_ratio,pict_type,coded_picture_number,display_picture_number,^
interlaced_frame,top_field_first,repeat_pict,color_range,color_space,color_primaries,color_transfer,^
chroma_location > "%%~na_ffprobe.csv"
echo No.  pts_time  type > "%%~na_AllFrames.txt"
@echo. > "%%~na_keys.txt"
ver > nul"
set /a Number=0
ffprobe.exe -v quiet -select_streams v:0 -print_format csv -show_entries frame "%%~nxa"  >> "%%~na_ffprobe.csv"
for /F "tokens=4,6,18,19 delims=," %%b in ('findstr "video" "%%~na_ffprobe.csv"') do (
set "x=%%b"
set "y=%%c"
set "z=%%d"
set "w=%%e"
set "v=%%f"
set "u=%%g"
set sort=!y:~0,-7!
::if !sort! GEQ 0 if !sort! LEQ 10 echo !count!  !y:~0,13!  !x! !w:~0,1! >> "%%~na_AllFrames.txt"
echo !count!  !y:~0,13!  !x! !w:~0,1! >> "%%~na_AllFrames.txt"
set "key=!w:~0,1!"
if !key! == I echo !count!  !y:~0,13!  !x! !w:~0,1! >> "%%~na_keys.txt"
set /a count+=1
 set /a ekko=count%%100
 if !ekko! EQU 0 echo Frame !count! processed
)
)
pause

[Attachment 49834 - Click to enlarge]

2. Play one video with shuttle and find glitch. (We'll assume frame 58)

3. Checking list I find 55 2.294000 0 P
53 2.211000 0 B
54 2.253000 0 B
55 2.294000 0 P
56 2.336000 0 B
57 2.378000 0 B
58 2.420000 0 B
59 2.461000 0 P
60 2.503000 0 B
61 2.545000 0 B
62 2.586000 0 B

B frames can not be left at the end of a segment because they are the difference between frame before and frame after (B as in both directions). Frame 55 must be at the end of the first segment because it is still ok in the original and it is the last good 'P' frame (P as in Preceding) before the glitch, comparing its difference with the preceding frame.

4. Find that frame 55 in original video using a script and avisynth FFMS2 filter.

ffmpegsource2("C:\Users\Bud\Desktop\[dp]Manjandani-1.mp4", atrack=-1)
x = float(Width) / float(Height)
last.LanczosResize(round(x * 480.0 / 4.0) * 4, 480)
ShowFrameNumber(scroll=true, x=10, y=127, font="Arial", size=24, text_color=$ffff80)
Subtitle("Actual Frame -0", x=10, y=128, font="Arial", size=24, text_color=$ffff80)
ShowTime(offset_f=0,x=72, y=169, font="Arial", size=24, text_color=$ffff80)
Subtitle("0.000000 Frame 0 Offset", x=10, y=171, font="Arial", size=24, text_color=$ffff80)
ScriptClip(Last, """Subtitle("[ "+Chr(FFPICT_TYPE)+" ]", size=(Height*56.0/720), align=2)""", after_frame=true)

[Attachment 49835 - Click to enlarge]

5. Find frame 56 in the good video and encode it from 56 through Frame 70.
6. Continue to Play Bad video (Repeat from Step 2)

And doesn't it cause audio glitches to simply merge two good sections before and after a bad one ? I remember being warned about that kind of issue years ago when I exposed a situation where I had to export / render a large movie in three parts.

If its done correctly, FLV likes to have 0.08 overhead which is 8/100 of a second, Hardly noticeable.

It seems complicated and I hope this helps but it is not that hard and can be done with CMD drag and drop or batch mode

[Attachment 49838 - Click to enlarge]

I cheat with my program. It gives an idea of the above scenario. Trim means re-encode, keep means copy. So 0-55 copied, 56-70 is Encoded, 71-end of file is copied.
Then just merge or join the 3 segments by any of many programs. I use ffmpeg to merge.

[Attachment 49840 - Click to enlarge]


Comparing gives the example above of Exam Diff, which says nothing of what frame, time or anything and as I pointed out, just trying to copy data would not work unless you had a program that copied on a byte basis and if the glitch happens to be in the middle of a byte, you would never know just looking a the last good 'byte/half byte'.
Example... last 'good' byte of glitched video shows 81, 9F B0. The good video shows 81, 9F, B8. Do you replace B0 with BF? And I know of no programs (Free anyway) that will copy videos on a byte basis. Most are based on a frame or time point and copying from a time not a key frame gives many results that are nasty to play. Everyone knows you have to cut on a key frame or re-encode but there are virtually none that explain how to cut exactly at the key frame and result in a PTS-DTS time of 0.00000. It can be done fairly easily (at least from my point of view) and byte by byte could be done but would take a specially written program that took a list of cut point and copied byte by byte, IF you know the points.

It is easiest for me to play, find the 1st bad FRAME visually , re-encode from last good 'P' frame up to next Key frame, and then copy the rest. View and mark