Years ago I tried to download a video on C-Span's website (a speech by George Carlin), using StreamTransport and Replay Media Catcher. I tried repeatedly with both softwares but each downloaded file had glitches caused by missing chunks at different spots. I kept 5 FLV files, each one having a different size (from 275635KB to 289770KB). When I went back to the same page months later, the video had been replaced by one of inferior quality, missing a part at the begining and/or end.
In order to complete the video as much as possible, by completing the missing chunks in one file which are not missing in another, I tried WinHex in “synchronize and compare” mode, but it proved excruciatingly tedious, as there are many differences, and it can't be automated, while WinHex's compare feature is not practical at all (for each search a menu has to be opened, a “report” file has to be created, and there's often a random error message saying that it can't write that damn report file, making it even more tedious). WinMerge can compare two files in an automated way and successfully match equal chunks located at different spots / offsets, but only in “text” mode it would seem, not in “binary” mode, so it's not usable either in a case like this. HexWorkshop should be able to compare two files in their entirety in “resynchronizing compare” mode, but it's extremely slow if the “resync window” is set higher than a few hundred bytes (it seems to be freezing without ever completing the analysis).
Is there a better method to perform such a task ?
Try StreamFab Downloader and download from Netflix, Amazon, Youtube! Or Try DVDFab and copy Blu-rays! or rip iTunes movies!
+ Reply to Thread
Results 1 to 7 of 7
Thread
-
-
Last edited by october262; 16th Aug 2019 at 02:49.
-
Well, thanks but that's a bit frustrating... I was trying to regenerate the video with the source(s) I have, which are probably in better quality than a video downloaded from YouTube which has suffered at least one generation loss (often more than that), and has been typically recompressed at a very low bitrate.
And a secondary purpose of a forum discussion is to expose possible solutions to a problem so that anyone having a similar problem in the future could use it as a reference and solve said problem without having to ask again. In this case, with that kind of answer, it could only generate the same kind of frustration in the future. -
FLV are usually not too difficult to repair, at least by cutting bad chunks since they start with Hex 12 with header information and 09, video and 08 audio. Replacing sections would be more difficult and comparing each other is definitely not going to work since minute changes in stream, hesitation points, etc. would possibly make similar sections not the same hex wise.
My best method, if I were doing it, would be to play 2 (or more) in multiple players with a script to show time to the decimal seonds and frame numbers and when area of glitch arises, find that area in another player and cut frame accurate and merge with the first. I do this a lot with MPC-BE since it is free, portable and allows skipping to key frame, frame by frame, and small, medium, large jumps. It also allows multiple processes to run at the same time.
I use FFMpeg and re-encode segment from B or P frame up to the next Key Frame (I) E.G.
Frame 0-49 IBBBPBBBP- GLITCH - PBBBPBBBPIBBBPBBBPBBBPBBBPI <--Bad Copy
Frame 0-49 IBBBPBBBPBBBPBBBPBBBPBBBPBBBPIBBBPBBBPBBBPBBBPI <--Good Copy
Frame 0-49 IBBBPBBBP<-Re-encode------------->IBBBPBBBPBBBPBBBPI < Cut Sections 1, 2, 3
Frame 0-49 I-CopiedBP|I-yyyyyyyyyyyyyyyyyyy-|I-CopiedBPBBBPBBBP-I < merged segments.
I know this is time consuming but it works and I do it a lot since I am not in a hurry most times and can watch an hour movie with 2 shuttles to move frames back and forth with multiple players.
[Attachment 49819 - Click to enlarge]
I use script for most everything. (Also am in middle of Creating A utility for this.) The below example, a GOP 250 (I-Frame every 250 frames) shows 'KEEP' checked for frames 0-58 (copied), 'TRIM check for Frames 59-249 (Re-encoded), All segments after that are consecutive 'KEEP' checks they they will all be copied as 1 segment. Then the 3 segments will be merged.
I hope this example is useful, but bear in mind Most editors will not cut at frame accurate so there may still be glitches or replays of a quick segment. This can be eliminated Easily with AVC and FFMpeg but FLV use a slightly different algorithm to cut precisely with FFMpeg.
[Attachment 49820 - Click to enlarge] -
[QUOTE=abolibibelot;2557695]WinMerge can compare two files in an automated way and successfully match equal chunks located at different spots / offsets, but only in “text” mode it would seem, not in “binary” mode, so it's not usable either in a case like this.
If WinMerge can compare and merge the files in "text" mode then you're in luck assuming what is being saved in the text file are hex values of the bytes. Once you have the text file you can convert it to binary, with Unix and Linux you can do it right from the command line, with Windows there's a bunch of programs that are capable of converting binary to text and back again.
Google "reversible hex dump". -
@ Budman1
FLV are usually not too difficult to repair, at least by cutting bad chunks since they start with Hex 12 with header information and 09, video and 08 audio. Replacing sections would be more difficult and comparing each other is definitely not going to work since minute changes in stream, hesitation points, etc. would possibly make similar sections not the same hex wise.
From what I could see in WinHex, the sections with no missing chunks seem to be identical, but the missing chunks are not aligned at sector boundaries.
In WinMerge it looks... more complicated (see below).
I use FFMpeg and re-encode segment from B or P frame up to the next Key Frame (I) E.G.
I know this is time consuming but it works and I do it a lot since I am not in a hurry most times and can watch an hour movie with 2 shuttles to move frames back and forth with multiple players.
I hope this example is useful, but bear in mind Most editors will not cut at frame accurate so there may still be glitches or replays of a quick segment. This can be eliminated Easily with AVC and FFMpeg but FLV use a slightly different algorithm to cut precisely with FFMpeg.
But have you been in a similar situation, where you had several files with corrupted / missing “packets” of data, which could be at least partially combined from one another ?
And doesn't it cause audio glitches to simply merge two good sections before and after a bad one ? I remember being warned about that kind of issue years ago when I exposed a situation where I had to export / render a large movie in three parts.
Quotes from Cornucopia :
“I would suggest that you remember to render a separate pass of audio-only at the full length (and then AudioDub() it in AVISynth). This avoids clicks/glitches at the join points. Have done this successfully many times.”
“Regarding the audio, it's your call, but if it were me, yes I'd do it. Glitches in silence will be very low, but will still be glitches unless you are extremely lucky to be exactly at digital 0 for a good length of time (on both ends). Those clicks are of unusual character so are quite noticeable even at low levels.”
@ sophisticles
If WinMerge can compare and merge the files in "text" mode then you're in luck assuming what is being saved in the text file are hex values of the bytes. Once you have the text file you can convert it to binary, with Unix and Linux you can do it right from the command line, with Windows there's a bunch of programs that are capable of converting binary to text and back again.
Here are three screencaps :
Last edited by abolibibelot; 17th Aug 2019 at 11:07.
-
Thanks for this in-depth reply. What do you mean by “minute changes in stream, hesitation points, etc.” ?
From what I could see in WinHex, the sections with no missing chunks seem to be identical, but the missing chunks are not aligned at sector boundaries.
In WinMerge it looks... more complicated (see below).
[Attachment 49841 - Click to enlarge]
BTW... FLV always start the same way FLV.........0x12 for Script data, 0x09 for video and 0x08 for audio. Youcan remove bytes 0x12 up x09 or 09 to 09 and video will still play although you lose data toSee Below
[Attachment 49832 - Click to enlarge]
With which parameters ? And how can it re-encode a segment with missing data ?
Now assuming each scene change starts with a Key Frame 'I' type. If you were missing frames 71-199, then you would just find a copy of the video with a good frame sequence 71-199 and copy it to the first video.
The more likely scenario is that your streaming glitched between Frames 58 and 75. That means you need to have good frames 58-75 in another copy of the video. And since Frame 58 is not a Key frame it will have to be encoded up to Frame 70 and then copy and add frames 71-199. Now you would have key frames at 0 (Original start) and 58 (because of the encode) and 71 (original Scene change Key frame and 200 (Original 3rd segment Key frame) This is where the 'Shuttle' comes in.
[Attachment 49833 - Click to enlarge]
It is a device to basically step forward and back or whatever you program it for for many programs. I have the 2 because they have left, right, Depress, Depress+left, Depress+right, and Long Depress. There are others with more function but with only 6 functions, one shuttle eats up Short jump, medium jump, and long jump both forward and back for 6 functions all by itself. I bought 2, Mainly because if I bought another with more functions, the original one would be wasted and my wife wouldn't go for that. (I did make her believe I needed 2 though, LOL)
Order of steps:
1. Obtain a list of all key frames and all frames and type with FFprobe
Code:setlocal enabledelayedexpansion @echo off ::for %%a in (*.mp4,*.mpg,*.flv) Do ( for %%a in ("%~nx1") Do ( set /a count=0 cd %%~dpa echo frame,media_type,stream_index,key_frame,pkt_pts,pkt_pts_time,pkt_dts,pkt_dts_time,best_effort_timestamp,^ best_effort_timestamp_time,pkt_duration,pkt_duration_time,pkt_pos,pkt_size,^ Width,Height,pix_fmt,sample_aspect_ratio,pict_type,coded_picture_number,display_picture_number,^ interlaced_frame,top_field_first,repeat_pict,color_range,color_space,color_primaries,color_transfer,^ chroma_location > "%%~na_ffprobe.csv" echo No. pts_time type > "%%~na_AllFrames.txt" @echo. > "%%~na_keys.txt" ver > nul" set /a Number=0 ffprobe.exe -v quiet -select_streams v:0 -print_format csv -show_entries frame "%%~nxa" >> "%%~na_ffprobe.csv" for /F "tokens=4,6,18,19 delims=," %%b in ('findstr "video" "%%~na_ffprobe.csv"') do ( set "x=%%b" set "y=%%c" set "z=%%d" set "w=%%e" set "v=%%f" set "u=%%g" set sort=!y:~0,-7! ::if !sort! GEQ 0 if !sort! LEQ 10 echo !count! !y:~0,13! !x! !w:~0,1! >> "%%~na_AllFrames.txt" echo !count! !y:~0,13! !x! !w:~0,1! >> "%%~na_AllFrames.txt" set "key=!w:~0,1!" if !key! == I echo !count! !y:~0,13! !x! !w:~0,1! >> "%%~na_keys.txt" set /a count+=1 set /a ekko=count%%100 if !ekko! EQU 0 echo Frame !count! processed ) ) pause
[Attachment 49834 - Click to enlarge]
2. Play one video with shuttle and find glitch. (We'll assume frame 58)
3. Checking list I find 55 2.294000 0 P
53 2.211000 0 B
54 2.253000 0 B
55 2.294000 0 P
56 2.336000 0 B
57 2.378000 0 B
58 2.420000 0 B
59 2.461000 0 P
60 2.503000 0 B
61 2.545000 0 B
62 2.586000 0 B
B frames can not be left at the end of a segment because they are the difference between frame before and frame after (B as in both directions). Frame 55 must be at the end of the first segment because it is still ok in the original and it is the last good 'P' frame (P as in Preceding) before the glitch, comparing its difference with the preceding frame.
4. Find that frame 55 in original video using a script and avisynth FFMS2 filter.
Code:ffmpegsource2("C:\Users\Bud\Desktop\[dp]Manjandani-1.mp4", atrack=-1) x = float(Width) / float(Height) last.LanczosResize(round(x * 480.0 / 4.0) * 4, 480) ShowFrameNumber(scroll=true, x=10, y=127, font="Arial", size=24, text_color=$ffff80) Subtitle("Actual Frame -0", x=10, y=128, font="Arial", size=24, text_color=$ffff80) ShowTime(offset_f=0,x=72, y=169, font="Arial", size=24, text_color=$ffff80) Subtitle("0.000000 Frame 0 Offset", x=10, y=171, font="Arial", size=24, text_color=$ffff80) ScriptClip(Last, """Subtitle("[ "+Chr(FFPICT_TYPE)+" ]", size=(Height*56.0/720), align=2)""", after_frame=true)
[Attachment 49835 - Click to enlarge]
5. Find frame 56 in the good video and encode it from 56 through Frame 70.
6. Continue to Play Bad video (Repeat from Step 2)
And doesn't it cause audio glitches to simply merge two good sections before and after a bad one ? I remember being warned about that kind of issue years ago when I exposed a situation where I had to export / render a large movie in three parts.
It seems complicated and I hope this helps but it is not that hard and can be done with CMD drag and drop or batch mode
[Attachment 49838 - Click to enlarge]
I cheat with my program. It gives an idea of the above scenario. Trim means re-encode, keep means copy. So 0-55 copied, 56-70 is Encoded, 71-end of file is copied.
Then just merge or join the 3 segments by any of many programs. I use ffmpeg to merge.
[Attachment 49840 - Click to enlarge]
Comparing gives the example above of Exam Diff, which says nothing of what frame, time or anything and as I pointed out, just trying to copy data would not work unless you had a program that copied on a byte basis and if the glitch happens to be in the middle of a byte, you would never know just looking a the last good 'byte/half byte'.
Example... last 'good' byte of glitched video shows 81, 9F B0. The good video shows 81, 9F, B8. Do you replace B0 with BF? And I know of no programs (Free anyway) that will copy videos on a byte basis. Most are based on a frame or time point and copying from a time not a key frame gives many results that are nasty to play. Everyone knows you have to cut on a key frame or re-encode but there are virtually none that explain how to cut exactly at the key frame and result in a PTS-DTS time of 0.00000. It can be done fairly easily (at least from my point of view) and byte by byte could be done but would take a specially written program that took a list of cut point and copied byte by byte, IF you know the points.
It is easiest for me to play, find the 1st bad FRAME visually , re-encode from last good 'P' frame up to next Key frame, and then copy the rest. View and mark
Similar Threads
-
Downloading video full of m4v chunks
By umbada in forum Video Streaming DownloadingReplies: 5Last Post: 17th Oct 2019, 03:28 -
Searching *.mp4 video editor which allows deletion of chunks without re-enc
By pxstein in forum EditingReplies: 5Last Post: 15th May 2017, 09:40 -
Sync of video from one source to audio from another source (same content)
By tbbttbbt in forum AudioReplies: 2Last Post: 5th May 2016, 15:10 -
Intensity Pro 4K video disappeared from XSplit Broadcaster
By sdsumike619 in forum CapturingReplies: 0Last Post: 29th Feb 2016, 10:31 -
How to capture video chunks frames of data in realtime from HVR 2250?
By rshapiro in forum CapturingReplies: 19Last Post: 27th Aug 2014, 05:41