? Cutting AVI in Bitwise Identical Manner (at non-keyframes)

[Synetech]

Hi,

I am trying to find a way to cut an AVI such that the bytes in the resulting video—the part that is kept aside from the header, etc.—are identical to the original. I have been using the v/nandub, etc. applications in direct stream copy mode to cut videos for a long time, but the videos that are made are not identical. In other words, when you do a binary comparison of the two files, you should get a few small differences in the header or whatever, and the original should be longer, but otherwise the files should be identical (ie, changes could be made with a hex-editor, like you can with an MP3 as long as you cut on a frame-boundary).

For example, let’s say that I have made a large video (eg 1GB), and am sharing it via FTP, P2P, HTTP, etc. (basically any resumable transfer method). I then decide to cut out the last minute of the video. Doing it like I have been doing causes the file to be drastically altered even if the resulting image and sound are identical. This means that people who have gotten only part of the video cannot use what they have and must start downloading it all over from scratch (and thus wasting my bandwidth).


Also, sometimes I run into the situation where I need to cut a video where there is no keyframe but cannot figure out how. I know I’ve done it a couple of times by chance with one of the dub apps and with another commercial app, but it never works when I purposely try to do it. Will re-deriving the keyframes work? If so, it will create a non-identical file right? This isn’t a high priority; I can keep a little extra.


Thanks a lot.

[poisondeathray]

I can make a suggestion for the transfer of large files: cut it into smaller pieces using archival software like winrar, winzip etc...The resultant re-assembled file will be identical.

You can specify an exact size, add a password, add .sfv and recovery records for checking.

Furthermore, there are many free hosting sites, some which have unlimited bandwidth and duration of hosting. You would only have to use 1x upload bandwidth.

[Synetech]

Thanks for the suggestion but you can’t really compress videos so there’s no point in using WinRAR, etc. over a standard file splitter.

[redwudz]

I don't think he is suggesting WinRAR for compression, just archiving into small pieces to be reassembled as a whole file when Un-RARed. If you add a .sfv file and a PAR file, you have redundancy and error checking. I have used QuickPAR with Usenet downloads of RAR and PAR files.

But, I more often cut them at keyframes with the keyframe icon button in VirtualDub. When you use 'File>Open video file...' in VD, you can check the 'Ask for extended options after this dialog' box at the lower left and in the next menu, select 'Re-derive keyframe flags'.

[jagabo]

Syntech, you have a bunch of misunderstandings. What you wan't basically isn't feasible.

Most P2P software uses hashes to identify files. If even one byte is different, or the length is different, the new file will not be identified as the same file. An incomplete FTP or HTTP transfer might be able to resume if only the end of the file is cut off but...

The AVI headers include lots of stuff that will vary depending on the program that created it and the nature of the video. So using a tool like VirtualDub or AVIDemux to trim the end off the video will create more changes than just cutting off the end of the file.

The end of an AVI file, with high compression codecs like Divx and Xvid, has a list of keyframes. The list will change if you remove frames from the file. If this list is missing, most players won't be able to play the file.

[Synetech]

I don't think he is suggesting WinRAR for compression, just archiving into small pieces to be reassembled as a whole file when Un-RARed.

I know, but since there is no compression to be gained, there is no reason to use WinRAR over a regular, non-compressing file splitter.

But, I more often cut them at keyframes with the keyframe icon button in VirtualDub. When you use 'File>Open video file...' in VD, you can check the 'Ask for extended options after this dialog' box at the lower left and in the next menu, select 'Re-derive keyframe flags'.

Yup, I asked that in the first post; thanks for confirming that it will work.

Syntech, you have a bunch of misunderstandings. What you wan't basically isn't feasible.

Most P2P software uses hashes to identify files. If even one byte is different, or the length is different, the new file will not be identified as the same file.

The AVI headers include lots of stuff that will vary depending on the program that created it and the nature of the video. So using a tool like VirtualDub or AVIDemux to trim the end off the video will create more changes than just cutting off the end of the file.

The end of an AVI file, with high compression codecs like Divx and Xvid, has a list of keyframes. The list will change if you remove frames from the file.

I was simplifying the question. Obviously frame-related information in the header and/or footer will change (eg length, number of frames, etc.), but the contents of the actual frames that remain should not since they are to be unaltered.

P2P apps don’t use a single hash per file, they hash and compare smaller blocks, so if the new file is mostly identical to the original, then only a bit from the beginning and/or end of the file needs to be re-transferred.

Ultimately, it is not even a transfer issue, I just want to be sure that the cut video is exactly identical in quality to the original, and if every single frame has been changed, then how can I be certain of that?

[jagabo]

I was simplifying the question. Obviously frame-related information in the header and/or footer will change (eg length, number of frames, etc.), but the contents of the actual frames that remain should not since they are to be unaltered.

But they may not be in the same place since the header information can be variable in length.

P2P apps don’t use a single hash per file, they hash and compare smaller blocks, so if the new file is mostly identical to the original, then only a bit from the beginning and/or end of the file needs to be re-transferred.

They use a hash to identify the entire file too.

Ultimately, it is not even a transfer issue, I just want to be sure that the cut video is exactly identical in quality to the original, and if every single frame has been changed, then how can I be certain of that?

If you don't trust the programs write your own to find the data for each frame and do you own comparison.

AVISynth has the tools for frame by frame comparison of video files. Here's one way you can compare frames:

# compare two videos

v1 = AviSource("video1.avi")
v2 = AviSource("video2.avi")

sub = v1.subtract(v2) 
substrong = sub.levels(112,1,144,0,255) 

return StackVertical(StackHorizontal(v1.subtitle("original"),v2.subtitle("encoded")),StackHorizontal(sub.subtitle("Difference"),substrong.subtitle("Difference amplified")))

[Synetech]

I was simplifying the question. Obviously frame-related information in the header and/or footer will change (eg length, number of frames, etc.), but the contents of the actual frames that remain should not since they are to be unaltered.

But they may not be in the same place since the header information can be variable in length.

That doesn’t matter. I have worked with many files that had differences and were of different lengths, comparing, editing, updating, restoring, etc. When I do a binary comparison, I align the files at the next identical spot, ignoring variances leading in and out.

P2P apps don’t use a single hash per file, they hash and compare smaller blocks, so if the new file is mostly identical to the original, then only a bit from the beginning and/or end of the file needs to be re-transferred.

They use a hash to identify the entire file too.

Yes, to identify. Besides, isn’t there a provision for padding? The header could just be padded so that the first frame is aligned.

Ultimately, it is not even a transfer issue, I just want to be sure that the cut video is exactly identical in quality to the original, and if every single frame has been changed, then how can I be certain of that?

If you don't trust the programs write your own to find the data for each frame and do you own comparison.

AVISynth has the tools for frame by frame comparison of video files.

You mean the decoded output frame right? Because when looking at the input and output frames with a hex-editor, they are not identical which is the whole problem.

Here’s an analogy: if the original first frame contained 12/6 giving a result of 2, I want the first frame in the output file to be 12/6 as well, not 18/9 even though it gives the exact same results after decoding.

[jagabo]

AVISynth has the tools for frame by frame comparison of video files.

You mean the decoded output frame right? Because when looking at the input and output frames with a hex-editor, they are not identical which is the whole problem.

Here’s an analogy: if the original first frame contained 12/6 giving a result of 2, I want the first frame in the output file to be 12/6 as well, not 18/9 even though it gives the exact same results after decoding.

Yes, the AVISynth script I gave will only compare the decompressed frames.

I think a better example might be that 5/2 and 4/2 will both give 2 as an integer result. But by using 4/2 instead of 5/2 some precision has been lost which might otherwise have been used. I don't know the codecs in enough detail to know what might change within the compressed data during a remux. But some of the remuxers can't even decode the audio and video they're working with so it seems unlikely they will change anything that effects the way a decoder will decode them.

[AlanHK]

I don't think he is suggesting WinRAR for compression, just archiving into small pieces to be reassembled as a whole file when Un-RARed.

I know, but since there is no compression to be gained, there is no reason to use WinRAR over a regular, non-compressing file splitter.

There is, because there is no standard file splitter. I've got three or four I had to find and download to put together various split files. Just about everyone already has Winrar (or something compatible), and can automatically reassemble files with it. It also provides error checking. You can set it to no compression when making archives and it is then extremely fast.

Obviously frame-related information in the header and/or footer will change (eg length, number of frames, etc.), but the contents of the actual frames that remain should not since they are to be unaltered.

Avi and mpeg and other video compression do not store single frames, (except in "uncompressed" mode, which gives gigabyte size files for minutes of video). There are key frames and intermediate frames are actually lists of changes from the key frames. So a video file cutter, if not cutting exactly on a key frame, has to build at least one new key frame and the differences for that. Compressed audio will also require some rebuilding.

Ultimately, it is not even a transfer issue, I just want to be sure that the cut video is exactly identical in quality to the original, and if every single frame has been changed, then how can I be certain of that?

The output can be almost identical, but the data will certainly not be. You just have to eyeball it.

[Synetech]

I don't think he is suggesting WinRAR for compression, just archiving into small pieces to be reassembled as a whole file when Un-RARed.

I know, but since there is no compression to be gained, there is no reason to use WinRAR over a regular, non-compressing file splitter.

There is, because there is no standard file splitter. I've got three or four I had to find and download to put together various split files. Just about everyone already has Winrar (or something compatible), and can automatically reassemble files with it. It also provides error checking. You can set it to no compression when making archives and it is then extremely fast.

True, but WinRAR is a commercial app so you cannot assume it to be present. By “standard file splitter” I mean a raw byte cut which can be reassembled with a binary appending (> copy /b piece1+piece2 wholefile) The one I liked best back in the days of floppies was Space Dolphin’s Split Files.

Obviously frame-related information in the header and/or footer will change (eg length, number of frames, etc.), but the contents of the actual frames that remain should not since they are to be unaltered.

Avi and mpeg and other video compression do not store single frames, (except in "uncompressed" mode, which gives gigabyte size files for minutes of video). There are key frames and intermediate frames are actually lists of changes from the key frames. So a video file cutter, if not cutting exactly on a key frame, has to build at least one new key frame and the differences for that. Compressed audio will also require some rebuilding.

I didn’t think it would be so easy. I was hoping it would be similar to MP3s where you can literally cut out a chunk with a hex-editor and as long as you cut out the whole frame it would not be corrupt.

Ultimately, it is not even a transfer issue, I just want to be sure that the cut video is exactly identical in quality to the original, and if every single frame has been changed, then how can I be certain of that?

The output can be almost identical, but the data will certainly not be. You just have to eyeball it.

So there’s no way to only cut a video? Cutting a video mandates modifying it too?

I am even more concerned about the dub apps’ output now because I just opened some test videos in Nandub and saved them—no cutting—with both audio and video set to direct stream copy (in other words, just copy the file, don’t alter it in any way), yet the resulting files were drastically different (and not just the metadata). Sure, the frames may be identical when you display them in a player, but the binary data is quite different, so you cannot compare it to check for corruption. Is there perhaps a setting or something that would give the same file (hopefully including not modifying the metadata)? I have attached one of the small ones to try.

catandceilingfan.avi

[jagabo]

So there’s no way to only cut a video? Cutting a video mandates modifying it too?

Only the cut GOPs need to be reencoded.

I took your little video and remuxed the video into a new AVI with VirtualDub called A.AVI. I then opened A.AVI with VirtualDub, cut the last frame, and saved as a new file, B.AVI. Both files were identical except for a few bytes in the header, a chunk size somewhere in the middle, and about 6000 bytes at the end.

I took your video and remuxed it with AVIDemux (audio and video). I took that file and remuxed the video into a new AVI with VirtualDub, as above. The output was identical to A.AVI.

[redwudz]

I'm not a expert with all this, but what jagabo said here seems important to me:

The end of an AVI file, with high compression codecs like Divx and Xvid, has a list of keyframes. The list will change if you remove frames from the file. If this list is missing, most players won't be able to play the file.

That would cause file changes when the video is split, as the end of the AVI in both parts would have to be changed after a split to correct that list. Maybe a minor point, but that seems one reason binary file splitters (Or joiners) seem to have problems with highly compressed video formats at times.

And I'm only suggesting a program like WinRAR for distribution of a video in smaller parts over the internet, especially with a slow connection. I dislike having to rejoin RAR files, but it is a viable method for internet distribution.

[AlanHK]

True, but WinRAR is a commercial app so you cannot assume it to be present. By "standard file splitter" I mean a raw byte cut which can be reassembled with a binary appending (> copy /b piece1+piece2 wholefile) The one I liked best back in the days of floppies was Space Dolphin's Split Files.

WinRAR has a trial version, and anyone downloading stuff these days needs it.
Or if you prefer, 7-Zip is freeware and understands RAR. Or Google for unrar to find many other, free, ways.

I can use DOS commands, but many have a deep phobia about it. You'll get a lot of people complaining and asking you to explain this over and over if you offer files like that.

I am even more concerned about the dub apps' output now because I just opened some test videos in Nandub and saved them--no cutting--with both audio and video set to direct stream copy (in other words, just copy the file, don't alter it in any way), yet the resulting files were drastically different (and not just the metadata). Sure, the frames may be identical when you display them in a player, but the binary data is quite different

If it looks identical, that's all that matters. Even if it keeps the video data the same, every video editor will make new headers and other metadata, including usually its signature, and most likely in a different way than the original creating app.

You really can't slice up a video file with a hex editor and expect to get a valid file, unless you know a great deal about the data structures.

[Synetech]

You really can't slice up a video file with a hex editor and expect to get a valid file, unless you know a great deal about the data structures.

That’s why I’m looking for an app that can do it cleanly without making all kinds of extra changes, making the cut with as little modification as possible.