Synchronising audio and video of 300 AVI clips

[sanlyn]

Well....the method I used in posts #17 and #18 seemed to work pretty well as far as "sighting" where a particular blip on the audio display and a particular event in a frame that might be matched up. If you have a scene in which you see a door slam, would you sit playing again and again and again changing a setting in 10 ms increments until the audio matches a visual pause in speech followed by a sudden wore start8ing with an "m". or would you look for a visual cue in the audio display that "looks like" that event? Why not use a graphical display and scan for a signal in the audio track's display that offers a clue? I can imagine someone sitting at a movie editor and replaying a film again and again until they can precisely match up the clapboard signal with the correct frame by sight and sound alone. I don't think it can be done. I did it years ago in film editing by matching up the "spike" in a picture of the audio sigtnal with the frame where the clapboard made impact.

I get the idea that these 300 clips are not going to be presented in their original sequence. I believe the O.P. searched through the captures for all shots related to certain persons or certain chains of events that occurred over a period of time. So these 300 clips were pulled out of the original captures and rearranged to a different pattern and sequence. This search-and-cut operation went on for some weeks -- I'm still trying to figure out how one would spend that much time editing a group of long captures without listening to the audio. But that aside, look at the church video in post #4. It begins with two people talking on the steps of a church, but you can't hear anyone talking. Next shot is a group of people walking down concrete church steps (no sounds of footsteps heard), one or more of them talking rather loudly it seems (no sounds of anyone talking), car doors opening and closing (no door sounds), and traffic passing in the street (no traffic sounds). Apparently the audio for that sequence exists in another part of the capture. It could be 2 seconds back, it could be 10 seconds ahead. How would one pull a scene that long from a capture and have no related audio at all, if the source from which it was cut had not been out of sync to begin with?

Anyway, that was my reasoning for matching those two short clips. I also feel that the origi9nal sync problem was due to audio being slower than the video. Naturally, if you take a continuous 15-minute clip with audio running slower than video, you could sync that up at the beginning of the clip but 15 minutes later the slow audio would be out of sync again. Here, however, we have clips who sync appears be off by 1/3 second to 2 seconds to several seconds as in the church video. I've begun to recognize some of the people in these videos and other posts, so it would seem to me that the separate scenes posted here span several years; none of them came from the same capture session. Each of them is out of sync to a different degree. It seems to me that they would have to have the audio adjusted individually. If you then manage to collect 20 different short clips, and each clip is from a different capture, and each clip is individually re-sync'd, and none of the clips is as long as a minute (and probably shorter), and if they are connected properly, why would they suddenly go out of sync again? Assume that the joining is performed correctly and no new mistakes are made.

The real problems arise with scenes like that church video. Where is its audio? Its likely been cut up into other scenes from elsewhere in the original capture, perhaps some of it discarded altogether.

[manono]

Well....the method I used in posts #17 and #18 seemed to work pretty well as far as "sighting" where a particular blip on the audio display and a particular event in a frame that might be matched up.

Oh, sorry. I thought you meant playing the video in VDub, not lining up the video with the audio waveform.

[sanlyn]

Well....the method I used in posts #17 and #18 seemed to work pretty well as far as "sighting" where a particular blip on the audio display and a particular event in a frame that might be matched up.

Oh, sorry. I thought you meant playing the video in VDub, not lining up the video with the audio waveform.

No, I figured that was the case. My long-winded explanation was primarily for avz10's benefit. Or for other readers who might not have kept track during more than a year of threads concerning this project -- which has been extremely interesting in terms of managing the usual catastrophes. Hopefully, it will all work one way or another.

[2Bdecided]

As VD does not have sound

What? It has sound here!


If it wasn't for the time you've spent splitting and organising these clips, I'd definitely say re-capture. I've tried re-synching audio and video. It might just be me, but I find it very difficult. Sanlyn has given you some great hints and ideas, but it's still difficult. If you want those initial and/or final seconds that have ended up in different files, it's going to be even worse.


You've got to think, honestly, what is going to be the best and most efficient use of your time.


I'd remind you of the advice to do one DVD properly, full process beginning to end, before starting on any others. I'm not trying to be a pain. I'm speaking from bitter experience myself here!

Cheers,
David.

[sanlyn]

Well....I have to agree. Wholesale capture of several tapes at one time is not out of the question (I have done it myself). But each capture should be saved somewhere until the processing glitches and complications have been analyzed and solved. If you are new to captures and don't know what to expect, the same mistakes can be made on every one of them and don't become apparent until later.

I have had poor captures. I can recall one 90-minute VHS tape that I captured that looked "OK" at first, until I spent a few weeks on the movie and got 20 minutes into the film. Alas! There was a big "blip" of 7 frames that played nothing but static, with a loud "Phhhfftt!" in the audio. I had watched the entire movie most of the time during the capture and had somehow missed that moment the first time. Dirt on the tape, apparently, had caused it. There was no other solution: I simply had to recapture at least that portion of the tape. And once I had all the equipment reconnected, I decided to recapture the entire movie anyway. I kept both captures on an external hard drive. And it's a good thing I did: the second capture had another dirty glitch that was not in the first one.

Attached is a copy of the bad frames in the second capture (A_LILI_capture2_glitch.avi). It is direct from the original capture, still telecined and unprocessed except for converting from YUY2 to YV12 to save a little forum space. The same frames in the attached B_LIL_capture1_OK.avi were made the same way. This scene is 1 hour and 17 minutes into the movie; there is too much movement in the scene to use frame interpolation to create replacement frames, and the audio in the glitch is missing. If I had not kept both captures, what do you think I would have been able to do? The answer: nothing. After weeks of processing, I would have been up the river without a paddle.

You're not alone here. Everyone -- and I mean everyone -- has made similar mistakes. I sometimes think I own the international world record for mistakes! Fortunately my experience as a programmmer taught me to save copies of scripts and code, and even to save copies (temporarily, at least) of processing scripts and filter setups that didn't work very well.

You can try to do as much as you can with the workarounds suggested. It's not impossible. But it's going to take time. There would be nothing unusual about spending a great deal of time on tapes that have personal value. I have done it for years. Frankly I've never regretted the effort, and neither have others who valued the videos as much as I did. Certainly, I have a lot of "junk" tapes on which I spent very little time or simply grew tired of the hassle with them. But those are junk. You seem to have some very nice shots of family history, and I'd be proud to own something similar myself. But I never owned a movie camera !!!! Now I wish I had, but it's too late.

[avz10]

It seems as if Sanlyn is questioning my integrity and honesty in #31. I want to clear this up and see this as an unnecessary post.

I get the idea that these 300 clips are not going to be presented in their original sequence. I believe the O.P. searched through the captures for all shots related to certain persons or certain chains of events that occurred over a period of time. So these 300 clips were pulled out of the original captures and rearranged to a different pattern and sequence. This search-and-cut operation went on for some weeks -- I'm still trying to figure out how one would spend that much time editing a group of long captures without listening to the audio.

I will try to provide as much detail as I can, so that the question "why did he not listen to find out if the audio is in sync" can once and for all be resolved.

My first daughter, Marli, was born in 1989. At that stage I did not have a video camera. I took one roll with my father’s super 8 camera. A friend bought a video camera and I often used it. I started off by making two VHS cassettes, called “Marli 1” and another one, named “Family”. Over the years the collection grew and I eventually had “Marli 1; 2;3 till 9”.

The big mistake that I made then, was that I did not record scenes from the VHS-C to the VHS that were not related to Marli. I should have made master VHS cassettes that included everything that I thought was worthy to keep.

My second daughter, Lize, was born in 1991. I started with a series “Lize 1”, and did the same- cut out pieces of video that was not applicable to Lize and so it carried on over the years. I did the same with my son, Erik, born in 1995.

1989 has only three clips- Marli was just born, no major cuts were done and there were no duplications, as Lize was not born at that stage, etc:

This is in contrast to 1991, when Lize was born and I had duplicate cassettes, but not complete (as Marli would have more scenes in some instances (birthday parties that she went to, but not Lize) and Lize more in other instances (e.g. ultrasound of her when my wife was pregnant, etc):



From 1997 until 1999, I studied in Australia and I returned with 15 VHS-C cassettes. Back in South Africa, I decided to stop cutting inappropriate pieces from the individual children’s videos. So the videos for each child had the same content. I carried on like this and stopped in 2005.

Note if a clip says 1C or 2C, it means that I used the VHS-C cassette in this project.




Last year I decided to make one “master” video, combining all three videos. This is when this project started. I had the Marli, Lize, Erik series; one family tape; quite a few VHS-C cassettes and videos from family members that I borrowed.

I initially captured the videos, but with the errors that I made, I started again in July 2013 #22 (from the "Advice on what filters to use...)

I have thought about the project a lot over the weekend and if I could have started a project of this magnitude once, I can just as well have a second attempt at it.

I set my settings during capturing so that the “volume meter” showed.

During capturing, I never noticed a problem (example in the screen shot above). I never even thought of “sync” problems, to be quite honest.

With the editing, I did the following:

I opened VirtualDub so that two screens showed on the desktop. I would then load videos of approximately the same year in each e.g. Marli 3 in the one and Lize 1 in the other.

I would then go through both, checking which parts I have saved in one cassette and not in the other. I would save that piece of video as a separate clip. I would also cut the other video so that I knew where to insert it- creating therefore 2 clips.



I numbered them in sequence and saved them per year in chronological order.

#37 in thread: Advice on what filters to use when converting VHS to AVI:

I have progressed from 1986 to 2001 and have edited a bit with VD-especially deleting some errors as well as some of the duplicate scenes. I have also been able to sort the clips/videos in years, but a lot of work still needs to be done.

I gave the VHS-C cassettes preference above the already copied videos. In this way, I was basically able to reduce 3 videos to the size of 1 video, but have not yet linked the clips.

Perhaps due to the size of the clips, VD never plays my videos smoothly on my computers and I would not have noticed a sync problem. I did not load them into Vegas then.

I then started posting videos and you guys worked on the clips.

But that aside, look at the church video in post #4. It begins with two people talking on the steps of a church, but you can't hear anyone talking. Next shot is a group of people walking down concrete church steps (no sounds of footsteps heard), one or more of them talking rather loudly it seems (no sounds of anyone talking), car doors opening and closing (no door sounds), and traffic passing in the street (no traffic sounds).

This is odd, as surely Sanlyn, with all your years of experience, you must have noticed this discrepancy when you worked on the clip (before 19 July 2013 #51) and warned me that you have found a sync problem in this clip?

I find it strange/mysterious/odd that comments/assumptions like

I get the idea that these 300 clips are not going to be presented in their original sequence.

I believe the O.P. searched through the captures for all shots related to certain persons or certain chains of events that occurred over a period of time.

So these 300 clips were pulled out of the original captures and rearranged to a different pattern and sequence.

are made.

Why would I do one thing and present something else in the forum?

[sanlyn]

It seems as if Sanlyn is questioning my integrity and honesty in #31. I want to clear this up and see this as an unnecessary post.

Well, I did get a little shock from that one, but I wouldn't question your integrity. I can't imagine why anyone would.

Many of the clips I worked on had no identifiable way to tell if they were in sync or not (no one talking, or the person on screen not saying anything). I did have the speakers on at least once when first viewing each clip I downloaded -- not not always during later processing, but always after encoding. Today I reviewed all of the clips I worked on. I can't report on all of them because, as mentioned, some of them had no visible speaker as an identifiable source for the audio. I still have questions on a couple of them:

1.1.avi, post #1 6/29/2013: appears to be in sync
3(1).avi, post #1 6/29/2013: appears to be in sync
3.avi, post #22 7/07/2013: unable to determine if precisely in sync, lip movement is too blurred. Language problem. Seems to be correct.
4.avi, post #27 7/09/2013: appears to be in sync
6.avi, post #30 7/02/2013: appears to be in sync
b.avi, post #42 7/18/2013: not certain (language), but the father appears to be in sync.
d.avi, post #42 7/18/2013: no audio ? Track in Audio View shows no modulation. I assume that means audio was not recorded.
e.avi, post #42 7/18/2013: in sync, but could possibly be very slightly off ?
h.avi, post #42 7/18/2013: can hear woman talking. Is the boy making a sound?
i.avi, post #42 7/18/2013: Man and someone else is talking. Can the woman be heard? (appears to be speaking softly).
j.avi, post #44 7/18/2013: appears to be in sync
k.avi, post #44 7/18/2013: Is the photographer talking ? ? are we supposed to hear the other people? I assumed the microphone is not close enough to pick up all the voices. Is the male voice the man on the right, or is it the voice of the photographer behind the camera? If the others are supposed to be heard, this is not in sync. I didn't finish working on the cleanup for this. Ran out of time, so likely I did not actually hear it more than once.
LC.avi, post #44 7/17/2013: Can't say. Too many people talking.
n.avi, post #44 7/18/2013: cannot see the person making the sound

I'm sorry to say that I'm unable to fully comprehend every detail of your methods. I don't understand if you want everything in historical sequence, or if you want a sequence for each person, child, or group of children, etc. In any case, the actual sequencing doesn't really matter, because people frequently capture family tapes to a PC and rearrange the scenes. I do understand that you spent much time sorting and rearranging, and deleting duplicates (how did you get duplicates on VHS tape?. Were they professionally duplicated, or made from VCR to VCR?).

If there were early signs of bad audio sync, it seems to have appeared on only one of the above clips, and that one is nearly unintelligible. You would have to be the judge of whether it has good sound. This is only my second hearing of it and it appears out of sync to me. But perhaps it's OK: I don't know that much about what was going on or how each person's voice should sound.

[avz10]

Well, I can only answer/comment on what I read. It might be that because English is my second language; that I misunderstood what you were saying; but you might have at least asked twice when the first time was when I listened to the sound of the recording.

I get the idea (my bold avz10) that these 300 clips are not going to be presented in their original sequence. I believe the O.P. searched through the captures for all shots related to certain persons or certain chains of events that occurred over a period of time. So these 300 clips were pulled out of the original captures and rearranged to a different pattern and sequence (??? where did you get this idea from? avz10). This search-and-cut operation went on for some weeks -- I'm still trying to figure out how one would spend that much time editing a group of long captures without listening to the audio (my bold avz10).

To me, this quote does sound that I "have a hidden agenda" or not revealing everything that I am doing. I am obviously wrong???

You saw approximately a small part of 6% of my clips, but on analysis you did not clearly point out one that was clearly out of sync. You did not mention the church seen previously, of which there is no comment now and I simply asked how you could have missed such a bad sync. In that particular post, you said quite a lot about that clip, but did not mention it previously???

But that aside, look at the church video in post #4. It begins with two people talking on the steps of a church, but you can't hear anyone talking. Next shot is a group of people walking down concrete church steps (no sounds of footsteps heard), one or more of them talking rather loudly it seems (no sounds of anyone talking), car doors opening and closing (no door sounds), and traffic passing in the street (no traffic sounds). Apparently the audio for that sequence exists in another part of the capture. It could be 2 seconds back, it could be 10 seconds ahead. How would one pull a scene that long from a capture and have no related audio at all, if the source from which it was cut had not been out of sync to begin with?

I don't understand if you want everything in historical sequence, or if you want a sequence for each person, child, or group of children, etc.

I have explained this previously in short and now at length. I really cannot understand that you cannot see that I want to make a master copy and if you look at my screenshots and what I have written, it should be clear that I want everything in historical perspective.

Were they professionally duplicated, or made from VCR to VCR?

No, much easier. I connected the video camera to the VCR. I inserted e.g. Marli 3 (a VHS cassette) in the VCR and recorded what I wanted, leaving things that Lize was doing, out. I removed that cassette and for example put Lize 2 in and recorded what I wanted, etc.

These two comments I really do not understand:

If there were early signs of bad audio sync, it seems to have appeared on only one of the above clips, and that one is nearly unintelligible (my bold avz10)

And then:

This is only my second hearing of it and it appears out of sync to me

All I can say is that I have no idea what you want to say to me.

If one actually analyze your summary, then these are your findings:

Appears to be in sync: 8 clips
Unable to identify speaker: 8 clips
Uncertain: 3 clips

But then the last quote above states that " it appears out of sync to me"

I have now watched a number of clips and many are out of sync. A setting during my work in VirtualDub must have been wrong, either during capturing or during editing.

In any case, thanks for all your trouble with this project.

At this stage, I feel like just quitting. My goal was just decent digital video for my children to watch one day, but the focus got totally out of hand with scripts for a green line in this clip and a script for four bad frames in another clip.

To my children I have said on numerous occasions that this project, with photos that were supposed to be a next step, will be my legacy to them.

[2Bdecided]

To me, this quote does sound that I "have a hidden agenda" or not revealing everything that I am doing.

I think it's a language thing. I didn't read sanlyn's words like that. He could equally well have said "I can't believe you didn't listen to the audio", but that would not literally mean he doesn't believe you. It would mean "it's such a shame that you didn't listen to the audio" or "it's so unfortunate that you didn't listen to the audio", not "you're lying, you must have listened to the audio".



I'm really sorry, but I never listen to the audio when someone posts a video for advice. Like yours, my older machine can't always play audio properly when watching results in VirtualDUB, but unless someone mentioned an audio problem, I wouldn't turn my headphones up anyway.

At this stage, I feel like just quitting. My goal was just decent digital video for my children to watch one day, but the focus got totally out of hand with scripts for a green line in this clip and a script for four bad frames in another clip.

To my children I have said on numerous occasions that this project, with photos that were supposed to be a next step, will be my legacy to them.

I think you should take a break from these videos. It's really sad and disheartening what's happened after putting so much work in. The only comfort is that all you've lost so far is time, though the older I get the more I value that over some other things.



I think you're really lucky to have had access to a video camera when your kids were so young though. Next year you can (if you want) copy them one tape at a time and do nothing to them. You'll still have preserved them. They don't have to be perfect.


Cheers,
David.

[sanlyn]

Sorry, I'm away for a while. My wife checked the forum for me and told me that avz10 and others had posted new info. So....

Ahh, 2BDecided has the correct idea about the language problem. I definitely do not think anyone was "lying". This misunderstanding is similar to the one about the phrase "working too quickly". By "working too quickly" I meant that time was not taken to test or re-check the work at different phases.

You had audio problems in the past, and so have I. Some time ago I did a 90-minute capture during which the audio went progressively out of sync. I believe I mentioned that incident earlier. It wasn't noticed until the capture was finished and was quickly browsed for review. After 20 minutes the audio was slightly off. At 1 hour, it was 2 seconds off. By 90 minutes, audio was 4 seconds too late. It didn't take long to scroll quickly through the capture to detect problems. Since that time, I never accept a capture without a quick scroll through it, usually in its entirety. Yes, it does take about half an hour to quickly scroll and advance through a long capture. And, yes, it takes about half an hour to copy that capture to an external drive -- where I verify the copy itself, again, in the same way.

I have done absent-minded things such as forgetting to rejoin audio to a video, or to reset the audio rate after removing telecine when the audio was stored elsewhere. I've replaced frames in long sections of video, not realizing that the way the frames were replaced would cause a "bubbling" distortion in the audio -- and had to go back to an earlier version of the file to retrieve its audio and mix it with the new video. I ask again: what could I have done without the original, all because of 1 short minute of user error?

You saw approximately a small part of 6% of my clips, but on analysis you did not clearly point out one that was clearly out of sync. You did not mention the church seen previously, of which there is no comment now and I simply asked how you could have missed such a bad sync. In that particular post, you said quite a lot about that clip, but did not mention it previously???

How could I mention the church scene "previously" when it wasn't posted until much later. I believe someone else discussed that scene before I saw it and posted my comments. As for the very dim family scene posted last July, I heard the video for only the second time just 2 days ago. I heard the audio track only once in July and have not touched it since August 6, when I worked on color and contrast masking in Photoshop without playing the clip. I've already explained why the sound error was not immediately obvious. I don't think it would have been apparent in a brief, cursory initial hearing by someone unfamiliar with what the people are saying.

When I first saw that scene I made a single run with a set of standard filters but stopped the script because it didn't work well. I changed filters and tested a few frames without audio, so I didn't save the output; rather, I copied some of the partially denoised frames into Photoshop and spent a late evening trying to fix the horrible color. According to my copy of that forum work (I archive almost all work that I submit, on two 2.5" USB hard drives), I last saved the Photoshop filter settings for "k.avi" on 5 and 6 August, 2013, between the hours of ~9:00 P.M. and ~2:30 A.M. (you might note that this indicates more than 5 hours of work on color samples from a single set of still frames, taken from a video that plays for only 2.7 seconds in 69 frames). When have you spent that much initial time on one of your short clips? I did that because the video looked so ugly that it was a challenge, and because I realize the material's importance for you. I finally gave up at the time, and other problems intervened. I heard the audio for only the second time 2 days ago. I had to repeat it a few times in a media player to realize what was happening. All of my media players are set to "Repeat forever" for every clip I play. If I had completed work and encoded it, the finished clip would have been replayed completely many times in multiple players before being posted. I don't you think realize how many script revisions, tests, and encodes are used before many members here submit their ideas to users.

You might look at that specific clip another way: you must have watched it many times over the years. You must have reviewed it before posting it (or you should have). Surely you viewed it several times in your editing and in spot-checks of results. You would be much more familiar with the original sequence than would someone who views 69 frames of it for the first time, completely out of context.

I really cannot understand that you cannot see that I want to make a master copy and if you look at my screenshots and what I have written, it should be clear that I want everything in historical perspective.

The sorting order is not the point. The point is that the original captures, which have been disassembled and sorted into 300 separate clips, were the real "master copies" -- and they were discarded before completing the project. Sorting and editing are intermediate steps, not original masters. I can't count the number of times we've seen this story; there are countless versions from people who complain 5 years later that they destroyed both the original PC capture AND the original tapes!

Were they professionally duplicated, or made from VCR to VCR?

No, much easier. I connected the video camera to the VCR. I inserted e.g. Marli 3 (a VHS cassette) in the VCR and recorded what I wanted, leaving things that Lize was doing, out. I removed that cassette and for example put Lize 2 in and recorded what I wanted, etc.

[/QUOTE]Good heavens! I had suspected something like that. This explains why some of the clips look so bad. It's too late to complain about that now, but are the original camera recordings still available? That's what you should use, not badly flawed second-generation copies. Motion problems and many other artifacts in those duplicates can't be repaired, not even by playing the duplicates with better equipment.

I have now watched a number of clips and many are out of sync. A setting during my work in VirtualDub must have been wrong, either during capturing or during editing.

This is a frequent problem. And because it has happened to me, I know exactly how it feels. How or where the problem developed is impossible to determine without earlier versions. But at least it's not impossible to make new captures. What if something happens to your tapes or to some parts of them, rendering them unplayable? As for "legacy" -- something that all of us can appreciate -- you no longer have a reliable, true-to-source, lossless capture of your tapes. Tapes deteriorate and grow brittle over time, whether they are played or not. Eventually they won't be playable. Or your VCR could suffer a breakdown.

At this stage, I feel like just quitting.

Some people do. I don't think you will. You need a break. You can save your processing scripts and filter settings. They can be used again. You can even save the clips with bad audio: if you recapture (which you ought to do anyway for a safe copy of the tapes), you can extract the audio and overlay it into work you've already done. Clips that play properly should not need more work.

My goal was just decent digital video for my children to watch one day, but the focus got totally out of hand with scripts for a green line in this clip and a script for four bad frames in another clip.

To my children I have said on numerous occasions that this project, with photos that were supposed to be a next step, will be my legacy to them.

Did you want to present your legacy with most of those defects repaired? Or would you prefer to permanently leave them as they existed when you began?