Program / Plug-in to identify TBC errors (but not correct them)

[robjv1]

It's often the case that when I work with tapes with a TBC engaged (ie JVC/Panasonic TBC), that inevitably there are at least a few frames (sometimes more) where having the TBC engaged will cause a slight jump in the picture. Looking at it on a frame by frame basis, typically what happens is that the whole frame shifts directly upward, for a duration of usually around 4 frames of video, causing a blur to the picture before the frame returns back to it's normal position.

It is a minor problem of course, offset by the benefit of having the TBC engaged -- but in the cases where one wants absolute perfection, there aren't too many options for correcting it.

I end up having to resort to a sort of brute-force method -- where I watch the entire duration of the tape, make note of the locations where the TBC has caused an issue, and then correct those parts by editing in non-TBC engaged versions of the various clips. This is fine for tapes that I personally cherish, but it is not ideal for other tapes.

I've tried the VirtualDub deshaker filter for trying to correct these segments without having to redub -- and while it works pretty well, it's not perfect. (Although perhaps with some tweaking of the settings...)

What would really be ideal for me, would be to come up with a tool that can just identify the portions where one of these types of errors occur, generating a list of times that I could in turn use to identify which segments need to be redubbed without having to watch every second of each video like a hawk. I'm sure it would be nearly impossible to get perfect -- it would miss other problems with the footage and there would certainly be false positives, but it would be nice to have something that can approximate a very specific error and cut down on my overall time spent examining it by a significant margin.

Does something like this already exist? If not, what sort of an algorithm do you think would be needed to deal with it? It seems to me that you'd have to identify it based on some % change from one frame to the next over a series of frames, obviously excluding cuts where the picture is 100% different. I emailed a professor at school about this idea, and he suggested that Python/OpenCV might be a good language to program something like this in.

Any thoughts, ideas?

I guess this is what extremely anal people daydream about in their free time

[poisondeathray]

Just an idea -

you can use the log file generated from deshaker or depanestimate in avisynth , import into excel or spreadsheet program, and use a custom search string or sort by ascending or descending; this will help to identify large jumps (either + / - in the Y-axis or X-axis) with the corresponding frame #

      1    0.02   -0.02  -0.011 1.00018
      2   -0.01    0.01  -0.002 1.00008
      3    0.02   -0.01  -0.007 1.00011
      4    0.38    0.16  -0.015 1.00023
      5    0.10    0.78   0.045 1.00022
      6    0.30    0.19   0.012 0.99999

I think the syntax is difference between frames for x, y, rotation, and zoom. The 1st column is frame# (starting with 0, so "row 1" is the difference between frames 0 and 1 for those parameters)

If you don't need zoom or rotation, you can turn those off in the deshaker settings (or depan settings) , and the search will be faster

For example, if you wanted to search for vertical translation (y-axis jumps) , you could highlight the 3rd column and sort by descending to identify the large jumps

I think it looks to differences between frames, so if you have 4 frames that are shifted up in a row, it might only flag the 1st one as "shifted" (because the difference between the 1st shifted frame and 2nd shifted frame is zero). You can do some tests and figure it out

Depanestimate has a many more parameters you can adjust for the search, for example you can limit the range for detection (say 6 frames, or 10 frames), or dxmax (max x translation), or dymax etc... Adusting some parameters will cause the search to slow down significantly. You can read the documentation for more info

[robjv1]

Very clever idea! I didn't even realize it generated a log file. Thanks for responding, I'll mess around with this and repost later.

[poisondeathray]

I just did a little test; it works as advertised

I made a testclip, where frames 180-183 shift up and blurred as you described

Here is an excerpt from that log

 
    176    0.09    0.04   0.000 1.00000
    178    0.10    0.07   0.000 1.00000
    180    0.17    6.60   0.000 1.00000
    182    0.08    0.00   0.000 1.00000
    184    0.33   -7.60   0.000 1.00000
    186    0.62    0.46   0.000 1.00000
    188    0.59    0.18   0.000 1.00000

I uploaded the testclip and logfile

The script used is below, and you just have to play the .avs file in mpc or vdub

AVISource("test.avi")
mdata = DePanEstimate(range=1, log="test.log")
DePanStabilize(data=mdata,dxmax=0,dymax=0)

I set dxmax, and dymax to zero for depanstabilize, because you don't want to stabilize the picture, just want to print the log file to see the shifts

There might be a smarter way to sort it in excel, but if you sort by ascending and by descending you should get the large + and - shifts . Or maybe someone can write a little application to simplify this

[robjv1]

That's great, a very good simulation! I'll try it out with a real world example as soon as I can dig one up and see what the log says. After seeing your clip though, I think the shifted TBC effect is somewhat less clean -- it might be more like it shifts up and down a little bit both within that span of time.

This is great though -- I'm sure I could write a macro (or find a way to parse the log file) to find consecutive or clustered frames of an absolute value of |6| or |7| or whatever the common range ends up being.

[poisondeathray]

Since you probably have interlaced source, you need to deinterlace it first for the algorithm to work correctly. You don't need a super slow high quality one; a simple bob() will do because all you're looking at is x,y translational movements

If you use bob() , it will convert to double rate (each field becomes a frame), so it will return 59.94 frames per second. So you if you want the frame number (not field number), use selecteven() or selectodd()

e.g.
AVISource("interlaced.avi)
Bob()
SelectEven()
mdata = DePanEstimate(range=1, log="test.log")
DePanStabilize(data=mdata,dxmax=0,dymax=0)

[robjv1]

Ah yes, good point. They are all interlaced sources. I'll have to see how long a quickie deinterlace will take on this system -- I'll just go with whatever algorithm is the very fastest. I'm sure it will add to the time a bit, but still way less time than for me to sit down and watch it all

[poisondeathray]

bob().selecteven() or yadif is pretty quick

If you want to speed it up (if you have multiple cores), use multiple instances of vdub so you process several clips simultaneously - but don't forget to change the scripts and the output log file name . You may run into other bottlenecks like HDD I/O

[luigi2000]

This process is a lot of possibly unnecessary work. I recommend trying a different VCR and/or a different TBC to eliminate as many framing disturbances as possible before resorting to video filters. Detection of off tape vertical sync from a jittery or noisy signal is sometimes problematic.

[robjv1]

Re poisondeathray:

I've been messing around with it with some real clips and so far the results are looking good! Most of my outliers are around +5 to -5, although I had a few that a bit more or less. Over the weekend, I'll try it with a longer video where I already know the position of all of the displaced frames and see how it does.

This process is a lot of possibly unnecessary work. I recommend trying a different VCR and/or a different TBC to eliminate as many framing disturbances as possible before resorting to video filters. Detection of off tape vertical sync from a jittery or noisy signal is sometimes problematic.

Point well taken. I don't plan on using this tool to fully automate the process though, just more or less supplement it.

Primarily, I'd like to use this tapes that from all outward indications have very little in the way of jitter, but occasionally exhibit the "jumpy TBC" syndrome.

Remember, I'm already watching these tapes straight through from end to end, from the first frame to the last frame, to correct what amounts to maybe (on average) 5 or 10 seconds of displaced frames over a two or three hour program.

Something like this (if reliable) would save me a great deal of time in that scenario.

[robjv1]

One other question -- what is the most efficient way to enumerate all of the frames for purposes of getting results to the log? Should I just export it as a file, or would it be faster to scan the video stream for errors, or something else?

[poisondeathray]

One other question -- what is the most efficient way to enumerate all of the frames for purposes of getting results to the log? Should I just export it as a file, or would it be faster to scan the video stream for errors, or something else?

I'm not sure I understand what the question is ? Can you clarify what you're asking ?

[robjv1]

Nevermind, you addressed it above with the mention of using multiple cores. I'm just wondering what modes I should run it in to prevent as many bottlenecks as possible. I may have an excuse to buy a new computer for this... 30 seconds for a 42 second video is probably not so great

[poisondeathray]

I guess you could run it overnight

I suspect the bottleneck is the either depan/deshaker. Some filters are multithreaded, but I think depan isn't (or very poorly) , and deshaker is quite slow too (especially if you use accurate settings). So even if you have multiple cores, it may be using low cpu%. So the only way you can speed those up is to use multiple instances on separate clips.

BTW , if you have black borders, they should be cropped beforehand for better analysis (it's the same for stabilizing in deshaker or any motion analysis)

[robjv1]

Yeah, it's not a big deal. I've got lots of these captured to my hard drive already, so I could easily run one each night and build up a good size collection of stuff to fix once I know it's a reliable way of doing it.

[poisondeathray]

You may have to adjust other parameters, esp. the "trust" value if you're using depan . It has automatic scenechange detection, but you may want to incr. or decr. it . Have a look at the documentation to tweak some of the other parameters.

Definitely run some more tests before you commit to this

[robjv1]

Absolutely and thanks for all of your help!

I'm looking at this as a long-term experiment at this point.

I figure I'll continue on using my normal method for the next couple of months and cross-check it with the log results to see what % of stuff it's getting, what % of stuff it's missing, how tweaking it changes that and all the other factors I didn't consider to begin with. (One thing I can think of right off that bat, is when the TBC error occurs at a cut between two shots in the video).