Nero Video Grabber vs Hauppauge WinTV-USBlive2 vs August VGB300

[Alwyn]

While it is relative simple and reliable to discover duplicates by means of a script

Do you have a script for spotting dupes I could try?

[Sharc]

While it is relative simple and reliable to discover duplicates by means of a script

Do you have a script for spotting dupes I could try?

The DeDup avisynth plugin comes to my mind:
http://avisynth.nl/index.php/DeDup

The first pass 'DupMC' will create a log.
A very low difference (~0%) between 2 frames indicate a duplicate. The log can be evaluated in Excel for example, or evaluated and processed in pass2 ('DeDup') for removing the duplicates.

[jagabo]

AviSource("filename.avi"
WriteFileIf("Dups.txt", "YDifferenceFromPrevious<0.5", "current_frame", flush=true)

Open the AVS file in VirtualDub then use File -> Run Video Analysis Pass. When it's done you will have a file called Dups.txt that has a list of the duplicate frames. You can change the threshold to reduce false positives or false negatives. Also note that it only compares the Y channel (not U and V).

[Alwyn]

Thanks guys, I finally got DupMC working after seeing Jagabo's comment re running the video analysis pass. That isn't mentioned in the wiki or readme that I can see.

@Jagabo, I'm still experimenting with your script. I haven't worked out how the "difference" parameter affects things yet.

[jagabo]

YDifferenceFromPrevious is the average difference of the Y value of each pixel of the current frame and the previous frame. If the current frame and the previous frame are identical (as far as the Y values are concerned) the calculated value will be zero. It will be non-zero if any of the Y values is different.

Note that compression artifacts (when using lossy compression) or other noise may cause originally identical frames to no longer be identical. That's why a non-zero value is used in the sample code. It is no longer possible to detect for certain which frames were identical before the lossy compression.

[Alwyn]

Thanks, I'm working with analogue AVI captures.

I've set the "Diff from Prev" to 0.0 but am getting just a list of numbers (I assume the frame numbers, looks like very frame) with no other details, as per the attached.

The DupMC report does show the % difference between each frame.

[jagabo]

I've set the "Diff from Prev" to 0.0 but am getting just a list of numbers

Yes, that's all it shows, the frame number of frames that are duplicates of the frame before them. Oh, it also always shows frame zero as a duplicate, a false positive (as there's nothing to compare that frame to). It shouldn't show every frame is a duplicate.

[Sharc]

Thanks, I'm working with analogue AVI captures.

I've set the "Diff from Prev" to 0.0 but am getting just a list of numbers (I assume the frame numbers, looks like very frame) with no other details, as per the attached.....

The clip must be in planar format, so

AviSource("your.avi")
converttoYV16(interlaced=true) #convert to planar
WriteFileIf("Dups.txt", "(YDifferenceFromPrevious)<0.5", "current_frame",flush=true)

If you want to see the % as well:

AviSource("your.avi")
converttoYV16(interlaced=true) #convert to planar
WriteFileIf("Dups.txt", "(YDifferenceFromPrevious)<0.5", "current_frame",  """ ":" """, "YDifferenceFromPrevious",flush=true)

[Alwyn]

Excellent, thanks Sharc and Jagabo. That works.

[jagabo]

You can include the chroma channels using UtoY() and VtoY(). For YV16:

StackHorizontal(last, UtoY(), VtoY())

before WriteFileIf().

For YV12:

StackHorizontal(last, StackVertical(UtoY(), VtoY()))

[rgr]

While it is relative simple and reliable to discover duplicates by means of a script

Do you have a script for spotting dupes I could try?

The DeDup avisynth plugin comes to my mind:
http://avisynth.nl/index.php/DeDup

The first pass 'DupMC' will create a log.
A very low difference (~0%) between 2 frames indicate a duplicate. The log can be evaluated in Excel for example, or evaluated and processed in pass2 ('DeDup') for removing the duplicates.

A very unreliable script, like all the others. With compressed material and fast scenes, it can produce a high value for duplicate frames, while in still scenes, it can produce a low value for non-duplicate frames.

In fact, you should compare it with neighboring values, i.e.:
- 40, 3, 50 - in subsequent frames indicate a duplicate
- 3, 2, 3 - indicate a non-duplicate
- 3, 0.5, 3 - indicate a duplicate

Dfttest with high sigma (up to 64) increases the number of correct detections but eliminates easy recognition of perfect dups.

RT_FrameMovement (RT_Stats) is better than DupMC.

However, duplicates in lossless capturing should not differ at all (YDiff=0).

[rgr]

If you send me this file, I can check it with my program based on the above assumptions.

[rgr]

You can include the chroma channels using UtoY() and VtoY(). For YV16:

StackHorizontal(last, UtoY(), VtoY())

before WriteFileIf().

For YV12:

StackHorizontal(last, StackVertical(UtoY(), VtoY()))

Or use RT_FrameDifference (RT_Stats).

[Sharc]

A very unreliable script, like all the others....

It is not an "unreliable script". DupMC does exactly and reliably what it claims to be doing. It's up to the user how to interpret the data which it delivers.

[Alwyn]

DupMC is verifying that the AmarecTV report is correct WRT dupes (Sharc, I'm using = as the delimiter ).

And yes, I finally got AmarecTV to report two actual drops (not dupes). Never seen drops before in an Amarec capture report. And they were in the recording, at the frame count the report said. Were there other drops (without dupes)? I don't know, but with the audio in sync after 3 hours, I'm happy.

[rgr]

A very unreliable script, like all the others....

It is not an "unreliable script". DupMC does exactly and reliably what it claims to be doing. It's up to the user how to interpret the data which it delivers.

If it's supposed to show the difference between frames, then it can be considered effective.
If it's supposed to detect duplicates, then it performs average to good.

[rgr]

DupMC is verifying that the AmarecTV report is correct WRT dupes (Sharc, I'm using = as the delimiter ).

And yes, I finally got AmarecTV to report two actual drops (not dupes). Never seen drops before in an Amarec capture report. And they were in the recording, at the frame count the report said. Were there other drops (without dupes)? I don't know, but with the audio in sync after 3 hours, I'm happy.

Have you checked the frame counter between VDub and AmarecTV captures? For a 3-hour movie, AmarecTV should drop about 60 frames, or ~1 second.

[Alwyn]

For a 3-hour movie, AmarecTV should drop about 60 frames, or ~1 second.

?? 60 frames, even at 29.97, is 2 seconds. That would obviously be immediately noticeable if the audio wasn't corrected during capture. Since I've never had that sort of audio error, are you saying that Amarec adjusts the audio to compensate for the alleged masses of dropped frames?

[rgr]

Of course.
https://forum.videohelp.com/attachments/87807-1752419717/rating.png

Sync means nothing.

[Alwyn]

That's just your table, nothing particularly credible about, based on what I see.

What do you mean, "sync means nothing"? Are you suggesting that, throughout the capture, the audio just randomly drifts in and out of sync with all those 60 dropped frames but miraculously at the end it's perfectly in-sync?

If Amarec was dropping 20 frames in an hour without replacing them, the out-of-sync would be blindingly obvious.

[Sharc]

What do you mean, "sync means nothing"? Are you suggesting that, throughout the capture, the audio just randomly drifts in and out of sync with all those 60 dropped frames but miraculously at the end it's perfectly in-sync?

Hmm, what do these ~1s fluctuating fps in Amarec's status bar indicate? The variation of the duty cycle of ....-slow-fast-slow-fast-.... could be a strategy to keep A/V in sync. Just speculating.

[rgr]

That's just your table, nothing particularly credible about, based on what I see.

What do you mean, "sync means nothing"? Are you suggesting that, throughout the capture, the audio just randomly drifts in and out of sync with all those 60 dropped frames but miraculously at the end it's perfectly in-sync?

If Amarec was dropping 20 frames in an hour without replacing them, the out-of-sync would be blindingly obvious.

Apparently, it also leaves out parts of the audio. Or resamples audio (VDub can do this).
The Chinese grabber adds 66 frames, yet the audio is synchronized.

But you didn't answer the question. You haven't actually tested anything...

[Alwyn]

Latest test.

Workflow: HR-S5700AM>S Video>Pioneer 645H>S-Video>DVK-200>S-Video>IOData GV-USB2>Lagarith

Using VDub 2: 7 Duplicate frames, total frame count 86224

Using AmarecTV: 1 Duplicate frame, total frame count 86219.

I don't have the ability to detect dropped frames and there was no audio accurate enough to check the sync.

[rgr]

I'm almost certain that where VDub duplicated the unreadable frames, AmarecTV permanently dropped without reporting as usual. However, due to the "interesting" workflow and my lack of knowledge about VDub's settings, it's difficult to say for sure.

[Sharc]

I don't have the ability to detect dropped frames and there was no audio accurate enough to check the sync.

Detecting dropped frames reliably can be difficult and tedious. Duplicates detection is pretty trivial.
Also keep in mind that playing the same tape twice in different runs will usually not deliver identical results, even using the same equipment. It's the analog realm.

[aramkolt]

I always speculated that the only real way to know where frames were being dropped is if you used a timecode generator that also burns a unique timecode into each frame while also producing LTC audio that you capture along with it - the idea would then be to see if the difference in time drifts between the "audio" and "burnt in" timecode. You'd almost have to go frame by frame to then see if right after a drop that you weren't just getting an inserted frame. An inserted frame would show two of the same "visibly burnt in" timecodes on two neighboring frames whereas a drop would skip a "visibly burnt in time code" possibly then immediately following with a duplicate of that good frame. This is not ideal for capturing where you don't want a burnt in timecode to be visible on each frame, but might be useful in troubleshooting to determine which chains are most prone to dropping frames on certain difficult tapes.

You could also make a fresh "test" recording from something stable like a DVD and have the timecode burnt into that stream while recording onto a VHS tape with the LTC audio being one of the audio channels. From there, You'd capture as normal and look for drops in the capture. You could also just try putting the timecode generator inline with the output of a real capture as well and then just look for drifting in the LTC audio compared to the actual number of frames captured up until that point. If you then check for duplicate frames and don't find any and there are the expected number of frames between the start and end of the capture, I think it would be safe to say that no frames were dropped. Every duplicated frame you find likely represents a dropped Frame that had an insert placed in to keep timing correct. If you see a duplicate frame at a very regular interval, that to me says there's a slight difference in the VCR's playback clock/servo compared to the recording framerate.

You might also be able to make a test recording with a visible timecode and LTC audio present from within a nonlinear editor as the starting source and then record that to a tape as a test source.

Something like this I think could work for these types of "inline" tests: https://www.ebay.com/itm/224737578141

[Sharc]

And the test should be field based IMO, as analog capturing is field by field by field ...
Interlaced frames are built by the capturing device/software only.

[rgr]

I always speculated that the only real way to know where frames were being dropped is if you used a timecode generator that also burns a unique timecode into each frame

This is how I performed this test, unfortunately the screenshots from the posts have already disappeared from the image hosting server.

Edit: post #49.

[aramkolt]

Ah, I missed quite a bit on this thread which is all very interesting and systematic! Did you ever try the "DVA-Fidelity" thing? That looks pretty interesting as well. It'd be nice if it was more user friendly in the form of say a DVD that one could then record to a VHS or other tape as a first step. From there, I"m not sure if the analysis part could be made any easier or not as most users probably don't regularly use linux. Does DVA fidelity do anything with LTC audio, or is it purely looking at frame drops based on the series of known-ordered frames?