The old "war of opinion" - Capture 8mm tapes in DV or lossless?

[DB83]

Way back in this thread I mentioned that the ffmpeg port of huffyuv was somewhat cavalier when it came to bitrate etc. So allow me to present a comparison between one of my sample vids to your recent one. Note the significantly higher bit-rate (even without audio) and you now appear to have a progressive video.

Also I do not see any response from you re the BM and selecting the other quoted capture source in vdub >> Decklink IIRC. No idea if it would make any difference.

[Sharc]

@JohnMeyer

As promised, here is a sequence of our ski holiday trip in 1996 in Austria. It's about 1 minute long. I captured - as usual - uncompressed then converted it using VDub into FFMPG Huffyuf 4:2:2, no RGB.
Like I said in my prior post - watching the capture in my preview monitor I can never see any abnormalties such as dropped frames or anything else.

PS: since the file size exceeded the maximum file size of the forum by only 5 mb, I uploaded it on WeTransfer: https://we.tl/t-XGcTx2ulOl

Duh ... these strong blue colors....
I know we should only discuss dropped frames now, but this capture is an example of what has been discussed earlier regarding clipping and out-of-gamut RGB as I see it.
The captured Y,Cb,Cr (aka YUV) values are valid and legal, however when converted to RGB (for viewing on a display) the levels (mainly the blue) are badly clipped. The Y,Cb,Cr (YUV) of the capture are partially outside the RGB cube shown in post#208. So the capture as such is valid IMO, but for distribution one may want to apply color corrections (desaturate, white balance .....).
Sorry if going off topic. Colors are also a matter of personal preference.

[Marvolo]

Way back in this thread I mentioned that the ffmpeg port of huffyuv was somewhat cavalier when it came to bitrate etc. So allow me to present a comparison between one of my sample vids to your recent one. Note the significantly higher bit-rate (even without audio) and you now appear to have a progressive video.

I have no idea how and why this is... All I did was select "normal recompression" then "FFMPG Huffyuv and changed the color space to 4:2:2, no RGB. Should I have used a different lossless one?

Also I do not see any response from you re the BM and selecting the other quoted capture source in vdub >> Decklink IIRC. No idea if it would make any difference.

Neither the WDM BM driver nor Decklink made any changes. It just stays black.

[DB83]

Like I said, your results are consistent to what I received when I tested it some time ago. I appreciate you have little choice in the matter. So your files will be smaller than the uncompressed equivilent and probably (but do not take my word for that) more manageable in your NLE. The 'pleasures' of using proprietory hardware.

[Marvolo]

Sadly, I cannot open them for playback when converted to Huffyuf. Not with VLC, not with Windows Media Player. It seems to be a very incompatible codec. Also, the progressive flag should not be correct?? I have no idea how that came about.

[DB83]

Play them in vdub2. But as stated, the clip reminds me of a Status Quo album "Blue For You"

[Sharc]

Sadly, I cannot open them for playback when converted to Huffyuf. Not with VLC, not with Windows Media Player. It seems to be a very incompatible codec. Also, the progressive flag should not be correct?? I have no idea how that came about.

Open it with ffms2 or LWLibavVideoSource.
Your capture is interlaced, TFF. There is motion with every field advancement.
Some .avi codecs/containers do not set an interlace flag, and the analyzing tool may simply default to progressive then.
Just make sure that your NLE interprets the video as interlaced (usually a source parameter one can select/overwrite).

[Marvolo]

Play them in vdub2. But as stated, the clip reminds me of a Status Quo album "Blue For You"

Must be due to the S-Video output. I can check if with my neighbor's cam it will be the same...

Here's the DV 2008 "reference" file. It doesn't seem to be quite as blue.
However, the blue saturation only seems to be in the very beginning of the clip. After a while I don't really see a difference to the 2008 file.

Anyway...
But what about drops? That's what I was actually trying to find out. Does the lossless capture have any drops? Does the 2008 "reference" have any?

[Brad]

Some .avi codecs/containers do not set an interlace flag, and the analyzing tool may simply default to progressive then.

Possibly it's reporting that FFVH has been set to Progressive (frame-based) compression mode, unlike DB83's HFYU clip. The dialog box in VDub2 doesn't offer any setting for it.

As you know, this is unrelated to the type of flagging some codecs can do that actually allows a player to see the video as "interlaced".

HYMT config dialog contains a tooltip regarding its own internal flag. The default in HFYU before other authors started patching the original source was that inputs with 288 lines & below were compressed frame-by-frame while 289+ were compressed on a field basis. My DFAQ post on this topic.


Note: there is no PQ impact from configuring this setting incorrectly; the output is mathematically lossless regardless. "Interlaced" mode will just compress better for content with significant motion between fields.

@Marvolo: FFVH defaults to Prediction: Left. Change to Median for somewhat higher compression ratio.

[Sharc]

Must be due to the S-Video output. I can check if with my neighbor's cam it will be the same...

Here's the DV 2008 "reference" file. It doesn't seem to be quite as blue.
However, the blue saturation only seems to be in the very beginning of the clip. After a while I don't really see a difference to the 2008 file.

Well, both have a blue tint. Apparently this is how the camera took it (auto white balance, some have a setting for "snow scenes", "beach", "portrait" ....). If you don't like it's something for post processing. Easier than tweaking the capture levels.

[lordsmurf]

You should monitor captures. Look at the logging. You should know if the capture was fine at 1:00 (one hour), but you saw dropped frames when you came back at 1:10. Stop capture, start again from just before 1:00.

How do you recognize dropped frames just by monitoring the capture? Whenever I capture I constantly watch the preview monitor and everything seems to look fine and flawless.

VirtualDub has a dropped frames counter. Look at it.

Avoid cards that do not report dropped frames, ie HD cards that "also does SD" (and poorly) such as Blackmagic cards.

Drops/inserts (repetitions) produced by external devices such as TBC, DVD recorder used as ADC or in passthrough mode will normally not be reported by the capturing SW.

Yes and no.

- Almost all drops are from capture cards. This is a reason that "internal" (which includes USB) capturing is suggested, because you can monitor what is happening.
- Actual TBCs rarely drop/insert, while "TBC(ish)" devices (DVD recorders, etc) can as well (and often do).
- External capture boxes, such as DV boxes, or even SDI boxes (but less so), do drop/insert as well internally, and you have no idea what is happening without frame-by-frame scanning of out put video.

[lollo]

@Marvolo: FFVH defaults to Prediction: Left. Change to Median for somewhat higher compression ratio.

Even better, stop using VirtualDub2 and its hacked HuffyYUV codec, and install original HuffYUV codec to be used with original VirtualDub (not 2)

[lollo]

Avoid cards that do not report dropped frames, ie HD cards that "also does SD" (and poorly) such as Blackmagic cards.

LS, we are in a specific workflow here.

The capture card is the Panasonic device. BM is only transferring the digital stream. Everything "happpening" at capture process is between the Player and the Panasonic device, and AFAIK there is no way to report that with this workflow.

[Sharc]

Here's the DV 2008 "reference" file. It doesn't seem to be quite as blue.
However, the blue saturation only seems to be in the very beginning of the clip. After a while I don't really see a difference to the 2008 file.

There are differences. FWIW here a comparison. See the dark blue sky on the top/right of the side-by-side pictures. (The comparison relates actually to the original question of this thread )

[Marvolo]

@Sharc

if you compare it like that, I can definitely see differences, yes.

is "V1" the lossless capture and "V2" the D8 2008 capture? If so, then my impression is that the V1 has more powerful colours than V2. Do you mean that in your opinion, the lossless approach "wins" over the D8 one as it has the stronger / more powerful colors?

Or is this just what we have already found out, that is, the lossless capture has a better saturation than the D8 capture?

[poisondeathray]

RE: Skiurlaub

There is a drop between field 7 and 8 for FFVH . This time there is no compensatory duplicate field inserted

If the mechanism dropped a field, then always inserted a compensatory duplicate, the field count would be the same between DV and "lossless" capture. (They would line up at the beginning and end in your NLE, but could differ in the middle)

This and the earlier example demonstrates your setup has different variations - so it really depends on "what else is going on"

Was this the "new" capture setup, with everything else turned off ? The capture guys really don't like FFVH, try one of the "magical" huffyuv variants

[Marvolo]

There is a drop between field 7 and 8 for FFVH . This time there is no compensatory duplicate field inserted

Do you mean frame 7 and 8? If so, I cannot see a missing frame. Or maybe I just don't know what I'm supposed to be seeing there.

Was this the "new" capture setup, with everything else turned off ? The capture guys really don't like FFVH, try one of the "magical" huffyuv variants

Yes, I reinstalled Windows and only installed the necessary drivers for the hardware as well as the BM capture software. Everything else is turned off or has been uninstalled (like programs/apps that came with Windows). The computer has never been connected to the internet since Windows was reinstalled. There are no other programs or anything else running except for the BM capture software. I even tried to defrag the HDD drives, but apparently, Windows seem to be doing this on its own from time to time. There was nothing to defrag there according to the analyses.

If you tell me how I can see what you see, then maybe I can find them myself in the future, but I actually don't see anything between frame 7 and 8.

Also, I do not know what else I could do to my system to keep it from dropping frames. I mean, it's basically running on a freshly installed Windows with only the BM capture software and everything else has been disabled or uninstalled.

[poisondeathray]

There is a drop between field 7 and 8 for FFVH . This time there is no compensatory duplicate field inserted

Do you mean frame 7 and 8? If so, I cannot see a missing frame. Or maybe I just don't know what I'm supposed to be seeing there.

Fields not frames. Otherwise you're only seeing 1/2 the information , or it will weaved into a frame (combined) so that you cannot see clearly

If you're looking at frames, you need to look at 50fps, where each field is separated and interpolated to a frame

A way to do that in most NLE's is drop it onto a 50p timeline. It will be double rate deinterlaced to 50p. I prefer avisynth/avspmod , but everyone has their preferred methods

The field order matters, the DV is BFF, the FFVH is TFF . If you interpret the wrong field order, you will get jerky fwd/back motion

[Marvolo]

Also, I do not know what else I could do to my system to keep it from dropping frames. I mean, it's basically running on a freshly installed Windows with only the BM capture software and everything else has been disabled or uninstalled.

I also believe that my system, even though 15 years old, should still be strong enough to capture 576i without dropping information?? People had been doing that way before 2009 when I got my current system.

[lordsmurf]

Avoid cards that do not report dropped frames, ie HD cards that "also does SD" (and poorly) such as Blackmagic cards.

LS, we are in a specific workflow here.
The capture card is the Panasonic device. BM is only transferring the digital stream. Everything "happpening" at capture process is between the Player and the Panasonic device, and AFAIK there is no way to report that with this workflow.

I was just referencing in general, not specific to the thread.

That said, and as I've mentioned many times in the past, "closed loop" capture systems (which is what this essentially is here) can be a problem, because you have no idea what is happening. Dumb boxes for dummies. And yes, that includes "pro" boxes for pro dummies. "Magic" boxes that "do everything", with little to no checks or safeguards to the I/O data. Putting faith in a machine. That always ends well, right?

Use proper tools for the specific task at hand, not random tools for another task. (Some people stubbornly like to whack screws with a hammer, makes no sense to me. Wrong tool.)

[DB83]

For the record, you can still install a regular huffyuv codec in both vdub and vdub2 and use that for the conversion rather than the ffmpeg port available in vdub2

[Marvolo]

For the record, you can still install a regular huffyuv codec in both vdub and vdub2 and use that for the conversion rather than the ffmpeg port available in vdub2

I will do, but I believe this will not get rid of the dropped frames / fields problematic?

[Marvolo]

Use proper tools for the specific task at hand, not random tools for another task. (Some people stubbornly like to whack screws with a hammer, makes no sense to me. Wrong tool.)

What would be your suggestion as for lossless capture of analogue video which at the same time is still according to the video norm (many of those out-of-the-box solutions e.g. grabbers etc. produce nothing which meets the standards or the norm --> see those long threads on testing and analysing in the German forum. That's the reason why they ultimately came up with my current set-up as this is one of the few capture chains that captures according to video standards and norms.

[johnmeyer]

For software DV codecs, for instance, it was well known that the DV codec built into Windows was absolutely awful, whereas professional DV codecs, like the one sold by the Germain company Mainconcept, were exceptionally good. I've often suspected that those who dump on DV never used the Mainconcept, Canoupus, or Sonic Foundry DV codecs.

Contradictory statement, John. While capturing you have no access to the DV codec, because it is implemented in the device. On reading, or decoding for further processing, and eventually encode back to DV, I also experiemented all available DV codecs available at that time. And found Canopus and Sony to be the best, not Mainconcept (personal opinion). But this has no relation to the original question: better DV or lossless approach?

I tried to differentiate between those two things: 1) The A/D converter and 2) The compression performed on the bits handed to the DCT algorithm by the A/D converter. So, yes, I understand that when digitizing analog tapes using a Digital8 camcorder (or doing the same thing for DV tapes) there are two separate things going on and BOTH of them can have differing degrees of quality.

I do agree that when digitizing using a camcorder, you have no separate access to the A/D and the DV codec because they are built into the camera.

When digitizing, it is possible to get better results by spending more time looking at nearby pixels and making different decisions. This is similar to what many people reading this have experienced when compressing music which has been digitized to a lossless format like WAV and then compressing to a lossy format like MP3: the encoder usually has a "quality" slider where you can tradeoff encoding time for quality. By being "smarter" about how DCT blocks are formed, more quality can be maintained.

Finally, when playing back the DV video, there are no choices to be made, and all decoders will produce the same result (same thing with MP3 audio). There is no way to magically improve already compressed digital video unless there is some post-processing done (like what is discussed in these forums about using AVISynth filters to improve video). However, the actual act of retrieving video from DCT storage is strictly a set of "assembly instructions," leaving no room for choices or improvements.

However, if the resulting digital video has to be changed back to analog with a D/A, then that operation can indeed further screw up the video, depending on the quality of the D/A conversion.

[lollo]

many of those out-of-the-box solutions e.g. grabbers etc. produce nothing which meets the standards or the norm --> see those long threads on testing and analysing in the German forum.

They do, if you use the recommended workflow: high-end player with Y/C output and (line)TBC correction -> one of the recommended capture cards, eventually a (frameTBC) in between.
The HDMI approach is a good alternative, has the advantage to avoid lossy A/D conversion, but relies on the quality of the Panasonic device digitizer and is not easy/impossible to have a report of dropped/inserted frames as in the classic workflow.

[lollo]

For software DV codecs, for instance, it was well known that the DV codec built into Windows was absolutely awful, whereas professional DV codecs, like the one sold by the Germain company Mainconcept, were exceptionally good. I've often suspected that those who dump on DV never used the Mainconcept, Canoupus, or Sonic Foundry DV codecs.

Contradictory statement, John. While capturing you have no access to the DV codec, because it is implemented in the device. On reading, or decoding for further processing, and eventually encode back to DV, I also experiemented all available DV codecs available at that time. And found Canopus and Sony to be the best, not Mainconcept (personal opinion). But this has no relation to the original question: better DV or lossless approach?

I tried to differentiate between those two things: 1) The A/D converter and 2) The compression performed on the bits handed to the DCT algorithm by the A/D converter. So, yes, I understand that when digitizing analog tapes using a Digital8 camcorder (or doing the same thing for DV tapes) there are two separate things going on and BOTH of them can have differing degrees of quality.

I do agree that when digitizing using a camcorder, you have no separate access to the A/D and the DV codec because they are built into the camera.

When digitizing, it is possible to get better results by spending more time looking at nearby pixels and making different decisions. This is similar to what many people reading this have experienced when compressing music which has been digitized to a lossless format like WAV and then compressing to a lossy format like MP3: the encoder usually has a "quality" slider where you can tradeoff encoding time for quality. By being "smarter" about how DCT blocks are formed, more quality can be maintained.

Finally, when playing back the DV video, there are no choices to be made, and all decoders will produce the same result (same thing with MP3 audio). There is no way to magically improve already compressed digital video unless there is some post-processing done (like what is discussed in these forums about using AVISynth filters to improve video). However, the actual act of retrieving video from DCT storage is strictly a set of "assembly instructions," leaving no room for choices or improvements.

However, if the resulting digital video has to be changed back to analog with a D/A, then that operation can indeed further screw up the video, depending on the quality of the D/A conversion.

?????

Are you repeating the same concept I expressed when I replied, and then contradicting your first statement, or am I misunderstanding something?

Does not matter, it's an Off Topic anyhow...

[Sharc]

@Sharc

if you compare it like that, I can definitely see differences, yes.

That has been the purpose, to show that there are differences. See also the color flicker in the sky. Difference in gradients and banding.

is "V1" the lossless capture and "V2" the D8 2008 capture? If so, then my impression is that the V1 has more powerful colours than V2. Do you mean that in your opinion, the lossless approach "wins" over the D8 one as it has the stronger / more powerful colors?

V1 is the lossless, V2 is the DV variant. Just a comparison, not to declare a winner.

Or is this just what we have already found out, that is, the lossless capture has a better saturation than the D8 capture?

You decide how much blue the blue should be. The blue is clipped in RGB for both variants. As long as only one of the RGB is simultaneously clipped (mostly BLUE in your case while RED and GREEN are sane) it's not that problematic. It is from the camera, not primarily a capture issue.
It's a side-by-side comparison between the 2, no less and no more.

[Marvolo]

If only it would capture without dropping frames or fields, then I'd keep the lossless ones next to the DV ones as archive.

At the moment, I am not sure if I should continue capturing all those tapes not knowing if they end up fine on my HDD or not. I don't want faulty files for archive or mastering purposes.

That said, has anyone found dropped frames or fields in the 2008 DV files so far?
I'm trying to get Vdub to capture from my BM card. Maybe if I stop capturing uncompressed, it will improve the situation and avoid drops completely?

I assume, the reason why it won't capture with Vdub is the same reason why it won't capture with my MAGIX NLE. Even though it recognizes the BM card and I can choose between Decklink and WDM drivers, the preview window just remains black. No signal coming through.

BTW: interestingly, MAGIX has a counter for dropped frames whenever capturing analogue. Right beneath the preview window. It doesn't for DV.

[DB83]

Apart from what has already been written, I guess the reason why 'capture' software does not report dropped frames for DV is because there is no actual capture. In it's purest form a DV stream merely transfers what is already stored in digital form on the tape. The only choice is how you treat audio either by Type 1 or Type 2.

In the less pure form the device, in this case a D8 camera, converts the analogue content on the tape to a DV signal. If there are 'drops' then they are out of the user's control since the conversion is done internally, again, that DV signal is transfered as-is by firewire with no further conversion.


Now I will leave this thread since I do not seeing it going anywhere in the present circumstances. But I will leave with one thought that the OP may wish, if he knows what he is writing about, to expand upon the notion of 'video standards and the norm' which can only be achieved by the method employed. Methinks, and I certainly am, most on here would take great offence that the methods we employ in converting analogue video to digital are not done utilizing ' video standards and the norm'. In my book, such a comment stinks of eliteism.


Rant over.

[Marvolo]

In the less pure form the device, in this case a D8 camera, converts the analogue content on the tape to a DV signal. If there are 'drops' then they are out of the user's control since the conversion is done internally, again, that DV signal is transfered as-is by firewire with no further conversion.

True. But doesn't the same principle apply with my current set-up? Now, instead of the D8 camera, the Panasonic Recorder does the conversion and feeds it to the BM card as a pure digital signal. If there are drops, then it must be done internally (maybe the Panasonic?)?

As I understand it, the same principles are in effect here. The panasonic feeds a purely as-is digital signal without further conversion to the PC / BM card...