DV tape transfer to computer: completeness and metadata

[ave1965]

I am trying to create an archive of multiple miniDV tapes on my hard drive and have read a lot about various ways how that can be done. Would appreciate any help with the following questions:

- As I understood from multiple posts, many programs can be used to import the data in the original DV format without transcoding: Final Cut Pro (FCP), iMovie, Quicktime (I am a mac user). Do those programs transfer EVERY bit of information recorded on the tape, or they leave some technical data behind? I am particularly interested in the metadata, such as date and time when a particular shot was taken.

- If the date/time information is transferred to FCP or other programs, how can you display them?

- My camera manual (Canon Elura 50) states that there are additional data recorded on the tape, such as exposure and ISO information that can be read by special software. I have NEVER been able to find a way to extract those metadata. Are they taken along in the usual import procedures, e.g. in FCP? If yes, how can I see them? If not, is there any special software that can extract them?

Thanks in advance.

[Richard_G]

No idea about Mac software but when using WinDV or similar on a PC over Firewire the transfer is just that. A bit for bit transfer of data from the tape to hard drive so all data stored on the tape is transferred. This is a transfer to DV-AVI. If you use something that converts to a different format, it may very well not.

[Cornucopia]

Pc format depends on the capping app. Most cap to AVI, a few cap to MOV and a few cap to MXF or even just the raw streams.

On a Mac, because QT is the underlying MM engine, most are capped into a MOV file, but even then it isn't universal. Imovie usually caps just the raw stream. AVIDs cap to OMF or MXF container.

Regardless of app or fileformat, assuming they are going through Firewire, they ARE getting the full stream info at the point of capping. However, they may not be SAVING all the data. For example, Many apps when given a dv stream that contains 4 channels of audio will only save 2 (depending on the app you may be able to choose, or not).

But it is trivial to save the metadata (in terms of size), and is probably trivial to save the VAUX userdata (programmingwise), so there is no reason NOT to keep it.


However, I note that you are talking about Exposure & ISO. While these come into play in video, that is something that was taken up into the metadata in hardware much more early with still cameras and the EXIF specs. Video does not have its own universal equivalent (though the rise of DSLR video may change that). Pro cameras have metadata, but lots of consumer ones don't . (Mp4 + h264 + Solidstate/file-based recording is changing this). But DV or HDV? Not likely much there.

If you want to be sure, take your material to a Pc, cap to avi with windv and view with enosoft processor. If there is any metadata there, it should be able to show you.

Scott

[ave1965]

Thank you for the information. I may follow the PC route, but I have also found some interesting info in this link:

http://www.avpreserve.com/dvanalyzer/dv-metadata/

[Cornucopia]

BTW, my mistake on the enosoft product: it wasn't the DV processor, it was the Enhanced DV Decoder ($129USD) I was thinking of. And it specifically states that it can show camera info (if it exists in the stream).

That DV Analyzer is good for showing timecode discontinuities, and it can do more, but it's mainly about the timecode. And it's main purpose here is to get around the difficulty of FCP to (normally) pass through metadata. Other NLEs don't necessarily have that shortcoming anyway (if they do smart-rendering for DV streams).

Scott

[LightWeightProducer]

Unless the recording program goes to the extraordinary length of splitting the DV stream up (or of course recoding) the video part of the recording file will contain everything that came across from the camera. Even when only two audio streams are saved (which really means been extracted from the DV stream and muxed into the output file as separate streams) the four channels are still present in the video stream and can be extracted.

[Cornucopia]

Re: Audio in the core stream, that's true.
Re: Splitting up the DV stream, firewire capping/transfer programs may remux but they specifically do not re-encode. The VAUX data in the stream is appended to EACH frame of the video stream (static info being repeated). Thus, even if the stream gets CUT UP, the metadata is retained (whatever was there to begin with). Only in the case of NLEs & converters that re-encode/transcode (or capping programs that don't use the firewire method) will the metadata be lost. It is of course usually much better if the capping program does NOT split up the DV/HDV stream unless specifically warranted (scene change/TC discontinuity, for example).

Scott

[fvisagie]

The DVInfo Avisynth plugin can extract both timecode and datecode, while DVdate by Paul Glagla can extract (and do some manipulation of the file's) datecode data.

[ybarra]

I ran through this process a while back with my DV library of 60+ home movie tapes. From a PC perspective, this was my workflow:

1) Purchase 2 1-TB external hard drives and set them up as a mirror for my archiving
2) Use Scenalizer or equivalent to capture type-1 DV files to the hard drive array
3) Run AVPS DV Analyzer on the captured stream to validate there were no transfer errors. In my case, approximately 10% of my tapes had 1 or more frames with DV concealment because the metal tape shelf life is really not as long as people would like you to believe (I was using Maxell XR Metal and Sony Metal tapes).
4) Recapture areas with concealment until I could get as many corrected blocks as possible and splice the frames together if necessary.
5) Cut DV files down if necessary.
6) Extract metadata with DVdate
7) Run custom written VB.NET app I wrote to convert DVdate metadata to Adobe Encore Subtitle file
8) Encode DV files to h.264 using x264-based app (in my case MeGui) using a blu-ray compliant setting
9) Import h.264 files and Subtitle files to Encore and author and burn Blu-ray
10) Result is home movies that look identical to source on blu-ray and that have date/time stamp via subtitle button.
11) Run ICEECC on my DV archive and create 10% parity files in case of future data corruption.
12) Store hard drives in external cases on shelf in needed.

[fvisagie]

From a PC perspective, this was my workflow:
<< snip >>

Nice and solid .

Out of interest, why did you write a utility for converting DVdate output to Adobe Encore, while on the face of it subtitle tools like Aegisub and Subtitle Workshop support that conversion?

[2Bdecided]

3) Run AVPS DV Analyzer on the captured stream to validate there were no transfer errors.

Thank you for sharing that tip - I hadn't heard of that software before, and it is very useful!

Cheers,
David.

[ybarra]

From a PC perspective, this was my workflow:
<< snip >>

Nice and solid .

Out of interest, why did you write a utility for converting DVdate output to Adobe Encore, while on the face of it subtitle tools like Aegisub and Subtitle Workshop support that conversion?

I've not tried Aegisub. When I wrote the utility 3+ years ago, Subtitle Workshop did not create accurate Encore files from the DVdate output. My camera generates NTSC drop-frame (Sony) and SW kept screwing up the calculations. I finally decided to write my own app. I can't remember exactly what was wrong back then. It might have been that the subtitles were not constant and SW was having a frame where the title would disappear and then the next come on, but my memory is vague.

[fvisagie]

I understand. And amen to David, thanks for the AVPS DV Analyzer tip!

[fvisagie]

- If the date/time information is transferred to FCP or other programs, how can you display them?

In my post above I forgot to address this point. The easiest and most common way is to extract the data using one of the utilities mentioned here, and then to display them as selectable subtitles as ybarra mentions. I've used this approach with DV datecode when making DVDs from home movies and it works great.

[ybarra]

Glad I could help. I actually ran into AVPS because Scenalyzer would sometimes tag soft errors (yellow mark) and hard errors (red mark). I was getting some random yellow marks and wanted to know what was up because I couldn't SEE the error. Once I learned about concealment and AVPS I was able to find the frame and the actual blocks that were erroring (you could barely see the block used the previous frame's). At that point, I added it into my workflow and found tons of errors. I even went so far as borrowing a 2nd deck with less wear on the heads and found that the errors were actually tape dropouts and not because of my own deck. Sometimes subsequent passes will get just enough ECC bits to correct the blocks. I only had one tape out of the 60 that had about a 2 minute segment with severe errors that I couldn't correct, but the footage isn't critical or I could just crop and scale the damaged part out (right side damaged blocks). Bottom line, DV tapes can start developing errors around 10 years on the shelf or less, and AVPS DV Analyzer can help you pin them down quickly when batch capturing large amounts.

[zgwforum]

I have MiniDV tapes from a Panasonic Elura, circa 1999. I had them converted to MP4, but completely forgot about the metadata, most importantly date and time stamp. Ugh. (In my humble opinion, I should have been told the conversion service would lose this information).

Forgive my lack of knowledge in this area.

Is there a way to add the metadata, even if I have to do it manually, to the MP4 files?

Thank you in advance.

[aedipuss]

next time don't dig up a old dead post to ask your own question.

if all you have is the mp4 files you are out of luck. you can change the filename of the mp4 and add date/time if you know what it should be if you wanted.