Who knows how to determine the transfer rate (firewire + HD)

[itandsport]

I'm capturing via firewire from a digital cam and experience about
1% of dropped frames, allthough I followed all the rules given in this
forum. In order to solve this problem I like to get the following information:

What is the transfer rate (MByte/s) for data transfer from digital cam via firewire? And how is it calculated?

(Some say 3,6 MByte/s; My calculation is
720 x 576 x 25 (fps) = 10,368 MByte/s
according to what I read DV is a 1:5 compression => 2,07 MByte/s

What is correct?

My second problem is, how to determine the writing speed on the disk.
I have a WD1200JB (120 GB, 7200/s, 8MB cache) dedicated to take
the capture files (there are no programs or anything else on the disk).
It is set up as the slave on the first IDE channel. The boot disk is the
master. I checked the device manager and I'm using Ultra DMA mode 2
which should be equivalent to the UDMA/33 with a transfer rate of
33 MByte/s. During capture it really uses the DMA, as the CPU load
is always below 10%.

How can I determine the real transfer speed, and where is the bottleneck?
Test tools?

The bottleneck should not be the firewire card. I run the benchmark
test of VirtualDub and got a result of 17,5 MB/s (is it MByte/s or MBit/s?)

Is this enough?

My PC:
- P3 with 600 MHz (Asus P3B-F, Intel chip set)
- 256 MB Ram
- WD1200JB (120 GB, 7200/s, 8MB cache) dedicated for capturing
- Ultra DMA mode 2


Thank's for your help

[Dragonsf]

You forgot, that RGB in fact calls for 3 Bytes.So you end up with:
720*576*3*25=31MBytes/sec.The compress rate of DV is not 5 but tending more to 10, so 3.6 MByte/sec is the correct Bitrate.
The problem is, for correct transfer, you need this as constant rate.An UDMA-2 will do (this should at least have a rate of 12MByte/sec- good enough).
But I doubt, a 600MHz will be good enough for DV capturing, as my 1.2MHz Athlon gives me a dropped frame here and there (from DCR around 1 in 5 Min, from HDB one per min).1% is quite bad (depending on your footage of course).
The HD part for capturing should be clean (defragged) before capturing.
And use NTFS for capturing!

[Thwiipp]

Well capture through a DV cam and havent had a dropped frame yet, i have a P3 866 MHZ, 512MB RAM, im using Vegas Video 3 for capturing and it does a wonderful job.

[itandsport]

Dragonsf, thank's. Now I understand the calculation.

However, where is the problem with the 600 MHz cpu?

Does it mean, that the firewire card fills an input buffer?
(where is this input buffer?)
And does it mean that the cpu is not fast enough in sending
a command to transfer it via DMA to the hard disk?

Can someone explain the complete process?

[ktnwin]

I have been able to capture via Firewire on a Pentium2-266 Mhz with only one single drop frame (right at the beginning of the capture).
It's not the CPU speed, firewire or hard drive.
I believe your PC setup may have a little too much in there (drivers, clock, ...). These TSR programs may interfere with the capture once in a while causing dropped frames.
Try to simply remove those TSRs (but keep enough for the system to function). For example, you do not need the clock, remove it.
On my new 1.2GHz, I do not have to remove them.

[Dragonsf]

Yes, the background processes do this kind of interruption.As Win2K is a presumptive OS, you can't prohibit other tasks from getting CPU.You can alter priority to Real-time for the capturing process, but I can live with 1 dropped frame in 5 min.The dropped frames with HDB are a different isssue, the droppedf frames are independant of the PC.

[You_Are_Alive]

Your system is definitely fast enough, and with the WD JB edition, that's a screaming drive. It could be that some other software or hardware is not behaving like it should and is trying to access that same pci slot that your card is on. Your problems are caused most likely by bad PCI DMA transfers. It's just the nature of mobo's, some perform better then others. Just look at my specs, I never dropped a frame. I've captured clips that were just over an hour long with no problems. On my brother in laws computer, its an Athlon 1400 Thunderbird. Guess what, he drops frames. I capture via a USB2/IEEE1394 combo card and no problems with that one either.

A few corrections and then I'll get to tips: Source videoguys

The DV spec is a 720x480 image size, at roughly a 5:1 compression. More accurately, it is compressed at a constant throughput of 3600 kilobytes/second, which averages out to 5:1 compression. This works out to 13GB per hour of footage.

And the formula you used with a data rate of 1:1 or uncompressed video is calculated: width in pixels x height in pixels x 2 bytes per pixel (luminance and chromanince) x number of frames per second. of course ntsc and pal, 486 and 576, respectively. Honestly, I've never heard of using 3 bytes per pixel for r,g,b but I maybe wrong. Not sure though, any technical ppl out there?

First, see if mobo has reported problems with your card and/or updated bios drivers that may be necessary. Make sure drive is not heavily fragmented, and don't capture to your system drive, capture to a different drive altogether. Check your memory usage to see that its not using disk memory to swap files. Now, when I capture in win2000, I have about 21 processes running when looking at task manager on bootup. I don't have any other programs running like virus protection, reminders, etc.. If you've done all that then look at your device manager and look for any exclamations. See if those are related to your IEEE1394 card. Try re-installing the card's driver> uninstall, then scan for hardware changes, and use an updated driver if possible. If still having problems, then re-install software that may have come with the card as sometimes this can also be a problem. I did this one time and it fixed a problem I had.

Big tip here: when it asks to restart, the best thing to do a is complete shutdown. Then re-boot, it almost guarantees that your changes will take this way. Finally, if none of this helps, open case and if available, put it on a different pci slot that is dedicated, and not shared. Hopefully this will work, if not, maybe you got a lemon.

[Dragonsf]

In reality, the resulting DV stream is neither RGB nor YCrYCb, but a representation of the DCT-transformed and Huffman compressed macroblock -data:
http://www.adamwilt.com/DVvsMJPEG.html
Only thing, which is constant are the 25 Mbits/Sec Bitrate and the 1:5 compression rate.
But my first calculation was based on RGB raw data (uncompressed) and not YUV (which indeed is 2 Bytes only for each pixel if 4:2:2.

[itandsport]

I'm using Windows XP, might that be a problem?

- I stoped nearly all services, only those which couldn't be stopped
remained.
- The firewire card is already at a location with a non shared interrupt.
- VirtualDub benchmark shows me a transfer rate of more than 17 MB/s
=> this should be enough
- I haven't updated BIOS for the ASUS P3B-F yet, because I cannot
find any advantage in the documentation of the updates.
- The cpu load is like a wave with a cycle time of about 8 seconds
(min 2%, max 28%), when transferring with DVIO.EXE

Sometimes I don't loose a frame for 2000 frames, sometimes I loose
4 in a row.

I more and more believe, that for Windows XP I would need a faster
cpu, whereas it would be enough for Windows 98.

Any opinions?

[deadmonty]

Your capture drive should not be a slave to your boot drive, it should be the master on the second IDE channel. There are a couple of possibilities for your dropped frames.

1. Your capture is being buffered in your pagefile on the boot disk and then sent to the capture drive. This would effectively stall the capture process.

2. Something is writing to the boot drive, thus temporarily interrupting the flow to the capture drive.

3. Your capture program is using your boot drive as a temporary store - I think Movie Maker does this unless you tell it otherwise.

Paul

[itandsport]

Paul, how can you figure out, whether the boot disk is used during
capturing process (item 1 and 3 of your comment). Is there any
software tool to verify that? I would like to know that before, because
swapping the drives means a lot of effort (IDE cables are too short, setup
changes etc.)

I don't think that item 2 can be a problem, because the writing speed
(17,5 MB/s) is much higher than what is neede. So, most of the time,
writing should not be active.

[Dragonsf]

Get filemon for monitoring any file operation on your system.You can filter some programs out of the display if you like.

[deadmonty]

It may also be that your capture drive is recalibrating during a write operation. Unless a drive specifically says it is designed for AV work it will periodically recalibrate itself to account for things such as platter expansion and shrinkage as the drive warms up and cools down. I know this has been known to cause problems, which is why some companies sell drives specifically for AV use.

Paul

[itandsport]

I ran filemon (and regmon, diskmon as well) and could verify,
that writing to disk is ok. Every 40ms two writing actions take
place (first one to adjust NTFS-table 5kB, second one to append
the data to the file 144kB). Both actions together take about
10ms. So, no problem there. And all actions are always in time
(I verified the timestamp). No other processes are active. Neither
on the capture drive, nor on my master drive.

If a dropped frame occurs, the corresponding actions are just missing
and the gap between the starting point of two sequencial actions is
80ms instead of 40ms.

I assume, that the problem is on the other side (firewire, cpu, etc.)

Does anybody know a tool to verify what arrives at the firewire ports?
How exactly is a dropped frame detected, what parameters are compared?
Does anybody know about a debug version of a DV capture software,
where the complete transfer process can be performed step by step
to verify where exactly the problem is?
Is there a DV-capture software with source code?

Many questions, I hope somebody has some answers.

[Dragonsf]

Yes, there is: DVAPp comes with complete SouzrceCode in MSSDK (DirectX SDK).But IMHO frame by frame (or slowed transmission) is impossible, becaus of the serial nature of DV stream.IIRC, the stream is controlled by the source and not the receiver.
Anyway, look at the source of DVApp, maybe you find an answer.
And yes, the 40ms are the frame time.Only between frames, other processes (even the OS) can use the resource HD.If you don't have the Source (or don't want the complete SDK), send me a pm.

[donpedro]

Maybe repeating information, but this is from Help file:

The DV format uses a fixed 5:1 compression ratio, which implies a data transfer rate for real-time capture of approximately 3.6 megabytes per second (MB/sec). The transfer rate of your capture drive must be at least 4 MB/sec to allow for any variations across the drive.

Full quality DV capture, which uses about 200 MB of disk space per minute of video.

[Bizuser]

itandsport:

720 x 576 x 25 (fps) = 10,368 MByte/s
according to what I read DV is a 1:5 compression => 2,07 MByte/s

I think it would be worth clarifying some basics - is 10,368 MByte/sec 10 thousand 368 MByte/sec, or, 10 point 368 MBytes/sec? In the second line, you say 2,07 MByte/sec - presumably that's 2 point 07. I know some countries/languages use comma as a decimal separator, but the majority use period (.) for decimal, comma (,) for thousands.

Also, I'd like to suggest everyone spell out MBytes and MBits to avoid confusion - eg, Donpedro says "The transfer rate of your capture drive must be at least 4 MB/sec to allow for any variations across the drive. " - presumably 4 Megabytes/sec.

In theory, if all you need is 4 megabytes/sec, then even an older 33 Mbyte/sec drive should be more than adequate, and should not drop frames if the occasional other program accesses the drive. However, even on my 100 Mbyte/sec drive (on a P4, 1.6 GHz, 533MHz FSB system), I see dropped frames, so I'm wondering if it's unrelated to the drive performance directly.

I've been testing with a new USB 2.0 external hard drive. USB 2.0 spec is 480 MBit/sec - 60 MByte/sec. I've tested transfer rate using stopwatch and large files, and determined the 'actual' rate is closer to 20 MByte/sec, but that's still pretty good. But when I save video to it, I see more dropped frames. Interestingly, with this drive simply 'on' and recognized by the system, even saving to my regular hard drive incurs more dropped frames - I'm guessing that windows is spending time 'polling' or checking on it's various drives on a periodic basis. Shutting down this external drive definitely seems to help improve my dropped frame results.

A lot of posters recommend paying attention to IRQs. This has merit to me. However, Windows 2000 just loves lumping everything on one IRQ, then using IRQ steering. My system is generally very, very stable (runs for months and months without reboot) so I'm reluctant to start messing with IRQs too much, but ... maybe that's where I need to look next.

Beware of what you see in filemon. I love this utility - filemon and regmon are probably the most useful tools I've ever used, over the past 10 years in IT and support roles. But filemon reports a lot of accesses that are not true physical accesses, I believe. Even though files are cached, I think filemon reports such accesses - so don't assume an entry in filemon is related to a real drive access.

[donpedro]

Also, I'd like to suggest everyone spell out MBytes and MBits to avoid confusion - eg, Donpedro says "The transfer rate of your capture drive must be at least 4 MB/sec to allow for any variations across the drive. " - presumably 4 Megabytes/sec.

I have to defend myself I did exact copy from Help file. So what they mean there ??? Who knows. Isn't one lowercase "b" and another uppercase "B" based on norms ?

Well I know one think. I finisched "mechanical engineering" on University in Europe and "kilo" was alwas "k" and not "K" that I can see here all the time.

[Dragonsf]

in real world terms:k means kilo means 1000, in IT K means kilo too, but means 1024.ÍMHO b stands for bits and B for Bytes.
so 1MB are 1024 KB are 1024*1024 Bytes.

[Bizuser]

Hey, Dragonsf and Donpedro - I was not trying to offend anyone or split hairs ... I have a degree in engineering too, and sort-of remember that there are standards for this sort of thing, but I'd say that just about everyone - vendors, help-file writers, and everyone else - tend to be sloppy in their usage, so if one is to engage in a discussion about data rates, explicitly spelling out the terms can help reduce confusion. My main focus was, though, the use of comma and period for separators and decimal indication ... if we can't be sure where the thousands are, we're in deep trouble!

I just re-tested my video captures and got no problems this time while saving to my external USB drive. Dropped only 5 frames in 20 minutes. This time, I shut down everything (virus scan, etc) on my pc other than essentials. Then I did something different - I moved a window - that is, I clicked on the 'top bar' of a window, and dragged it around the screen ... my dropped frame rate went through the roof! So even on a fast machine, using the video subsystem greatly affects the capture ability. Thus surprised me.

[donpedro]

Yeah ! Isn't that funny ? 1.000,00 or 1,000.0 ; 02/08/03 or 08/02/03 ; k or K ; b or B ; m (meters) or ft (foot) ; C or F ; l or oz .... Why couldn't everybody agree what to use (like that did in Europe) and we would know what is what.

[itandsport]

I just wanted to let you know about the result of my trials during
the weekend. I had a look at the source code of AMCap and DVCap
and noticed that there is no chance for further diagnostics. They all
use the DirectShow api, where the transfer is just set up and started.
So, no improvements on my standard PC.

However, I took a second one same motherboard P3B-F, but only
an PIII 500 MHz instead of the PIII 600 MHz I was using. I cleaned
the disk completely (an old disk) and reinstalled XP on it. Used the
same firewire card, plugged in the same slot (to avoid IRQ sharing)

Guess what, no dropped frames within 60 minutes.

Summary (my assumption):
- drives are good enough if they make 10 MBytes/s or more
(filemoon can be used to check time and duration)
- IRQ sharing should probably be avoided (I have no proof that
this would create a problem)
- What is most important: Reduce the influence of the OS. With the
brand new installation everything worked fine, even with a slower
cpu and a slower hard disk.

Has anybody ever checked the internal XP activities during capture?
Is there any tool?

So far, thank's all of you for your help with during the last days