Dropping frames in transmitting DV to external HDD on XP, Shared IRQ 9

[wayney]

I am new on video stuff and am having problem of transmitting video from camcorder to WinXP with external HDD. I searched the forum and tried a few things still not working. My info:

WinXP SP2 on Sony PCG-FX250 (PIII, 800MHz, 20G HDD, 256RAM) Notebook.
External HDD on USB 2.0 PCMCIA card.
Canon MiniDV camcorder on FireWire port by the Notebook directly.
Nero 6, Nero 7, DVGate to capture video in DV Type1

When I transmitted video to internal HDD, it may drop only 100 frames at the beginging some time. Other time, it was fine. In the exact same condition (turnning off most applications including those run in background), I tried to transmitted it to external HDD because internal was loaded. It continuously dropped frames in ratio 1/1.5 (captured/dropped).

I know 800MHz is not high performance. But it seems that most DVD burning software takes that as the minimum and other people had been doing fine on similar ones.

Learned from the forum here on IRQ issue. I checked mine and the IRQ9 is shared by quite a few devices including FireWire port and VIA Enhanced Host Controller for USB 2.0 Card. My external HDD connected through the USB 2.0 PCMCIA card. But I guess the sharing is because of ACPI for PnP on XP. It takes reinstall or disable ACPI completely to make them use different IRQs. In that case, all the devices shareing the IRQ will not be PnP any more. This might be too much to do.

Also, there were opinions suggesting the sharing is fine. So, I did not try this one. Many of the experts suggested external HDD is not an issue either. Is it my USB 2.0 card does not really deliver the 480Mbps claimed? Any tool to test that?

I tried to connect HDD to USB 1.1 ports built-in with the notebook. Same kind of frame dropping behavior.

Any other suggestions I can try to solve the problem?

I planned to transmit it in DV and burn it to DVD later.

Thanks a lot for any suggestions.

[Richard_G]

I doubt it is an IRQ problem, earlier versions of Windows used to suffer but by XP that seems to not matter any more. Although your processor is slow, that shouldn't matter too much as a DV transfer isn't processor intensive. You could do with more RAM though, 256 Mb is an absolute minimum for XP, you really want at least 512 Mb if not twice that.

It could simply be that your hard drives can't cope with the transfer rate. As you have Nero 6, open up Nero Burning Rom, go File, Preferences, Cache and, with your external drive connected, click on Test all Drive Speeds. This will test the sustained data rate for your drives. Anything down into single figures (less than 10Mbs) and that is almost certainly your problem.

Download WinDV and try to transfer with that.

[wayney]

Thanks for prompt reply.

Good, I dont mess up with IRQ stuff. I dont have Nero 6 any more. I bought a DVD burner first and that came with Nero 6. It did not work and I thought it was the burner or Nero 6. I returned and bought a different one with Nero 7 now. Same problem. So, I drilled down to the point that it had nothing to do with the burner or Nero 6. It just could not transmit to the external HDD in DV.

Any other free tool allows me to test the sustained data rate?

[jagabo]

The problem may be a combination of the USB port and the low CPU speed. The USB 2.0 raw bitrate is 480 Mb/s. First divide that by 8 (bits per byte) and you get 60 MB/s. In the real world with serial stop and start bits, CRC data, and shared bus protocol overhead, a USB 2.0 port will max out around 20 to 30 MB/s. DV's requirement of 3.6 MB/s is far less than that but when you add in the high CPU usage of USB and the low speed of your processor you may exceeding what your computer can handle.

Start up Task Manager and watch the CPU usage graphs while you capture to the intern and exteral drives. Any difference?

IRQ conficts usually lead to much worse problems like system or process lockup.

Nero Burning ROM has a drive speed test program: File -> Preferences -> Cache -> Test All Drive Speeds. I believe it only tests read speads (write speeds are usually lower) but it should give you some idea of how fast the USB port is running.

[wayney]

I ran Nero Vision 4.9.6.6 in the Nero 7 again. I checked the Task Manager. Before running, CPU is at 10%.

During transmission, it went between 30% to 66%. 70% of the time, it was between 30% to 45%. Available RAM was 78MB. Captured/Dropped ratio was about 1/2.

I tried to find the drive speed test you suggested and could not find it. Maybe it was because I am using Nero 7 (Nero Burning ROM 7.9.6.0)? I checked the Nero 7 suite and could not find one to test the external HDD. There was something called Nero DriveSpeed in Nero Tools. But it was only for checking CD/DVD drive speed.

Any other freeware tool I can download to test the data rate?

[wayney]

Sorry, I forgot to report the data from transmitting DV to internal HDD.

Before transmission, it was about the same.

During transmission. CPU went between 30% to 68% and most of the time was between 45% to 50%. Available RAM was 80MB. At the very begining, there was 115 frames dropped. After that no more frame dropped.

[edDV]

I am new on video stuff and am having problem of transmitting video from camcorder to WinXP with external HDD. I searched the forum and tried a few things still not working. My info:

WinXP SP2 on Sony PCG-FX250 (PIII, 800MHz, 20G HDD, 256RAM) Notebook.
External HDD on USB 2.0 PCMCIA card.
Canon MiniDV camcorder on FireWire port by the Notebook directly.
Nero 6, Nero 7, DVGate to capture video in DV Type1

When I transmitted video to internal HDD, it may drop only 100 frames at the beginging some time. Other time, it was fine. In the exact same condition (turnning off most applications including those run in background), I tried to transmitted it to external HDD because internal was loaded. It continuously dropped frames in ratio 1/1.5 (captured/dropped).

I know 800MHz is not high performance. But it seems that most DVD burning software takes that as the minimum and other people had been doing fine on similar ones.
...

I used to capture DV to a similar Sony PCG-XG29 notebook (PIII, 750MHz, 512MB max memory) so I'll share my experience. It was like walking on eggs to get an hour of transfer without dropped frames. I'd usually do it in smaller increments.

After many attempts and tweaks here was my solution.

1. I used Partition Magic to make a separate 14GB video partition at the fast end of the drive. Next came the swap file and XP OS. The main reason for the video partition was to make defrag easy.

2. I created a separate hardware profile for capture that removed all drivers and services not required for capture. No antivirus, no USB, no networking, no modem, minimal background services. Only display card drivers, mouse and IEEE-1394.

With that hardware profile and using simple DV capture programs like DVIO or WinDV, I was able to get reliable transfers from either my DV camcorder or Canopus ADVC-100. I could not get consistent results using Premiere for capture because it required too many system resources and seemed to create background tasks*.

After capture, I'd reboot to the normal hardware profile and use Premiere or ULead Video Studio to edit without issues. I could then use external IEEE-1394 or USB hard drives.


* note that a DV "capture" is a stream capture not a networked file copy. If the OS or other program takes priority on the C: drive, frames are lost in the DV stream transfer. Since buffering is minimal, data is lost. The trick is to prevent OS activity on the drive.


PS: An external USB2 hard drive places a heavy load on the notebook CPU since the CPU is acting as hard disk controller (a software task for USB) and is managing memory flows from the IEEE-1394 port to the external drive. It is this CPU activity that you want to avoid during DV capture. The internal drive uses the hardware disk controller on the motherboard which demands little CPU support.

[wayney]

I really appreciate all the analysis you guys had.

I also thought it could be because of CPU after reading all yours. But the numbers I got seems not support that, isn't it? I was also surprised to see CPU were not maxed up in both cases which had dropped and not dropped frames. If CPU is really the one, it should at least 90% or around.

For my case, when there was no frame dropped (to internal disk), CPU usage tended to be even a little lower, based on the Task Manager. It made me feel more likely to be hard disk speed. One way we usually debug performance is to look at CPU usage. If it is 100%, it might be software has infinite loop since it is always CPU cycles. If it respond slowly while CPU usage is low, it is more likely because of disk I/O. What do you guys think?

In my case, my internal disk is a single physical disk with C, D, and E partitions. Because I have only 3 GB left on it, it is not good enough for transmitting 1 hour DV. That is why I bought the external HDD 160GB and hoped to solve the problem. Theoretically, USB 2.0 should be fast enough.

I thought when I used external HDD it will be different drive and can avoid the problem of having too many activities on one disk. In this way, all OS activities will be on internal and "captured" will be on external. But it did not help.

[edDV]

I don't think you are understanding the cause of frame drop. The camcorder is delivering a steady 25Mb/s (plus audio) stream. There is no file transfer protocol. No request for lost packets. If the notebook can't keep up and the buffer fills, then pixels, lines or frames are lost. On a slow machine, the path to an external usb drive requires significant continuous CPU load since the CPU is acting as disk controller. Frame loss occurs when an application or background task interrupts the data flow. The trick is to quiet background interrupt activity from applications or the OS.

This isn't an issue with a desktop machine with two internal drives. PCI Bus Mastering separates the IEEE-1394 to capture drive activity from the OS drive background tasks. A notebook with external USB drive has no such task isolation. OS interrupts can directly cause frame loss even if the CPU load is less than 100%. Faster notebooks are less likely to drop frames. Desktop computers with two drives rarely drop frames.

[jagabo]

As edDV pointed out, it's not a matter of overall CPU usage but a matter of the CPU being available when it's needed. In my experience, when CPU usage gets over 50 percent you are in danger of dropping frames.

Maybe Nero 7 has dropped the drive speed feature, I'm running version 6. In case it helps, I get there via Start -> All Programs -> Nero 6 Ultra Edition -> Nero Buring ROM. Then from Nero Burning ROM select File -> Preferences -> Cache. I wouldn't be surprised if the Preferences option moved to another menu.



Check to see if the internal drive is running in DMA mode (the external USB drive won't have this setting) so you know it's working optimally. Right click on My Computer, select Manage. From the Computer Management dialog select Device Manager -> IDE ATA/ATAPI Controllers. Double click on Primary IDE Channel, go to the Advanced Settings tab. Device 0 should be your hard drive. Be sure Transfer Mode is set to DMA If Available. The current Transfer Mode should show a DMA mode, not PIO.

[wayney]

Yep. It makes sense. I guess the CPU curves and numbers shown in the Task Manager give general idea instead of every moment. It simply samples the CPU every x.x second. It could be that CPU is not available for a moment while the moment was not sampled by the manager. So, it does not show up.

I checked my Primary IDE Channel. Device 0 is on Ulltra DMA Mode. The Device 0 in the 2ndary IDE Channel is on PIO.

I guess the Nero 7 took out the "Test Drive Speed". I could see similar dialog window without that Test button and the dropdown list allowing you to select the drive. I also tried searching around in menus for that kind of button or feature. I could not find it. I thought they might have moved it to be an independent utility in the Nero Tools which is part of the suite. But it seems that none of them is for testing the HDD speed.

[wayney]

BTW, I also tried WinDV and it also had frames dropped continuously. During the time, CPU usage is 20% to 35% and most of the time it is between 25% to 30%.

Any other suggestions?

[jagabo]

Too bad Nero dropped the drive speed benchmarks (or hid them really well).

It does sound like you have speed problems with your external drive. Try the free version of HDTach. It only shows read performance but that will tell you what ballpark you're in. I have an external USB 2.0 hard drive currently hooked up to a 2.8 GHz Pentium 4 machine. It reads at a constant 30 MB/s.

[edDV]

The solution is to clear enough space on the internal drive. My XG29 drive only had 40GB total. I cleared a 14GB partition. I think the original hard drive was only 18GB.

Today, swapping the notebook drive for a larger one is more cost effective.

Newegg is showing 120GB notebook drives @~$75, 160GB @~$120
http://www.newegg.com/Store/SubCategory.aspx?SubCategory=380&name=Notebook-Laptop-Hard-Drives

[wayney]

I will try the HDTach tonight and report the results.

Sure. A new internal HDD will solve the problem. But I was reluctant to do if there is any other cost effective alternatives and this is what I am looking for. I have both W2K and XP installed and have a lot of development tools installed in W2k. Replacing with a new internal HDD means A LOT of reinstall for all of them. Many of those need special configurations. Basically rebuilding the systems will be a major project.

That was why I bought an external HDD. I wish it had allowed me to install 2nd internal HDD. But I guess notebook is intend to be compact. It will not leave the room. I did not realize that USB uses a lot of CPU and thought some on board chips will take care the I/O there just like dealing with regular on board drives.

[wayney]

I ran a long benchmark, with HDTach, for my external HDD . Here are some results:

Random Access: 18.3 ms
CPU Utilization: 18 %
Average Read: 2.1 MB/s

For my local HDD:

Random Access: 18.1 ms
Average Read: 14.1 MB/s

Umm. Sounds like the external HDD does have significant lower date rate.

Will this mean it is not CPU and it is HDD?
Was the problem of USB card? Could it be the USB card not good enough? I tried a few different cards and only this one was compatible.
Or, it does not depend on which card because the difference should not be that significant?
Or, it is not the problem of USB card and it is the external HDD?

[edDV]

That seems very slow. You need >3.6MB/s average write.
I could toss a USB2 drive on my XG-29 Vaio and see what I get.

Something just occurred to me, these PIII Vaios only have USB 1.1. That is why I used a Firewire external hard drive.

[wayney]

Yep. They only have USB 1.1 ports built in. I connect my external HDD through PCMCIA USB 2.0 expansion card. I am thinking whether it is because the card is not good enough.

I will bring another computer, later model, to test this card and the HDD with the HDTach again tomorrow. I guess it may help to figure out something by putting them on a 1.5 GHz notebook. That one has built in USB 2.0 also. We will see.

[TBoneit]

to quote your opening message "When I transmitted video to internal HDD, it may drop only 100 frames at the beginging some time. Other time, it was fine."


Get yourself a larger internal hard drive with a higher RPM and a larger buffer. Yours is probably a slow drive to extend battery life. Plus Sony didi funny things making the boot partition small. I've seen Sony laptops where the C: was full and the D: was empty.

Boottom line you need to capture to a internal drive. Best solution get a cheapo Desktop for capturing if you can. Laptops are full of compromises in performance to extend the battery life.

Example a less than 1 year old HP, DVD burner is 8X desktops these daya at least 16X

[edDV]

to quote your opening message "When I transmitted video to internal HDD, it may drop only 100 frames at the beginging some time. Other time, it was fine."


Get yourself a larger internal hard drive with a higher RPM and a larger buffer. Yours is probably a slow drive to extend battery life. Plus Sony didi funny things making the boot partition small. I've seen Sony laptops where the C: was full and the D: was empty.

Boottom line you need to capture to a internal drive. Best solution get a cheapo Desktop for capturing if you can. Laptops are full of compromises in performance to extend the battery life.

Example a less than 1 year old HP, DVD burner is 8X desktops these daya at least 16X

I agree with that. The only reason I used the PIII 750MHz Vaio to capture at all was for location and travel capture. In those days, my minitower 300MHz Pentium II with a two ATA 66 drives had no dropped frames capturing DV format even when using Premiere. The Pentium II + Intel chipset was the first to support PCI bus mastering. The Pentium III laptop never matched the desktop Pentium II accept for pure computation (e.g. encoding).

[edDV]

Last night I ran some sustained throughput tests on the Vaio internal notebook drive. I used a utility supplied with the Canopus Raptor card to qualify hard drives for DV capture.

The OS partition scored 6.9MB/s for read and 5.8MB/s for write.
The capture partition scored 10MB/s for read and 8.3MB/s for write

The Canopus RapTest program qualifies a drive for DV capture (green zone) if it can sustain write >7.2MB/s or two times base DV rate.

I may try HDTach later if I get time.

[wayney]

Here are some testing results and I have not had time to think through on how to interpret them yet.

Background: I used a new notebook (IBM T42, 1.7GHz, 1GB RAM, 40GB internal HDD, built-in USB 2.0 ports) I could borrow. I mounted the same external HDD to it through its own built-in USB 2.0 port, the USB 2.0 expansion card used earlier for the other computer, and even another scenario which I put another USB 2.0 Hub between the HDD and the USB expansion card. The system was also XP SP2 with Ultra DMA for device 0. I used HDTach again with lthe ong bench option.

Here are the results:

Using its on-board or built-in USB:
Random Access: 13.7 ms CPU utilization: 0% Average Read: 28.3 MB/s

Using USB expansion card:
Random Access: 13.8 ms CPU utilization: 0% Average Read: 18.8 MB/s

Using built-in USB + USB Hub:
Random Access: 13.8 ms CPU utilization: 0% Average Read: 27.7 MB/s

Using USB expansion card + USB Hub:
Random Access: 13.9 ms CPU utilization: 0% Average Read: 15.8 MB/s

I also tried to run HDTach against its own local HDD. Here are the results:
Random Access: 17.0 ms CPU utilization: 0% Average Read: 26.3 MB/s

[wayney]

Some information which may be useful:

IBM's internal HDD info: HITACHI, ATA-6, 40GB, 8MB Cache, 5400 RPM, 12ms avg seek time.
external USB HDD info: WesternDi, Ultra ATA/100, 2MB Cache, 7200 RPM, 8.9 ms seek time.

In my opinion, with built-in USB 2.0 ports, the differences between internal HDD and external one are marginal.

[jagabo]

There are different levels of PCMCIA ports with different speeds:

http://www.pcmcia.org/faq.htm#data
http://en.wikipedia.org/wiki/Pcmcia

It could be that your laptop has an older/slower port. Or it could be a USB driver issue.

[edDV]

Some information which may be useful:

IBM's internal HDD info: HITACHI, ATA-6, 40GB, 8MB Cache, 5400 RPM, 12ms avg seek time.
external USB HDD info: WesternDi, Ultra ATA/100, 2MB Cache, 7200 RPM, 8.9 ms seek time.

In my opinion, with built-in USB 2.0 ports, the differences between internal HDD and external one are marginal.

Seek time is something to be prevented (through defrag) but not relevant to the discussion. A video frame takes 16.7ms so without buffering, those seek times you mentioned would cause loss of half to 2/3 of a video frame. The critical issue is sustained throughput from tape playback to the hard disk head. The hard disk buffer doesn't help because the data flow is continuous and non-repetitive like it may be in a server.

Data flow from the camcorder tape to the PCI bus is more or less continuous. The OS needs to manage dataflow from the PCI bus, through memory to the software hard disk control application which supports the USB drive. Data must be converted from DV format to addressed hard disk clusters and sectors and then formated for USB transmission to the drive. This is a continous flow. If suddently the antivirus program, or the disk indexing program or windows update wants to assume prority and load its application, data flow may be interrupted*. This is how frames are lost.

It has nothing to do with the raw hard disk performance which far exceeds the requirement. The USB drives on my Core2Duo hit 24-32MB/s routinely. We are talking here about an 800MHz PIII with 256MB memory and in typical Sony fashion, a ton of background applications that need to be silenced.

The first thing to do is add another 256MB Ram. My XG-29 has max 512MB.

As for internal vs. USB2 hard drives, do a test. Test an EIDE 7200 RPM drive under internal hard disk control and compare the same drive in a USB2 housing. My tests have shown 52-64MB/s sustained transfer when mounted internally vs. 24-32MB/s in a USB2 enclosure. IEEE-1394 goes a bit faster than USB2 but much slower than internal disk control. The main advantage of IEEE-1394 is low CPU utilization vs. USB2.


* This data interruption isn't as much of a problem if the OS is directing a packetized file transfer protocol as is done for disk copy or network file transfers. In those cases there is an accounting scheme for each data packet. If a packet doesn't show up at the destination, a request is made to resend the packet and data recovery is 100%. This doesn't happen for a DV stream transfer. If a pixel, line or frame is lost, there is no request to resend data. The data is gone.

[wayney]

There are different levels of PCMCIA ports with different speeds:

http://www.pcmcia.org/faq.htm#data
http://en.wikipedia.org/wiki/Pcmcia

It could be that your laptop has an older/slower port. Or it could be a USB driver issue.

Mine is USB 2.0 and it claims 480Mbs as its max rate. When you said different levels and speeds, you meant USB 1.0, 1.1, 2.0, or even 2.1 and their corresponding speeds, right?

BTW, I have to correct my numbers for the Sony FX250 now. I just realized there was a mistake when getting the numbers. Sorry about that.

My external HDD through USB 2.0 expansion card:
Random Access: 16.0 ms CPU: 44% Average Read: 7.4 MB/s

My external HDD through built-in USB 1.1 port:
Raondom Access: 17.8 ms CPU: 6% Average Read: 1.0 MB /s

My local HDD (DJSA220, 20GB, 4200 RPM):
Random Access: 18.0 ms CPU: 6% Average Read: 14.3 MB/s

I dont really know how to interpret the random access and CPU. So, I include them here any way.

The mistake I made was that I did not realize there was another instance of HDTach still running when I ran a new instance of it to get those numbers. Actually when I finished one run and tried to "Save Results", it caused some "access violation" error after saving the data to file. The window then disappeared. But it actually was still running and I discovered it through task manager. The orphant process cost a lot of CPU and screwed the numbers.

I need to think over on what edDV suggested. When I tried all these, I did close antivirus stuff. I might need to find out the names of the process for the indexing and window update stuff. I did not know those and not sure which one to kill. If you know the names, could you share all those here and I can see whether they are running and terminate them for capturing.

[edDV]

One comment. Sustained bitrate is a continuous measurement with small tolerance for gaps. Average bitrate might include many dropped frames.

[jagabo]

It could be that your laptop has an older/slower port. Or it could be a USB driver issue.

Mine is USB 2.0 and it claims 480Mbs as its max rate. When you said different levels and speeds, you meant USB 1.0, 1.1, 2.0, or even 2.1 and their corresponding speeds, right?

Yes, the USB chip's spec is 480 Mb/s. But I meant there are several versions of the PCMCIA spec with different transfer rates. A slow PCMCIA port would become a bottleneck for a USB 2.0 transfer since all data has to pass through the PCMCIA port.

BTW, I have to correct my numbers for the Sony FX250 now. I just realized there was a mistake when getting the numbers. Sorry about that.

My external HDD through USB 2.0 expansion card:
Random Access: 16.0 ms CPU: 44% Average Read: 7.4 MB/s

My external HDD through built-in USB 1.1 port:
Raondom Access: 17.8 ms CPU: 6% Average Read: 1.0 MB /s

My local HDD (DJSA220, 20GB, 4200 RPM):
Random Access: 18.0 ms CPU: 6% Average Read: 14.3 MB/s

I dont really know how to interpret the random access and CPU. So, I include them here any way.

Those numbers sound more like what is expected. The 7.4 MB/s with high CPU usage would correspond to the 16 bit PIO rate given at the pcmcia.org site.

You probably know this much but:

The random access time is how long it takes to seek from one random place on the disk to another. This includecs the time it takes to move the read/write head(s) and waiting for the platter(s) to spin so that the data passes under the head. As long as your disk isn't too fragmented and there isn't some other program using the hard drive at the same time, this shouldn't be an issue during video capture.

CPU usage obviously is an issue. All the CPU cycles spent writing to the hard drive are cycles where the CPU isn't available to handle other tasks. Since the max throughput via the PCMCIA card is 7.4 MB/s with 44 percent CPU usage you can estimate that 3.6 MB/s of a DV transfer is going to consume about 20 percent of the CPU just writing to the drive.

It's hard to place any definitive cut-off on where max transfer rate, random seek time, and CPU usage will start causing frames to be dropped. But I think your numbers are in the gray area -- as your constant dropped frames would attest.

I need to think over on what edDV suggested. When I tried all these, I did close antivirus stuff. I might need to find out the names of the process for the indexing and window update stuff. I did not know those and not sure which one to kill. If you know the names, could you share all those here and I can see whether they are running and terminate them for capturing.

I would expect Windows' Indexing and Update to cause intermittent dropped frames.

It's possible to replace your hard drive without reinstalling Windows and all your software. I do it all the time. It's easy on a desktop where you can plug both drives in and use a mirroring program to copy one to the other. It's harder on a laptop though. I'm not sure if you can put the new drive in a USB enclosure, mirror your internal drive to it, then swap the drives. It's worth a try if you decide to go that route. And you'll still have your original drive intact if it doesn't work.

[wayney]

Can any one give a list of "usual suspects" (indexing, antivirus, etc.) to be taken out to avoid competing for CPU? Also, how to turn them off? I know I can "End Process" the applications if I can get their names usually shown in Task Manager.

But, is there any way like setting a user account/profie dedicated for capturing which will only load basics so that I dont have to terminate them one by one every time? Or, hardware or device profile? Can get some instructions on how to set them up?

I would like to try this way first since this will be the easiest to achieve. Also, I agree you all on the dumb way of transmitting DV frames is demanding on CPU. From the numbers I had, faster CPU did have higher data rate even for exactly the same USB card and external HDD. Also, CPU utilization on faster system was almost zero while on slower system it was 44%.

[edDV]

As said above I did a separate hardware profile for capture and shut off near everything in My Computer - Manage - Services and Applications.

Hardware profile setup is under System Properties.