VirtualDub Running Slow, Completely Stalling With MCTD

[koberulz]

VirtualDub is having an absolutely awful time running AviSynth scripts.

Running the attached script on the attached AVI takes me about five and a half minutes. Apparently it should take less than three. Over the course of a full encode, this works out at a lot of time.

I'm trying to run MCTD, by itself, on a clip lasting an hour and 45 minutes. It starts out its estimates at approximately a day, then it just increases, slowly, until it finally completely stops working (the current frame doesn't change) and the estimated time rockets up.

Doesn't appear to just be MCTD though, other scripts have run really slow. Never completely stopped though.

Any help?

EDIT: intel Core i7, 16GB RAM, Windows Vista, VirtualDub 1.9.11, AviSynth 2.60.

EDIT 2: VirtualDub also keeps crashing when I use ColorTools: "out-of-bounds memory access (access violation) occurred in module 'ntdll'...writing address [bunch of zeroes]...while stopping filter "Color Tools [1.4]" (FilterInstance.cpp:1214)

[hello_hello]

Deleted

[koberulz]

The attached script is purely a "stress test" of sorts; it's put together specifically to take a significant amount of time on the one-second sample. Bit difficult to figure out if anything's faster or slower if the whole thing runs in under 20 seconds anyway. The actual script I'm using runs EzDenoise with QTGMC, amongst other things, which I then save as a Lagarith AVI. That AVI is then put through MCTemporalDenoise and saved, then that AVI is combined with the previous AVI via MergeChroma.

Works fine with clips of a couple of minutes, albeit really slowly. But when I try to run the MCTemporalDenoise script on the full capture, it completely freezes up.

x264 command line.

If it's lossless I'd consider muxing the video into another container to avoid AviSource and use the command line x264 encoder rather than the VFW version, if you're not already.

Where did all this x264 stuff come from?

Is the source a DVD or mpeg video or is it the same as the lossless sample?

The original source is a VHS tape, of which I've made a Lagarith capture.

[hello_hello]

The attached script is purely a "stress test" of sorts.

That would have been nice to know. I see you did mentioned the problem occurs when running MCTD by itself, but you also have the same problem with other plugins, and it appears when you referred to running the script you're not actually using over the course of an encode I failed to realise it was a hypothetical one.

Where did all this x264 stuff come from?

I pulled it out of thin air due to the lack of specific information regarding your workflow, the sample being lossless, and not knowing the script you posted was a time waster, so I thought it likely you were going from sample.avi to IntroTestScript.avs to x264 encode, given there was nothing to indicate the "encode" was of another type . My apologies.

The original source is a VHS tape, of which I've made a Lagarith capture.

So... the sample script you're not actually using was accompanied by a sample huffyuv AVI that's different to your source.
I must have had a bad run at guessing, or I was having a below average comprehension day, so I'll delete my post as it appears it's probably not going to be of much help.

You're welcome.
Oh sorry.... I did it again. I must have just assumed you said "thanks anyway" for trying help.

[koberulz]

So... the sample script you're not actually using was accompanied by a sample huffyuv AVI that's different to your source.

Is that Huffyuv? Oops. Sorry, I've been working through the actual restoration process for a month or two now, and it's become apparent that my PC is an issue, so I'm trying to go back and remember things. I needed a Huffyuv sample for someone who didn't have Lagarith installed, so the Huffyuv is a simple 'fast recompress' from the Lagarith. I'd completely forgotten about that.

I was also apparently in a rush when posting. Initially forgot to mention my specs and the VDub crash issue relating to Color Tools, and obviously didn't sufficiently explain what was going on. Like I said, I've been working through it for a while now with others, so I'm not used to having to explain everything from the beginning.

But yes, the script attached to the OP is basically a 'stress test', although based on a script I'm actually using. I'm capturing VHS losslessly, then doing restoration work losslessly.

That being said, you could have asked questions instead of making assumptions.

[hello_hello]

I asked if you were aware QTGMC has a noise filter. You answered that question.
I did assume you were using the script you posted but had I not done so would you have explained it's real purpose?
I asked which version of MCTD you were using. It might have been relevant, but it doesn't matter if it needs to remain a secret.
I asked about the source source type and you then supplied specific information.
I assumed x264, as "encoding" generally refers to lossy (although it's not a rule) and x264 is more likely to contribute to a slowdown than lossless compression, and probably even more so when VFW is involved.

That being said, there's also a third option of not replying if it means playing 20 questions and the effort appears not to be appreciated.

[koberulz]

I asked which version of MCTD you were using.

...I could have sworn I answered that question.

1.4.20. Although I'm not sure any specific filter is the issue; I recall having issues with scripts that didn't include it as well. Might not even be a VDub/AviSynth issue, it could be a more general issue with my computer.

I assumed x264, as "encoding" generally refers to lossy (although it's not a rule) and x264 is more likely to contribute to a slowdown than lossless compression, and probably even more so when VFW is involved.

Well, for that definition of 'encoding' none is happening. Lossless in, lossless out. I've always used 'encoding' to mean 'the process of turning something into a video file', whether it's an Adobe Premier project, or an AviSynth script, or anything else. What word should I be using in this context?

[LMotlow]

I assume the following from the thread so far:
- You have a lossless YUY2 AVI capture from a VHS tape.
- The sample AVI furnished is a direct cut from that tape, and except for recompression from Lagarith to huffyuv the sample is otherwise unprocessed from the original capture. - You're using the sample script to go from lossless working file to another lossless working file containing the results of the script, not re-encoding directly to another codec.
- You're running this script in VirtualDUb. I assume you have Virtualdub somewhere in the process because you mention ColorTools. I've seen that ColorTools doesn't work in most setups past XP (I see you have Vista).

I ran that script a handful of times on my 5-year-old i5 3.5GHz Intel, by no means a breakneck speedy machine, where it never took more than 35 seconds. If it takes 5 and a half minutes to run this short script, either something's dead wrong or some info about your workflow is missing. Usually I see people running heavy duty plugins like this in separate scripts, outputting one file for part of the process and using a second step on that output file. This supposedly avoids memory and drive swapping, and other gridlocks, or something like that (gurus can explain this with numbers, I just got that general idea from reading what they say). I broke that script into two steps, where neither step took more than 6 seconds, or 12 seconds total for the whole shebang.

That script is only a start for the problems I see in your sample. I take it MCTemporalDnoise used as shown intends to address that hard chroma clipping and flicker, which appears to be an integral part of what looks like a multigeneration tape. This tape is yours, or handed off to you for repair? Another assumption I'm making is that you're looking for a one-step process or script that does anything and everything in one step for almost 2 hours of capture. I'd suggest you break that into two separate steps, even with a faster machine. How old is this PC of yours? Custom build? Retail?

[koberulz]

I assume the following from the thread so far:
- You have a lossless YUY2 AVI capture from a VHS tape.
- The sample AVI furnished is a direct cut from that tape, and except for recompression from Lagarith to huffyuv the sample is otherwise unprocessed from the original capture. - You're using the sample script to go from lossless working file to another lossless working file containing the results of the script, not re-encoding directly to another codec.
- You're running this script in VirtualDUb. I assume you have Virtualdub somewhere in the process because you mention ColorTools. I've seen that ColorTools doesn't work in most setups past XP (I see you have Vista).

Yes, the final AVI usually goes through VirtualDub. Depends on the tape.

What's the issue with Color Tools? Is there an alternative?

I ran that script a handful of times on my 5-year-old i5 3.5GHz Intel, by no means a breakneck speedy machine, where it never took more than 35 seconds. If it takes 5 and a half minutes to run this short script, either something's dead wrong or some info about your workflow is missing.

I opened that AVS file in VirtualDub, the File->Run video analysis pass. Five and a half minutes later, it finished.

Usually I see people running heavy duty plugins like this in separate scripts, outputting one file for part of the process and using a second step on that output file. This supposedly avoids memory and drive swapping, and other gridlocks, or something like that (gurus can explain this with numbers, I just got that general idea from reading what they say). I broke that script into two steps, where neither step took more than 6 seconds, or 12 seconds total for the whole shebang.

That's what I'm doing for the actual work, I just needed a script that would take long enough for speed differences to be clear.

That script is only a start for the problems I see in your sample. I take it MCTemporalDnoise used as shown intends to address that hard chroma clipping and flicker, which appears to be an integral part of what looks like a multigeneration tape. This tape is yours, or handed off to you for repair?

The tape is older than I am, and only came into my possession a couple of months ago.

Another assumption I'm making is that you're looking for a one-step process or script that does anything and everything in one step for almost 2 hours of capture.

God no. Long story short, it's a basketball game. Three cameras. All will need different settings, but they cut between each other so frequently it's quicker to run the whole thing through for each set of filters, then arrange them in Premiere later. Since I'll be going through everything there, I'll be able to spot any issues that crop up in individual shots and patch them up later.

How old is this PC of yours? Custom build? Retail?

Six or seven years, I think. Went into a PC shop, had a discussion about what I was intending to do with the machine, and they came up with a list of components and put them all together. I've since upgraded the RAM (from 8GB to 16GB, replacing the previous two sticks with four new sticks) and the video card (because the old one died).

Here's what I'm actually running on that short in-studio introduction to the main program:

Script 1:

AVISource("..\Captures\Capture Panasonic.avi")
Trim(1313,2147) ++ Trim(81511,82004) ++ Trim(152304,153380)
ConverttoYV12(matrix="rec601",interlaced=true)
ColorYUV(cont_v=-40)
AssumeTFF().QTGMC(preset="medium",Border=true,Ezdenoise=10.0,denoiser="dfttest",shownoise=false)
Dehalo_alpha()
StabMod()
FixChromaBleeding()

Script 2:

AVISource("..\Process\Studio.avi")
MergeChroma(MCTemporalDenoise(settings="very high))
ChromaShift(C=10,L=-4)
MergeChroma(awarpsharp2(depth=30))
SmoothUV()
LimitedSharpenFaster(edgemode=2)
AddGrainC(2.0,2.5)
StabMod()
StabMod()
StabMod()

Script 3:

AVISource("Studio2.avi")
BadFrames(588,589,blend=true)
AssumeTFF().SeparateFields().SelectEvery(4,0,3).Weave()
AssumeTFF().SeparateFields()
a = last
e = a.SelectEven().RemoveSpotsMC3()
o = a.SelectOdd().RemoveSpotsMC3()
Interleave(e,o)
Weave()
ConverttoRGB32(matrix="rec601", interlaced=true)

And then finishing touches in VirtualDub (in this particular case, simply a crop and add borders).

So, yeah. There's a whole lot being done on the restoration front, I'm pretty much across that. It's the speed that's at issue here.

[LMotlow]

Yep, after some digging around I found a similar thread at digitalfaq. The scripts in that thread are by a former member here I used to know from AVS Forum. But your version has some odd changes. stabmod() four times? Have you noticed how two of the frame's borders flutter around after using that stabilizer? The MergeChroma line apparently works on the awful reds. But what happened to the original code idea that killed oversaturation in the V channel, which is the main color problem ? Well, whatever, you might want to lower EZDenoise to about 8 or 6, the value 10 seems like overkill IMO. That won't save a whole lot of time, though. You might consider moving LimitedSharpenFaster to near the end of the third script. Why sharpen spots before cleaning them?

Those scripts are a slowdown on any PC, but if you have a slow Vista machine to begin with you might want to hang around and see if members here concoct some different ideas. I think the other forum discounted RAM as a problem if I read it correctly. as far as I can see, the original scripts were cleaning the hell out of that sample.

[koberulz]

I'm trying to get the scripts to run as fast as they should be running, rather than work on what the scripts are actually doing to the video, at this point. As you say, the script I posted in the OP shouldn't be taking five and a half minutes.

[shekh]

Where did you get ColorTools from?
I have a fixed version here https://sourceforge.net/projects/vdfiltermod/files/plugins/ColorTools/1.5%20update1/

[koberulz]

Not sure where I got it but I'll give that one a shot, thanks.

[koberulz]

That Color Tools does seem more stable.

Just had to reboot my computer, and after shutting down it stuck on a black screen with an underscore in the top left corner, before the BIOS stuff happened, for ages. So there definitely seems to be a computer issue here.

[koberulz]

Anyone?

[koberulz]

RAM previews in After Effects that used to render at full speed, or slightly faster, are now rendering at three or four frames per second.

EDIT: Just tried to work with some HD video, which locked up my entire computer and I had to reboot.

It took 5:48 to get to the login screen, 2:25 of which was the black screen with the underscore, prior to the intel splash screen and the BIOS Settings/Boot Options prompts. I stopped timing at that point but it was another 15-20 minutes after logging in before I could actually open and use any programs.

[Nelson37]

Power off PC. Disconnect ALL cables to ALL drives and power on, enter BIOS, compare times.

Next, shut down, connect power and data to ONE drive, test as above. Repeat with each drive, individually. INCLUDE OPTICAL DRIVES.

Probabilities are bad or failing drive, loose cable connection, bad cable, possible mobo problem, possible RAM problem.

Also remove and re-seat video card, remove other cards in system.

Symptoms indicate a basic communication or functionality problem with hardware.

[koberulz]

How would I backup and replace my C: drive? CrystalDiskInfo has it listed as 'Caution'. I've done it with data drives just fine; it's simply a matter of copying things across. But given I need to be using the operating system to copy C: in the first place...

[Nelson37]

Best procedure for dealing with a suspect OS install on a suspect drive is NOT repeat NOT to make a backup. Re-install on a fresh drive. IF you make an image copy, then do a repair install of the OS, and be prepared to deal with multiple flaky issues. Why copy corrupted files? And, no, you don't need a functional OS to copy the drive, but, that is a bad idea based on current data, which is incomplete and likely inaccurate. ISOLATE and IDENTIFY, proper diagnostic steps almost always eliminate significant parts of the system for a specific purpose. You have failed to observe this standard.

There is a reason I gave a specific, detailed set on analysis steps that will give an absolute, categorical answer instead of a guess from the software flavor of the week. Do as you wish.

[koberulz]

Well, I don't really want to get halfway through testing and have C: fail with no backups, even if there are other solutions in the longer term.

Even without this issue backing up C: is probably a good idea.

[Nelson37]

You are going to unplug it and later plug it back in. If it fails that fast you are not going to be able to back it up anyway. Also, this can identify a bad cable connection, of which there are decent odds that is your entire problem. Procedure will also identify if an entirely separate component is the cause of the issue.

You have already spend significant time and effort chasing a total red herring when the problem was something completely different. Now you want to make a precise copy of a drive that cannot reliably read the data on it. Verifying whether or not the drive itself is at fault is step one, if it truly is, then an understanding of the nature of the problem is necessary. An occasional error in a data file is bad, but can simply be avoided. A similar issue involving files belonging to the operating system is very much more serious, the time and effort spent attempting to resolve these can be significant, especially for those who do not recognize an impending drive failure in the first place.

I'm a pro, I do not do this as a hobby. You copy all the data you can, after verifying carefully the true nature of the problem, then you get another good drive, and do a re-install of the OS from scratch, because that gives you the best odds of a workable functioning system, instead of a crapfest built upon an unstable foundation, which is what you have now. It would probably help your understanding of this situation to understand that the methods necessary to make a functional OS backup circumvent many of the data validation systems in place for simple data copy, in short, you will copy the drive errors to the new drive.

Do as you wish, I'm not the one that spent two weeks chasing the wrong problem. Some folks enjoy that sort of thing, imaging an OS from a clearly failing drive is a very good way to achieve more of the same.

[LMotlow]

You copy all the data you can, after verifying carefully the true nature of the problem, then you get another good drive, and do a re-install of the OS from scratch, because that gives you the best odds of a workable functioning system, instead of a crapfest built upon an unstable foundation, which is what you have now. It would probably help your understanding of this situation to understand that the methods necessary to make a functional OS backup circumvent many of the data validation systems in place for simple data copy, in short, you will copy the drive errors to the new drive.

Agreed. A dysfunctional drive likely has read/write errors anyway, which many imaging apps will refuse to backup or will do so only with warnings to proceed "at your own risk". It might take time, but copying desired data to another location first will insure you have that data if a fresh reinstall is necessary. In my own practice I've done both, just in case -- copy data as well as make an image backup when possible. But nothing beats a fresh install on an old, well-used PC, even if it does take me 3 or 4 days to re-install my own heap of software.

In the future you might consider keeping documents, pictures, and that sort of thing on a different drive anyway, so copying won't be needed with a new install. In this hobby, apps like Avisynth and VirtualDub and their plugins don't need to be installed on the OS drive. Mine are installed elsewhere and simply require a fresh quick install, not a copy and hunt down-and-restore project. VirtualDub doesn't even need an install; on a fresh system it "installs" simply by running VirtualDub.exe. No copying of plugins and downloads, etc., is necessary if kept on a different drive. Avisynth itself installs only two dll's in the Windows system folder and makes some registry entries. The Avisynth program folder and plugins don't have to be on the C: drive. I'd just give up if I had 10 years of Avisynth/VirtualDub plugins, downloads, and whatnot on my C drive and lost that drive. They're all on a D: drive and periodically backed up (copied) to an external drive for safety.

The OS drive is for the operating system and installed software packages. It should not be used for mass storage. I could have saved hours or even days of system re-install time if I didn't have to first copy all the docs, pictures, videos, and internet downloads that customers keep on their OS drive, and then copy it all back again after the re-install.

[koberulz]

I do have all my data - but not AviSynth - on a different drive, for the most part. There are a couple of files on my desktop.

But other than that...open Firefox tabs, bookmarks, history, logins, etc. Any other program settings, or installs, or whatever else goes on.

I found Paragon Backup/Restore 14, but I got an I/O error on my newest external drive, which I was saving the backup to.

As I said, I currently have no backups of C: at all, so to me it makes sense to make one so at least I have something. Then we can get onto troubleshooting. If it turns out to be a bad cable or whatever, I can make another backup then.

[Nelson37]

Sure, spend hours or days trying to make a backup, instead of 30 seconds checking the cable. That makes perfect sense.

You sure that IO error was related to the external drive, right?

Good luck in finding what the actual problem is.

[koberulz]

It said an I/O error on drive 4. Drive 4 is the external, C: is drive 0.

Time to intel splash screen with no drives connected is seven seconds.

For the next step, do I progressively add drives, or do I plug in one, test it, unplug it and plug in another, etc.?

EDIT: Time to intel splash screen with just C: plugged in is six seconds.

EDIT 2: Realised I forgot to time it to the login screen, so I shut it down from the login screen to try again. The 'shutting down' screen stayed up for a while so I decided to time it. The screen went black at 1:06. Was probably a good 10-15 seconds before I started the timer. That's absurdly long, right?

Time to the login screen is still 1:43, a lot of which is a lengthy black screen between the loading screen and the login screen.

EDIT 3: Okay, just doing each of them individually rather than adding, time to intel splash screen:
C: 6 seconds
D: (main data) 5 seconds
E: (BluRay burner) 5 seconds
F: (secondary data) 5 seconds
G: (DVD burner) 5 seconds
L: (external data) 16 seconds, booted to Windows XP loading screen and then gave a BSOD. Apparently there's a semi-functional version of XP on there.

Untested: J, K, M, N, O. All external data. The problem long predates O and M, and probably N. Can't tell which is J and which is K just by looking at them.

H and I are virtual drives.

[koberulz]

J and K were 9 and 14 seconds, although as above I don't know which is which.

M was 39 seconds, N 49 seconds, O three minutes and 26 seconds.

But the issue with VirtualDub predates O coming into my possession.

[Nelson37]

Describe your internal power supply, and whether or not all external drives have their own power supply. That is a LOT of drives, which, unfortunately, means a LOT of possibilities for failure. Draw a circle around L drive and a big one around O.

You could have a small problem with the script and a big one with the drive, regardless, that is a LOT of drives. Do not connect those that are not needed.

Now - repeat individual drive tests. Next, connect C plus all NECESSARY drives, with no UN-NECESSARY drives, test several times. O is likely the problem but you have a LOT of drives.

Short version would be to permanently remove L and O and re-run script tests, resume normal operation and check for boot failures as previously observed. Connect additional external drives only as needed.

Did I mention that is a LOT of drives? Issues with total power available, interactions among drives, cumulative errors, etc. ISOLATE and IDENTIFY. The externals should have been unplugged FIRST THING without a second thought.

Also describe type of connection for externals. USB hard drive enclosures are exceptionally unreliable and prone to failure long term. Connecting a half-dozen or so of them is a near certain guarantee of problem, not to mention performance issues, etc. One at a time, and that only when necessary. Not for regular operation. If they are all drawing power from the USB connection then stop, go get a large stick, and beat yourself with it.

[koberulz]

Describe your internal power supply, and whether or not all external drives have their own power supply.

Describe what? It's an Antec Truepower 650, I think? It has that written on it, at least.

N is plugged in at the wall, the rest are just portable drives.

Draw a circle around L drive and a big one around O.

The problem predates O, though. That one just has some stuff on it I'm working with temporarily. M is only plugged in as needed. N is what I'm working off for the AviSynth script. J should probably be unplugged, K holds all my stock footage.

Now - repeat individual drive tests.

Just for extra data, or am I overlooking a variable that's supposed to change?

Also describe type of connection for externals. USB hard drive enclosures are exceptionally unreliable and prone to failure long term.

You mean to convert an internal drive to an external drive? I'm using one of those for L. It's not that old though, I don't think. A year, maybe?

EDIT: Plugged in C, D, and E. Six seconds to intel splash screen, followed by a Windows Boot Manager screen saying Windows failed to start, and a hardware or software change might be the cause.

File: \Windows\system32\winload.exe
Status: 0x000000c
Info: The selected entry could not be loaded because the application is missing or corrupt.


EDIT 2: Okay, it was actually F instead of C, because all hard drives look the same and I got confused. Apparently F has a partial Vista installation on it for some reason.

With C, D and E plugged in this time: five seconds to intel splash screen, but still 1:51 to the login screen.

EDIT 3: 17 seconds from login screen to desktop. As soon as the desktop showed, I double-clicked on an OpenOffice Calc file that was there. It opened after 2:47.

I'm trying to run my script, after plugging in N:, but I keep getting '(Not Responding)' in Windows Explorer when I try to right-click on it. I got it open in VDub once, but that stopped responding. It took the Windows Explorer window about seven or eight seconds to even open in the first place. After getting the script to run, aborted at 2:30 with an estimated total time of 5:23.

So, no improvement on any front except boot time, and that's still not good.

[Nelson37]

If you do not know what to describe about your power supply, that is part of the problem. The 650 is for 650 watts, which powers the mobo, and ALL the external drives, except the one with wall power, with the externals drawing from the USB bus. **** me.

You have files loading from a multitude of external drives at boot time. ELIMINATE THIS, AND NEVER LET IT HAPPEN AGAIN.

OK, eliminate ALL external drives. Boot C ONLY. Remove any and all references to external drives, copy the files to C, do whatever, however, get a stable boot to C. The reason to repeat the tests is because ONE repeat ONE measured instance is just not enough to establish a reliable pattern, a reliable pattern consists of several more than ONE, and so far, ONE is all you got.

BUT, with the total goatscrew you have made of your system, that is a secondary problem. Primary issue is get a stable boot to C that does not reference external USB drives, because they are unreliable, TEMPORARY spawn of the devil, and all startup references to them must be removed.

READ AGAIN my comment about O drive, and the probable existence of more than one problem, and repeat READING IT AGAIN until it makes some sense to you, and then REMOVE THE FREAKING DRIVE that causes an over 3 minute delay in boot time, because, maybe, after that, there might be some slight chance at solving the multitude of additional problems that are no doubt present. Jesus.

Download CCleaner at piriform dot com and run the file clean and registry clean. Reboot and do it again. Then download malwarebytes, run it, reboot, and run it again. Do these with ALL external drives removed, then again with individual drives connected, one at a time.

DO NOT CONNECT a half-dozen USB drives using USB power again unless you have a strong desire to virtually guarantee a ****ed up system with massively corrupted files, because that is a supremely dumbass idea.

After achieving a stable boot to C, with that drive, and that drive only, then and only then, re run the script test, with NO repeat NO reference to, or use of, any additional drives of any kind whatsoever, unless they are internally connected.

Oh yeah, and these drives with partial OS installs on them, the "some reason" is because some goddamn human installed an OS on them, then decided to use them for storage drives, with no understanding whatsoever of just how, why, when, or how such drives can, and will, at some point be the cause of a significant ****up of some sort, sooner or later.

[koberulz]

I'll get through that over the course of the day but just quickly: that boot was with only C (system), D (internal data), and E (Blu-Ray burner) connected. I plugged in N just to do the AviSynth test, N has its own power supply.