Virtualdub Batch Question

[thecoreyburton]

I have a question regarding Virtualdub's queue and multiple instances.

I often do a lot of AVISynth processing and need to run this particular set through Virtualdub. Usually when I'm running Virtualdub I can have several instances running without any real slowdown (usually equal to the number of cores my processor has - so in this case 4). I'm also familiar with the batch queue and for jobs I have to run one-at-a-time, it's been extremely useful. I'm going away for a few days and want to leave my computer doing a collection of encodes so when I return my computer's not out of commission whilst encoding (or alternatively, it's useable and I do one at a time which takes significantly longer).

Is there a way to have separate queues for separate instances of Virtualdub or use the same queue for each instance (with each instance taking the next item to be encoded in the queue, so two instances aren't encoding the same thing)? Or is there a way to make Virtualdub's queue do four encodes at once on separate threads? I'd love to get these all out of the way at once.

Thanks for reading my question!

[hello_hello]

If you specifically need to use VirtualDub for some reason this post won't help much, however......

I can't answer your question but have you tried MeGUI? You can add as many scripts to the job queue for encoding as you like and specify how many to run simultaneously (via the "Workers" menu). I think the audio is limited to one job at a time (due to unwanted artefacts when encoding audio simultaneously with Avisynth at some stage), but other than that it should step through the job queue running "x" number of jobs at a time (whatever you set).

You can also right click on any job(s) in the queue and select "run in temporary worker" and they'll run independently of the normal job queue and whether it's running or not, at least after adding individual encoding jobs via the video and audio sections. The OneClick encoder works a bit differently in respect to how jobs appear in the queue, but it can be used for batch encoding and it'll accept Avisynth scripts as the input. It can do automatic resizing and de-interlacing etc but I'd imagine you'd want to create a OneClick preset with that sort of thing disabled.

Are you using slow Avisynth filtering? I ask because without any especially slow filtering, encoding a single script usually keeps my 4 core CPU pretty busy when encoding with the x264 encoder. If the filtering is quite slow it might drop down to around 50% in which case I sometimes run two at the same time.

[thecoreyburton]

Thanks for the reply!

Unfortunately I have to sick with Virtualdub at this point as I need lossless RGB32. This is only an intermediate step and the output file needs to be processed further down the track and the source is currently RGB32, so I'd like to avoid compression or unnecessary colorspace conversion until the end of the filter chain. If MeGUI can do this, I'm just not aware of it at the moment!

Usually I use MeGUI, Handbrake (via AVFS) or even a pipe to x264 depending on the situation, script and intention but this is one of those cases where I don't think that'll cut it. That being said, I didn't know about the temporary worker function, so even though it didn't solve the problem it's going to be a useful piece of information to have in the future.

The scripts are very slow and vary a little depending on the input. I usually batch generate a bunch of scripts that fix a majority of the problems in the source as a whole and then tweak them individually from there. There are several stacked temporal functions and there's a repair function in there too. The encoding rate sits at about 0.78fps, but the clips are relatively short. Stacking four at once has some slowdown, but not enough to warrant running them one at a time. The speeds drop to about 0.6fps, but given that four clips will be done at once the net FPS is pushed to around 2.4fps - making it very valuable (as far as I can tell) to do multiple encodes at once.

To further add, the reason I want to save them losslessly is because of the stacked temporal functions. To finish my script, some of QTGMC's progressive features are required (followed by a separate chroma pass, hence the want to retaining all the chroma data) and adding that on top of a script already going that slow pushes it down exponentially because of wanting to temporally filter something that's already being temporally filtered. As far as I can tell, it's much quicker to save the file to a lossless intermediate location and then do the final encode off of a script that loads that up and does the final filtering.

I'm considering investigating Virtualdub command lines - I'm not fully familiar with virtualdub's options but because I'm loading an AVISynth script all I need is codec choice and destination file so I don't think it'll be too tricky. The problem would be making a batch with all of the commands (I think there's around 120) and telling it to launch four jobs but not the fifth until one of the first four is done. I could do four separate batches, but that wouldn't account for the fact that some of the videos finish much faster than the others, so potentially one or more batches would finish a lot faster, leaving only one running in the end (and negating this effort).

I really appreciate your post in terms of MeGUI usage - I know it's not exactly what I asked for and it didn't solve the problem, but I've definitely taken quite a lot from it for future use and it's going to be more than useful in my encoding life.

[JVRaines]

Have you read "Batch operation - Distributed mode" in the VirtualDub Help file?

[hello_hello]

MeGUI can do lossless HuffYUV as long as it's YUY2. Unfortunately the command line it uses isn't compatible with an RGB script output and it'll result in an error.

Try AnotherGUI.
Stick ffmpeg.exe in the same folder.
Create a new AnotherGUI preset for ffmpeg. The following command line will work for an RGB output, but you might want to check if ffmpeg is capable of other types of lossless compression. I don't know off the top of my head. Change the file exention for an output that's not an AVI if need be.

-report -i "<FullSourceFileName>" -y -threads 1 -c:v ffvhuff "<OutputPath><OutputFileName>.avi"

Save the preset.
Drag and drop your scripts onto AnotherGUI.
If need be, click on one of them in the GUI and use CTRL+A to select them all and choose a new output location.
Click "Go".

If you leave -report in the command line ffmpeg should save a log file for each job. Probably in the same folder as ffmpeg.exe. It might be handy if any jobs fail for some reason.
AnotherGUI has it's own option for setting the number of simultaneous processes. You could adjust it in combination with -threads in the command line.

I can't remember for certain how AnotherGUI does things when a single job fails. For instance if you load 20 jobs into the queue and number 12 fails for some reason, I think it'll just move onto jobs 13 to 20, but I don't know for sure so it might pay to check. I know the above command line works for video encoding and I'm sure ffmpeg can handle both video and audio output from a script, but I can't remember if any special command line options are required for that.

[_Al_]

check this out: http://www.virtualdub.org/blog/pivot/entry.php?id=20

To set up command line for VirtualDub first you need to make vcf file, it is script like text, you can create it yourself in notepad and save as something.vcf. But of course you do not know how. So there is method to sort of "record" it yourself, so open virtualdub, manually load your dummy (any - avisynth script), you manually set up options, filters in correct order , codecs or uncompressed whatever , then you store all of this by pressing [ctrl + S] or file/save processing settings ... store that vcf. That would give you my_workflow.vcf file. So then you can use this command line to run it with avisynth script, or more instances:

virtualdub.exe /s my_workflow.vcf /p avisynth.avs, new_file.avi /r

that is a theory, in reality, you have to write something like this:
"C:\tools\VirtualDub\vdub.exe" /s "C:\tools\plugins\my_workflow.vcf" /p "your_avisynth_script.avs","C:\destination\new.avi " /r

or if having three avisynth scripts (ergo three videos) you can just drop avisyth scripts one after another onto this Batch script:

@echo off
SETLOCAL
SET virtualdub="C:\tools\VirtualDub\vdub.exe"
SET load_script_vcf="C:\tools\plugins\my_workflow.vcf"
SET destination="C:\destination"
if not exist "%destination%" MD "%destination%"

"%virtualdub%" /s "%load_script_vcf%" /p "%~1","%destination%\%~n1_new.avi" /r
ENDLOCAL
pause
goto :eof

and that would run more instances at a time, just my guess, if not, VD would be somehow really badly written ...


if wanting to drop always unique vcf file and unique avisynth script, you drop both on this Batch file (make sure you avs is selected first ):

@echo off
SETLOCAL
SET virtualdub="C:\tools\VirtualDub\vdub.exe"
SET destination="C:\destination"
if not exist "%destination%" MD "%destination%"

"%virtualdub%" /s "%~2%" /p "%~1","%destination%\%~n1_new.avi" /r
ENDLOCAL
pause
goto :eof

[shekh]

Is there a way to have separate queues for separate instances of Virtualdub or use the same queue for each instance (with each instance taking the next item to be encoded in the queue, so two instances aren't encoding the same thing)?

As JVRaines said.
Distributed mode should to exactly that, you start server instance with shared jobs list and any number of slave instances to consume shared jobs list.

[thecoreyburton]

Distributed mode looks like it'll work a charm, I'm doing a dry test run now to make sure I've configured everything correctly but it looks to be more or less what I was after. AnotherGUI, the command line parameters, batch examples and everything else mentioned here are good fallbacks if for whatever reason this doesn't work and if not knowing more about these types of things will definitely come in handy with streamlining the encoding process in the future in different scenarios - so thank you all for your help and information!