Distributed video encoding

[exiled350]

Is there any apps or plugins for avisynth that would allow for distributed batch encoding? I am converting tv episodes from dvd into individual h264 files, and it takes a long time to run a job ~24h for 3hr of content. Id like to utlilize my other pc's that are on the network to help encode these files to reduce the time it takes. Other than taking the video to the other computers is there a way to do this?

[guns1inger]

What type of script, running on what PC system, takes 24 hours for three hours of content ? Even on my old system (Athlon 1800+) I could do very heavy restoration of VHS source (heavy denoising and other filtering) and get through 2+ hours of footage in around 11 - 12 hours.

It is a difficult problem, especially when using a frameserver to process the video prior to encoding. You gain little if only one part of the process is distributed. i.e. if the encoder can run across a render farm, but the avisynth script cannot, then you will bottle neck at the script end, leaving the farm waiting on content to encode. Likewise, if the script can run distributed by the encode cannot, then the avisynth will sit idle much of the time waiting on the encoder to call it.

I would suggest your solution will lie in two directions. First would be to post your script and computer specs to see what might be done to optimise your current processing. The numbers you cite seem large and I suspect could be brought down with some changes to either the script or the process. The example I cited (above) of 11 hours was a reduction from well over 30 because my first attempt included a 2-pass encode. This meant the video was run through an 11 - 12 hour filter process twice when being encoded. By encoding to a lossless codec after one pass, then doing a 2-pass encode on the resulting file, the entire process was reduced to around 14 hours. And this on an older, single core machine.

The second option is to simply manually run simultaneous jobs on multiple machines, or to use the Trim() statement to process multiple parts of one file on multiple machines, then join the results later.

However we need much more detail on your current script and process before offering definitive suggestions.

[jagabo]

Just encode individual episodes on different computers. No special software will be needed.

[exiled350]

Im using ripbot264 to encode m2v/ac3 clips to h264. Its scaling it to 1280x720 using spline36, denoising with fft3dgpu and deinterlacing. Typical run time is 3.5h for 23min of video on a 2-pass 6k. it may sound overkill but i tested the crap out of different settings and this looks the best with the wdtv on a 50" 1080p panny plasma.

My system is pretty old, athlon x2 4400+ @ 2500mhz with 2gb ram running vista x64 and 2x 6600gt 128mb. but it does the job. The time it takes really isnt that big of a deal, as i have another computer to use while this one is runing.

Unfourtnitaly, i cant run encodes for a while as i found out the fan on the first video card died causing it to run at 92°c under full load.

From what ive seen these encode times seem inline for my system, untill i can afford a x58/i7 setup, it will have to do.

[framebuster]

Media Encoding Cluster
this is the first Open Source, Plattform Independent Video Encoding Cluster
currently for Linux and Windows (Mac currently in planning).
please Visit our Webpage

http://www.codergrid.de/

any suggestions are Welcome,

Thanks for any input or replies.

[lordsmurf]

What type of script, running on what PC system, takes 24 hours for three hours of content ? Even on my old system (Athlon 1800+) I could do very heavy restoration of VHS source (heavy denoising and other filtering) and get through 2+ hours of footage in around 11 - 12 hours.

Try that with NeatVideo.

It's not uncommon to take 40-50 hours for a 2-hour video, with a heavy chain of filters, and that's on a quad-core.

[crazyant]

[removed: unrelated post]

[Noreen]

Hello!
I am a phd student. Can anyone send me the code for wyner-ziv or slepian wolf codec????????

[Cornucopia]

One neat way to do an Encoding "render farm" is to take a video and run scripted editor which cuts clips to multiple small lengths (like 2-5 min. each, although best to use exacting lengths equal to I-frame boundaries, also using a timecode naming structure), route each to a separate drive or folder.

Then, have these drives/folders be the source for watch scripts which process and encode each of these small clips to the final formats. (These scripts would ALL have to have identical settings for each clip)

Then, concatenate/assemble the clips (using TC numerical order) using a joiner or muxer, to a final, complete clip (later muxing in the audio, which should have been done separately).

It's really just a macroscopic version of the normally "microscopic" rendering paradigm. Since you are working with somewhat larger segments (than just Frames or GOPs), you usually get a reasonable bitrate setting, etc.
Works well with 8-32 machines.

Scott