Distributed Network Processing --> FrameServing!?!

[Trenton_Net]

Hey guys,

I was recently trying to get several computers to do distributed network processing of files. This involves having several computers grabing source data from a master computer, and encoding their own segment or CD of a movie.

The problem im having is that when you do something like frameserving, how can you get the other computers to see the file correctly? It appears that frames can only be served to the computer which actualy runs the frame server? It appears frames don't get served to clients who access the same file over the lan.

Is there a way to overcome this problem?

[kwag]

The problem has to do with the wrapper. As you say, you can read in the local machine, but not from a remote machine, because you don't have a RPC/wrapper to frameserve via TCP/IP. If you were dealing with U*nix to U*ix, you could have an IPC communication without a problem, but you are either frameserving with Vdub or AviSynth on Windows platform, and there's no way you're going to read from a client machine, even if you have the same .dll's installed on the client and the server.

-kwag

[Vapor]

I'll do some tests tonight and report back.

Which frameserver are you using?

It might be possible to use avisynth on the client PC's and in the script reference the source file from the server PC?

eg:

AVISource("\\serverpc\Drive\File.avi")

or maybe if that does work map a drive on the client PC's and:

AVISource("X:\\File.avi")

I'll let you know if it works later tonight.

[Trenton_Net]

Any luck? I use Virtua Dub --> Tmpgenc

[Vapor]

My apologies - had no time to check.

I will make the effort this evening.

[bunyip]

Frameserving across a network would indeed be a great time-saver, however it seems it is not to be, just yet - here is an extract from the VirtualDub user's guide which says it all.

Signpost files

A .vdr signpost file only holds the full name of the frameserver. This includes your NetBIOS computer name if you have one. However, there is no support yet for cross-computer frameserving. The signpost serves only to tell the frameclient software which frameserver to connect to. The frameclient then communicates with the server directly for all other information. Signpost files are not needed for the VCM driver, only for the AVIFile driver.

In case you get any smart ideas about sharing .vdr signposts across the network, it won’t work. The client on the other end will notice the computer name is incorrect and will post an error.

[Vapor]

It can be done - sorry for the extended delay.

Setup your encode using your AVISYNTH script as normal on the server PC and just install AVISYNTH (+any DLL's you use for it) and CCE on the client.

Drop the .avs file into CCE on the client and bingo The client does all frameserving and encoding - the server is basically used as a file server.

To make things easier I did change the drive letter which hosts the file on the server PC to R: and on the client I mapped that network drive to R: on the client also. That way when you drop the .avs into CCE all of the input and output files are read and written to the server PC Not a single file is needed on the client PC

This is tested and works fine for VOB > D2V > AVS > CCE.

I did find one odd thing when frameserving DivX encoded avi's using the above method though - CPU usage on the client PC was very low - 5-20%. Not too much of a problem though as most DivX files are 700meg or so and can be copied locally.

I have not tested with any uncompressed sources - if someone tries this method with such I would appreciate a post back on your findings.

I have missed out some obvious steps early on but I'm sure most of you know about those parts. Any questions simply post back.

On a final note about performance - local processes have higher priority than network ones so hammering the same HD used for the source/destination files on the server PC does affect the performance of the client PC's encoding (CPU usage visibly drops) as their accesses are given lower priority. I have a 100mbit full duplex, full switched network and do not know whether a slower network would affect performance. I do however doubt it due to the relatively slow rate of data transfer. Unless you have some stonking PC's to use the actual data more or less dribbles.

I know this is not in the vain of your requirements trenton but does show it can be done. What source are you working with so I can have a closer look at your exact scenario?