Hey Everyone,

I spent some time researching Xgrid this afternoon with an eye towards figuring out just exactly how helpful it could be to making ffmpegX faster. The results were fairly interesting, and the primary answer is there are two ways it can help - one would be easy to implement, but would only help in a small percentage of situations, and the other would help all of the time, if it worked at all, but whether it would work comes down to the details of the output video formats.

I'm not very knowledgable about video transcoding, so I had hoped to engage others here to figure out whether the encoders themselves can do what they would need to for this to work.

First, and Xgrid overview. Basically, all Xgrid does is farm out CLI tasks, and the files needed for them, to other CPUs. It doesn't just make it like your computer has more processors. If you can ssh to another machine and start a task in parallel with one you already have running, then Xgrid can do that for you. That's it.

The easy but not so useful way:

So, the simpler way that Xgrid could help ffmpegX is if you have several files to transcode. It would be a trivial matter for ffmpegX to submit these tasks to Xgrid as jobs, instead of just running them in the as normal Unix commands. This way, if you have 3 macs, you would not have to manually copy files to each of them to transcode, you would not have to monitor the progress, or copy the result files back - and if a job completed, Xgrid would immediately send the next job to that computer, there would be no down time while you finished dinner, unaware that one of your jobs was done.
In fact, all it would take is a checkbox and a text field for "Submit jobs to Xgrid controller [name]". Then if checked, ffmpegX would merely have to structure it's CLI differently to use the 'xgrid' cli command to launch the encoder, plus a couple of options for telling it where the input files are, and what to do with the output files.

The question is, how often do the users of ffmpegX need to run large batches of encodings. I would go so far as to say that this isn't worth Major4's time unless it is frequent to queue up *more* encodings than you have CPUs. Otherwise you should just run them all manually, one on each CPU you have access to.

The hard but better way:

The more exciting way that ffmpegX might make use of Xgrid would be to take a single file, and break it up into X pieces, where X is the number of CPU's in your grid. The problems I see with this are:

1) How would it break up the files?
2) How would it put the results back together?

Let's talk about 1) first.

People have suggested breaking files up at chapter boundaries. First of all, personally, pretty close to *none* of the source I ever have to use with ffmpegX has chapters. But even if it did, it still doesn't answer the question of how it would actually break up the files?

One option is not to break them up, but instead ship the entire source file to every CPU, even if it's only going to use a fraction of it. You could simply tell the encoder to start at a certain point in the file, and stop a certain point after that. The downside of this is that if you want to use 4 CPUs, you have to copy the file 4 times. In my experience, the copying of the file is up to 5% of the time it takes to encode it. That's ok, but bumping that up to 20% is kinda pricey.

But if you do anything else, you're going to need to use a tool to process the file to figure out exactly where you can split it, and that tool's going to take probably 50% as long to run as the transcoder, so why bother.

Now onto question 2:

Let's say you have 4 CPU's, and you've somehow gotten them to each produce an output file with a different 1/4th of the result. How do you splice them together? If they are VOB files, you can just cat them together, that's not so bad. But for just about every other format, the question is, is there an easy way to splice pieces together into a single whole file, or does that basically take as long as processing the whole file?

And there is a secondary question: If the position in the stream where ffmpegX split the file is not an exact frame boundary, then won't the resulting spliced output file have a single frame glitch at each splice point? Maybe this isn't that important, I'm not sure how happy people would be if their grid-processed videos had X glitches in them, one per CPU in the grid. I don't really know how noticeable a single duplicated or missing frame would be, or if it would cause a jolt to have an I-frame where it normally wouldn't be, such as very soon after another I-frame. I'm assuming here that if you can splice them together, you're going to somehow take the existing I-frame at the beginning of the second stream and put it directly after the last frame in the first stream.


What it boils down to is simply, is there a tool that can split and join video streams, and how fast is it compared to doing the entire transcoding? If it takes half as long as transcoding, then there is not enough benefit to doing any of this. But if a tool can quickly skip say 50 MB worth of a stream, and then dive right in at that point, figure out where in the stream format it is, and split the file at a logical breaking point, then we have something to talk about. Likewise, we need this, or another tool, to be able to do a similarly efficient job of splicing the multiple files together.

So, who out there knows a lot about the capabilities of the included, and other, video processing tools out there?

And for that matter, who things it would be useful to just do the simple version - submit entire transcoding tasks as jobs to Xgrid?

Thanks,
Avram