[Seeking Devs] Cloud Transcode: Distributed FFMpeg in the Cloud

[koxon]

Hi Guys,

I would like to advertise the "Cloud Transcode" project I am working on for my startup.

The goal of this project is to provide a cheap solution to run FFMpeg in the cloud in order to make it scalable and distributed.
Today's solution to transcode videos in the Cloud are very expensive and not affordable if you have large volume of videos to transcode.

Our solution is to use Amazon web services to run a distributed version of FFMpeg on several machines and allow parallel transcoding.
It's working ! But it still needs quite a bit of work.
We are seeking developers that would be interested in joining us to develop the first open source cloud transcoding platform running FFMpeg. It's very promising and can cut transcoding cost by hundreds. It is highly scalable, decoupled and can run locally, in hybrid mode (cloud/local) or entirely in Amazon Cloud on EC2 instances.

For more information about the project, check us out on GitHub at the following address:
https://github.com/sportarchive/CloudTranscode

Cheers !

Koxon

[Selur]

Just out of curiosity "distributed version of FFMpeg" -> didn't know there was a version of FFmpeg which allowed distributed encoding, got a link for that?
(I interpreted "distributed version of FFMpeg" as a version which allowed distributed encoding over at network or multiple ffmpeg clients.)

[enim]

OP must be looking for php, Object Oriented JavaScript, AJAX and YAML all-in-one Video Transcoding guru.

[Bjs]

Ive seen a few php scripts already in use on cloud services and all work well thou a bit limited, however

It's very promising and can cut transcoding cost by hundreds.

If this goes a head make sure clients are informed prior to using the service as they are the one's paying for upload and download data used ... isp data bandwidth is not free.

[Cornucopia]

Every processing or encode cycle has its bottlenecks. For most of us, it is the CPU algorithms (because they ARE complex processes). So this guy is going to farm out those processes into much smaller chunks and speed things up? Well, that would be great, except now the bottleneck is in the extended upload/download and in the data assignment breakout (and later subsequent reassemble). Seems like it's just trading one bottleneck for another.

A LAN-based version of this might be workable. Cloud-based?

Hint about the chunks: Scope should probably be GOP-level, not Frame-level. Think about it...

Scott

[Selur]

btw. Ripbot264 already supports network based distributed encoding.
-> only think interesting to me is the "distributed version of FFMpeg" (the rest doesn't seem that new)
Drom the locks of it it's mainly Symfony2, which is a nice framework do develop with (good choice!).

[koxon]

Hi Guys,

Thanks for the interest.

FFMpeg doesn't handle distributed processing. We do it. We dispatch transcoding jobs accross several workers, each running on a different machine.
Each worker process the input video provided and transcode it into one output.

Let's say you have a job that needs to transcode one file into 5 alternative files.
With Cloud Transcode, the 5 transcoding tasks will run in parralel on seperate machines. That offer scalability and total control. You can make it fast (more machines) or keep it cheap and slow (few machines).

Concerning the split in several chunks, this is a road we want to explore. There is a great paper about this, created by some students in a Finland University. This will be the next step. I already contacted them and we will hopefully start to work together.
See: This Link

The key of this project is to make transcoding cheap and scalable. IF you use commercial solutions to transcode large volume of video such as: Amazon Elastic transcoder, Zencoder, encoding.com, etc ... well, it's extremly expensive!
To give you an idea, 250TB of video to transcode in 3 alternative files, is approximatly $80,000 with Amazon. So obvioulsy very few can afford that.

The key with our solution is that you can control how fast you want to transcode. You can be super fast (and expensive) if you have 200 machines available in your cluster. Or slow and cheap if you have only a handful of boxes.

The beauty is that it can even run locally on your desktops and laptops thanks to Amazon cloud services. Any machine with an Internet connection can be part of your "transcoding cluster". The only requirement is to have an Amazon AWS account as we use the SWF (Simple Workflow) service. Your worker communicate with Amazon SWF over the internet (or locally in case of EC2 instances) to get the task to do. Obviously, for performance, you don't want your workers scattered around the globe, the Internet connection will be a bottleneck for file upload/download. But, you can test things out and for small files it may be ok.

SWF is super cheap, S3 too. So you will pay pretty much nothing if you runs workers locally.

For a more realistic business use case:
Let's say you are on a tight budget, you have tons of videos to transcode but don't need them soon, you have time. Then this solution is perfect.
You start few EC2 instances on Amazon, install the project and its dependencies on them. Upload your videos to your Amazon S3 bucket and then you roll !
You will get all your ouput videos uploaded back to another Amazon S3 bucket (all on Amazon local network so it's super fast).
The only bottleneck here is the initial upload.

Now let's say you quickly need to transcode many videos. Then start 50 instances, upload your videos, and your roll ! You will be done in no time. Once all transcoding jobs are over, you shutdown all your instances. The only thing you will pay is the price of the instances per hour. Relatively cheap, few cents an hour.

Our goal is to make it as a service and for hackers. Provide an API, and maybe later, a web interface to monitor the jobs.

Let me know if you have other questions

Koxon

[Selur]

Let's say you have a job that needs to transcode one file into 5 alternative files.
With Cloud Transcode, the 5 transcoding tasks will run in parralel on seperate machines. That offer scalability and total control. You can make it fast (more machines) or keep it cheap and slow (few machines).

okay, that's not what I had in mind when I was thinking about distributed encoding. I thought you would split the encoding job for one file and thus speed up the encoding of large files. What you are doing is simply encode the whole file with different settings in different instances. That's not really new and not an ffmpeg enhancement, but simply a UI feature.´

Thanks for clearing that up.

[koxon]

simply a UI feature.

Note that there is no UI (User Interface) to this project.
You just need to execute 2 scripts:

• Decider: Make the decision based on the workflow information: Initiate new task, cancel task, look out for task timeout, etc.
• ActivityPoller: Listen for new incoming task to work on, process and send heartbeat back to the workflow.

How can you submit new jobs ?
You just create a JSON input that describes what needs to be done. You send it to the transcoding stack using Amazon SQS (messaging system). The transcoding stack listen to a SQS channel and if a new JSON input (with the right format) comes in, it will start a new workflow using Amazon SWF and initiate the necessary activity tasks to be processed.

Transcoding Job distribution is not new, that's true. What's new is an open source platform using Cloud Services to do the distribution.
That's what commercial solutions (zencoder, Amazon transcoder, encoding.com) are doing. We try to be the open source alternative to that

[Selur]

We try to be the open source alternative to that

I wish you good luck with that. I just misunderstood the initial post.
It's nice to see someone is trying to make an open source alternative to the existing Cloud based encoding services.

[koxon]

I wish you good luck with that. I just misunderstood the initial post.
It's nice to see someone is trying to make an open source alternative to the existing Cloud based encoding services.

Thanks Selur. Hopefully it will come to completion pretty soon.

Updates:
We are developing several transcoding modules. So pretty soon, not only you'll be able transcode videos but other media files as well:
Audio, Images and Documents.

We are still looking for experienced developers with good PHP skills. If you think you have what it takes to join two experienced hackers, contact me

Best,

Koxon

[hrasadi]

The key of this project is to make transcoding cheap and scalable.

I was impressed with this part an I also took a look on your Github pages, checking the performance benchmarks. But I cannot see how Cloud Transcode can outperform Zencoder or Amazon in price.

I have also checked the prices on Zencoder.com and in the best case your method would costs a double of what Zencoder offers.

I am missing something?

[koxon]

The key of this project is to make transcoding cheap and scalable.

I was impressed with this part an I also took a look on your Github pages, checking the performance benchmarks. But I cannot see how Cloud Transcode can outperform Zencoder or Amazon in price.

I have also checked the prices on Zencoder.com and in the best case your method would costs a double of what Zencoder offers.

I am missing something?

Hi,

Thanks for your interest.

First, there was a bug in the spreadsheet and I forgot to push it out to git. I just did it, thanks for that !
Second, I readjusted the global pricing as AWS dropped their prices recently, big savings!
I attached the benchmark to this post for anyone interested

Now, let's compare our solution with Zencoder:
If you look at Zencoder pricing, they charge based on the number of minutes of output.
Let's take the "pay as you go" pricing model: $0.05/minutes of output.

In my benchmark file, calculations are based on 5000 minutes of input and output.
Let's say you need to transcode these 5000 minutes of videos in WebHD 16:9 (960x540):
With Zencoder, you would pay: 5000*0.05*2 = $250 (If real HD (720p), minutes count double in Zencoder so that would be $500)
With our solution, let's say you go for a single instance 'c3.xlarge' (most cost effective one), you'll pay $11.694 and it will take 3341.17 minutes to transcode.

Now, let's transcode the same 5000 minutes but in SD (480x360):
With Zencoder, you will still pay: 5000*0.05 = $250
With our solution, same instance, 'c3.xlarge', you'll pay $6.164 and it will take 1761 minutes to transcode.

With our solution you can control how fast you will do your processing.
So, imagine you have many input videos and need many outputs. Well, you can start several c3.xlarge instances and cut down your processing time.
But it will increase you cost.

Whatever you do, the single cost of transcoding "one minute" is far less than the commercial solutions out there.
Take a look at the benchmark again, particularly the PROJECTIONS at the bottom.

I hope this help and thank you for taking a look! Much appreciated.

Best,

Koxon

[hrasadi]

Hi,

Thnx for the update in the spreadsheet. I can now see the difference in price. By the way there is one another defect in the file: in "Benchmarking Results" part the 'Cost / second transcoded' cells calculation is wrong. you must divide this number to 30 (the length of the video you transcoded), so that you can calculate the cost based on the length of the original video.

Back to the context of the thread, I have some questions:

- Have you performed any performance comparison between Zencoder (or Amazon) & FFMpeg? This way you can compare the ROIs of each way. i.e. is the lower cost paying back for the performance drawbacks?

- Are you serious on using PHP? I always thought of PHP as an advantage in web programming.

Best,
Hamid

[koxon]

Hi,

Thnx for the update in the spreadsheet. I can now see the difference in price. By the way there is one another defect in the file: in "Benchmarking Results" part the 'Cost / second transcoded' cells calculation is wrong. you must divide this number to 30 (the length of the video you transcoded), so that you can calculate the cost based on the length of the original video.

Back to the context of the thread, I have some questions:

- Have you performed any performance comparison between Zencoder (or Amazon) & FFMpeg? This way you can compare the ROIs of each way. i.e. is the lower cost paying back for the performance drawbacks?

- Are you serious on using PHP? I always thought of PHP as an advantage in web programming.

Best,
Hamid

Hi Hamid,

Thanks for the second pair of eyes here! I fixed what you mentioned, and will update the file on github tonight.

I have not tested Zencoder. I tested Amazon Elastic Transcoder though. I couldn't test on lenghty videos because the cost is prohibitive ... However, transcoding is fast from what I saw. I think they must have a cluster of high performace machines doing the work.
However I don't think they split the videos to perform distributed transcoding on "video chunks". I presume one video is processed by one machine.

Chunking videos is another aspect I'm exploring. I am in talk with students from the University of Finland to see if we could implement their thesis into CloudTranscode. See: Thesis

If you have some benchmark on Zencoder let us know.

Fast transcoding is what all commercial solution are offering. But as a user you can't control how fast you want it too be. I'd love to be able to pay less for slower transcoding. Today you can't do that, you pay premium price for high performance and that's it.
With our solution, you scale up and scale out on demand. So if you want fast, then start several c3.8xlarge EC2 instances and you will transcode much faster

Thanks a lot,

Koxon

[koxon]

Hi,
- Are you serious on using PHP? I always thought of PHP as an advantage in web programming.

Haaa PHP. Always a source of debate

I chose PHP for several reasons:
- Our main project, which will use this transcoding stack, is in PHP. Keeping this in PHP allow continuity, easy integration and developers can participate in both projects.
- Amazon provides a well documented PHP SDK for its services
- PHP is a scripting language before anything else. Much like Perl you can use it for many purposes.
- PHP is not bad It's simple, faster and more secure than it used to be.

I could have done it in Java, but there was more overhead and I am not a great Java developer.
I needed to bootstrap this project quickly and PHP was the best choice at the time.

Koxon

[koxon]

I am happy to say that the stack is fully functional.
You can now transcode videos, generate thumbnails and insert watermarks. Next step is to handle audio, images and documents.

The project documentation can be found here:
https://sportarchive.hackpad.com/Cloud-Transcode-project-poG8vKTC16J

For adventurous people with an AWS account, please test it out!
I would love to get some feedback and have people testing it.

Edit: I am working in creating a JUJU Charm for CloudTranscode so you can easily deploy CloudTranscode in the Cloud.
See JuJu: https://juju.ubuntu.com/

Best

Koxon

[micronix]

Any updates on parallelism Development of CloudTranscode ?

Regards Sascha

[koxon]

Any updates on parallelism Development of CloudTranscode ?

Regards Sascha

What do you mean by "parallelism Development" ?

[micronix]

Chunking videos is another aspect I'm exploring. I am in talk with students from the University of Finland to see if we could implement their thesis into CloudTranscode. See: Thesis

This is parallelism method ^_^

[koxon]

Chunking videos is another aspect I'm exploring. I am in talk with students from the University of Finland to see if we could implement their thesis into CloudTranscode. See: Thesis

This is parallelism method ^_^

That's what I presumed.

That's not the priority for now but it's on the roadmap.
We are planning to implement a basic split/merge for h264 as a start.

[micronix]

Where i can view this Roadmap ?

Thanks Sascha

[koxon]

Where i can view this Roadmap ?

Thanks Sascha

It's not publicly available, and mostly in my head still.
However you can find more information on: https://sportarchive.hackpad.com/Cloud-Transcode-project-poG8vKTC16J#:h=Quick-start-with-Vagrant

[koxon]

Hi all,

I am happy to announce that we created a Vagrant box to easily bootstrap Cloud Transcode for development and testing.

Vagrant provides a way to start a local (or remote) Virtual Machine and load a "box" in it. A box is a complete OS with all the dependencies and soft pre-installed.
We created a Vagrant box with all the requirements to run Cloud Transcode: FFmpeg, imagemagick, PHP and more.
For those not familiar with Vagrant see: http://www.vagrantup.com/

You can easily set the box up, and provide your AWS credentials to it. It will bootstrap testing and development and insure consistent environment no matter how your host is locally setup.
It is a great way to quickly test the stack on Mac, Windows or Linux!

Please give it a try if you get a chance, and send me your feedback and issues on GitHub: https://github.com/sportarchive/CloudTranscode

Next step:
- Create an AMI for Amazon Ec2
- Allow Image transcoding with ImageMagick
- Improve thumbnails creation
- Allow Document transcoding with Unoconv
- Improve WorkflowTracker

Cheers,
Koxon

[micronix]

Interesting article.
The chunking method is the most wanted feature by all encoding engineers. It saves a lot of costs on every side.
Take a look http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Streaming-Media-Eas...Lag-96981.aspx

[koxon]

Interesting article.
The chunking method is the most wanted feature by all encoding engineers. It saves a lot of costs on every side.
Take a look http://www.streamingmedia.com/Articles/Editorial/Featured-Articles/Streaming-Media-Eas...Lag-96981.aspx

Thanks for the article Micronix! Very interesting. It's reassuring to see that I am going in the right direction: FFMpeg, Amazon, splitting, workfow ...

Again if some of you have spare time, please try to implement this! For now I am alone on this and I am being pulled in many different directions so I don't have as much time as I would like on the project. I am a basic version that works now so I am putting development on hold for a bit in order to focus on my start-up product (that will use CloudTranscode).

There are many implementation of the split out there, small scripts, detailed ffmpeg commands, merge,, etc. It's just a matter of implementing a SWF workflow in CloudTranscode that uses those.

Best,
Nicolas

[koxon]

Hey Guys,

Quick update on the project. We're making good progress. We are 3 guys working part time on this now.
My employee will start putting some hours as well in the coming days.

We'll come up with a Docker image so you can spawn a fully functional stack with one single command.

We are creating a much better Decider engine that will allow execution of arbitrary workflows. This will enable the split/merge implementation.
Still to be implemented, the split/merge is a tough piece of engineering. I am preparing the ground and documentation to enable the community to implement it.
If we have a good framework and a good base, I'm sure some of you will do miracles

We're also creating a "real" SDK that will allow your application to communicate with the stack. The protocol will be well documented.
We'll do a PHP implementation first. I hope the community will do other implementations in future.

I will post here again once the project is in Beta state.

best
koxon

[koxon]

Hey Guys,

The Beta is out.

We have splitted the project in two:

- CPE: Cloud Processing Engine. As the framework is quite generic we thought it could be used for other purposes than just transcoding. So we made it a separate project that you can use to process any self-contained tasks at scale in the Cloud.
- CT: Cloud Transcode. Retains only the transcoding activities responsible for converting videos. They are loaded by the CPE worker.

We're still working on the Docker images. They should be ready soon.

The documentation is not yet finalized but contains enough information to get started:

- CPE: http://sportarchive.github.io/CloudProcessingEngine/
- CT: http://sportarchive.github.io/CloudTranscode/

Give it a shot if you can and any feedback would be much appreciated. Contact me on GitHub.

Thanks
koxon

[koxon]

CT now supports:

- Arbitrary FFmpeg commands
- HTTP/HTTPS input file. Probe and Transcode files over HTTP on the fly. Big gain in processing speed and performance!

Cheers

[apueee]

Hello Koxon. It's great that you working on media transcoding and making it opensource. I need a media streaming system where i have to implement adaptive video streaming on user's uploaded video. Currently i'm using aws s3, elastic transcoder, cloudfront and lambda to create new job on transcoder. Still it's expensive solution though we are in development stage. I'm looking for alternate solution using ffmpeg and found this thread. Exactly what i was looking u already implemented those. I will look more on your repo. Do u have the Docker image ready now? or is there any step by step documentation to run the whole system?