Encoding/Transcoding LARGE files...help!

[PsychOfMSE]

Hey guys, I'm currently using a Poweredge 2900 server with dual quad core xeons to do video encoding as I don't have a desktop that is really up to snuff.

I'm working with 20-100gb video files and am ripping commercials out and transcoding them for archival.


I have a budget of around $3000-4000 for a rig to make this process much faster. Right now it's taking upwards of 3 days to transcode a file.


My question is, what route would be best for me to go?

1. New Highest End Core i7 (4960x or similar)
2. Mid-Range Core i7 and a couple beefy GPUs (maybe dual titans or a beast of a professional video card?)
3. New server (8-10 core hyperthreaded xeon processor (E5-2670 or similar)

Any option that I go with will have 2 Samsung 840 500gb SSDs in Raid 0 and 6-8 2tb drives in Raid 10 for storage.



For $3-4K what would you guys do?


Thanks!!

[poisondeathray]

I woud find out what's wrong if it takes you 3 days to encode a single file ...

What format are the source files ?

What are you encoding them to ? Doesn't sound like an archival format to me

[PsychOfMSE]

I've actually been using MCEBuddy to rip commercials out.

I capture the video in TS at the highest resolution/quality I can from a Happauge HD PVR 2.


I think load it into MCEBuddy and it will pull the commercials out and transcode it to MP4 at the highest quality I can.

[deadrats]

i would go with option 1, your budget isn't anywhere near enough for a 10 core xeon and i don't think it's enough for an 8 core setup either.

[poisondeathray]

Well, you're re-encoding the video and losing quality. Not only that, you're probably taking 2.99 days longer than you should be (you're probably using unnecessary crazy slow settings). Upgrading might still take you 2 days per video the way you're doing it now...

A better strategy IMO would be to use a smart editor that re-encodes only a few frames around your cut sites around GOP boundaries e.g. solveigmm video splitter, videoredo beta, tmpgenc mpeg smart renderer

It might only take a few minutes, not days, because the segments in between your cuts are just copied through (no quality loss in the segments passed through, very fast, you're basically limited by disk I/O speed) . Better quality, much faster, less expensive than upgrading your HW

BUT - The problem is I have heard the HDPVR2 streams can be buggy , and some firmware versions make the streams either more buggy or less buggy - so I'm not sure of how compatible they will be with these software. You might try one of the trial versions out or ask in the Hauppauge forums for software compatiblity issues

[jagabo]

A better strategy IMO would be to use a smart editor that re-encodes only a few frames around your cut sites around GOP boundaries e.g. solveigmm video splitter, videoredo beta, tmpgenc mpeg smart renderer

I agree. And "highest resolution/quality I can from a Happauge HD PVR 2" isn't all that great to start with. Don't make it worse.

[PsychOfMSE]

i would go with option 1, your budget isn't anywhere near enough for a 10 core xeon and i don't think it's enough for an 8 core setup either.

Well I could definitely get an 8 core E5-2450 rig for around $3500. There are tons of 8 core E5 series servers on eBay that are refurbished/used that would give me 8 cores/16 threads in my budget.


Would it be a better idea to ditch the HD PVR 2 and pick up something better, and then go with a new rig as well?


I guess my question is what would be the best way of capturing 1080p video, ripping commercials out, and uploading it to an Amazon CloudFront server for distribution in the highest quality possible?

[poisondeathray]

What are you capturing from ? What is the source ?

[PsychOfMSE]

What are you capturing from ? What is the source ?

It's local football/basketball games. I live stream & capture UK sports for friends and family and also keep an archive of all games on my home server. It's being captured from Dish.

[PsychOfMSE]

I am open to any suggestions from capture hardware, to software, to encoding hardware & software. I just want to provide the highest quality recordings for friends & family all while keeping as high quality recordings for myself.

[poisondeathray]

I am open to any suggestions from capture hardware, to software, to encoding hardware & software. I just want to provide the highest quality recordings for friends & family all while keeping as high quality recordings for myself.

Realistically you probably don't mean "highest quality" , because that means uncompressed captures, with something like the intensity pro or or other blackmagic devices/cards/decks

What data rates does the HDPVR record at 1080i ? 12-16 Mbit/sec ?

If you think your files are "large" now, uncompressed captures will be about 50-60x larger. Your same 20GB file will be over 1TB . Your friends/ family in the UK will love you for sending those huge files LOL



Also, you mentioned 1080p video, but in the US, local sports basketball/football games will be broadcast at 1080i59.94 fields per second on most affilates (I think some might be 720p59.94)

[PsychOfMSE]

I am open to any suggestions from capture hardware, to software, to encoding hardware & software. I just want to provide the highest quality recordings for friends & family all while keeping as high quality recordings for myself.

Realistically you probably don't mean "highest quality" , because that means uncompressed captures, with something like the intensity pro or or other blackmagic devices/cards/decks

What data rates does the HDPVR record at 1080i ? 12-16 Mbit/sec ?

If you think your files are "large" now, uncompressed captures will be about 50-60x larger. Your same 20GB file will be over 1TB . Your friends/ family in the UK will love you for sending those huge files LOL



Also, you mentioned 1080p video, but in the US, local sports basketball/football games will be broadcast at 1080i59.94 fields per second on most affilates (I think some might be 720p59.94)

I don't mind large files. I want the highest quality recording that I can store on my local server. I've got 17tb of storage at home so I've got plenty of space right now for high quality recordings.


I'd like to be able to convert these uncompressed recordings into various qualities. One quality to upload to cloudfront for streaming, another smaller quality to send to friends/family, and my uncompressed quality for archival at home.


What would be the best route to go to get uncompressed high quality captures from a dish 1080i signal and then be able to time-effectively work with these files into different quality files?

[jagabo]

I don't mind large files. I want the highest quality recording that I can store on my local server.

Uncomrpessed YV12 video at 1920x1080 at 30i runs about 336 GB per hour. With lossless compression you could get that down to 1/2 to 1/3 that size.

[poisondeathray]

Probably one of the Black Magic products, like the intensity pro

You can edit in something like vdub (If you're only deleting commercials out)

You can use lossless compression to reduce your filesizes (~ 1/2) eg. huffyuv, ut video codec . That edited lossless compressed file can also serve as your archived master, and as the template for your multiple versions

[PsychOfMSE]

What about a setup like this:

Dish signal through HDMI into
HD Fury 3 HDCP Stripper through HDMI into
Black Magic Intensity Pro through PCI-E into


A Desktop with:
Core i7-4960x processor
Asus x79 Deluxe mobo
64gb of Ram
4 Samsung 840 500gb SSDs in Raid 0


Would this give me ultra high quality and the ability to relatively quickly manipulate the files?