Removing adds from 5,000 MPEG2 files.

[Phil H]

Hi,

I have a library of about 5,000 files recorded off Free To Air television. I am interested in somehow removing adds from all 5,000 files.

What would be the best way of going about this? I wouldn't also mind converting the video in MPEG4.

Thanks.

[redwudz]

No real reason to convert them. You could try a program like Comskip. It takes a bit of 'tuning' but is about the closest you may find for automating the procedure. Even with a program like that, it will take a long, long time.

Or if you want to do it manually, maybe VideoReDo or MPEG-VCR or MPEG StreamClip.

[Phil H]

Okay I will look into those thanks.

[PuzZLeR]

If you don't mind doing it manually, and don't mind paying for them, the best are: Womble MPEG Video Wizard DvD, VideoReDo TVSuite and TMPGEnc MPEG Editor. Each do it losslessly (no re-encoding except the very few frames where you made the cuts) so that means full quality and less time exporting the completed file.

Cuttermaran is free and can do it manually as well if you don't mind demuxing and remuxing streams.

Ulead VideoStudio (not free) is excellent with their Multi-Trim feature. However, you have to re-encode the final result since their "Smart Render" is buggy and broken with MPEG-2 IMO.

As for automatically removing the ads, I only know of VideoReDo's "Ad-Detective" and VideoStudio's "Ad-Zapper". But they take some processing time, are sometimes not perfect and you still have to check the results anyway in the end to make sure - certainly not a feature for me when I want life to be easier. I personally haven't had much success with them.

Maybe it's me, or maybe I'm obsessive or maybe I'm not doing it right, but using the apps mentioned, it's very easy to get into a "rhythm" when you get used to them manually click-by-click and now I can do it much faster, and more reliably, without need for these automatic features. Experiment on your own first.

As for MPEG-4, keep in mind you still have the option to keep them in MPEG-2 - easy to process, author and play on any DvD player. As well, you retain full quality. But if you need to compress to MPEG-4, your best bets IMO today are:

DivX and Xvid: both high quality and fast encoding and play on many devices.

H.264 (using x264 or Nero Recode): even better quality for the file size but demand more processing to encode and still as of today a rarity on devices (although that will change, especially when DivX 7 comes out).

[redwudz]

I probably should have added when I said 'No real reason to convert them', I meant for editing. Highly compressed formats like MPEG-4 are much worse for editing than MPEG. Most compressed formats like Divx/Xvid have keyframes located every 300 or so frames. And if you cut other than on keyframes, you would have to re-encode at least at the cutpoint. This makes frame accurate editing very difficult. And with commercials, you want accurate editing if you don't want ad remnants at each edit point.

If you want to convert them to save space, then do your editing first, then convert.

But if you remove all the ads, they are likely to end up quite a bit smaller anyway. And you would save a lot of encoding time and quality loss.

[PuzZLeR]

If you want to convert them to save space, then do your editing first, then convert.

I absolutely second that for the O/P. Whenever going from MPEG-2 -> MPEG-4 it is highly recommended all edits are complete before the final encode. MPEG-4 will be a major pain to edit, even if it is only a few cuts, so avoid any need to later.

But if you remove all the ads, they are likely to end up quite a bit smaller anyway. And you would save a lot of encoding time and quality loss.

I dumped a bunch of unfinished stuff from my DVR and camera last month to my PC's hard drive for later edits. After finishing off the commercial cuts, the trimming of the ends and the removal of other unwanted stuff I was pleasantly surprised how many GBs I actually shaved off. It's amazing sometimes how unwanted stuff clutters your drive.

[bbhank]

OK, already. Anybody wondering why its so difficult to find programs that remove commercials automatically? First, the codes that tell where the program starts and ends and the cuts for commercials are not recorded with the MPEG stream. Once these are sent to the station there is no need for them to send them further. The only use anyone else would have for them would be to power an automated deletion or recording process (for those who actually record the commercials and not the shows, such as those ofus who work in advertising).
Second and probably even more important is the fact that advertisers don't want you to be able to delete their commercials. They pay a lot of money to the stations to put up their ads. Stations live off that revenue along with the revenue from their broadcast through cable and the like. They want you to see that stuff whether or not you want it.
If these programs were readily available, many of us would never look at commercials. The general consensus has been for many years that commercials are garbage, necessary baggage to make television affordable. As with junk mail, I don't care about any of that. I refuse all junk mail and have no desire to look at commercials, especially since they are attempting to disregard ones desire not to receive them.
DVDMF is one of the better tools for cutting video and getting the disk out in one package. Much improvement has been made in the latest versions. Such is not true with the AdZapper. It relies on finding blank spots and does not do well at that. These spots are not always easily found even manually in some MPEGs due to their brevity. The program has no "smarts" when it comes to recognition of content and once it is run a few times its ability to perform the cutting decreases to nil.
All the other programs listed here require manual intervention - not satisfying the request for automation at all. When requesting "automated" commercial removal, we mean - input uncut MPEG recording of program with commercials, output MPEG without commercials cleanly without further intervention.

Bottom line is that there is a need and desire for a reliable, clean, TOTALLY automated way to remove commercials from MPEG files and if anybody has concise, succinct input on where such programming can be located and obtained - please......

[MOVIEGEEK]

Bottom line is there's no automatic way of removings ads,I would just leave ads in the video because if you converted three a day it would take you 5 years to do all of them.You have to ask yourself these questions:
Are these videos available on DVD?
Will I want to watch these videos 5 or 10 years from now?

[bbhank]

I have to disagree on what is bottom line. Here's why:
The ability to automate the processes is a matter of programming. This is being worked on as we speak. It is taking a variety of applications designed to work together assembled by people whose expertise is strong in each area. Once this done, a combined customizable suite is the logical outcome.
Today's software and hardware are faster than ever such that final rendering/processing times are disk write speeds. Software to produce final product now ranges from free to high end.
The very question of availability of content and especially, price, drive this project. And yes, I do and will want to look at these in 5 years or 10 years or whenever.

[PuzZLeR]

@bbhank:
There are indeed programs that remove ads, but in theory, they are not very practical. For example, VideoStudio has something called Ad-Zapper in its Multi-Trim interface. However, it's not very accurate unless you increase the sensitivity parameter. Increasing the sensitivity parameter slows speed. Then at this slow speed, it suddenly becomes much more practical to do it yourself then, especially with the good tools available.

Honestly, using a three hour sporting event as an example, it would take me less than 5 minutes to do this manually - scrub, click, scrub, click, scrub, click, scrub, click, ..., and I'm done. Womble, TMPGEnc MPEG Editor and VideoReDo have scrub features that make this a breeze for MPEG-1/2. If it happened to be AVI, the latest VirtualDub(Mod) tools, with their latest features like Smart Render and batching make this as well too easy.

It's not a complicated venture at all like it once used to be a few years ago. I remember the "old days"... ughhh...

@MOVIEGEEK:
As for leaving the ads in, I agree with you totally if you are just going to watch it, once or twice, and not keep it, or if available, or will be available soon, on DvD. Why bother? Fully agree.

Even before I make the decision of whether I keep it, I still will watch it with the ads, and cut them out only if I want to archive it.

However, whenever I do make the decision to archive something, it's always without the ads given the excellent tools available out there and the extra space (and re-watch time later) I save.

[bbhank]

AdZapper is also a tool in the latest DVDMF, and yea it royally sux. It has some other problems as well making it not a viable option. In addition, again it is not fully automated. One still has to go in and manually remove its choices and/or correct its mistakes making full manual faster. Our goal is to automate this step. We do not want to prescreen anything. Everything is archived first.

[Supreme2k]

Edit: God Damn it! Sucked into another dead thread.

[bbhank]

With automation, at today's machine speeds, 5000 files is not an issue. Even though we are not dealing with 5000 files in one batch, the machine speed and disk space are available.
Know that it can be done!!! We're looking at it using tools that are readily available, some free, and a little geeking of our own.

[MOVIEGEEK]

With automation, at today's machine speeds, 5000 files is not an issue.

Hardware isn't the issue,it's the software that's lacking.
I would rather go to a store and rent or buy the video instead of spending time removing ads.

[bbhank]

Read previous posts, please.
As stated these are not movies or sporting events. Neither are the Free To Air programs.
I found some of the answers in this forum along with some Google searching. And yes, there is some various code being written. The solution's been out there all along.
Don't be discouraged.

[Supreme2k]

With automation, at today's machine speeds, 5000 files is not an issue. Even though we are not dealing with 5000 files in one batch, the machine speed and disk space are available.

Most people (including me) aren't referring to you, but rather the thread starter. 5000 files is definitely an issue for him.

[ks47]

I don’t know how these television shows are archived. I had an occasion to convert to DVD (MPEG-2) from VHS format. They were about 100 ½ hour episodes of anime off the TV with commercials. It consumed almost a month of my free time. I used TsunamiMPEG DVDAuthor to cut away unwanted portions of video and chapters were created at the same time for authoring purposes.

In order to do editing on the computer, you first need to copy the video to the hard drive with analog to digital conversion, whether it be using TV/Video capture card, converter device, camcorder with pass through or whatever. I used a DVD recorder to -RAM/RW then into the hard drive.

There are other freeware like AviDemux, to help you along. At any rate to do it manually requires a lot of your time for editing and conversion time.

TS

Edit: Ignore the parts of the 2 paragraphs above, I didn't realize they were already in the form of MPEG-2.

[bbhank]

These posts have been place to help the person who has 5000 files to convert and not directed to the rest of the forum unless the information is needed. I am trying to emphasize that this is NOT as daunting a task as some would have one believe and there is no use in being discouraged by anyone who does not feel up to the challenge but would rather complain than offer valid, succinct, concrete, solutions or directions to find such solutions.
If you who have files to convert want more information on how we are progressing and what processes we use to get this done, including machine specs, please contact us directly, as this cannot and will not be posted on this or any other forum.
As I have stated, it's not rocket science and the information is readily available. You have to massage it to get it to work.

[Number Six]

The last time the OP'er came to this site was on Aug 23, 2008 09:40 - so, either he is BUSY with this project, or he is in cyber oblivion.

[jagabo]

The ability to automate the processes is a matter of programming.

In order to fully know what is an ad and what is program software will have to understand the real world. We are many many years away from that. Automated software that is 99 percent accurate isn't good enough if it occasionally removes part of the program.

[bbhank]

Thank technology it is not "years" away.
Nuff sed.

[bbhank]

[guns1inger]

Thank technology it is not "years" away.
Nuff sed.

Thank you for posting a non-answer answer. As helpful as you may feel you have been in offer the sage advice that "it can be done", by offering no concrete or useful solution, you have in fact offered nothing.

However you obviously feel very smug about you cryptic cleverness, so good for you.

[AlanHK]

It might be helpful to note that ads aren't placed at random. If you're doing it manually, you will notice that each station has a pattern: maybe a block of 2-3 minutes at 12, 28, 42, 50 minutes past the hour, for instance. And ads are usually exact multiples of 30 seconds in duration. So you can just jump to that zone and scan back and forth and delete each ad block fairly quickly. Not automatic, but takes a couple of minutes per hour of video. I usually delete the "previously on" and "next week" segments while I'm at it -- either redundant or unwanted spoilers in my view.

[jagabo]

I usually delete the "previously on" and "next week" segments while I'm at it -- either redundant or unwanted spoilers in my view.

I hate those spoilers too.

[guns1inger]

It depends on the country and the station. Over here the shortest commercial is 15 seconds. Breaks are not spaced at even intervals on the fully commercial stations, although you will usually find a pattern after a while. That pattern does vary by show, however, with the ads coming more frequently as the show progresses. The semi-commercial station has three ad breaks per show if the running time is longer than 30 minutes. These breaks will be either 2 minutes, or 2 minutes 30 seconds. The non-commercial station runs promos and news updates between programs (never during) and these can run for up to 7 - 8 minutes.

[jagabo]

Thank technology it is not "years" away.
Nuff sed.

All those other guys who tried it before were morons, right? And we'll soon have flying cars that drive themselves, robotic butlers to do all our household chores, computers that do everything via voice commands, a cure for the common cold... Oh, and world peace. How hard could that be?

[zoobie]

you can't even record a TV movie these days without some guy jumping up and down in an ad at the bottom of the screen...those you won't be able to remove...but the best way to go about it is to have a geeky guy with coke bottle glasses work endlessly on them as you sip iced tea by the pool...making sure to yell and curse him at least once an hour

[Kayembee]

Thank technology it is not "years" away.
Nuff sed.

All those other guys who tried it before were morons, right? And we'll soon have flying cars that drive themselves, robotic butlers to do all our household chores, computers that do everything via voice commands, a cure for the common cold... Oh, and world peace. How hard could that be?

Aw, give the guy a break. Perfection may be impossible, but "damn good" can be done. I've used
Comskip a couple times experimentally, with no tweaks, and it was near-perfect: no program
deleted, and only a couple seconds of ad missed.

I've never archived commercial TV, so I just >>> thru ads on time-shift recordings, but if I
needed to delete commercials, even on 5000 shows, I bet Comskip would be good enough.

Kayembee

[erik1958]

Why not use Comskip to detect where the commercials are.
Let comskip generate a .edl file or some other format that is recognized by a player to automatically skip over the commercials without cutting them out.
You could then even convert the 5000 recordings to a higher compression format and still have the playback without commercials without the need to cut them out.

Example players that can skip the commercials are mplayer (using the .edl file) and zoomplayer (has its specific format of which I forgot the name, its in the comskip documentation)

In the gbpvr forum there is a long thread on "automatic cutting and transcoding" http://forums.gbpvr.com/showthread.php?t=10606 that proves many people are actually doing what you want.