cleaning video, does this make any sense?

[deadrats]

ok, i just had a potentially silly thought and want to see if there is any merit to my reasoning:

assume you are "cleaning" video using consumer grade software, specifically either movavi's or tmpg express and you are primarily denoising and color enhancing/adjusting (and at times de-interlacing), would i get better results if i first converted the video to dv or perhaps to a lossless avi using something a codec like huffy? my reasoning is that when i run a video that's in h264 or mpeg-2, i will have b and p frames in addition to i frames and rather than having the editor applying the video denoise/color filters to p and b frames, i will get better quality if i can have the editor apply said effects to only the i frames. now i'm guessing huffy and it's brethren don't use i frames but they would be losslessly compressed frames, so part of me thinks i would get better results.

while i'm at it, if i'm encoding to mpeg-2 and the encoder supports manually choosing, what is the preferred settings for color primaries, transfer characteristics and matrix coefficients? i'm currently using bt.709 for all 3 all the time, are there better settings?

thanks.

[edDV]

No editor of filter is going to work directly on the b and p frames. MPeg can be cut on the GOP (I frame). Otherwise, all subGOP editing and all filtering requires an MPeg decode to fields/frames.

Editors like Premiere will also convert YCbCr to RGB except in rare cases.

The workflow required depends on what you want to do.

[lordsmurf]

@deadrats
Those are not the right tools for restoring video.

[jagabo]

The process of filtering video (except for some DCT domain filters) involves completely decompressing the frames to YUV or RGB. So you will get exactly the same thing filtering a compressed source directly, or recompressing to uncompressed or a losslessly compressed format first.

[deadrats]

@deadrats
Those are not the right tools for restoring video.

what would you recommend?

[CogoSWSDS]

If there were a "FixEverythingThatsWrongWithThisVideo" filter, everyone would use it. However, there's no such thing. Different filters do different things and can be used in different ways to make your video look better. Without more details though, it is impossible to say which filters you should use.

You can post a short 2-second clip to this forum, or can upload to a file sharing site and then link to it. Don't post it to youtube and then link, youtube generally recompresses your video and can introduce even more problems than what you're trying to illustrate.

LordSmurf does video restoration professionally and can definitely point you to free (or commercial) tools that will do what you want.

The more details you can share, the more we can help you.

CogoSWDS

[lordsmurf]

If there were a "FixEverythingThatsWrongWithThisVideo" filter, everyone would use it.

Sure there is! Just press CTRL+ALT+ANYKEY

Finding that tricky anykey is always the issue for most folks. :P

[jagabo]

AviSynth, VirtualDub

For DV AVI, simple levels/colors adjustments without reencoding: Enosoft DV Processor.

[deadrats]

The more details you can share, the more we can help you.

ok, here's the deal: i have a bunch of legit, 100% legal dvd's of main stream movies and tv shows that are quite frankly encoded by what i have to surmise were a bunch of drunk chimps. i bought these dvd's back in the early days of dvd, from best buy and the quality sucks beyond belief. some of the worst offenders are my collection of "21 jump street", specifically the episodes with johnny depp (those were some great stories), most of them are so dark it's almost as if all the scenes were shot in a room with the lights turned down, some of them have significant levels of noise, if it wasn't for the fact that they were shrink wrapped and bought from a legit retailer i would swear that someone had captured them from tv using the crappiest capture card available, encoded them with the worst possible settings and then sold them to me. some of the movies are the same (especially my collection of the kung fu movies they used to show on saturday afternoons on channel 5 here in the ny area before it was fox) and i also have a number of adult dvd's from the '90's that have tons of noise and are dark as hell (if anyone is familiar with adult dvd's from the '90's they were, hands down, the worst quality dvd available anywhere, forget about the 6 hour compilations with all the compression artifacts, i'm talking about atrocious lighting, poorly adjusted contrast and color balance and bucket loads of noise, it's funny that things have come a full 180 and now adult blu-ray's are among the highest quality blu-rays you can buy, perhaps because they are shot on high quality cameras, some are shot using sony's "red one" and then authored straight to blu-ray with little processing in between).

anyway, basically i've been using the 90's era adult dvd's as guinea pigs, i have been experimenting with cleaning up the movies as much as i can and i figure once i perfect the technique i'll start the real project with my 21 jump street and kung fu movie collection.

i like movavi's software for ease of use, it's denoise filter is pretty good and it has a "magic enhance" filter which automagically corrects color levels and saturation but even though the encoder is obviously based on main concept's encoder, it is f'en slow, 20 minutes of video can take 2.5 to 3 hours to finish and the thing that really pisses me off is that if you stop the process before it's completed the app deletes the partial file, so the only way to see the results is to wait for the entire encode to finish.

tmpg express on the other hand has an excellent denoise filter and thanks to the fact that decode of mpeg-2 and all the filters are gpu accelerated, i can take a dvd source, use the most aggressive de-interlace filter (inverse telecine), set the denoise filter to the most aggressive settings, add an audio denoise filter (which works great, if your source's audio is noisy) and no matter how many built in filters i chain together, the encode is still done in real time with my cpu never going above 25% used. best of all if i interrupt the encode after a couple of minutes it doesn't delete the partial file so i can experiment with different settings.

so there you have it, my source is dvd spec mpeg-2, in all cases poorly encoded, noisy, in most cases poorly lit, in most cases horribly interlaced, and i'm trying to find the method that produces the best results.

[jagabo]

Sounds like you're happy with Tmpg Express.

To answer your original question again, it's not possible to avoid decompressing b frames if you're filtering. In theory, since MPEG 2 uses DCT compression you could adjust brightness without fully decompressing the macroblocks (somewhat like the way DVD Shrink doesn't fully decompress frames when shrinking) but I'm not aware of any software that does this.

[poisondeathray]

post a sample


of the adult dvd


....joking


But seriously, if you are deinterlacing, highly recommended to use some avisynth based ones. I know for certain TMPGenc's deinterlacer sucks big time. It may be faster, but the quality is much much worse. I can post comparisons if you need examples or proof...

You also have more choices and filters with avisynth. Generally you would target specfic types of noise in the attempt to retain as much detail as possible. Using a generic denoiser usually gives you inferior results

Color correction is usually better done in "pro" software like premiere / vegas which allow for 10-bit precision (if you do it correctly) , or after effects which allows for 16 and 32-bit. Avisynth is limited to 8-bit, which can result in banding when you are color correcting or even a simple levels adjustment.

But for simple corrections, avisynth is sometimes better, because you can stay in the same YUV colorspace, while most of the other "pro" and bought programs will do a RGB conversion, especially as soon as you add a filter.

Weigh your pros/cons for everything, there's usually tradeoffs in speed/quality depending on the route you choose

To answer your original 2nd question, the color primaries you would choose would be the one the original camera shot with or your original source was. This should be used thoughout the chain of processing from acquisition, to editing, to distribution. For some programs this is metadata only (ie. has no effect), but for others this actually changes the color coefficents and can change the appearance of colors. If you don't know what it was, and you have a HD source, BT.709 is usually a safe bet, but SD sources can vary a lot.

Good luck

[deadrats]

Sounds like you're happy with Tmpg Express.

for the most part i'm satisfied with the results, i also like the results movavi produces but it's just so freaken slow that i just want to kick someone in the nuts, however simply because i'm happy with what i have been able to achieve i'm still interested in improving the technique, tweaking it here and there, and if i can improve the quality a few more percentage points, then i'm willing to try it.

it's kind of like good bbq, you may be happy with the way the ribs or chicken come out but you still play around with cooking at different temperatures, using different charcoal, different sauces, seasonings, different marinades, to see if you can get it just a bit better.

basically before i commit myself to fixing my dvd collection i want to make sure that i can't possibly improve the quality anymore than i already can.

[deadrats]

But seriously, if you are deinterlacing, highly recommended to use some avisynth based ones. I know for certain TMPGenc's deinterlacer sucks big time. It may be faster, but the quality is much much worse. I can post comparisons if you need examples or proof...

you must know by now, of course i want to see proof. tmpg express offers about a dozen different de-interlacing options, the most aggressive method is inverse pulldown (reinforced anti-stripes) and the most aggressive mode is "24 fps (prioritize motion).

what does avisynth offer that's "much, much better". btw, i don't suppose there are any virtual dub or avisynth filters that are gpu accelerated, are there?

[poisondeathray]

you must know by now, of course i want to see proof. tmpg express offers about a dozen different de-interlacing options, the most aggressive method is inverse pulldown (reinforced anti-stripes) and the most aggressive mode is "24 fps (prioritize motion).

what does avisynth offer that's "much, much better". btw, i don't suppose there are any virtual dub or avisynth filters that are gpu accelerated, are there?

If we are just limiting our discussion to deinterlacing filters, all of tmpgenc's choices suck, I've tested them all out. If you want be to provide any more examples or different test clips let me know.

"What does avisynth offer that's better?" The answer: better quality and configurable options.

Here is a sample that cowboy satelitte uploaded for another thread. It's a good sample for deinterlacing, because guitar strings reveal the deinterlacing artifacts and jaggies nicely as they move at different angles. It's important to check all frames, because some frames maybe worse than others. But if you look at every frame, you will find almost every frame is significantly worse with TMPGEnc. These were encoded losslessly, so the only effect we are testing is the deinterlacer, not compression.

Sample
http://www.megaupload.com/?d=YSNG55F4

TMPGEnc - inverse pulldown (reinforced antistripes)


TMPGEnc - interpolation adjusted


YadifMod+NNEDI2


When you do this for yourself, and watch the clips in motion, you will notice the jaggies combine on subsequent frames to produce "shimmering". Very annoying. There is significantly fewer deinterlacing artifacts by using yadifmod+nnedi2. If you want an even slower, even less shimmer, tempgaussmc_beta1 is an option too. The point is avisynth has many more options that produce better quality. But yes, it takes longer, and is harder to use. Again, trade offs.

[poisondeathray]

BTW we could also do this exercise for noise reduction filters too... Avisynth has many more choices, better quality etc...

Why don't you give it shot and learn the basics? To be honest, I'm sure glad I did. I started out hating avisynth because I knew nothing about it. I asked lots of stupid questions and eventually learned enough to get by from the help of folks here and on doom9.

But if you're happy with TMPGEnc, then that's all that matters

Cheers

[jagabo]

The best noise reduction is Neat Video's filter. It's very slow though.
http://www.neatvideo.com/

That guitar MPEG video is interlaced. IVTC is the wrong deinterlacing method.

[deadrats]

The best noise reduction is Neat Video's filter. It's very slow though.
http://www.neatvideo.com/

you're not kidding!!! i just downloaded the demo for virtual dub and it is awesome, i'm blown away by the results. i have to admit i thought the job tmpg express did was good, now looking at it's results compared to neat video, i can see tmpg's short comings. the only problem is that now i'm left with trying to figure out the proper workflow, here is what i came up with:

load the vob into virtual dub mod, under filters i first select de-interlace (blend) and then neat video, i then export it to lagarith wrapped in an avi container.

the results are great, i just 2 "concerns":

1) it is slow!!! i'm talking a max of 5 fps.

2) the file sizes are huge. a full dvd takes about 40 gigs. this isn't that much of a problem considering that i have 4tb of space, but i'm just worried that when i try to feed the resulting avi into an encoder the encoder might choke.

one more question: there's no open source plug in comparable to neat video, is there? the home version of the virtual dub plug in is only $50 but it only supports a max resolution of 720x576, the pro version that supports unlimited resolution sizes (i may want to clean up some high def video) cost $100. i may just have to bite the bullet.

i also had another idea: after i'm done cleaning up each dvd, i'm thinking about encoding to h264 with ac3 audio wrapped in m2ts, in order to have the results by blu-ray compliant, that would be smarter than re-encoding back to dvd spec mpeg-2, no?

thanks for everyone's help, it looks like i'm going to be able to move forward with my project pretty soon, with the number of dvd's that i want to process i figure it's going to take me a while.

[jagabo]

now i'm left with trying to figure out the proper workflow, here is what i came up with:

load the vob into virtual dub mod, under filters i first select de-interlace (blend)

VirtualDubMod's blend deinterlace is one of the worst choices. Use DgIndex to build an index file, AviSynth's Mpeg2Source() to open it. Use one of the good AviSynth deinterlacers or IVTC (if appropriate). Here is a comparison of some AviSynth deinterlacers on some very difficult material:

https://forum.videohelp.com/images/guides/p1934885/stockholma_0-520_q3_yadif_mvbobmod_t...mca4_tdtmm.avi

If you really want to use VirtualDub for deinterlacing try Donald Graft's SmartDeinterlace. Or the Yadif deinterlacer in the newer versions of VirtualDub.

Open the AVS script with VirtualDub (or use AviSynth's ability to use VirtualDub filters) to filter with Neat Video.

Oh, and make sure your levels are correct before VirtualDub. VirtualDub will lose blacker-than-black (Y<16), and brighter-than-bright (Y>235).

i then export it to lagarith wrapped in an avi container.

That's your best bet if you are going to do a 2 pass MPEG 2 encode for DVD. If you're doing a single pass encode you can frame serve directly to your MPEG encoder (if it supports it).

the results are great, i just 2 "concerns":

1) it is slow!!! i'm talking a max of 5 fps.

Yes. If you want really slow combine AviSynth's TempGaussMC_beta1() deinterlacer with Neat Video. You'll probably get 1 or 2 fps! But you'll find that combination very hard to beat.

To speed things up a bit you might try a multithreaded build of AviSynth and SetMtMode(2). Some filters aren't thread safe though.

2) the file sizes are huge. a full dvd takes about 40 gigs. this isn't that much of a problem considering that i have 4tb of space, but i'm just worried that when i try to feed the resulting avi into an encoder the encoder might choke.

That shouldn't be an issue with modern MPEG 2 encoders.

one more question: there's no open source plug in comparable to neat video, is there?

Not that I've seen.

the home version of the virtual dub plug in is only $50 but it only supports a max resolution of 720x576, the pro version that supports unlimited resolution sizes (i may want to clean up some high def video) cost $100. i may just have to bite the bullet.

Sucks.

i also had another idea: after i'm done cleaning up each dvd, i'm thinking about encoding to h264 with ac3 audio wrapped in m2ts, in order to have the results by blu-ray compliant, that would be smarter than re-encoding back to dvd spec mpeg-2, no?

Yes, if you can live with that.

[poisondeathray]

In my experience, you can get similar results to neat video by using the right combination of avisynth filters, but there is no easy to use GUI. In some cases, you can get better results (more detail retention, less noise). But it takes a lot more fiddling around to get the right combo

Some avisynth filters can use GPU. e.g FFT3DfilterGPU , and MCTemporalDenoise have portions that are accelerated by GPU. MCTemporalDenoise is an aggregate of about 30 avisynth functions and highly configurable

For true SD blu-ray compliance, it has to be interlaced (or 60i signal). You lose a lot of your compression advantage with h.264 vs. MPEG2 when encoding interlaced. And if you are using x264, it hasn't been optimized for interlaced encoding at all. You would be better off getting HTPC or media box like wdtv in my opinion. Donald Graft has a DGAVCPulldown which supposedly functions like DGPulldown for MPEG2, but I'm unsure if it works with blu-ray
http://neuron2.net/dgavcpulldown/dgavcpulldown01.zip

Word of caution on tempgaussmc_beta1() - it blurs a bit in order to combat the deinterlace shimmer but you can adjust the settings - so you may have to dial down your noise reduction

[jagabo]

By the way, try deinterlacing poisondeathray's guitar video with TMPG Express and compare it to this TempGaussMC_beta1() deinterlace:

short.avi (Only the first 10 seconds and no audio to make sure it came in under 6 MB.)

You'll see there's no comparison.

[deadrats]

In my experience, you can get similar results to neat video by using the right combination of avisynth filters, but there is no easy to use GUI. In some cases, you can get better results (more detail retention, less noise). But it takes a lot more fiddling around to get the right combo

Some avisynth filters can use GPU. e.g FFT3DfilterGPU , and MCTemporalDenoise have portions that are accelerated by GPU. MCTemporalDenoise is an aggregate of about 30 avisynth functions and highly configurable

For true SD blu-ray compliance, it has to be interlaced (or 60i signal). You lose a lot of your compression advantage with h.264 vs. MPEG2 when encoding interlaced. And if you are using x264, it hasn't been optimized for interlaced encoding at all. You would be better off getting HTPC or media box like wdtv in my opinion. Donald Graft has a DGAVCPulldown which supposedly functions like DGPulldown for MPEG2, but I'm unsure if it works with blu-ray
http://neuron2.net/dgavcpulldown/dgavcpulldown01.zip

Word of caution on tempgaussmc_beta1() - it blurs a bit in order to combat the deinterlace shimmer but you can adjust the settings - so you may have to dial down your noise reduction

this kind of throws a monkey wrench in the works, based on what you said, if i'm planning to encode the final output to blu-ray compliant video then there's no point in de-interlacing the source vob's, is there? furthermore, is it possible to denoise interlaced content?

while i'm at it a question popped into my mind: how the hell do these dvd's get in this condition in the first place? specifically how many generations removed are they from the original film?

from what i thought i knew, first a movie was shot on film, say 35mm, 70mm, whatever, then all the footage was sent to the producer who basically put the story together, then a backup was made and the original was archived for safe keeping. the backup copy was then used to transfer to digital, then it was sent for all the special effects to be added, then when everything was done it was encoded into a movie and sent to an authoring house. i remember when blu-ray was first announced the sales pitch was that blu-ray was what used to be called the master (distributed on "pro-disk") and it was what dvd's were encoded from, is this a fairly accurate representation of the process from the scenes shot on film to the final dvd?

what about tv shows? what is the process from being shot to appearing on tv?

the reason i'm curious is because as i have mentioned a number of times i have come across some adult content, on various web sites and from other countries specifically italy, hungary and france, where the image is so crystal clear, you can see details that you would only expect to see if you were actually standing right next to the performers, and the one that really shocked me was i ran across a 720p clip about 20 minutes long (maybe a bit more, i seem to have lost it in all the hdd space i have) that was only about 500mb. at first i thought the file must be corrupt or an incomplete download (for such a small file size) or i figured the quality was going to be abysmal but to my surprise it was crystal clear. so i checked the bit rate and i was shocked to see it was vc-1 at 1mb/s. at first i wondered what encoder and parameters they were using that allowed them to drop the bit rate all the way to 1mb/s and still have such high quality and that's when i remembered that while walking through an electronics store some time ago i saw what looked like a semi-pro grade high end camera that had a built in hdd, recorded at 720p straight to vc-1, it had a wide screen lens, a audio mic on top of the handle and it cost about $2500.

that's when i realized why i was seeing all these really high quality adult movies and clips: they shoot the scenes straight to a digital format, in the case of the various web sites that offer hi def content they shoot the scene and then upload the footage with little to no editing or processing, so what the end user is getting is of the highest quality, in the case of the adult dvd's from europe they must be filming the scene with those hi end camcorders that have built in hdd and record in mpeg-2, they probably film the various scenes, edit them into one movie and author the dvd.

it kind of annoys me that after spending all that dough buying those dvd's i now have to sit and try to fix them. and it also annoys me that some of the commercial blu-rays of main stream movies from major studios suck so much that the same movie shown on hdtv is actually crisper than the blu-ray, i'm specifically talking about batman begins but there are others, how is it that the tv broadcast is of higher quality than the blu-ray? when a station wants to broadcast a movie, say in hi def, what do the use as the broadcast source? it can't by a blu-ray, perhaps an earlier generation version on tape?

ok, i'm done, it's just starting to piss me off that no one has ever created tools to easily clean up video or at least a nice easy to use avisynth gui that someone can just fire up and get the job done without having to jump through hoops. and why the hell are there so many de-interlacing methods? why have a dozen or more methods when only one or two actually produce good results?

i'm done ranting, complaining, thanks for all the help, it looks like i have to start digging through some avisynth documentation, maybe upgrade my cpu to a 32 core doesn'texistyet so that i can get some real time performance.

[poisondeathray]

this kind of throws a monkey wrench in the works, based on what you said, if i'm planning to encode the final output to blu-ray compliant video then there's no point in de-interlacing the source vob's, is there? furthermore, is it possible to denoise interlaced content?

If they are truly interlaced, not telecined, I would keep them interlaced. You can leave them as MPEG2 (or if you are doing some restoration/filtering , encode them to MPEG2).

Some filters work on interlaced content, the majority don't or don't work very well. Neat video has an interlaced setting, as do some avisynth filters like MCTemporalDenoise. Usually filters work much better on progressive content , especially temporal ones (that work on future and past frames). Interlacing screws up this temporal relationship. You usually have to separate the fields and work on individual fields, but that isn't ideal for image processing

while i'm at it a question popped into my mind: how the hell do these dvd's get in this condition in the first place? specifically how many generations removed are they from the original film?

It depends on the studio and the quality of the transfer. Very few do it properly, even the big name studios. You can take almost any blu-ray title and encode your own dvd with HCEnc or CCE for example, and 99% of the time it looks better than their offical DVD release, even using a lower bitrate! I've done this for several titles, and many others have as well. There are plenty of threads on this showing examples here and on other forums.

from what i thought i knew, first a movie was shot on film, say 35mm, 70mm, whatever, then all the footage was sent to the producer who basically put the story together, then a backup was made and the original was archived for safe keeping. the backup copy was then used to transfer to digital, then it was sent for all the special effects to be added, then when everything was done it was encoded into a movie and sent to an authoring house. i remember when blu-ray was first announced the sales pitch was that blu-ray was what used to be called the master (distributed on "pro-disk") and it was what dvd's were encoded from, is this a fairly accurate representation of the process from the scenes shot on film to the final dvd?

I don't think the blu-ray being a "master" for the DVD is true. A lot of corners cut if that were true

what about tv shows? what is the process from being shot to appearing on tv?

There were a couple of threads on this, but it varies a lot. Newer series are not being shot with digitally instead of film. EdDV , jagabo and manono had lots of good info on several threads. If you're really interested, start a new thread and edDV will probably fill you in.

the reason i'm curious is because as i have mentioned a number of times i have come across some adult content, on various web sites and from other countries specifically italy, hungary and france, where the image is so crystal clear, you can see details that you would only expect to see if you were actually standing right next to the performers, and the one that really shocked me was i ran across a 720p clip about 20 minutes long (maybe a bit more, i seem to have lost it in all the hdd space i have) that was only about 500mb. at first i thought the file must be corrupt or an incomplete download (for such a small file size) or i figured the quality was going to be abysmal but to my surprise it was crystal clear. so i checked the bit rate and i was shocked to see it was vc-1 at 1mb/s. at first i wondered what encoder and parameters they were using that allowed them to drop the bit rate all the way to 1mb/s and still have such high quality and that's when i remembered that while walking through an electronics store some time ago i saw what looked like a semi-pro grade high end camera that had a built in hdd, recorded at 720p straight to vc-1, it had a wide screen lens, a audio mic on top of the handle and it cost about $2500.

I don't know of any that record straight to VC-1. EdDV might know. More than likely it was re-encoded to VC-1, I'm 99.9999999999% certain of that.

DVCPro-HD, AVC-Intra, XDCAM are more commonly used "pro" acquisition formats, but none of them come for that low price.

that's when i realized why i was seeing all these really high quality adult movies and clips: they shoot the scenes straight to a digital format, in the case of the various web sites that offer hi def content they shoot the scene and then upload the footage with little to no editing or processing, so what the end user is getting is of the highest quality, in the case of the adult dvd's from europe they must be filming the scene with those hi end camcorders that have built in hdd and record in mpeg-2, they probably film the various scenes, edit them into one movie and author the dvd.

I seriously doubt it. Most of them are edited and re-encoded

it kind of annoys me that after spending all that dough buying those dvd's i now have to sit and try to fix them.

You haven't seen anything! Ask manono about his "Indian movie field blended garbage sources"

and it also annoys me that some of the commercial blu-rays of main stream movies from major studios suck so much that the same movie shown on hdtv is actually crisper than the blu-ray, i'm specifically talking about batman begins but there are others, how is it that the tv broadcast is of higher quality than the blu-ray? when a station wants to broadcast a movie, say in hi def, what do the use as the broadcast source? it can't by a blu-ray, perhaps an earlier generation version on tape?

Good question, I would like to know this too. I'm sure edDV or someone else knows

ok, i'm done, it's just starting to piss me off that no one has ever created tools to easily clean up video or at least a nice easy to use avisynth gui that someone can just fire up and get the job done without having to jump through hoops. and why the hell are there so many de-interlacing methods? why have a dozen or more methods when only one or two actually produce good results?

No gui. 100's have tried making a NLE based on avisynth, but none have really succeeded. As for filters - they all have pros/cons. There isn't 1 best for everything. TempGaussMC_Beta1() for deinterlacing is just about the best in general but this is source dependent. Also it's very slow, and I've seen cases where other filters are better on some sources. It also tends to blur (which may or maynot be a good thing).

If you want "close to perfect", or the best you can do, you often end up using various filters for various segments, and even mask different parts of a single frame and apply different effects/filters. For example, when you color correct dark scenes, usually "garbage" shows up when you brighten the image. You might only denoise parts of each frame , instead of losing important details like actors faces etc....There is a continuum of how far you are willing to go & how much effort you are willing to put in

i'm done ranting, complaining, thanks for all the help, it looks like i have to start digging through some avisynth documentation, maybe upgrade my cpu to a 32 core doesn'texistyet so that i can get some real time performance.

avisynth and filters more often than not are the bottleneck. Parallelization ususally doesn't help, but clockspeed does. Avisynth isn't very well multithreaded and very few of it's filters are. Avisynth-mt is very buggy. Even with 32 physical cores, maybe 1 or 2 would be fully used.

Even with conventional software like premiere/vegas/after effects, filtering is usually the bottleneck

[jagabo]

GUI for AviSynth: AVSP It's probably not as easy to use as you want though.

There are many different deinterlacers because:

1) There is no perfect way to deinterlace. Developers are working all the time on newer and better deinterlacers.

2) Different algorithms work better with some material than others.

3) Just because a particular deinterlacing method gives the best image quality with a particular video doesn't mean you will want to use it. TempGaussMC_beta1() often gives the best results but it's very slow. One might find Yadif() is almost as good for a particular video and, since it's 50 times faster, decide to use that instead.

[poisondeathray]

^ just to add, even tempgaussmc_beta1() has different interpolation and edimodes . I've seen it do a garbage job on some sources, and switching the edimode from eedi2 to nnedi2 made it pristine again. Sometimes it's as simple as fiddling with a single filter's settings, instead of stacking filters or switching filters

and deadrats, for your "HD euro porn" clips, aren't most of them watermarked? That would strongly suggest that they were re-encoded. And a 20min clip @ ~500MB is closer to ~3-3.5Mb/s depending on audio , not 1Mb/s

[deadrats]

for your "HD euro porn" clips, aren't most of them watermarked? That would strongly suggest that they were re-encoded. And a 20min clip @ ~500MB is closer to ~3-3.5Mb/s depending on audio , not 1Mb/s

nope, no watermarking, why would they be? as for the file size i'm probably off with the length, i'm sure it's 1 mb/s, i have to look through my hdd and find it, when i do i'll cut a short non-rule violating segment with movie maker and up it so that you can see it for yourself.

btw, what do you think of this filter i found:

http://www.risingresearch.com/en/denoise/

it's a vdub filter that supposedly denoises and de-interlaces at the same time.

edit: i just tried that gpu accelerated msu denoise vdub filter, now we're talking!!! now, what did we say was the finest vdub de-interlacing filter? is there a gpu accelerated one?

i think i have come up with a sweet workflow, let me know what you think: load vob into vdub mod, first filter will be a de-interlace filter, second will be msu, then i will use the gpu accelerated mpeg-2 encoder plug in for vdub, gmpeg2, mux everything into an mkv and done.

how does that sound?

[poisondeathray]

nope, no watermarking, why would they be?

If you download clips from websites, adult content or not, they are usually watermarked

as for the file size i'm probably off with the length, i'm sure it's 1 mb/s, i have to look through my hdd and find it, when i do i'll cut a short non-rule violating segment with movie maker and up it so that you can see it for yourself.

I would use asfbin.

btw, what do you think of this filter i found:

http://www.risingresearch.com/en/denoise/

it's a vdub filter that supposedly denoises and de-interlaces at the same time.

I haven't used it, but I am always looking for something better, faster, etc.. I highly doubt it will beat avisynth filters

Jagabo really bought up an important point. No 1 tool will be the best at everything. It really is source dependent. This is true for everything, even tmpgenc filters. It' s just that having a bunch of high quality options in the toolbox (like avisynth filters), enables you to tackle more situations more effectively.

TempGaussMC_Beta1() deinterlaces and denoises too. This is what I meant by blurring. The blur is to reduce the deinterlace flicker. This may or may not be a good thing depending on your goals and the type of footage. Often MPEG2 footage is low quality and full of macroblocks and actually benefits from a mild blur. Compare the detail in Cowboy Satellite's the hands/knuckles, and the pattern in the guitar

TGMC


YadifMod+NNEDI2

[edDV]

what about tv shows? what is the process from being shot to appearing on tv?

the reason i'm curious is because as i have mentioned a number of times i have come across some adult content, on various web sites and from other countries specifically italy, hungary and france, where the image is so crystal clear, you can see details that you would only expect to see if you were actually standing right next to the performers, and the one that really shocked me was i ran across a 720p clip about 20 minutes long (maybe a bit more, i seem to have lost it in all the hdd space i have) that was only about 500mb. at first i thought the file must be corrupt or an incomplete download (for such a small file size) or i figured the quality was going to be abysmal but to my surprise it was crystal clear. so i checked the bit rate and i was shocked to see it was vc-1 at 1mb/s. at first i wondered what encoder and parameters they were using that allowed them to drop the bit rate all the way to 1mb/s and still have such high quality and that's when i remembered that while walking through an electronics store some time ago i saw what looked like a semi-pro grade high end camera that had a built in hdd, recorded at 720p straight to vc-1, it had a wide screen lens, a audio mic on top of the handle and it cost about $2500.

that's when i realized why i was seeing all these really high quality adult movies and clips: they shoot the scenes straight to a digital format, in the case of the various web sites that offer hi def content they shoot the scene and then upload the footage with little to no editing or processing, so what the end user is getting is of the highest quality, in the case of the adult dvd's from europe they must be filming the scene with those hi end camcorders that have built in hdd and record in mpeg-2, they probably film the various scenes, edit them into one movie and author the dvd.

it kind of annoys me that after spending all that dough buying those dvd's i now have to sit and try to fix them. and it also annoys me that some of the commercial blu-rays of main stream movies from major studios suck so much that the same movie shown on hdtv is actually crisper than the blu-ray, i'm specifically talking about batman begins but there are others, how is it that the tv broadcast is of higher quality than the blu-ray? when a station wants to broadcast a movie, say in hi def, what do the use as the broadcast source? it can't by a blu-ray, perhaps an earlier generation version on tape?

ok, i'm done, it's just starting to piss me off that no one has ever created tools to easily clean up video or at least a nice easy to use avisynth gui that someone can just fire up and get the job done without having to jump through hoops. and why the hell are there so many de-interlacing methods? why have a dozen or more methods when only one or two actually produce good results?

Those cameras that go straight to VC-1 are relatively new and limited to cuts editing since VC-1 and h.264 are not optimal for recode. A better example is a new class of camcorders that shoot direct to Apple's ProRes 4:2:2 digital intermediate format on hard disk for Final Cut Pro. Earlier camcorders shot to AVID's digital intermediate. These bypass typical DVCProHD, HDV/XDCAM or HDCAM compression.

At the digital cinema end, it has been typical to shoot 1920x1080 RGB HDCAM-SR at 440 or 880 Mb/s data rates. Compare that to 20-35 Mb/s for Blu-Ray. HDCAM-SR is usually loaded over uncompressed SDI at 2.7 Gb/s data rates to huge RAID servers for workstation editing.

These ProRes422 cams shoot to about 220 Mb/s so can be handled on smaller RAIDs and MacPro workstations.

Red cameras can also capture to digital intermediates.

Other workflows such as film transfer use uncompressed data for uncompressed editing or conversion to digital intermediates (e.g. Cineform, AVID or other).

Compression is most effeicient directly from these low compressed masters.

[poisondeathray]

edit: i just tried that gpu accelerated msu denoise vdub filter, now we're talking!!! now, what did we say was the finest vdub de-interlacing filter? is there a gpu accelerated one?

i think i have come up with a sweet workflow, let me know what you think: load vob into vdub mod, first filter will be a de-interlace filter, second will be msu, then i will use the gpu accelerated mpeg-2 encoder plug in for vdub, gmpeg2, mux everything into an mkv and done.

how does that sound?

vdub doesn't have any good deinterlacers. Newer ones come with yadif, which is fast and decent quality, but you still get the "marching ants" or jaggy artifacts. The only GPU deinterlacer I know of is purevideo deinterlacer , and you could access this by using DGNVTools (not free) and avisynth

If you're happy with that workflow, then great. But I think you could improve the deinterlacer. I have never used gmpeg2 so I can't comment. Also, why are you deinterlacing if you are planning for SD blu-ray? Or are you making these mkv's for another purpose?

[poisondeathray]

Those cameras that go straight to VC-1 are relatively new and limited to cuts editing since VC-1 and h.264 are not optimal for recode. A better example is a new class of camcorders that shoot direct to Apple's ProRes 4:2:2 digital intermediate format on hard disk for Final Cut Pro.

Out of curiosity , which cameras shoot VC-1 and ProRes directly? I'm aware of Ki-Pro for ProRes, but that's more of an add on device, not a camera, and not really that portable due to size.

@Deadrats - And even if you could shoot VC-1 directly and upload clips, it would be rare that it wouldn't be color corrected, edited etc.. Also cameras in that price rance you mentioned have lower quality smaller sensors, lower quality lenses etc... Hardware recording is usually fixed GOP 1-pass VBR, and less efficient at the same bitrate if done with software 2pass and dynamic GOP sizes and rate control. You would get better end quality from using a more expensive camera like the ones edDV mentiond, with better sensors, glass, then re-encoding with a 2pass VC-1 encode

[edDV]

Those cameras that go straight to VC-1 are relatively new and limited to cuts editing since VC-1 and h.264 are not optimal for recode. A better example is a new class of camcorders that shoot direct to Apple's ProRes 4:2:2 digital intermediate format on hard disk for Final Cut Pro.

Out of curiosity , which cameras shoot VC-1 and ProRes directly? I'm aware of Ki-Pro for ProRes, but that's more of an add on device, not a camera, and not really that portable due to size.

@Deadrats - And even if you could shoot VC-1 directly and upload clips, it would be rare that it wouldn't be color corrected, edited etc.. Also cameras in that price rance you mentioned have lower quality smaller sensors, lower quality lenses etc... Hardware recording is usually fixed GOP 1-pass VBR, and less efficient at the same bitrate if done with software 2pass and dynamic GOP sizes and rate control. You would get better end quality from using a more expensive camera like the ones edDV mentiond, with better sensors, glass, then re-encoding with a 2pass VC-1 encode

I don't think direct to VC-1 would be a good option for high end production since VC-1 is a heavy compression. It would be used more for live transmission of news or events or as an alternative to MPeg2. Ikegami did the first AVID cam some years ago. It had hard disk packs that could be editied from a PC directly. The trend is to cap to a digital intermediate format at the camera to reduce storage space vs. uncompressed and reduce conversion loss. This would be more for TV production than movies. TV usually pioneers these techniques then film takes it up. 4kx2k digital cinema is so big, some kind of compression is needed but they shoot short takes and have union rules that limit work flow.