VHS to DV to DVD, Macroblocks on final DVD

[hysteriah]

I'm trying to capture some VHS tapes but the final DVD compatible MPEG-2 file ends up with macroblocks. I really need some help to make a final file with deasent quality.

This is what I do:
- Playing VHS tape on my JVC HR-S7600 (with built in TBC)

- Capturing to computer using Canopus ADVC300 with Adobe Premiere.
So far so good. When watching the captured DV file the quality is absoluteley "perfect" *ironic*... as "perfect" as these old tapes can be, I guess?

- Converting the DV file to DVD compatible MPEG-2 using Canopus ProCoder 2
This step makes the file look terrible. The final MPEG-2 file is full of macroblocks at any settings Even with video bitrate at 7500KBPS and max bitrate at 8000KBPS (at 720x576 resolution) with 2-pass encoding at "Mastering Quality" the file ends up with lots of macroblocks. I've also tried the HalfD1 resolution (352x576) but this resolution makes the macroblocks even worse at normal bitrates. With HalfD1 resolution I must set the bitrate as high as 7500KBPS to get rid of all the macroblocks.

I've also tried other apps than ProCoder with even worse results. I've tried converting the DV file to DVD with Adobe Premiere, Pinnacle Studio, TMPGEnc and Ulead DVD MovieFactory. I get the best result with Canopus ProCoder.

I would prefer to use the HalfD1 resolution at normal bitrate (3000KBPS to 4000KBPS) so I can get up to 3hrs on a disc. Anyone that can help me with this?

I guess the macroblocks appear because the VHS tapes are in bad shape with some noise in the image. But I think it's kind of strange that it only appears in the final MPEG-2 file. In the DV file there's absoluteley no signs of macroblocks. Is there any filters I can add to avoid this problem? Or is the MPEG-2 format simply not a suitable format for archiving files like these? So what format should I use to get the best quality on these files that can be archived on discs?

Please help.

[edDV]

This shouldn't be happening at those bitrates. Could you post some frames showing the problem?

Something is wrong in the encoder settings. Did you try the Adobe MPeg encoder for comparison?

Are you certain the field order is set correctly? DV is bottom field first. Reversed field order will cause problems during motion that may look like blocking.


PS I see you've tried other encoders. It would help to post a "good" DV frame and the same frame capture after MPeg2 encoding.

[Soopafresh]

Would you post 10-20 seconds of your SOURCE DV file?

[hysteriah]

Hi guys... and thanks for trying to help me out here

Here are the files:

File 1 (kylie_dv.avi):
http://www.alp.no/test/kylie/kylie_dv.avi
7 seconds of the source file in DV quality

File 2 (kylie_dvd.mpg):
http://www.alp.no/test/kylie/kylie_dvd.mpg
The same sequence in HalfD1 MPEG-2 quality with video bitrate at 3500KBPS and max bitrate at 5000KBPS with master quality. Converted using Canopus ProCoder 2. Take a look at the second B/W scene of kylie in slow motion. Look carefully at kylies face when she turns her head in that scene. There you can see lots of macroblocks.

There's a lot of scenes like these on all my tapes that always ends up with lots of macroblocks after converting. I have to increese the bitrate to 7500KBPS at HalfD1 resolution to avoid all the macroblocks. That's simply TO MUCH... There must be a better way to acheive deasent quality?

PS: I know that this tape is a tv capture at absoluteley garbage quality. This is just one of the worst examples I could find right now

[jagabo]

With all that noise and partial field blended NTSC -> PAL interlaced conversion I don't think you're going get macroblock free results unless you go with really high bitrates.

[hysteriah]

Thanks jagabo

So what you're really telling me is that the MPEG-2 format is not suitable for archiving these tapes (unless I'm willing to go with verry high bitrate)? Isn't there any good filters for cleaning them up and avoid the macroblocks?

Or... is there any other formats than MPEG-2 that can archive clips like these with deasent quality (similar to the DV file) without oversizing the files?

I guess the BEST way is to go for the DV format archived on tapes... but DV tapes are expensive... so I really need a cheaper way to archive these clips on discs. Is there any format that makes it possible to archive 5 mins of video that takes up less than 200MB for 5 minute clips?

Is there any QuickTime format that may be an alternate? Or Windows Media?

Anu suggestions?

[edDV]

The DV file looks problematic to me. Was this originally NTSC? If so, it should be archived as NTSC.

Maybe I'm missing something. I loaded it into a Vegas 4x3 PAL DV project. It shows extreme deinterlace artifacting with double images on most frames. If this was film source NTSC, it didn't get inverse telecine before the PAL conversion.

Conversion techniques will improve but this conversion isn't worth saving. The MPeg encoder is getting garbage and doing what it can. The problem here is with the DV file.

A typical DV frame

[hysteriah]

edDV: As you can see, I've captured it from a TV broadcast (MTV) to VHS. I guess it was filmed and edited as NTSC video at 29.97 fps on BETA (or something like that) and converted to PAL when it was broadcasted in europe. I don't belive it has never been a progressive 24fps film source, but I don't know this for sure because I have no clue what was done to it before it was broadcasted

All I know is that I captured it to VHS straight from the broadcasted signal (PAL, Cable TV) in the early 90's using a terrible PAL VHS recorder. Now I've captured it straight to DV using my Canopus ADVC300. I have not deinterlaced anything, not either done any PAL/NTSC conversion, so if that's a problem, it must have been done before it was broadcasted.

OK, I know the quality is terrible... but when I watch the captured DV file on my TV (through firewire from my PC), I think it looks "ok". It looks exactley as good as the tape I captured it from and I'm quite satisfied with that. I don't notice any deinterlace artifacts when watching it on my TV. All I want is to be able to store it without noticeable quality loss compared to that DV file... but how?

[edDV]

edDV: As you can see, I've captured it from a TV broadcast (MTV) to VHS. I guess it was filmed and edited as NTSC video at 29.97 fps on BETA (or something like that) and converted to PAL when it was broadcasted in Europe. I don't believe it has never been a progressive 24fps film source, but I don't know this for sure because I have no clue what was done to it before it was broadcasted

All I know is that I captured it to VHS straight from the broadcasted signal (PAL, Cable TV) in the early 90's using a terrible PAL VHS recorder. Now I've captured it straight to DV using my Canopus ADVC300. I have not deinterlaced anything, not either done any PAL/NTSC conversion, so if that's a problem, it must have been done before it was broadcasted.

OK, I know the quality is terrible... but when I watch the captured DV file on my TV (through firewire from my PC), I think it looks "ok". It looks exactley as good as the tape I captured it from and I'm quite satisfied with that. I don't notice any deinterlace artifacts when watching it on my TV. All I want is to be able to store it without noticeable quality loss compared to that DV file... but how?

We are into crime scene investigation at this point. The most likely steps were 24p film to telecined NTSC 29.97, to a 90's Snell and Wilcox level NTSC to PAL standards converter to MTV PAL broadcast, then VHS recording and finally capture to DV. So the file is showing the damage from this gauntlet of conversion.

My point is the MPeg2 encoder is being given a hopeless task of squeezing 60% more compression out of this stream primarily by motion change compression. If the encoder motion analysis finds blocks of interlace artifacts and double frames changing, it will average those into block movements. If it gives up on motion compression, it still has to average 7500 Kb/s so it then blurs out the I frames until that rate is forced.

If you present the same encoder with a clean PAL VHS recording it should do much better. If you give it a PAL broadcast to DV capture, it should do much better at half the bitrate.

I'm just saying in a long winded way that I agree with jagabo. Further compression of that source means less picture quality.

[FulciLives]

Well I downloaded the sample clip entitled "kylie_dv.avi" and I did a conversion to MPEG-2 DVD spec using Half D1 resolution (352x576) at 5000kbps CBR using CCE SP and it looks fine to me ... considering the source material.

I uploaded my clip to the rapidshare.com website. The link is listed below:

http://rapidshare.com/files/56050494/kylie_dv.mpg

The file (called "kylie_dv.mpg") is only 4.96MB in size. Also I should point out I used 256kbps MP2 audio because I was "lazy" and didn't want to do the extra steps to make AC-3 audio. Now I would always recommend AC-3 audio as opposed to MP2 audio but we are more concerned about video quality here than audio quality so I kept it simple.

If you think it looks good I'll tell you the encoding settings I used etc. but really I didn't do anything all that special ... I didn't even use AviSynth because I was being "lazy" yet it looks good to me *shrug*

Please note that at a Video bitrate of 5000kbps with an audio bitrate of 256kbps that you can fit about 1 hour and 55 minutes per single layer DVD although if you want enough overhead for DVD authoring you might want to keep it at around 1 hour and 50 minutes to be safe.

- John "FulciLives" Coleman

[edDV]

A closer look at the highest motion frames shows the encoder doing a reasonably good job. I see only slight blurring in the MPeg file (right) vs. DV file (left). These are the files sent by hysteriah.


click on the frame to see full size


click on the frame to see full size

[hysteriah]

I think this is the same result as I get with Canopus ProCoder at 5000KBPS. I can still see macroblocks in some frames. I guess you have to increase the bitrate to about 7500KBPS to remove them completeley?

FulciLives: You mentioned Avisynth? What filters and settings would you use for this file? I'm not really into this Avisynth thing. But I found a easy guide for using a plugin called MipSmooth that was quite easy to setup. But I'm still not quite satisfied with the result... so I guess I need something else/better.

Does any of you guys have a URL to a good site with guides on how to use filters/noise reduction with AviSynth... some filters that may improve my clip... for newbes, please?

[FulciLives]

To me it boils down to this:

You have crap ass footage. I mean really horrible looking stuff.

Compared to how it looks (the DV AVI file) I think the CCE SP MPEG-2 I did looks pretty good.

I did compare some frames of your MPEG-2 and my MPEG-2 and I see less macroblocking with mine but like I said I did nothing special ... I just think that (at least in this case) that CCE SP is superior to Procoder although the method CCE SP uses to resize could have had something to do with it I suppose.

In short I don't think you can do a whole hell of a lot better with this footage. You are being way too critical.

- John "FulciLives" Coleman

Some Examples [Procoder then CCE SP]

Procoder:


CCE SP:



Procoder:


CCE SP:

[hysteriah]

FulciLives: Well... It would've surprised me if the CCE didn't looks better than procoder now... because they're not really compareable... because the CCE file is 5000kbps while the Procoder file is only 3500kbps If I go up to 5000kbps in procoder I guess I will get the same result...

But I still hope it would be possible to avoid macroblocks completeley with a bitrate as high as that...

[edDV]

There are filters that can remove the interlacing and double frame artifacts but they leave more noise such as the Photoshop deinterlace filter. DV in (left) and Photoshop Deinterlace Filter out (right). These tend to work better for still captures since they may introduce jerkyness in 1x playback. These and other tricks are used for serious frame by frame restoration work.





This filter analyzes the offset field fragments and removes lines from one field and then fills the hole by interpolating from surrounding pixels.

[hysteriah]

edDV: Does this mean I have to deinterlace the video? I find it hard to belive that this filter alone will remove macroblocks in the final MPEG... because there's the same amount of noise in the frame...

[edDV]

edDV: Does this mean I have to deinterlace the video? I find it hard to belive that this filter alone will remove macroblocks in the final MPEG... because there's the same amount of noise in the frame...

The interlace issue happened way back in time when it was NTSC (MTV problem). They should have inverse telecined before conversion from 640x480/29.97 to 768x576/25 so those lines were in the picture. The still that I gave to the Photoshop filter was a progressive image. It detected the lines and motion offsets, then interpolated new pixels.

I don't see where it removed macroblocks, it just detected and removed the offset double image. I see more noise in the processed frame.

If you step frame by frame through this clip you will see odd patterns of progressive and time offset interlace frames. This must have been the result of standards conversion and would be difficult to reverse.

[hysteriah]

edDV: There is no macroblocks in the DV file so it could not possibly have removed any

I really don't understand what you are talking about. I'm sure you're right that there's odd patterns of progressive and time offset interlace frames... but I don't see the point in reversing that. Simply because it doesn't anoy me

The only thing that really anoys me is the macroblocks in the final MPEG... so what's the reasons why they appear... and is it possible to remove them or make them become less visible by filtering the video in any way? I really don't care if the image gets softer or blury or whatever if that's what it takes to avoid the macroblocks in the final MPG...

[edDV]

All you can do is trial and error with encoder bit rate and full size vs. 352x576.

Start with highest bit rate and move down until unsatisfactory.

[vhelp]

Good morning every one.

I started to replay (below) last night before my dial-up hung up on me due to no
activity, (I was typing) and I gave up in discuss and called it a night.

---

This is the usual scenario of archived vhs video's, waiting to be transfered
to its new format, mpeg, but has many problems ahead.

Music video's are the worse to work with

This is the tipicle case of the music video Butcher Editing, and in this video's
regional case, Region Conversion Butchering.

What I'm saying here is that if you are familiar with the NTSC editing after
the 24p -> Telecine[29.970 fps] for music video's (as well as other usual
videos like this) then you know how poor looking and imposible to revise or
IVTC the telecine back to 24p when these are originally film origin.

The problem is compounded when the source video is from another region and
is converted to that region's format. In this case, (assuming that the video
is NTSC origin) then we could be talking NTSC -> PAL (frame rate) conversion,
which seems to be an additional issue with your problem of encoding to mpeg.

To transfer these recorded to vhs video's to MPEG presents a challenge, though
a messy one. But, to keep the video playing smoothly, you would have to encode
them as they are, which in your (regional case) is PAL, 25 fps.

Its one thing to deal with such sources when its an telecine within the region.
But, when the source is from another region which may have different format specs,
this can present a problem with the transfer to mpeg process, because you have
to deal with, edits; cuts; telecine; frame rate conversions (ie, ntsc -> pal, or
pal -> ntsc) field blends; scene changes; and so on. Most of the music videos
out today are 24p -> telecine. But it becomes a problem when those who own the
source at the time do further editing to the now telecine videos. This is usually
the case, and then becomes 24p -> telecine -> editing -> mpeg. You see,
you never want to do any editing/cutting of video on a telecine source. You only
want to do this on a progressive source, like a 24p film source. Then, after you've
edited the 24p film source, you run it through the telecine process for the region
it will be intended for.

In the example demo source you uploaded, your source is mostly interlace. It is
from an Telecine and Region Converted video. The reason I say region converted
is because there are noticable Field Blends, which is usually found in NTSC->PAL or
PAL->NTSC frame rate conversions. And part of your reason for the noticable
macroblocks (pixelations) is because the majority of the video is interlaced and
your mpeg encoder and encoding process put them there. So, unfortunately, it is
partly your fault they got there in the first place.

---

Today ...

The video you presented has enough nonsense to fill a whole page worth of how-to
and why-not-to and so on and so forth. I mean, describing the blended frames and
messy telecine and region conversion and so on, that you could spend the day or
two, sweating it out to make it (back to its original) progressive again, if you could,
though with this (MTV) video, I truely believe it is impossible. In any case..

Now, to try and reverse the [s:ed20c9652d]damage[/s:ed20c9652d] video's circus of interlacing would require the
popular AVIsynth and its many filter plug-in functions. But, even with this direction,
it could prove worthless because just in this video, its a mess. And, if you have a
bunch of these video's, it would require a skillfull person at the helm, and one with
great patiance and drive/motivation to make it all work, flawlessly for all of them.
And, even if someone were to tell you .. yeah, just type this script in and run it
through all the videos .. will not be enough -- for the chances are that your vidoes
will have various turning points that will most likely require manual adjustments that
only a skillfull user at the helm might be able to fix most of them, given his/her
energy levels and purpose, etc.

fwiw.. for marginally best results, I would suggest keeping the videos as they are,
interlaced and 25 fps frame rate and encoding them that way, but with a
high enough bitrate and prob less "minutes" per disc.

I too put minor effort into making an attempt to just de-interlace it for XviD or H264.
But the clip was too messy for me to want to put the effort into trying to make it work.
I found it to be better leaving it at interlace and 25 fps frame rate for dvd mpeg.
However, for xvid/h264, de-interlacing would not be such a bad idea if you can find
a good method that yields good results.

I would like to make a few more suggestions ...

Consider evaluating your vcr settings. You see, the video you posted left me with
the conclusion that your vcr had applied some "softening" filter to the video. And
if you want to clean them up to make them "better" looking, then IMHO, you would
want to keep all the detail that are allready on the tape and capture it as it is, and
later on, trial n error your filter applications on them before archiving them to the
MPEG format on disc. There is no point in double-filtering a video in some if not,
most cases. That has been my experience, so far.

-vhelp 4399

[hysteriah]

Hi again... and once again thank you verry much for trying to help me out her I must say that I'm really impressed by all your knowledge, guys

vhelp: I must say that you are verry observant when you say that my vcr had applied some "softening" filter... well, you're right about the softening... but it's not my vcr. I guess it's because I made a minor adjustment to the sharpness slider on my ADVC300 I adjusted it to avoid some of the noise in the contours that I found a bit anoying... But I guess you want me to adjust it back and capture it again? Will not this make the picture even noisier and make the macroblocks even worse in the final MPG?

Ok... so you guys have convinced me that I just have to accept the macroblocks in my videos... But there's still one thing that I wonder:

I also have a Hauppauge PVR 150 capture card (MPEG hardware)... and I'm not sure If I've ever tried it with this exact clip of kylie... but my overall impression is that when capturing with that card makes me able to go a lot lower on the bitrate. I've captured some VHS tapes with that card and always at 3000kbps videobitrate on the HalfD1 resolution... some of the clips that I've captured with that card looks even worse than this one from kylie... but I have never noticed macroblocks so anoying as this with the Hauppauge card. I guess the 3000kbps clips from the hauppauge card is comparable to the 4500-5000kbps clips with the ADVC300. I guess it's because the hauppauge card applys some noise filtering to the video (Temporal and Spatial noise reduction filter). The noise filter is a bit anoying in some scenes and the Hauppauge card also suffers from other errors that makes me don't wanna use it... But this makes me think that there must be a way to filter the video in one way or another to achieve deasent quality and avoid the macroblocks. But I guess you guys have convinced me now that it's best to just live with them from now on But maybe some of you can explain for me why they are not so anoying with my hauppauge card?

Anyway... once again, THANK YOU VERRY MUCH for all your wonderfull help here

[edDV]

The PVR-150 can capture higher bitrates than that but this is near the worst case source for real time MPeg2 capture. Try it and see.

The premium non-real time encoders are doing fairly well with this source. Let's see what Conexant can do and compare.

[FulciLives]

Try a capture using the Hauppauge and set it at Half D1 with a CBR of 5000kbps and see how it compares. Usually going higher than 5000kbps with Half D1 yields no better quality but you probably found one of the worst clips around ... one that really "tests" the abilities of an encoder or perhaps the MPEG-2 format itself.

- John "FulciLives" Coleman

[hysteriah]

Hehe... I tried capturing this same clip with my PVR150 now... and yes, you're perfectley right... It really can't compare with the ADVC300 procoder or CCE files in any way The 5000kbps clip with PVR-150 is even worse and has more macroblocks than the 3500kbps advc300/procoder file I just had the feeling that I've never seen so anoying macroblocks before on anything captured with the PVR-150 card.

So... I guess that's because I've found one of the worst piece of crap video ever to work with right now... I just get a feeling that I do have quite many of those "worst videos ever" in my collection of old VHS tapes *grrrrrr* Wish me luck... hehe... :P

Thanks again

[jagabo]

If you want to spend money and a lot of processing time you can try Neat Video's noise reduction filter for VirtualDub. They have a free demo.

http://www.neatvideo.com/

Before and after (I only have the demo which forced me to reduce the frame size to 640x480):

[hysteriah]

Wow, jagabo

It looks like it does wonders to that sample pic... and the best of it all is that it also excist as plupin for Premiere :P I guess I have to check it out and see if it will avoid all the macroblocks in the final mpeg

Thank you verry much

[jagabo]

I also suggest you mask all that junk and in the overscan region with black bars. There's no useful picture there so there's no point in letting it consume bitrate.

[FulciLives]

I was able to gain access to Neat Video 2.0 (2.1 is the newest version) and I applied it to the clip then encoded it to a MPEG-2 DVD spec file using CCE SP.

I should point out that I didn't notice this before but this clip appears to be 16x9 WS ... the aspect ratio looks more proper that way than 4:3 Full Screen so I encoded it at 720x576 16x9 WS at a CBR of 7500kbps with 256kbps MP2 Stereo audio sound.

Here is the link: http://rapidshare.com/files/56473983/kylie_neat_video.avi.mpg

Whatcha think?

- John "FulciLives" Coleman

Here are some sample still images:

Original AVI file:


Final MPEG-2 DVD spec file:



Original AVI file:


Final MPEG-2 DVD spec file:

[hysteriah]

Thank you verry much, FulciLives

That plugin really looks like a MUST HAVE for VHS cleaning. It's verry slow and CPU demanding, but on important footage it's absolutley worth the wait, and simply a MUST. I had no idea that a old VHS tape could ever look that good :P

I see on Neat Video that there's a Home version that only supports 8-bit video data, and a Pro version that supports both 8-bit and 16-bit. I must admit that I don't have a clue what that means. Anybody who knows what the difference will be? Will I ever need the 16-bit version for VHS cleaning on AVI DV files?

[Soopafresh]

Nice results, Fulcilives!

Did you deinterlace first before applying the filter?