Need help converting/encoding a DAT file for better quality

[coxanhvn]

Hi, I have a DAT file from an old VCD and I would like to convert it to MP4 (or any common format) with some adjustments from the software that could make the video looks better. I don't know if this is possible but I've seen people doing that and "encoding" is what they call. Could anyone please suggest me any software that could do it and what are the settings in the software that I can play around to get the best quality? Thank you very much in advance.

[Wolfen]

This will help you out for anything VCD it's older works on XP machines and it's FREE http://vcdgear.com/ Use this to convert the dat to an mpg and just use any program you want for editing etc.

[coxanhvn]

Hi Wolfen, "just use any program you want for editing," this is the exact stage that I am stuck at.

I've just read a few posts from this forum and it looks like it does require some "script" processing which I complelely have no idea how to do it. Now I'm asking if anyone can please give this a try and help me making it look better, I will pay (via PayPal) in return.

Below is my source video (A) and the second one (B) is an "encoded" version by someone (using a different source which has a "scratch" scene and they couldn't fix). My wish is to encode video A or apply any script or settings to it so that the result will look at least like video B or better.

Video A (Original): https://mega.nz/#!64UmxDhS!LSV5KzRvxHrgFvo19-FCNmgU1hGDilbFpe7yD25qd-k

Video B (Encoded): https://mega.nz/#!SlsHVY4Q!B442VDPO0KlwDDo7I2jI1UW8Acmk8ar1XQ_ycDM0aj4

Thank you very much in advance,

[manono]

My wish is to encode video A or apply any script or settings to it so that the result will look at least like video B or better.

You can forget about that because, as you said, theirs is from a different (and probably hi-def) source. Yours is from a crappy VCD source. Yes, it can probably be improved but you're living in a dream world if you think it can be made to look as good as (or better than!) the MP4 (which itself looks pretty bad).

Can't you just keep the second one and call the project finished? Or did you want the 'scratch' part at 3:18 fixed? It can be fixed fairly well with frame interpolation.

[jagabo]

This AviSynth script makes A look pretty close to B:

Mpeg2Source("videoA_original.d2v", CPU2="xxxxxx", Info=3) 
Crop(0,34,-0,-30)
ColorYUV(gain_y=15, off_y=-8, gamma_y=75, cont_u=-60, cont_v=-0)
TemporalDegrain()
nnedi3_rpow2(4, cshift="Spline36Resize", fwidth=1280, fheight=720)

[poisondeathray]

This is one of those rare instances where mpeg2source has decoding issue - or maybe it's my dgdecode.dll version ? Can you guys check around frame 2757-2770 (~1:32), there is some pixellation

The version of FFMS2 I'm using decodes that section correctly with rffmode=0, seekmode=0, but has problems with the first few frames in the intro sequence (which you can use dgdecode for)

But I agree with manono, there are severe macroblocking and deterioration in some frames in the VCD version. Might be better to "fix" the frames in version B with frames from version A

[manono]

MPEG2Source/DGDecode works here. There's no corruption at the place you mentioned. Did you convert the DAT to MPG first? When I don't do that I sometimes have the problem you describe.

[jagabo]

Yes, I saw problems around frame 2757+ with Mpeg2Source. I didn't remux to MPG, used the original DAT file.

A FixBlendIVTC() and TDecimate() will get rid of most of the blend deinterlace ghosting and restore the video to 23.976 fps. Though it leaves a few jerks here and there. Playing with the TDecimate() parameters might help.

[poisondeathray]

MPEG2Source/DGDecode works here. There's no corruption at the place you mentioned. Did you convert the DAT to MPG first? When I don't do that I sometimes have the problem you describe.

Thanks, that works - never knew about that one. I wonder why that would happen in the first place ? DG is "retired" now

[coxanhvn]

Hi all, thank you for all your replies and discussions, now I know there's a thing called "frame interpolation" as manono mentioned.

@manono it is my fault not to mention that the original B video was taken from a VCD too as I asked the uploader, what different are the video is retouched some time long ago by someone and it was "upscaled" to 720p. If you resize the window while playing it, it looks quite similar to video A except less "noise" and brighter. This is why I'm having the impossible dream as you said because I just simply think we can "smooth" out the pixels to make the whole frame looks "cleaner" and then... add some "light" to make it easier to look at. But anyway, can you try to fix the bad part from video B and let me see just that couple of seconds of video?

@jagabo thanks for sharing the codes which I believe "scripts"? can you please give me a sample result of just a few seconds?

@poisondeathray I used to try mix 2 videos together like you said but it didn't turn out good as I don't know how to fix it first, could you please try if you got time or suggest anyone else who seems to play with these a lot so I could ask for help?

Thank you all,

[jagabo]

Here's the video (no audio) after my earlier script. A little more noise reduction was applied to match the other clip better. This includes the glitch that poisondeathray noted in post #6.

[manono]

@manono it is my fault not to mention that the original B video was taken from a VCD too...

But not your NTSC VCD as his is 25fps. I mean, unless he knows lots of AviSynth tricks then it's not from the same source. But maybe I was mistaken in thinking it was from a much higher resolution source.

But anyway, can you try to fix the bad part from video B and let me see just that couple of seconds of video?

I'm including that scene where the corruption is. It's not perfect, but pretty decent, I think. Or you could do as pdr suggests and replace the bad parts of the MP4 with good parts of the VCD filtered to make it look like the MP4. Or use jagabo's one.

FFVideoSource("videoB_encoded.mp4")
ReplaceFramesMC(4968,13)
Trim(4934,4989)

[poisondeathray]

Also, if you were going to fix the "encoded" version, there's also a yellow splotch on the top right corner ~ 3:06. But that one is pretty easy to fix

[Selur]

I wonder why that would happen in the first place ? DG is "retired" now

more like: only available through http://rationalqm.us/board/index.php

[coxanhvn]

Thanks poisondeathray, I did notice that but didn't want to mention in the first place, however can you please show me the steps to fix it since it seems not too hard?

@manono OMG, if I knew you could get it looks so good like this, I wouldn't bother asking anything about the A video, please check you inbox before reencoding the whole thing and by the way, do you think the frames or pictures from 1:27 to 1:28 could be fixed also because it looks like there's a problem there?

@jagabo thanks a lot for uploading the whole video (I didn't know we could attach such a big file in here?), the video is exactly what I have pictured in my head when I started this thread to ask for help, although the picture color doesn't look as natural as B, the noise reduction is perfect! Please check your inbox and could you please show me the tools and steps to do it so I could adjust the color myself? It looks like I have to start learning AviSynth from now on.

To all, these 2 videos are surely taken from 2 different sources, however I noticed these 3 frame rates 23, 25 and 29 were mentioned in this thread. Is there any way you could tell which is the original setting of any video when it's first "produced"? What happen when you reencode and change it to a different setting? Any disadvantage or advantage in quality gain?

[manono]

I wouldn't bother asking anything about the A video, please check you inbox before reencoding the whole thing...

I checked the inbox. Thanks, but I'm not reencoding the entire video. That short section was just a 'proof of concept'. I have other projects of my own requiring my attention.

...do you think the frames or pictures from 1:27 to 1:28 could be fixed also because it looks like there's a problem there?

I didn't really see much except for a sort of bright 'flash' which could also easily be fixed by interpolating from the frames just before and after.

...however I noticed these 3 frame rates 23, 25 and 29 were mentioned in this thread. Is there any way you could tell which is the original setting of any video when it's first "produced"? What happen when you reencode and change it to a different setting? Any disadvantage or advantage in quality gain?

That can be difficult to figure out sometimes. It can help to know where the group is from which might provide a clue as to whether the 'original' source was NTSC or PAL. As for it being film or video (23.976 or 29.97fps), with this one I don't know although maybe some of the others here have already decided. Yes, there's always an advantage to getting the framerate (and often the audio pitch) correct.

[jagabo]

I only saw one bad frame around 1:27 in videoB, frame 2191. That's easily replaced using the same method manono used earlier, ReplaceFramesMC(2191,1). That function replaces one or more frames by interpolating motion between the frames that bracket the sequence. So in my example 1 frame is replaced, frame 2191, using frames 2190 and 2192 as the references. This technique works when motions aren't too complex or too large. But it doesn't work well with the smudge that appears at 3:06. The motions there are too complex. The generated frames end up with weird artifacts. I'd use an alpha mask and only replace the part of the picture where the smudge appears.

Regarding colors, I wasn't shooting for natural colors. I was trying to make videoA look like videoB. It's very hard to get one video to look exactly like another I just tried to get in the ballpark with:

ColorYUV(gain_y=15, off_y=-8, gamma_y=75, cont_u=-60, cont_v=-0)

You can play with all the arguments (there are a few others) to get whatever brightness, contrast, and colors you want. There are several other methods of dealing with colors in AviSynth, Levels(), Tweak(), RGBAdjust(), etc. There's even a filter that attempts to automatically make the colors of one video similar to the colors of another video, ColourLike(). You'll also want to look at tools like Histogram() and VideoScope() to analyze the videos.

Regarding frame rates: the video was shot on film at 24 fps. For NTSC film is normally slowed to 23.976 fps and 3:2 pulldown applied to make 29.97 fps interlaced. For PAL it's often sped up to 25 fps progressive. That's where all those number some from.

To get started with AviSynth I suggest you download and install 32 bit AviSynth (there is a 64 bit version but it's not as stable and many third party filters are not available in 64 bit versions.) To run the functions mentioned in this thread you'll need several other third part filters like MVTools2, ReplaceFramesMC(), VideoScope(), the ffmpeg source filter, DgMpegDec, etc. I create and edit scripts with Notepad. But some people like to use AvsPMod. I use VirtualDub to view the results of my scripts before encoding (usually with the x264 CLI encoder).

http://avisynth.nl/index.php/Getting_started

Oh, the heavier filter I did was:

TemporalDegrain(SAD1=600, SAD2=450)

Instead of the defaults (SAD1=400, SAD2=300) in my earlier script.

[poisondeathray]

Thanks poisondeathray, I did notice that but didn't want to mention in the first place, however can you please show me the steps to fix it since it seems not too hard?

jagabo addressed it - you cover it up with a mask, so you only affect that tiny background area, not the singers or foreground. you can use the interpolation method suggested earlier, or if you use a static frame "patch" - here is some camera movement so you need a slight animation if you use a static frame, or and apply . It sounds very difficult but it's actually easy to do in a video editor or compositor.

If you were going to "fix" the encoded video B , instead of taking jagabo's approach, I noticed a few other glitches
~00:00:10 , frame 252 there is a small "block" glitch on the mountain
~00:01:02 , frame 1582 there is a tape dropout near the mountains

Those are easy to fix with any of the methods too

[poisondeathray]

And the audio sucks on both those versions. It sounds like listening with a tin can

I suspect you are a fan and have their CD already, but there are better audio versions on youtube you can replace it with, including some "official" release versions so that shouldn't be a problem with copyright etc...

[jagabo]

You use an "alpha mask" to blend areas of 2 different videos together.



Where the image is white pixels come from the first video, where the image is black pixels come from the second video. Shades of grey in between are a weighted blend of the two.

amask = ImageSource("amask.png").ConvertToYV12(matrix="PC.601")
source = ffvideosource("videoB_encoded.mp4") 
patch = source.ReplaceFramesMC(4653,14)
Overlay(source, patch, mask=amask)

In this script the image is loaded and called amask. The source video loaded and a copy of it is made (patch) with the problematic frames motion interploted with ReplaceFramesMC. Then Overlay is used to merge frames of the two videos.

Attached is a short video that shows a bit of the original video with the glitch, followed by the same segment after the fix. My mask wasn't quite perfect -- if you look closely you'll see a little of the green blotch still shows up in a few frames.

[jagabo]

And here's why you don't want to use motion interpolation for that entire segment.

[manono]

Or you can interpolate just up around the green rectangle without having to go through the trouble of making a mask. It's by no means perfect but also avoids the artifacts you get on the girl's face from having to interpolate the complete frames for all frames.

FFVideoSource("videoB_encoded.mp4")

A=Last
B=ReplaceFramesMC(4653,14)
B=B.Crop(1052,0,0,-684)
Overlay(A,B,1052,0)
Trim(4607,4684)

Edit: But jagabo's is better. What was done is less noticeable, although I don't think anyone just watching the video would notice.

[coxanhvn]

Wow thanks a lot guys for all the great info, and please excuse me because I will definitely need some time "processing" before posting any further comments.