Joining, deinterlacing, and encoding MOD source files to progressive h264

[nicholasserra]

Hey everyone, thanks for looking. I have some questions regarding proper process for a video i'm trying to create.

For a long time now i've been encoding my videos in h264 progressive via Premiere and i've been okay with the results. But now i've stumbled down the rabbit hole of correctly deinterlacing and doubling frame rate for progressive videos. I've recently learned that Premiere is terrible with deinterlacing and basically throws away half the content.

So I have several source files in the MOD format from a JVC hard drive camcorder that I want to join together into one video, deinterlace, and export as progressive h264 for youtube upload.

The files are interlaced, 29.97frames, NTSC. Here's a sample: https://s3.amazonaws.com/public-nicholasserra/original.MOD

I'm guessing i'll need to be using avisynth for this process.

My questions:

- Premiere can't even read these MOD files. Even after a straight container convert via FFMPEG to mpeg2. What should I be using for simple video joining? Could this be part of an avisynth script? Does avisynth support this?
- What is the field order of these MOD files? I assume I need to know this for my script?
- If I deinterlace this footage is the resulting output now 59.94?


So basically asking what is the best way to take these MOD files, join them, deinterlace, and output progressive.

Thank you!

[Budman1]

Try changing extension to mpg. http://www.makeuseof.com/tag/convert-mod-camcorder-video-format-mpg-instantly/

[nicholasserra]

Try changing extension to mpg. http://www.makeuseof.com/tag/convert-mod-camcorder-video-format-mpg-instantly/

I went that route without success in the beginning. Still wasn't able to import into premiere. But now that I know premiere won't do correct deinterlacing, i've abandoned the idea of using it at all. The mpg rename may get me a good starting point though. Thanks!

[jagabo]

You can join your clips with DgIndex (part of the DgMpgDec package). Save the resulting index file (d2v). This will also export the audio as an AC3 file. Use Mpeg2Source() in AviSynth to open the index file. Apply the QTGMC() filter to deinterlace to 59.94 fps.

Mpeg2Source("original.d2v", CPU2="ooooxx", Info=3) #reduce DCT ringing, flag field order
QTGMC()

Add any other filtering you want to the script (levels, colors, noise). Use whatever encoder you want to encode the video (I use the x264 command line encoder). Mux the AC3 audio with the new video.

[nicholasserra]

You can join your clips with DgIndex (part of the DgMpgDec package). Save the resulting index file (d2v). This will also export the audio as an AC3 file. Use Mpeg2Source() in AviSynth to open the index file. Apply the QTGMC() filter to deinterlace to 59.94 fps.

Mpeg2Source("original.d2v", CPU2="ooooxx", Info=3) #reduce DCT ringing, flag field order
QTGMC()

Add any other filtering you want to the script (levels, colors, noise). Use whatever encoder you want to encode the video (I use the x264 command line encoder). Mux the AC3 audio with the new video.

Awesome thank you!

Can you elaborate on the steps after the avisynth stuff? I've never actually used avisynth, what will it output for me? Assuming I use that output for the x264 CLI encode. Does the muxing of the audio occur during the x264 encode or is that joined afterwards with something else?

[jagabo]

I usually use a batch file to encode AviSynth scripts. For the sample I drag/dropped the AVS script onto this batch file:

start /b /low "x264" x264.exe --preset=slow --crf=18 --keyint=50 --sar=32:27 --colormatrix=smpte170m --output %1.mkv %1

x264.exe only handles the video. So I used MkvToolNix to mux that video along with the AC3 audio.

[nicholasserra]

Thanks, i'll give that process a try today and see where I get

[jagabo]

Depending on where x264.exe is on your computer you may need to supply the full path name with quotes:

start /b /low "x264" "C:\program files (x86)\x264\x264.exe" --preset=slow --crf=18 --keyint=50 --sar=32:27 --colormatrix=smpte170m --output %1.mkv %1

Note that the start /b /low "x264" is there to start the program a low priority so it doesn't bog down the system.

[_Al_]

...............   --output %1.mkv %1

I'd recomend using:

...............   --output "%~1.mkv"  "%~1"

instead,
Using of %1 sometimes does not work if there are spaces in filenames, which is quite common. That last %1 as input would work but in case of that output, Windows would put quotes in the middle of filename, not sure how that would turn out

[jagabo]

Yes, the behavior of %1 and %1.mkv will vary depending on which version of Windows and which programs you are using. It's safer to use the %~1 syntax with quotes.

[nicholasserra]

Wow, getting avisynth installed and all the plugins working is rough. I finally got past all the verbose errors. Now when I run the script in windows media player I get a "Windows media player encountered an error". Vdub gives me "Mpeg2Source not found" when I run via File->Run Script (is that right?).

I got it working via QuEnc, but it feels weird to me that I need to fire up an mpeg2 encoding program to run this. And i'm also presented with quality options which I don't like, because I assumed this wasn't doing an encode yet?

[nicholasserra]

Oh, nevermind. Now understanding that you're using x264 encoder to run avisynth. Now debugging errors there. So close :P

[jagabo]

With VirtualDub you open the AVS script as if was a video file with File -> Open Video File. Or you can drag/drop the script onto the VirtualDub window. File -> Run Script is for VirtualDub's own scripting language.

[_Al_]

Also you can use portable MPC-HC player, dropping your script on desktop shortcut would start to play that script. I think it is set up like that as a default. I use it a lot. It posts error on screen as well, if there is.

[nicholasserra]

So finally got everything encoded and joined together properly. Video looks great, correct frame rate. Only issue is that the audio has somehow drifted Starts off spot on, but at about 20 minutes it's probably a second off. Any ideas on how that could have happened?

[jagabo]

It's quite common for audio sync to be off when joining multiple files. In a video file the audio doesn't have to start or end at the same time as the video, nor does it have to be the same length as the video.

Is there just one shift in the audio? Or does it gradually get worse?

[nicholasserra]

Seems to get worse as it goes along. I would expect that DgIndex would output those demuxed audio video files in sync. Maybe i'll toss those together to try and pinpoint which step is causing the issue.

[poisondeathray]

I've recently learned that Premiere is terrible with deinterlacing and basically throws away half the content.

Not entirely true; Yes, PP's deinterlacing quality is not very good , but you can double rate deinterlace if you tell it to. (e.g. from a 720x480i29.97 timeline if you export 59.94p , nothing will be "thrown out" in terms of time sampling)

- Premiere can't even read these MOD files.

The mod file opens up in PP for me natively; what version of PP are you using ?





For sync, preview your avs script without qtgmc in mpchc to check for sync first. This will tell you if there was an issue with DGIndex joining. Add the audio to your script with NicAC3Source or FFAudioSource and AudioDub() . If there is a delay value in the demuxed audio name, add that value with DelayAudio. (the delayaudio cannot account for a progressive sync discrepancy, it's just for completeness. Progressive sync means A/V lenght mismatch)


It might look something like this

vid=MPEG2Source("blah.d2v")
aud=NicAC3Source("blah.ac3")
AudioDub(vid,aud)
#DelayAudio(somevalue)

[nicholasserra]

I'm running CS6. Maybe time to upgrade :P If the newer versions have acceptable deinterlacing that would save me a lot of time.

I'll give those sync troubleshooting steps a try. Thanks!

And again thank you everyone who has replied. This has been very educational at the least

[_Al_]

That clip you posted has a silence at the end in audio track. That might have something to do with it, if all clips are like that.

I would support that idea to rather work with non-linear editing software. Premiere or others.

fyi, Magix Vegas , old versions 8.0c or Vegas 10.0( so at that time Sony Vegas) loaded your example, but interpreted footage as 4:3, it needed to be corrected to 16:9 within Vegas (right click a clip/media/Pixel Aspect ratio change to widescreen DV), which would be tedious to fix in every clip (maybe there is a script for that), so testing updated Premiere would be a good idea.

Or batch change those mpg into lossless AVI instead, you have avisynth script working so it is not far fetched idea. Not joining them. Then load those AVI clips into Premiere.

[poisondeathray]

I'm running CS6. Maybe time to upgrade :P If the newer versions have acceptable deinterlacing that would save me a lot of time.

"acceptable" or not is a matter of opinion, but it's the same in terms of quality as older versions. It's a basic resize of fields; so there is no interpolation or filling in the missing spatial information. You're left with "jaggies" or aliasing artifacts , but in motion, usually its not that bad for most people


You should be able to batch re-wrap into mpeg2 transport stream to "convince" CS6 to import them . You might have to interpret the AR in PP to 16:9

Try 1 first

ffmpeg -i input.mod -c:v copy -c:a copy -f mpeg2ts output.m2ts

[nicholasserra]

Followed poisondeathray's advice and made a new script to check sync of the mpeg2 joining. Audio was still out of sync towards the end in that test, so that confirms that the issue occurred in DgIndex while joining. Maybe _Al_ is onto something with the silence at the end of the file.

Gonna start googling DGIndex audio sync issues and see if I can pinpoint which clip may have caused the audio shift.

In regards to the premiere stuff, I have been using premiere to output progressive for years, and it was always good enough for my eye. But seeing these avisynth deinterlace jobs with double frame rate got me down this path, they just look so good.

I did some comparison rendering with outputting 29.97 vs 59 and a friend told me that the 59frame file from premiere was simply double frames, so CS6 apparently isn't doing the "double rate deinterlace" in any meaningful way. Those files are here if anyone is bored and curious https://www.dropbox.com/sh/q1w3cfead6kwmvf/AAD6KRTO-bVw9HFgFyz9BI5-a?dl=0

[nicholasserra]

So the issue is for sure with one of the joined clips. It goes from spot on during one clip finishing to off starting the next clip.

[nicholasserra]

Went another route and merged all the videos together with VOBMerge and then just demuxed the one big file in DGindex after. Same result. So DGIndex is giving me sync issues even when using one file. Gonna find another demux application.

Now just thinking out loud for anyone who might google this ha.

[jagabo]

You might try trimming the one source clip with a tool like Mpg2Cut2 then rebuilding full video. Or, if it's just one discontinuity, adjust it with an audio editor (audacity?), and remux with the video.

[poisondeathray]

I did some comparison rendering with outputting 29.97 vs 59 and a friend told me that the 59frame file from premiere was simply double frames, so CS6 apparently isn't doing the "double rate deinterlace" in any meaningful way. Those files are here if anyone is bored and curious https://www.dropbox.com/sh/q1w3cfead6kwmvf/AAD6KRTO-bVw9HFgFyz9BI5-a?dl=0

Yes, there is a problem with your 59.94 clip , but it's more likely you didn't enter the settings correctly. Older versions of PP work the same way since at least CS3 . The sequence settings have to be set correctly as 29.97i and the export settings have to be set to 59.94p , then it will apply the double rate deinterlace

This is what it should look like from PP , no other filtering

(You could do other stuff like noise removal , chroma noise removal, adjust levels etc...., either in PP or in avisynth)

[nicholasserra]

I did some comparison rendering with outputting 29.97 vs 59 and a friend told me that the 59frame file from premiere was simply double frames, so CS6 apparently isn't doing the "double rate deinterlace" in any meaningful way. Those files are here if anyone is bored and curious https://www.dropbox.com/sh/q1w3cfead6kwmvf/AAD6KRTO-bVw9HFgFyz9BI5-a?dl=0

Yes, there is a problem with your 59.94 clip , but it's more likely you didn't enter the settings correctly. Older versions of PP work the same way since at least CS3 . The sequence settings have to be set correctly as 29.97i and the export settings have to be set to 59.94p , then it will apply the double rate deinterlace

This is what it should look like from PP , no other filtering

(You could do other stuff like noise removal , chroma noise removal, adjust levels etc...., either in PP or in avisynth)

Probably related to the fact that I couldn't even get the MOD (or renamed mpg) files into premiere in the first place. Still can't import them on mac or PC straight up. I'll try some of those conversions mentioned above when I'm done banging my head against the wall on this demux issue ha

[poisondeathray]

Premiere is usually pretty consistent with transport streams

This is the ffmpeg rewrap from above. Try importing it. If it works you can batch re-wrap , say all the files in a folder, pretty easily with ffmpeg

It's larger in filesize, because transport streams (ts, m2ts, mts) have larger overhead than program streams (mpg, mpeg) . But otherwise it's just a stream copy

In CC, the AR is incorrect and needs to be interpreted. (The native MOD needs to be interpreted for the AR too)

[nicholasserra]

Premiere will not accept an mpeg2 transport stream either. I'm beginning to wonder if I'm missing some mpeg2 libraries. This output would not import:

ffmpeg -i input.mod -c:v copy -c:a copy -f mpegts output.m2ts

I'm now going to install a new version of premiere.

[nicholasserra]

Premiere is usually pretty consistent with transport streams

This is the ffmpeg rewrap from above. Try importing it. If it works you can batch re-wrap , say all the files in a folder, pretty easily with ffmpeg

It's larger in filesize, because transport streams (ts, m2ts, mts) have larger overhead than program streams (mpg, mpeg) . But otherwise it's just a stream copy

In CC, the AR is incorrect and needs to be interpreted. (The native MOD needs to be interpreted for the AR too)

No go on yours or mine. Thanks for generating this though. Gonna upgrade and see where I get.