Batch processing AVI to MP4 (H.264)

[Selur]

Deinterlacing always will cost some quality.
If you playback equipment properly handles interlaced material, it should not get smoother, produce 'juny' edges and stuff, or the material is a wild mix of interlaced and progressive material, leave it as it is.
Otherwise:
For interlaced PAL Material deinterlace using a good deinterlacer (preferable: TGMC or QTGMC).
When you deinterlace you got two choices, use a same-frame-rate deinterlacer or a bobber.
A same-frame-deinterlacer will merge the two fields into and playback will stay at the input frame rate. (i.E. 25fps)
A bobber will interpolate a full frame out of each field and to stay synch double the playback frame rate. (i.e. form 25 to 50 fps)

down-sides:
- same-frame rate deinterlacer: you loose up to 50% of the motion informations, due to dropped frames.
- bobber: depending on the accuracy and pre- and postprocessing the bobber does you will have to double the bitrate, since the frame count per second doubled. (see: http://forum.doom9.org/showthread.php?t=128545 + iirc using qtgmc and slower only needed 50% more data rate)

What to do?

Personally prefer to deinterlace the content I encode since I really thing that interlaced content should be a thing of the past. But in the end you have to decide if and how you want/need to deinterlace your content or if you stay with interlaced content.

Cu Selur

Ps.: do not run deinterlace filters on progressive content
PPs.: Telecine material should be handled with IVTC.

[Atak_Snajpera]

Like I said before encode interlaced source as interlaced. Use --bff switch for x264 command line. GPUs these days can nicely (better quality than yadif) deinterlace on fly.

[Ahari]

It's great that I have 2 experts helping me with opposite opinions! It helps me to gain some perspective on both options.

It would seem I should leave the de-interlacing alone. It's just that the deinterlaced video was so much sharper, so I don't want to dismiss it just yet.

I don't want to open a whole new can of worms, but for argument sake, if I was to de-interlace the video, which of the Avidemux filters would be the best?

Selur, you said that the same-frame rate deinterlacer would lose 50% of the motion info - that makes sense and is consistent with what I am seeing. I was using the yadif filter. Which of the avidemux filters would increase the frame rate properly as you describe or would that have to be a whole separate operation?

I assume that the crt encoding would increase the bitrate accordingly to allow for extra frames?

How do you deinterlace?

[Selur]

I assume that the crt encoding would increase the bitrate accordingly to allow for extra frames?

yes, crf is frame raw independent,..

if I was to de-interlace the video, which of the Avidemux filters would be the best?

Sorry, but no clue, since I do not use Avidemux and do not know which deinterlace filters they offer.

How do you deinterlace?

When I use an Avisynth based tool I normally use QTGMC with preset slow and if it needs to be same-frame-rate and extrac SelectEven() or SelectOdd().
Other than that yadif is a decent alternative to deinterlace interlaced Material.

Cu Selur

[Ahari]

Thanks for all the help, people. You've given me a lot to think about!

Will do some testing when I get back from holiday - will be away for a week, so please forgive my lack of response while away!

[Ahari]

Well, I'm back from holiday and have sorted out my problems now!

My final settings:
I am using CRF 18 which gives me excellent quality and small file sizes.
The batch file runs perfectly.

SETLOCAL

set avidemux="C:\Program Files\Avidemux 2.5\avidemux2.exe"
set myjob="_avitomp4.js"
for %%f in (*.avi) do %avidemux% --force-alt-h264 --load "%%f" --run %myjob% --save "%%~nf.mp4" --quit
pause

ENDLOCAL

The name of my job file is titled like that so that it appears at the top of my folder when I run it - easier to find amongst my video files.
The "SETLOCAL" and "ENDLOCAL" are there to make sure the variables used are only used for this session and don't mess with the global variables in Windows - not really necessary, but keeps it clean.
I decided after much testing to deinterlace using the Yadif filter set to "Bob, Temporal & Spacial check, Bottom field first". This seemed to play back better on both my PC and TV and the file sizes were acceptably only a bit larger. By doing this the frame rate is increased to 50 fps and playback is very smooth and crisp.

For anyone who is interested these are the settings in my project file:

//AD  <- Needed to identify//
//--automatically built--
var app = new Avidemux();

//** Postproc **
app.video.setPostProc(3,3,0);

app.video.fps1000 = 25000;

//** Filters **
app.video.addFilter("YADIF","mode=1","order=0");

//** Video Codec conf **
app.video.codecPlugin("32BCB447-21C9-4210-AE9A-4FCE6C8588AE", "x264", "AQ=18", "<?xml version='1.0'?><x264Config><presetConfiguration><name>Alan's settings for converting Canon's DVCPro AVI files to H264 MP4 files for archiving</name><type>user</type></presetConfiguration><x264Options><fastFirstPass>true</fastFirstPass><threads>0</threads><deterministic>true</deterministic><sliceThreading>false</sliceThreading><threadedLookahead>-1</threadedLookahead><idcLevel>-1</idcLevel><vui><sarAsInput>true</sarAsInput><sarHeight>1</sarHeight><sarWidth>1</sarWidth><overscan>undefined</overscan><videoFormat>undefined</videoFormat><fullRangeSamples>true</fullRangeSamples><colorPrimaries>undefined</colorPrimaries><transfer>undefined</transfer><colorMatrix>smpte240m</colorMatrix><chromaSampleLocation>0</chromaSampleLocation></vui><referenceFrames>4</referenceFrames><gopMaximumSize>250</gopMaximumSize><gopMinimumSize>0</gopMinimumSize><scenecutThreshold>40</scenecutThreshold><periodicIntraRefresh>false</periodicIntraRefresh><bFrames>6</bFrames><adaptiveBframeDecision>2</adaptiveBframeDecision><bFrameBias>0</bFrameBias><bFrameReferences>normal</bFrameReferences><loopFilter>true</loopFilter><loopFilterAlphaC0>0</loopFilterAlphaC0><loopFilterBeta>0</loopFilterBeta><cabac>true</cabac><openGop>disabled</openGop><interlaced>disabled</interlaced><constrainedIntraPrediction>false</constrainedIntraPrediction><cqmPreset>flat</cqmPreset><intra4x4Luma><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value></intra4x4Luma><intraChroma><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value></intraChroma><inter4x4Luma><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value></inter4x4Luma><interChroma><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value></interChroma><intra8x8Luma><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value></intra8x8Luma><inter8x8Luma><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value><value>16</value></inter8x8Luma><analyse><partitionI4x4>true</partitionI4x4><partitionI8x8>true</partitionI8x8><partitionP8x8>true</partitionP8x8><partitionP4x4>false</partitionP4x4><partitionB8x8>true</partitionB8x8><dct8x8>true</dct8x8><weightedPredictionPframes>smart</weightedPredictionPframes><weightedPrediction>true</weightedPrediction><directPredictionMode>auto</directPredictionMode><chromaLumaQuantiserDifference>0</chromaLumaQuantiserDifference><motionEstimationMethod>multi-hexagonal</motionEstimationMethod><motionVectorSearchRange>28</motionVectorSearchRange><motionVectorLength>-1</motionVectorLength><motionVectorThreadBuffer>-1</motionVectorThreadBuffer><subpixelRefinement>10</subpixelRefinement><chromaMotionEstimation>true</chromaMotionEstimation><mixedReferences>true</mixedReferences><trellis>allModeDecisions</trellis><fastPSkip>true</fastPSkip><dctDecimate>true</dctDecimate><psychoRdo>1</psychoRdo><psychoTrellis>0</psychoTrellis><noiseReduction>0</noiseReduction><interLumaDeadzone>21</interLumaDeadzone><intraLumaDeadzone>11</intraLumaDeadzone></analyse><rateControl><quantiserMinimum>10</quantiserMinimum><quantiserMaximum>30</quantiserMaximum><quantiserStep>4</quantiserStep><maximumConstantRateFactor>0</maximumConstantRateFactor><averageBitrateTolerance>1</averageBitrateTolerance><vbvMaximumBitrate>0</vbvMaximumBitrate><vbvBufferSize>0</vbvBufferSize><vbvInitialOccupancy>0.9</vbvInitialOccupancy><ipFrameQuantiser>1.4</ipFrameQuantiser><pbFrameQuantiser>1.3</pbFrameQuantiser><adaptiveQuantiserMode>variance</adaptiveQuantiserMode><adaptiveQuantiserStrength>1</adaptiveQuantiserStrength><mbTree>true</mbTree><frametypeLookahead>40</frametypeLookahead><quantiserCurveCompression>0.6</quantiserCurveCompression><reduceFluxBeforeCurveCompression>20</reduceFluxBeforeCurveCompression><reduceFluxAfterCurveCompression>0.5</reduceFluxAfterCurveCompression></rateControl><accessUnitDelimiters>false</accessUnitDelimiters><spsIdentifier>0</spsIdentifier><sliceMaxSize>0</sliceMaxSize><sliceMaxMacroblocks>0</sliceMaxMacroblocks><sliceCount>0</sliceCount><hrd>none</hrd></x264Options></x264Config>");

//** Audio **
app.audio.reset();
app.audio.codec("Faac",224,4,"e0 00 00 00 ");
app.audio.normalizeMode=0;
app.audio.normalizeValue=0;
app.audio.delay=0;
app.audio.mixer="NONE";
app.setContainer("MP4");
setSuccess(1);
//app.Exit();

//End of script

May it be of use to someone - I derived the settings after much reading and they seem to work very nicely.

My batch file is running as I type - there are over 2500 video clips spanning 6 years totalling over 30 hours of video. At the rate it is going, it should take about 60 hours for everything! My machine is very fast! At least I think so!

These files will be my archive of all my home videos and will take up only about 20% of the space of the original DV camera files. They will play back better on TV and PC and my Mede8er media player will recognise and play them too! I am very pleased to say the least!

Unless there are any further posts with comments or constructive criticism about my chosen methods, this will be my last post on this thread.

So, huge thanks again to everyone for the patient help.

[yoda313]

These files will be my archive of all my home videos and will take up only about 20% of the space of the original DV camera files

I hope you are preserving your tapes. That way you always have your original sources to go back to in the future.

I browsed the thread but I didn't see any mention of it.

[Ahari]

I hope you are preserving your tapes.

Thanks for the thought, yoda313.

No, I won't be keeping the tapes. The whole purpose of this exercise was to get rid of the tapes and have the clips in a more user friendly format. Besides the camera that recorded the tapes is now broken and can only be used through the Windows software and only to play back the tapes. I just managed to get the tapes onto my PC in time!

Don't worry, I have 2 on-site daily backups and a weekly rotating off site back up as well. I was caught once and lost a years worth of photographs - never again!

I do realize that there is more "quality" in the original tapes and that if I edit and save, edit and save repeatedly that quality will rapidly deteriorate (much like Jpeg pictures). I put "quality" in quotes because in my tests, the new originals look better, play better, are more compatible and smaller - all round just better! I do also realise that this thought may ruffle a few purist's feathers! I have been as careful as I can to make sure that no unnecessary edits are performed (except for deinterlacing) and that the files are saved at a higher quality than I found acceptable (read perfect)- just for a little room to "breathe".

I intend to refer to these h264 files as my master files from now on and will go back to them for any edits that need to be made. I will be very careful not to degrade them by editing the master without keeping the new "original" intact and unaltered.

This format is the same as my new Sony HD camera records in, so I figure if it is good enough for the new camera, then why bother keeping the old format DV tapes lying around? I can't use them anyway and tapes don't last.

[yoda313]

@ahari - ok just so you know the risks and benefits. You can still of course buy the equipment for playback or take them to professional video shops for transferring in the future.

It might not hurt to take a little time to review your collection and double check for any "must keeps" for preservation. There might be some that are more special than others that you may someday wish you had the absolute originals for.

But as long as you aware of the decisions you are making that is important.

[Ahari]

Thanks for the heads up, yoda313! I would rather be told 100 times about something like that than think 5 years down the line "Ooooops! Why did no one tell me?!"