2-pass conversion using ffmpeg and Send To under Windows XP

[bidmead]

I've been converting (mostly) TV derived MPEGS to XViD for most of this century, and am still learning how to do it. A recent discussion with Big Matt of WinFF fame set me off on another tack. I've no doubt that some of the wise folks here may already know what I've recently discovered, but I thought it worth setting out here. I'm particularly keen to get feedback, as Matt, for example, hasn't been able to duplicate some of this stuff and we're not sure why.

So with the proviso that this "Works for Me", here goes.

In the past I've used VirtualDubMod for conversion of MPEG2 to XViD (and still do if I'm getting rid of logos with the DeLogo filter), but it's always seemed to involve an awful lot of mouse clicks just to do a routine conversion. I used AutoGK quite a lot -- until I discovered that the best way to edit is by creating an intermediate .avs file. Alas, although AutoGK uses Avisynth internally it doesn't accept .avs input. And again, all those mouse strokes. I messed with Super for a bit (but didn't much like the quality), and then lighted upon Matt's excellent WinFF, which introduced me to ffmpeg.

I got into a discussion about 2-pass with Matt, and he's now implemented that in version 0.3. There seem to me to be some suboptimal things about the implementation (version 0.3 is only a couple of days old as we speak, so, hey!) which I've raised with Matt and which he's looking into.

Matt's using the identical ffmpeg command line parameters for the first and second pass, changing only the pass number. This works, but:

Firstly, there's no need (I believe - correct me if I'm wrong) to encode the audio on pass 1. It's a time and cycle waster.

Secondly, the output from a regular first pass can be huge. It's not wanted for anything (you only need to collect the stats file) so it should be sent to /dev/null.

On this second point we seemed to be stumbling up against an immovable obstacle, because XP has no /dev/null. Is this why every encoder I've ever used on XP insists on taking up disk space with the output of pass 1? But I fished around on the Internet for a solution -- and found one.

A third cosmetic point was that it would be useful if the title of the command line window in which ffmpeg does its magic reflected what it was doing.

Here's an XP batch file that utilises all these points. You'll notice that my \path\to\ffmpeg is where Matt's software installs it -- your mileage may differ.

Name the batch file (eg) 2pass.bat. (there are just 5 lines in the batch file -- so watch out for forum wordwrap.)

=================cut here==============
Title %~n1 -- pass 1
"C:\Program Files\WinFF\ffmpeg.exe" -i %1 -vcodec xvid -b 930kb -bf 1 -an -pass 1 -y nul.avi
Title %~n1 -- pass 2
"C:\Program Files\WinFF\ffmpeg.exe" -i "%1 -vcodec xvid -acodec mp3 -b 930kb -ab 128kb -bf 1 -pass 2 "%~dp1%~n1.avi"
pause
=================cut here==============
(Later -- forget this -- there's a better batch file later in the thread)

You pass the source file as a parameter to this batch file (see below).

Line one puts the name of the source file, sans path and suffix, in the window title and adds the pass number.

Line two -- I gather that under Win XP any file named "nul.something" is a bit sink. (That's "nul" not "null" -- Bill was never too hot at spelling...)

The neatest way I've found to pass the source file into this is to put the batch file in the Win XP hidden Send To directory (usually C:\Documents and Settings\<username>\SendTo. If you do this, when a right mouse click on the source avs file brings up the context menu, the Send To section should include "2pass.bat". That should start the encoding, in a command line window called "<sourcename> -- pass 1", which changes appropriately to "<sourcename> -- pass 2" when the second pass starts.

The output file arrives in the same directory as the input file. The stats file should appear there too, and is preserved in case you want to rerun the second pass.

I know this approach isn't the be-all-and-end-all and that folks here will be able to suggest many improvements (eg, do we need to define the target bitrate in pass 1??. Would drag and drop be a better way than using Send To??). And it may be trickier to get this working than I've estimated -- I'm still trying to figure out why, for example, the Title stuff doesn't work on Matt's system.

So, feedback, please. Lots of it.

--
Chris




Reply Forward

[quake74]

Nice to know there's somebody out there as crazy as I am about batch files and ffmpeg Let me also share a couple of things: you can put the bat file anywhere ad just drag and drop files on it, and by changing a little bit, you can drag'n'drop a bunch of files and it will go through them one by one; or you can right click on a file (holding shift) and choose "Open with..." and choose the bat file from there, so that every other time you open the same file format (eve without holding shift) it will be one of the options.

If you like this sort of things, and a small solution to the windows piping problem, visit http://quake74.wikispaces.com/ for a similar issue dealing with x264.

quake74

[bidmead]

Nice to know there's somebody out there as crazy as I am about batch files and ffmpeg

Well, I didn't quite say that. Much prefer, eg, bash scripting. But I was looking for a simple way to test some features I hoped would make it into WinFF if they proved viable.

you can put the bat file anywhere ad just drag and drop files on it

Yes, I knew that, but the SendTo mechanism seems a lot neater way of adding a batch file to the general functionality of a Windows system.

and by changing a little bit, you can drag'n'drop a bunch of files and it will go through them one by one

I guessed this might be feasible (and might work with SendTo) but haven't the foggiest how to implement this as a Windows batch file. Please advise.*

I was particularly looking for feedback on whether the bit-sink to nul.avi for the first pass works for everybody.

I'd also be interested info about the -an parameter. If I add this to an ffmpeg command line that already includes instructions to encode the audio, I get no audio in the output (as you'd expect). But is ffmeg smart enough to know that it doesn't have to bother encoding the audio before throwing it away. This question is useful because I think Matt would like to have a single command line which is duplicated for the first and second pass and then just modified for the first pass by adding -an and sinking to nul.avi.

I'll check out the URL you suggest. Many thanks, quake74.

*LATER: Ok, many thanks quake74. Visited your own batch file and came up with this:

rem chb 1-Sept-07
rem structure inspired by quake94@videohelp; all errors are mine alone.
rem If it turns out that -an completely switches off audio processing
rem we can condense PASS1OPTS and PASS2OPTS to a single variable string.

set FFMPEG="C:\Program Files\WinFF\ffmpeg.exe"
set PASS1OPTS=-vcodec xvid -b 930kb  -bf 1 -an 
set PASS2OPTS=-vcodec xvid -acodec mp3 -b 930kb -ab 128kb -bf 1 

:main
if not "%~f1"=="" goto convert
pause
exit

:convert

set PASS1OUT=-y nul.avi
set PASS2OUT="%~dp1%~n1.avi"

Title %~n1 -- pass 1
%FFMPEG% -i %1 %PASS1OPTS% -pass 1 %PASS1OUT%
Title %~n1 -- pass 2
%FFMPEG% -i %1 %PASS2OPTS% -pass 2 %PASS2OUT%
shift

goto main

You can drop multiple source files on this (or handle them via Send To). What you can't do -- can anyone help? -- is issue multiple source files non-simultaneously and add them all to the same input stack, so that you can stoke the feed while processing is underway. What happens instead is that you generate multiple instances of the batch file, all trying to run simultaneously, which isn't usually what you want.

--
Chris

[quake74]

You can drop multiple source files on this (or handle them via Send To). What you can't do -- can anyone help? -- is issue multiple source files non-simultaneously and add them all to the same input stack, so that you can stoke the feed while processing is underway. What happens instead is that you generate multiple instances of the batch file, all trying to run simultaneously, which isn't usually what you want.

Well, the dump way has an added benefit: since ffmpeg is not SMP aware (at least for the H264, which I use), this way you get a cheap SMP version by running two (or more) processes at the same time.

The best help I can give you for the smart way, is advising you to look at http://gbpvr.com/pmwiki/pmwiki.php/Tips/Post3 . Very advanced batch processing for me, but it boils down to create a txt file which has the file names and then read from that for the postprocessing.

q74

[Soopafresh]

to process a list of files, you can try two different batch commands:


for %%a in ("*.mpg") do ffmpeg -i "%%a" %your settingsfor pass 1% -y >nul
for %%a in ("*.mpg") do ffmpeg -i "%%a" %your settingsfor pass 2% -y output-"%%a".avi


or

dir *.mpg >filelist.txt
for /f %%a in (filelist.txt) do call rocess %%a

rocess

ffmpeg -i %1 %pass1% -y >nul
ffmpeg -i %1 %pass2% -y output.avi

[bidmead]

Thanks, Soopafresh. I'll mess around with that. Except that what I'm after is the ability just to drop additional avs files (or send them with SendTo) over to the batch file as and when they're ready. I think if the user has to create a text list from existing files that rather defeats the object. While I'm editing together .avs number 4 I want the batch file already to be processing the .avs files 1 through 3. When 4's ready, I just want to be able to add it to the running process list.

So maybe:

A batch file that builds a list, continually adding new dropped/SendTo files at the bottom, and
A batch file that reads that same list as input (but doesn't keep starting again from line 1 every time the file changes).

I don't know if that's doable with Windows batch files.

But this is getting a bit out of hand. I was just looking for a simple way to run a proof-of-concept test for ideas that Matt might want to use in WinFF.

--
Chris

[bidmead]

@quake74

The best help I can give you for the smart way, is advising you to look at http://gbpvr.com/pmwiki/pmwiki.php/Tips/Post3.

As I say, a pre-cooked list isn't quite what I'm looking for. But I'm intrigued by comskip.exe. Does it work? I always cut ads by hand, because most of the time you can't get a proper invisible cut unless you pick your cut points very carefully with respect to video and sound. And often you need to resort to a dissolve.

--
Chris

[Soopafresh]

Yeah, comskip works incredibly well for the most part. But it does add a bit of time to your overall conversion process. Still, you can run it all overnight. Wrote a small How To with inclusion into a batch process here:

https://forum.videohelp.com/topic325734.html

[quake74]

@quake74

The best help I can give you for the smart way, is advising you to look at http://gbpvr.com/pmwiki/pmwiki.php/Tips/Post3.

As I say, a pre-cooked list isn't quite what I'm looking for. But I'm intrigued by comskip.exe. Does it work? I always cut ads by hand, because most of the time you can't get a proper invisible cut unless you pick your cut points very carefully with respect to video and sound. And often you need to resort to a dissolve.

I don't understand by what you mean by "pre cooked", but since the method is essentially the same that was described a couple of posts ago, it doesn't really matter.

Comskip has not been so good to me. I guess it's because I live in Europe and here the default way of inserting commercials are quite different. Even after a fair amount of tuning, at least once every ten shows, some part of the show was considered a commercial and cut out. But, on the good side, you can run it on the recording WHILE you are recording it, so it doesn't need any extra time at all in my experience.

[bidmead]

@Soopafresh
It's not so much the time I mind about, but the quality of the results. When I watch a movie I don't want to see ads (or logos) but I also don't want to see where ads (or logos) have been removed. IME absolute precision in the choice of cut points and control over dissolve duration (where I need to resort to dissolves) is necessary to make these edits invisible. In particular, you cannot rely on GOP-defined cut-points -- cuts must be frame-accurate (I used VideoReDo for several years), and even then you might need to substitute a dissolve (which is why I no longer rely on VideoReDo).

But perhaps this belongs in a different thread. Thanks for alerting me to comskip, Soopafresh. I'll kick it around some more.

@quake74
By "pre-cooked" I meant a non-dynamic list that says "here are the files I want to encode". If I've already got the files I want to encode then I don't need a list, I can just select them all and drag 'em or send them to the batch file.

What I want to be able to do is drop or send files one by one as I finish editing them, but in a way that will stack them up in a single input queue to the encoder while the encoder is processing the previous files in the queue. Sorry if that wasn't clear.

Yes, I guess you're right about UK ads. There tend to be a lot of video and/or audio fades at the ad in-points and out-points, and these often dictate the need for a crafty dissolve rather than a cut when you're fixing this. And there's one channel (forget which) that tends either to repeat a few of the pre-ad frames when it picks up again, or starts the post-ad sequence with four or five frozen frames. This looks ghastly if you do a straight cut at the ad in- and out-points. My aim is to repair the movie with edits that even the director wouldn't spot, although obviously there is some stuff you just have to fudge as best you can.

--
Chris

[bidmead]

@Soopafresh

ffmpeg -i %1 %pass1% -y >nul

Just spotted this. And I'm picking this up to remind myself that the chief reason for starting this thread was to make sure this sink to /dev/null works for everybody on Windows.

It's NOT a redirection to nul -- ffmpeg (my copy here, anyway) insists on a file name to output to. Hence "nul.avi", which "works for me" (tm) as a bit sink on my Windows XP, but I believe may need the NUL: non-plug-and-play device to be activated. The -y parameter (overwrite output) is there because oddly I found Windows complaining that nul.avi already existed, and this seemed to fix it, but of course logically this is a crock.

So I was asking for feedback about whether this works for other folks. If they try redirecting ffmpeg to nul on a Windows system, I believe this WON'T work. They might also want to try nul.avi without the -y param to see what happens.

--
Chris

[GTRBudda]

I hope I'm not butting in here, I tried some of this myself.

On my system (WinME), both "nul" (or "NUL") and "nul.avi" worked, IOW no file was generated.
Also, for grins I ran once with audio, and once without, using "-an", then ran the log files through WinDiff. Both were identicle, so audio on or off made no difference. Not using "-y" only stopped and asked for confirmation.

If this wasn't helpful, I'll sit down and shut up now.

[bidmead]

I hope I'm not butting in here...

Your butt is very welcome and useful, GTRBudda... :-)

Glad to hear "nul.avi" worked for you. I was thrilled to come across this -- first-pass output can be so large, and so unnecessary.

The -an question was possibly misphrased by me. Yup, I know that -an kills audio output even if you've got an acodec parameter set on the command line. The question is, does ffmpeg spend time processing the audio in this situation before throwing it away. If not, great. But if it's going to waste cycles then we need to make sure there's no acodec instruction at all in pass 1. Do the logs tell you this?

Not using "-y" only stopped and asked for confirmation.

Yup, that's what I got. The -y's there to keep the flow uninterrupted.

Does the Title stuff work for you? Is it useful to show the filename and pass number as the window title?

--
Chris[/b]

[GTRBudda]

Ah, glad to help. Sorry for the slow respose here, something shiny distracted me....
As for "-an" or "-acodec...", well, I did a test running the same batch file both with and without -an and timed the process. Both ran for about the same time, 1 min 4 sec on my 700 mhz cpu (it was only a five second clip). So from what I see, if it didn't make a difference in process time, it isn't useing any less cpu time(? cycles?) without audio in the first pass. Of coarse, the same test on a hotter machine with a longer clip (and the right tools to time/measure it with, instead of my eyeballs and the windows clock ) might show there is a difference after all. I would be guessing to say that ffmpeg reads/analyzes the audio weather it writes("NULs") it or not.

What I was saying about the logs (and if there's a mention of audio in there, I wouldn't know it unless it bit me) is that since they were identicle, there seemed to be no harm done to the output video when the 1st pass audio was turned off. If they'd been different, I'd have assumed 1st pass audio was essential.

As for the title stuff, no it didn't work, I got just the 'Bad command or file name' message on 'title', and the substitutions didn't seem to fall into place either. But this is on WinME, known for it's, um, diminished dos capabilites... hmm, probably I'm just missing something. That said, I wouldn't be looking for those anyway, the filename and pass number show up plainly in ffmpeg's text output. A "works for me" case there.

More helpful-er?

Edit: I had a 'duh' moment and re-ran tne test with neither -an or -acodec in the script. Again, the process time was the same, 1 min 4 sec. for both, and also the log files were still identicle. Apparently, audio happens.

[bidmead]

I've had a crack at creating a timed batch file to test what happens when you take the audio out of the first pass, either by negating the audio codec with -an or not running the codec (and also including -an).

@echo off

rem chb 04 Sept 07
rem Dependencies: needs timemath.exe 
rem http://gearbox.maem.umr.edu/batch/f_w_util

cls
echo This batch file calibrates the time to run an ffmpeg first pass
echo three different ways. 

echo All three passes are dumped to /dev/null

set FFMPEG="C:\Program Files\WinFF\ffmpeg.exe"
set   noaudio=-vcodec xvid -b 930kb  -bf 1 -an -pass 1 -y nul.avi
set dumpaudio=-vcodec xvid -b 930kb  -bf 1 -acodec mp3 -ab 128kb -an -pass 1 -y nul.avi
set withaudio=-vcodec xvid -b 930kb  -bf 1 -acodec mp3 -ab 128kb -pass 1 -y nul.avi

rem noaudio doesn't include an audio codec instruction
rem dumpaudio includes an audio codec instruction, but dumps it with -an
rem withaudio includes an audio codec instruction and doesn't use -an

set startna=%TIME%
%FFMPEG% -i %1 %noaudio%

set startda=%TIME%
%FFMPEG% -i %1 %dumpaudio%

set startwa=%TIME%
%FFMPEG% -i %1 %withaudio%

set finish=%TIME%

echo Pass 1 with no audio stream output > timelog.txt
timemath timeonly %startda% -%startna% >> timelog.txt

echo Pass 1 audio codec evoked but no audio stream output >> timelog.txt
timemath timeonly %startwa% -%startda% >> timelog.txt

echo Pass 1 audio codec evoked and audio stream retained >> timelog.txt
timemath timeonly %finish% -%startwa% >> timelog.txt

type timelog.txt
del timelog.txt

pause

My tests with this indicate that it saves significant time when you skip the audio processing on the first pass, and the fastest way to do this is by not evoking the audio codec and also using -an. If you do evoke the codec and then negate it with -an there's a slight overhead, but it's not very much.

Other folks might want to try this for themselves. The batch file just runs three pass 1s and dumps them to /dev/null, so it's not useful for actual encoding.

--
Chris

[erik1958]

Comskip has not been so good to me. I guess it's because I live in Europe and here the default way of inserting commercials are quite different. Even after a fair amount of tuning, at least once every ten shows, some part of the show was considered a commercial and cut out. But, on the good side, you can run it on the recording WHILE you are recording it, so it doesn't need any extra time at all in my experience.

I would welcome some example .csv files where comskip is making errors.
Could you follow this procedure?

http://www.kaashoek.com/comskip/viewtopic.php?f=5&t=34