Captured a 60gb 2hr .avi with Vdub/Huff, encoding to x264 takes 23+ hours??

[CZbwoi]

Captured 8mm tape with Huffyuv and Vdub, made a basic first time script with AvsPmod, and went to encode it into x264 by using their Tools > Script encoder (CLI). So I started the process and thought I was seeing thing, 11...10...hours eta. Let it go overnight. I wake up and it's still not over. It FINALLY finishes and now a second set of time appears on the command screen, 11 and a half hours again remaining. What? Is this how long it normally is when you encode these captured files to x264 or is something going wrong here?

I have a box of old tapes, 8mm, VHS-C, VHS, that will need converting. And I was going for the best quality and lossless to capture it with, makes sense, and I would get a huge file in an hour or two of capturing and that would be it after some filtering. I'd understand if converting to x264 would take an hour or 2 or 3, but a whole day's worth of 24 hours?

Please tell me I'm doing something wrong here or that AvsPmod's script encoder sucks and bugged out on me. This is insane. It will take me months or half a year to complete it all.

[sneaker]

AvsPmod allows you to choose different x264 presets. If it's currently too slow for you simply select a faster preset. If you don't want to do 2 passes choose from the "CRF" variants. (2pass is only recommended if you need to hit a specific bitrate e.g. because you want to fill a DVD-R fully.)

(x264 knows presets ultrafast, superfast, veryfast, faster, fast, medium, slow, slower, veryslow and placebo)

[CZbwoi]

I don't remember what I set that on before so now I have to play the waiting game and wait 10 hours for it to finish, but what would the recommended speed preset be and how much of a difference would a faster one do? Do you have any experience with encoding 45-60gb captured .avi files altered with Avisynth scripts, if so how long does it take you to encode those to x264 (with an mp4 container) and what settings do you use for it?

And would using MeGUI not change anything since it's just telling the x264 what to do like AvsPmod is doing or does it handle things better and faster..?

[sneaker]

MeGUI also simply passes the AviSynth script to x264. It will be as fast as AvsPmod given the same script and x264 settings.

Recommended preset? As slow as you are willing to accept. The speed difference between them is enormous:

[jagabo]

There is a 100 fold difference in encoding speed between x264 fastest (ultrafast) and slowest (placebo) presets. With SD material I usually work at --preset=slow -- crf=18. If there are no other bottlenecks that gives me about 100 percent CPU usage and 35 fps. On an Intel i5 2500K.

Of course, if you're using lots of slow filters like QTGMC and TemporalDegrain that will contribute to the encoding time. If AviSynth is a bottleneck using a multithreaded build of AviSynth can speed things up if you have a multicore CPU.

[CZbwoi]

(2pass is only recommended if you need to hit a specific bitrate e.g. because you want to fill a DVD-R fully.)

I still want these files (family videos) to be in the best quality possible, or right by it, they're for playback on PC primarily but will also be played through the TV/Blu-ray players. Does 2-pass really not make a difference in this case unless I want to 'fill up a DVD-R fully'?

Recommended preset? As slow as you are willing to accept. The speed difference between them is enormous:

Have you ever dealt with capture sizes as big as mine? What was your preset and your eta when all was said and done?

Jagabo: My system is Intel Core i7 - 920 @ 2.67GHz, Ram is 11 GB. Can I do the multithreaded method with mine? What's the fastest setting I can set it to while still retaining near lossless quality? Here's a picture, I suspect it was set to the same thing when I ran it last, other than maybe the aspect ratio. I guess I took the 2 pass time as how long 2 pass VBR in PP usually takes...



What should the settings here be (I also see that I can't select the CRF like you guys are)?


x264 is telling me that it's currently at 11.07 fps with an eta of 7 and a half hours still...and I don't even know if it's a good idea to start another capturing test while this thing is running. It's like I'm simply stuck here waiting a day for it to finish.

[sneaker]

I still want these files (family videos) to be in the best quality possible, or right by it, they're for playback on PC primarily but will also be played through the TV/Blu-ray players. Does 2-pass really not make a difference in this case unless I want to 'fill up a DVD-R fully'?

CRF is just as good (and saves some time). Again: do you need to hit a specific file size/bitrate? No? Then use CRF.

Jagabo: My system is Intel Core i7 - 920 @ 2.67GHz, Ram is 11 GB. Can I do the multithreaded method with mine?

Yes. But what is your AviSynth script? Like jagabo said looking into better multi-threading can make sense if filtering is a bottleneck. If it's not then it's useless. (Bottlenecks usually show as low CPU utilization in task manager.)

What's the fastest setting I can set it to while still retaining near lossless quality?

Every preset can achieve pristine quality. It's just a matter of how much bitrate you use. Faster presets need more bitrate to achieve the same quality as the slower ones. (see graph I posted earlier)

What should the settings here be (I also see that I can't select the CRF like you guys are)?

I can see the CRF modes in your screenshot.

x264 is telling me that it's currently at 11.07 fps with an eta of 7 and a half hours still...and I don't even know if it's a good idea to start another capturing test while this thing is running. It's like I'm simply stuck here waiting a day for it to finish.

If your system is already being used for a different job it won't make sense to test speed. Wait until your system is idle again.

[jagabo]

Use 2-pass mode when you want a specific file size. Use CRF mode when you want a specific quality. In 2-pass mode you pick the bitrate (and hence the file size) but you don't know what the quality will be. In CRF mode you pick the quality but don't know what the bitrate will be.

In general, the slower the preset you select, the better compression you get. In bitrate mode that means you get better quality with slower presets. In CRF mode it means you get smaller files with slower presets.

You can only set the CRF value if you select one of the CRF encoding modes. I suggest you start with "x264 - CRF Medium" and CRF=18. If that doesn't go much faster then AviSynth processing is the bottleneck.

Use TaskManager to view CPU usage while you're encoding. I suspect it will be pretty lowbecause you are using single threaded AviSynth and slow filters.

[theewizard]

this sounds like you are trying to turn old SD 8mm into 1080 HD
lossless or NOT
what did you use for a capture setting 60GB is God awfully large
and what did you use for an output resolution

it is useless 'make work' to capture above 640*480 or even dvd std 720*480
or convert above that settings
the detail is NOT there, no matter how high you set the capture and conversion

you gain nothing, but work, with extended time loss, that could be used doing more video conversions

[CZbwoi]

Yes. But what is your AviSynth script? Like jagabo said looking into better multi-threading can make sense if filtering is a bottleneck. If it's not then it's useless. (Bottlenecks usually show as low CPU utilization in task manager.)

AviSource("k:\record Old Hi8 haup.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)

*note that I still have little to no idea what I'm doing in there if something looks off or funny... Any bottlenecking here?

Every preset can achieve pristine quality. It's just a matter of how much bitrate you use. Faster presets need more bitrate to achieve the same quality as the slower ones. (see graph I posted earlier)

But then doing so (faster presets) increases size, right?

I can see the CRF modes in your screenshot.

Just realized I had to click the 3 options on the list that say CRF after the x264 to make that option un-grayed out...

If your system is already being used for a different job it won't make sense to test speed. Wait until your system is idle again.

Well...the 2hr capture process is done after I read your comment :/, and now this capture device's inserted frames sits at 20 while my other one sat at 8 when all was said and done. Do you think that the x264 encoding in the background may have had something to do with that 20 number, is that possible? In task manager x264 is only taking up 761,268 k of memory.

In CRF mode it means you get smaller files with slower presets.

So the slower time to encode for CRF equals only smaller files or does it also give you better quality as well? Or does quality stay the same with CRF and it all depends on where you place the 0-51 option?

With SD material I usually work at --preset=slow -- crf=18

In my options it only has very slow, medium and fast. I'm assuming my 'very slow' option isn't the same as your 'slow', right?

what did you use for a capture setting 60GB is God awfully large
and what did you use for an output resolution

Captured .avi in VirtualDub with Huffyuv compression (which I stated in the beginning), this is a pretty standard practice to get the best analog capture and that's an average file size for 2 hours from everyone I've talked to. Output resolution was a little less than 640x480 due to cropping with a 10:11 AR added on it.

it is useless 'make work' to capture above 640*480 or even dvd std 720*480
or convert above that settings
the detail is NOT there, no matter how high you set the capture and conversion

It captured at 720x480 because that's what the native feed is, so I didn't convert above that setting for the capture or after.

[poisondeathray]

A very useful diagnostic tool is avsmeter. It can tell you many thing including cpu usage, min/max/avg FPS ,GPU usage for GPU based filters, etc... it can help determine bottlenecks, help arrange /optimize scripts etc...

[jagabo]

In CRF mode it means you get smaller files with slower presets.

So the slower time to encode for CRF equals only smaller files or does it also give you better quality as well? Or does quality stay the same with CRF and it all depends on where you place the 0-51 option?

At the same CRF size changes a lot between ultrafast, superfast, and veryfast. Ultrafast will typically be about 3 times larger than veryfast. Beyond that the size usually doesn't change by much, usually another 5 to 10 percent smaller at placebo. But from veryfast to placebo quality increases slightly as well. The place to look is in dark areas and shallow gradients. You will start to see posterization artifacts there when your CRF+preset aren't good enough.

But you shouldn't take anyone's word for it. Just encode a short 1 minute clip at each setting and see for yourself.

[ackboo]

My system is Intel Core i7 - 920 @ 2.67GHz, Ram is 11 GB

The i7 920 paired with its stock cooler is known to overheat in some cases, check your cpu temperature with hwmonitor or an equivalent while running an encode, it might explain some slowdowns. Anything over 90°C is a problem.

[CZbwoi]

A very useful diagnostic tool is avsmeter. It can tell you many thing including cpu usage, min/max/avg FPS ,GPU usage for GPU based filters, etc... it can help determine bottlenecks, help arrange /optimize scripts etc...

I opened up both AVSMeter and AVSMeter64 and they both flashed on the screen closed in a millisecond...so I don't know what's happening there. Meanwhile x264 is still running with an eta 3hr5min.

I did some tests for a 10 second long clip while I'm waiting for the AvsPmod x264 encoding to stop...here's what I got:

1. The first one I did was use WinFF with this guy's preset he has for download at the bottom of the description. It's the first attached file and for some reason looks better than the 2 I got from MeGUI.

2. MeGUI, preset slow, 20 cbr. Second attached file.

3. MeGUI, preset slow, 16 cbr (the recommended absolutely perfect quality, according to MeGUI). Third attached file.

Here are my takeaways: why does the MeGUI pair look worse than the WinFF one? You can notice it in the face at the end of the clips, the MeGUI ones degrade and add so much noise and blockiness on the face at the end, WinFF does not. I also don't even see a difference between the 2 MeGUI ones despite it being 16 versus 20.

So what exactly is this WinFF guy doing in his preset that I'm not getting in MeGUI? I opened up his .wff script in notepad and the only thing I was able to take away from it is that he also set the cbr to 20 (which isn't even that high according to what I'm learning!), so how does it look so much better? Here it is opened in notepad:

<?xml version="1.0" encoding="utf-8"?>
<presets>
  <MPEG4VideoCapture>
    <label>MPEG-4 Capture</label>
    <params>-crf 20.0 -vcodec libx264 -pix_fmt yuv420p  -preset slow -acodec libvo_aacenc -ar 48000 -b:a 128k -coder 1 -flags +loop -cmp chroma -partitions +parti4x4+partp8x8+partb8x8 -me_method hex -subq 6 -me_range 16 -g 250 -keyint_min 25 -sc_threshold 40 -i_qfactor 0.71 -b_strategy 1 -threads 0</params>
    <extension>mp4</extension>
    <category>MPEG-4</category>
  </MPEG4VideoCapture>
</presets>

I'll also attach the original file and the script I used if that's of any help. I just don't get why 16 and 20 look the same in MeGUI but this guy makes it look better in WinFF.


Also, each time I encoded this 10 second clip it took me about an average of 1 minute and 25 seconds, so 85 seconds. I did the math, correctly I hope, and that comes out to 17 hours of encoding for a 2 hour clip...? I'm no mathematician, but is this right based on how long it takes me for 10 seconds...?

The i7 920 paired with its stock cooler is known to overheat in some cases, check your cpu temperature with hwmonitor or an equivalent while running an encode, it might explain some slowdowns. Anything over 90°C is a problem.

Here is what I got: https://forum.videohelp.com/images/imgfiles/dP7UQZn.png

[poisondeathray]

A very useful diagnostic tool is avsmeter. It can tell you many thing including cpu usage, min/max/avg FPS ,GPU usage for GPU based filters, etc... it can help determine bottlenecks, help arrange /optimize scripts etc...

I opened up both AVSMeter and AVSMeter64 and they both flashed on the screen closed in a millisecond...so I don't know what's happening there.

RE: avsmeter

It's a command line application

But you can use it as a drop & drag (drop and drag your avs script onto avsmeter.exe), if you edit the configuration.ini file. There should be an avsmeter.ini file in the avsmeter directory if you've run it once. Open that in notepad and where it says "PauseBeforeExit=0" set it to "1" instead of "0", save it . Then drop & drag your avs onto it again. 32bit avsmeter is for 32bit avisynth. You probably need a decent length test for results to be valid (10 sec clip might not be long enough, but go ahead and try)

[CZbwoi]

Alright here is what I got with the 10 second script clip (that I also attached in my last post). Note that the x264 long encode is still running in the background, but that might not do anything to this.



And in the process it looked like (ran it again):

[ziggy1971]

First off, we need more system specs. How many hard drives? How much space? What is the capturing device?

Next, I would get the GoPro CineForm codec. It's free with the GoPro software. It produces a much smaller file size (capture) and is quicker, I believe in both reading and writing (haven't tested it lately)

You're right, a 2 hour SD video should only take about 45-60mins to encode using a quad-core with hyperthreading. Obviously longer with more filters

applied in your script.

I did something very similar to this a while ago (used a VHS-DVD recorder instead), then ripped the DVD's.

If it would be possible, I'd like to see a short clip of the video.

There is no way it should take that much time to encode. I'd say 3 hours tops with absolutely the best settings possible. (Please note, the Placebo mode is for people who need to learn more about what they are doing IMHO.)

Edit: Slow to the game I guess...

[poisondeathray]

Not sure how valid those numbers are when running x264 in the background , but it matches your observations. It certainly suggests your script is the bottleneck. You're getting ~6 to 6.3 FPS, but have 59.94 FPS footage, so approx 1/10 realtime speed with the script only , and low ~15% CPU usage. Typically you can get about 2-3x speedup with QTGMC using MT

[jagabo]

QTGMC() running single threaded would explain 15 percent CPU usage on a 8 thread CPU.

[CZbwoi]

@ziggy1971: Did you read the rest of the thread? I think everything you asked about was answered other than the HD's. It's being stored and saved on a new 2tb hard drive, 300gb left. All the converters, programs, etc. are on the main C drive.

It certainly suggests your script is the bottleneck. You're getting ~6 to 6.3 FPS, but have 59.94 FPS footage, so approx 1/10 realtime speed with the script only , and low ~15% CPU usage. Typically you can get about 2-3x speedup with QTGMC using MT

How would I do this? I need to uninstall everything, or just QTGMC, or Avisynth and get another version again?

QTGMC() running single threaded would explain 15 percent CPU usage on a 8 thread CPU.

...The fact that this is my first week learning about any of this would explain why anything is happening lol. I do not know any of this, that I have an "8 thread CPU" or that now there's another way to run QTGMC, a filter I just learned of a couple days ago along with Avisynth. That is why I'm here for help, I'm trying to comprehend everything and learn it for the first time. The moment I just read you say that I Googled it and found this, which I'm still struggling to understand. I'm a week old into all of this, that's why every time I ask 5 more questions and await the answers...I'm sorry for trying to learn. I'm only saying this because it seems you're getting annoyed since this is all common sense for you and less and less of my questions are getting addressed.

[ziggy1971]

Like I said, slow to the game. I was multitasking and by the time I posted response others had already done so.
Enjoy.

[jagabo]

Typically you can get about 2-3x speedup with QTGMC using MT

How would I do this?[/QUOTE]
Download a multithreaded build of AviSynth. Copy AviSynth.dll over the one in your System32 or SysWow64 folder. Maybe this one:

http://forum.doom9.org/showthread.php?t=148782

You'll need to add SetMtMode command in your script. Usually something like SetMtMode(5,8) at the start, SetMtMode(2) after AviSource().

[CZbwoi]

Thank you, I'm trying to get it working to test it out but for the life of me I can't get AviSynth.dll to replace the one in SysWOW64. I'm the only account this computer has ever known, the only admin, it will not let me. I've tried to change permissions to the folder, no luck. Searching around the internet and I can't find any solution yet. It's weird how none of the users on Doom9 said anything about this problem and yet for me it's not that simple. It asks me to provide administrator permission to move to this folder, I click continue like always, "you need permission to perform this action" here's the Try Again button to press on a loop.

I'll have to figure this one out before I can get back to everybody, can't move the file to SysWOW64...

Edit: anyone in the future reading this that wants the solution, here you go.


So right now it's going at 8.6-8.8 fps in WinFF...doesn't seem like it did anything.

My script, I took the SetMTMode lines from examples in the Doom9 thread from others' scripts:

SetMemoryMax(512)
SetMTMode(5, 4)
AviSource("k:\record Old Hi8 haup.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
SetMTMode(2)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)

[jagabo]

Try replacing SetMtMode(5,4) with SetMtMode(5,8) or SetMtMode(5,12). The second value is the number of threads to use. And go ahead and move it to right after AviSource(). Depending on what encoder/editor you are using you might also have to add Distributor() at the end of the script.

[CZbwoi]

In AVSMeter the fps is currently an average of 15-16 with an eta of 7 and a half hours (and nope, now it went up to 8hr). I see the CPU usage is at about 50%, how do I smack it up to like 90% or nearly 100% to make it go even faster?

I do not know why WinFF is giving me it half as fast, and it frustrates me because the files comes out looking so much better over there than in MeGUI where it turns blocky at the end. If I could get MeGUI to match that preset's settings (possible?) it'd be great. Or if someone has a better preset they have for something like this for MeGUI, beautiful.

Last night I let it encode in MeGUI with this code:

SetMTMode(5, 24)
AviSource("k:\record Old Hi8 haup.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
SetMTMode(4)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)

I put 24 in the threads section at the bottom because someone online told me I have a possible number of 32 threads because I have 8 possible cores.

It's currently about to finish taking a total of 7 hours:


- scratch that, another window opened up with 'importing tracks' and 20 more minutes...

Try replacing SetMtMode(5,4) with SetMtMode(5,8) or SetMtMode(5,12). The second value is the number of threads to use. And go ahead and move it to right after AviSource(). Depending on what encoder/editor you are using you might also have to add Distributor() at the end of the script.

If I went too much up to 24 is that a bad thing, how do I know how much I'm able to use? And there's a difference if I add it after AviSource and not before? Can I leave Distributor() there at all times for the future if it does something, regardless if my script/encoder needs it or not?

[poisondeathray]

Here are my takeaways: why does the MeGUI pair look worse than the WinFF one? You can notice it in the face at the end of the clips, the MeGUI ones degrade and add so much noise and blockiness on the face at the end, WinFF does not. I also don't even see a difference between the 2 MeGUI ones despite it being 16 versus 20.

Because you set zones 343,599 q40 . This overrides x264 to increase the quantizer to 40 for frames 343-599 inclusive. It's inverse relationship (higher quantizer results in lower quality) . Open mediainfo (view=>text) and the meta data will show some of the encoding settings used

zones=343,599,q=40

It might be that you saved or used an old preset or something in megui and didn't know about it

Some programs require distributor() for MT for the threading to work properly . ffmpeg is one of them (thus ffmpeg libx264 will, which is winff uses - it's a GUI for ffmpeg). x264 CLI does not

[jagabo]

I put 24 in the threads section at the bottom because someone online told me I have a possible number of 32 threads because I have 8 possible cores.

The i7 920 has four hyperthreaded cores. So the CPU can run 8 simultaneous threads. You sometimes need more software threads to keep all CPU threads busy. Start with 8, if that doesn't give you near 100 percent CPU usage try bumping up to 10 or 12. Running more threads than necessary will slow down processing.

You usually don't want to switch to mt mode 2 until after the source filter.

[CZbwoi]

Here are my takeaways: why does the MeGUI pair look worse than the WinFF one? You can notice it in the face at the end of the clips, the MeGUI ones degrade and add so much noise and blockiness on the face at the end, WinFF does not. I also don't even see a difference between the 2 MeGUI ones despite it being 16 versus 20.

Because you set zones 343,599 q40 . This overrides x264 to increase the quantizer to 40 for frames 343-599 inclusive. It's inverse relationship (higher quantizer results in lower quality) . Open mediainfo (view=>text) and the meta data will show some of the encoding settings used

zones=343,599,q=40

It might be that you saved or used an old preset or something in megui and didn't know about it

That quantizer info is for MeGUI, right? I didn't use an old preset into there or anything, it was a new install. Can you share your MeGUI preset for old capture files (it's interface is more stable and friendly than WinFF) or make one that's like the WinFF one?


The encoding in MeGUI finished and here were the stats:


The finished file is 3.97gb, original was 58.4gb. Length is 2hr 3min and 46sec.

Here's a comparison, remember that super long 24 hour AvsPmod encode with their CLI using my x264.exe? That file is just the video since no audio was added to it, but it's 890mb compared to MeGUI's 3.97gb.

You usually don't want to switch to mt mode 2 until after the source filter.

It wasn't at mode 2 in the above script, but mode 4 (do you suggest 2?). I saw a user on Doom9 put his before the QTGMC so that's why I did it, but if you say to put it after I will.

edit: I added the distributor() at the end and moved the MT mode insertion after the QTGMC, moved in the timeline to change frames and now AvsPmod isn't responding. It's the distributor() code, opened it up again, inserted it and the same thing.

[jagabo]

Try something like this:

SetMemoryMax(500)
SetMTMode(5, 8) # adjust thread count as necessary
AviSource("k:\record Old Hi8 haup.avi")
SetMTMode(2)
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)

[CZbwoi]

Try something like this:

SetMemoryMax(500)
SetMTMode(5, 8) # adjust thread count as necessary
AviSource("k:\record Old Hi8 haup.avi")
SetMTMode(2)
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)

I'm getting near the same results with your script, eta 5hr 30min, as with this scipt:

SetMTMode(5, 16)
AviSource("k:\record Old Hi8 haup.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
SetMTMode(2)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)
Distributor()

Is there any benefit to setting a max memory like you did? And having the SetMTMode(2) before AssumeTFF() and everything else?