Captured a 60gb 2hr .avi with Vdub/Huff, encoding to x264 takes 23+ hours??

[poisondeathray]

That quantizer info is for MeGUI, right? I didn't use an old preset into there or anything, it was a new install. Can you share your MeGUI preset for old capture files (it's interface is more stable and friendly than WinFF) or make one that's like the WinFF one?

Yes, do you see how the bitrate graph falls off ? Because the zones command overrided what x264 would have normally distributed




I haven't really use megui for a few years, I tend to use x264.exe or ffmpeg.exe - but I know you can enter extra commandline options in MEGUI, also check the zones section. IIRC you have to checkmark/enable advanced settings to see the other parts of the GUI. If you have saved a preset with zones, that would explain what you are seeing. Or if you post some screenshots of your configuration setup I, or someone else, can try to walk you through it

megui is one of the more complex GUI's . If you want something with a more simple interface, maybe try ripbot, or simple x264 launcher . If you want to continue using megui, then provide more information - definitely there is some zones information in the preset you were using, or extra commandline box has those zones settings . You can also post the log file if you want more info

The finished file is 3.97gb, original was 58.4gb. Length is 2hr 3min and 46sec.

Here's a comparison, remember that super long 24 hour AvsPmod encode with their CLI using my x264.exe? That file is just the video since no audio was added to it, but it's 890mb compared to MeGUI's 3.97gb.

If everything else is the same in terms of the script, preprocessing, x264 settings and version, you will get exactly the same thing. So obviously something was different between them, probably teh settings

[jagabo]

Is there any benefit to setting a max memory like you did?

That limits how much memory AviSynth proper uses -- leaving more memory for other filters. QTGMC() can use lots of memory so limiting AviSynth to 500 MB leaves more QTGMC().

And having the SetMTMode(2) before AssumeTFF() and everything else?

Mode 5 is essentially single threaded. Putting SetMtMode(2) just before QTGMC() leaves AssumeTFF(), ConvertToYV12(), and Tweak() running single threaded. Those filters are pretty fast so it's probably not a problem here. But if you had some other slow filter there it would still run slow.

[CZbwoi]

poisondeathray here are the MeGUI settings, the ones you're looking for I hope. Everything is still at the default other than me selecting a different CRF with the "Quality" option. I didn't even touch the other tabs, so I don't know why my MeGUI's videos degrade towards the end like how I noticed and how your graph clearly shows...

Mode 5 is essentially single threaded.

So that's why you always place that at the beginning, to start things off slow and simple? Then things get kicked up right after you insert the avisource and you change it to 2. It's a little weird how you have to start it off with something and then disregard the setting immediately, but I don't really know how coding works I guess.


Thanks for your 2 edits to the code, I'll use your method now all the time and see how it goes.

[jagabo]

So that's why you always place that at the beginning, to start things off slow and simple? Then things get kicked up right after you insert the avisource and you change it to 2. It's a little weird how you have to start it off with something and then disregard the setting immediately

Some source filters don't work properly in mode 2.

[poisondeathray]

This is the encoding string of the "megui-muxed 16 cbr.mp4" file as read by mediainfo (view=>text) . (BTW it's "CRF" not "CBR" ; CRF is constant rate factor, CBR is constant bit rate)

Encoding settings : cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x113 / me=umh / subme=8 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=2 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=599 / keyint_min=59 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=crf / mbtree=1 / crf=16.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.40 / aq=1:1.00 / zones=343,599,q=40

On the "front page" when you open megui, there is a zones button. Push that and check if there are any entries there

If I look back at post#6, I know that's not megui, but you have zones specified there as well

If there are no zones specified anywhere, post the log file. There will be a folder "logs" , inside will be a record of all your encodes by date

[CZbwoi]

Some source filters don't work properly in mode 2.

Then what'd be the mode to always change it to where the majority of them work, 4? Because I don't know if this is correlated with what you just said, but last night I let an encoding go through in WinFF with the script you made and it looked fine in the end. Good file size, it finished in 5 hours and a half, but I scroll through it and notice these pink and green flashes through parts of the video, NOT present in the source .avi......so now here I am again, still without even a perfect encode. The attached file 2016-04-23-1049-08.flv shows it in action, please help me out...

On the "front page" when you open megui, there is a zones button. Push that and check if there are any entries there

Here is what I got:


I *may* have been seeing things, but I could've sworn the first time I opened that up before I closed it and opened it again it said 24. But it's the morning, so unless that's normal, nvm.

If there are no zones specified anywhere, post the log file. There will be a folder "logs" , inside will be a record of all your encodes by date

It's empty in there, only thing that is in the Log folder is Versions and Update detection. All the jobs are still posted in the Queue tab but I can't seem to pull out any sort of log from any of them. The Priority tab when the encode is happening is always to LOW or BELOW NORMAL for what it's worth.

I also just ran the encode again to pull up a new log for you and this time there's nothing wrong with the video. It's set at CRF 16. This time the file is 9.27mb, my original CRF 16 of the same file is 5.65mb. I'm just getting confused now, something was up last time.

[poisondeathray]

windows 7,8,10 don't like you writing anything on the c: drive, so if you have the megui folder on the c: drive, it might be a permissions issue writing log files to the log folder

For the FLV, post the script used, and if you preview the script, can you see the errors ? Post mediainfo (view=>text) on the source AVI file. This looks more like a decoding issue, but glitches, and mixed up frames can also occur with MT. The probably increases the more temporal filters you use. But QTGMC alone is usually pretty safe with MT

Try to narrow down the issue to see if it's repeatable. For example use Trim() to specify a segment to encode around that error and see if you can repeat the issue, or if the error is completely random. Or if it's related to something else like maybe high temps, corrupted memory

[CZbwoi]

For the FLV, post the script used

SetMemoryMax(500)
SetMTMode(5, 16)
AviSource("k:\record Old Hi8 haup.avi")
SetMTMode(2)
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)
Distributor()

and if you preview the script, can you see the errors ?

I am viewing it by pressing the Play button in WinFF, "Preview the selected source file with ffplay (good test to see if conversion is possible)", the video plays at like 1 frame per second but the green and pink flashes are there and WAY more than in my exported file. But yes, they are there too and more of it.

Post mediainfo (view=>text) on the source AVI file.

Complete name                            : K:\record Old Hi8 haup.avi
Format                                   : AVI
Format/Info                              : Audio Video Interleave
Format profile                           : OpenDML
File size                                : 58.5 GiB
Duration                                 : 2h 3mn
Overall bit rate                         : 67.7 Mbps

Video
ID                                       : 0
Format                                   : HuffYUV
Format version                           : Version 2
Codec ID                                 : HFYU
Duration                                 : 2h 3mn
Bit rate                                 : 66.1 Mbps
Width                                    : 720 pixels
Height                                   : 480 pixels
Display aspect ratio                     : 3:2
Frame rate                               : 29.970 fps
Standard                                 : NTSC
Color space                              : YUV
Chroma subsampling                       : 4:2:2
Bit depth                                : 8 bits
Scan type                                : Interlaced
Bits/(Pixel*Frame)                       : 6.383
Stream size                              : 57.2 GiB (98%)

Audio
ID                                       : 1
Format                                   : PCM
Format settings, Endianness              : Little
Format settings, Sign                    : Signed
Codec ID                                 : 1
Duration                                 : 2h 3mn
Bit rate mode                            : Constant
Bit rate                                 : 1 536 Kbps
Channel(s)                               : 2 channels
Sampling rate                            : 48.0 KHz
Bit depth                                : 16 bits
Stream size                              : 1.33 GiB (2%)
Alignment                                : Aligned on interleaves
Interleave, duration                     : 10 ms (0.30 video frame)

Here is also the media info for the finished file if you want:

Complete name                            : K:\record Old Hi8 haup2.mp4
Format                                   : MPEG-4
Format profile                           : Base Media
Codec ID                                 : isom (isom/iso2/avc1/mp41)
File size                                : 1.98 GiB
Duration                                 : 2h 3mn
Overall bit rate                         : 2 290 Kbps
Writing application                      : Lavf57.20.100

Video
ID                                       : 1
Format                                   : AVC
Format/Info                              : Advanced Video Codec
Format profile                           : High@L3.1
Format settings, CABAC                   : Yes
Format settings, ReFrames                : 5 frames
Codec ID                                 : avc1
Codec ID/Info                            : Advanced Video Coding
Duration                                 : 2h 3mn
Bit rate                                 : 2 149 Kbps
Width                                    : 624 pixels
Height                                   : 472 pixels
Display aspect ratio                     : 4:3
Frame rate mode                          : Constant
Frame rate                               : 59.940 (60000/1001) fps
Color space                              : YUV
Chroma subsampling                       : 4:2:0
Bit depth                                : 8 bits
Scan type                                : Progressive
Bits/(Pixel*Frame)                       : 0.122
Stream size                              : 1.86 GiB (94%)
Writing library                          : x264 core 148 r2638 7599210
Encoding settings                        : cabac=1 / ref=5 / deblock=1:0:0 / analyse=0x3:0x111 / me=hex / subme=6 / psy=1 / psy_rd=1.00:0.00 / mixed_ref=1 / me_range=16 / chroma_me=1 / trellis=1 / 8x8dct=1 / cqm=0 / deadzone=21,11 / fast_pskip=1 / chroma_qp_offset=-2 / threads=12 / lookahead_threads=2 / sliced_threads=0 / nr=0 / decimate=1 / interlaced=0 / bluray_compat=0 / constrained_intra=0 / bframes=3 / b_pyramid=2 / b_adapt=1 / b_bias=0 / direct=3 / weightb=1 / open_gop=0 / weightp=2 / keyint=250 / keyint_min=25 / scenecut=40 / intra_refresh=0 / rc_lookahead=50 / rc=crf / mbtree=1 / crf=20.0 / qcomp=0.60 / qpmin=0 / qpmax=69 / qpstep=4 / ip_ratio=1.41 / aq=1:1.00

Audio
ID                                       : 2
Format                                   : AAC
Format/Info                              : Advanced Audio Codec
Format profile                           : LC
Codec ID                                 : 40
Duration                                 : 2h 3mn
Duration_LastFrame                       : -4ms
Bit rate mode                            : Constant
Bit rate                                 : 128 Kbps
Channel(s)                               : 2 channels
Channel positions                        : Front: L R
Sampling rate                            : 48.0 KHz
Frame rate                               : 46.875 fps (1024 spf)
Compression mode                         : Lossy
Stream size                              : 113 MiB (6%)
Default                                  : Yes
Alternate group                          : 1

[poisondeathray]

Reduce the thread count (16 is probably a bit crazy), if that doesn't help, comment out the setmemorymax

[CZbwoi]

Okay, so in WinFF I had to drop the thread count all the way down to 6 to make it stop in it's ffplay source preview.

In MeGUI, with the threads staying at 16 as it originally was, if I click Reopen Video Preview (equivalent to what I'm doing in WinFF, that's how I play it w/ ffplay in here, right?) and press play the video plays flawlessly in slow motion there (albeit there's no sound whereas in WinFF there is frame by frame sound).

What exactly is happening here?

[poisondeathray]

Was that with distributor() for both ? When using x264.exe (such as though megui), you should comment out distributor()

A sane number of threads is required because you need threads reserved for x264 as well. There is a complex balance and you can get thrashing if you use too many.

Megui's preview uses VFW system, using draw dib vfw renderer . Different system than ffplay . But theoretically it shouldn't make a difference in terms of the glitches you are seeing





BTW, The choice of lossless codec also makes a difference in speed . Huffyuv was good back in the day, but it's been surpassed in speed and compression by ut video, magic yuv. The usual number quoted is ~5-10% faster in final encoding speed by the final encoder for faster lossless codecs like ut and magicyuv . They are faster in DEcoding speed as well. For example if you use in an editor, the scrubbing is faster

e.g. here are some observations

ffmpeg32 -i huffyuv_mt_distributor.avs -c:v rawvideo -an -f null NUL
28FPS

ffmpeg32 -i magicyuv_mt_distributor.avs -c:v rawvideo -an -f null NUL
32FPS

ffmpeg32 -i huffyuv_mt_distributor.avs -c:v libx264 -an huffyuv.mp4
26FPS

ffmpeg32 -i magicyuv_mt_distributor.avs -c:v libx264 -an magicyuv.mp4
29FPS

Both the script only, and actual final encoding speed with libx264 are faster with magicyuv. This was a SD PAL 4:2:2 test, but the final encoding speed was 11.5% faster just by using a more modern intermediate

SetMtMode(5,4)
AVISource()
SetMtMode(2)
ConvertToYV12(interlaced=true)
QTGMC(preset="slower")
Distributor()

Filesize for huffyuv was 7% larger than magicyuv . Basically there is zero reason to use huffyuv these days

[CZbwoi]

Was that with distributor() for both ? When using x264.exe (such as though megui), you should comment out distributor()

Yes, and isn't WinFF using x264.exe as well?

Megui's preview uses VFW system, using draw dib vfw renderer . Different system than ffplay . But theoretically it shouldn't make a difference in terms of the glitches you are seeing

How come they're appearing in WinFF and not in MeGUI then?

BTW, The choice of lossless codec also makes a difference in speed . Huffyuv was good back in the day, but it's been surpassed in speed and compression by ut video, magic yuv. The usual number quoted is ~5-10% faster in final encoding speed by the final encoder for faster lossless codecs like ut and magicyuv . They are faster in DEcoding speed as well. For example if you use in an editor, the scrubbing is faster

e.g. here are some observations

ffmpeg32 -i huffyuv_mt_distributor.avs -c:v rawvideo -an -f null NUL
28FPS

ffmpeg32 -i magicyuv_mt_distributor.avs -c:v rawvideo -an -f null NUL
32FPS

ffmpeg32 -i huffyuv_mt_distributor.avs -c:v libx264 -an huffyuv.mp4
26FPS

ffmpeg32 -i magicyuv_mt_distributor.avs -c:v libx264 -an magicyuv.mp4
29FPS

Both the script only, and actual final encoding speed with libx264 are faster with magicyuv. This was a SD PAL 4:2:2 test, but the final encoding speed was 11.5% faster just by using a more modern intermediate

SetMtMode(5,4)
AVISource()
SetMtMode(2)
ConvertToYV12(interlaced=true)
QTGMC(preset="slower")
Distributor()

Filesize for huffyuv was 7% larger than magicyuv . Basically there is zero reason to use huffyuv these days

When I first started out everyone here and on DigitalFaq was pointing me towards huffyuv...why is this? You're talking so highly of it and I've never seen anyone say anything about magic yuv to me before, I think. What's with the trialware $5 and 0 reviews 5/10 score on the VideoHelp page, no one has even reviewed it? Sorry if I seem skeptical but that's pretty odd...

[poisondeathray]

isn't WinFF using x264.exe as well?

No, it's using ffmpeg libx264. They are based on the same libraries but slightly different implementation. Some features are not available in ffmpeg libx264, for example --qp file

Megui's preview uses VFW system, using draw dib vfw renderer . Different system than ffplay . But theoretically it shouldn't make a difference in terms of the glitches you are seeing

How come they're appearing in WinFF and not in MeGUI then?

Not sure, but if you do a small test encode in megui, you might see glitches too

When I first started out everyone here and on DigitalFaq was pointing me towards huffyuv...why is this? You're talking so highly of it and I've never seen anyone say anything about magic yuv to me before, I think. What's with the trialware $5 and 0 reviews 5/10 score on the VideoHelp page, no one has even reviewed it? Sorry if I seem skeptical but that's pretty odd...

A lot of the information you read might be old or outdated. Huffyuv was/is stable and used for many years. Still works well. A lot of people are of the "if it ain't broke, don't fix it" mind set. It's just that there are faster, more compressed codecs now. Think about it, the code base of huffyuv hasn't changed in 15 years. It uses MMX from the pentium (!) era, but none of the new instructions that modern processors use like AVX etc...

UT Video Codec was the "king" for a few years but magicyuv is the new kid on the block. It's the new "king" in terms of speed, but hasn't been as thorougly tested in all scenarios as something like UT

Some "negatives" of magicyuv are it's not open source, there is no ffmpeg derivative. If the developer quits for exmaple, its a dead duck. No more development. No more bug fixes. It's on gumroad, so you "pay" whatever you want or think it's worth, even zero, or 1 cent

UT Video codec however is open source, there are ffmpeg, quicktime implementations. It will forever be around. Other people can contribute and add fixes. It's battle tested in NLE's and production environments, and even the recommended lossless intermediate in the Adobe forums. But it's a hair slower than magicyuv, but slightly more compressed (~0.5% smaller)


EDIT: here is the same test revised with UT results.

ffmpeg32 -i huffyuv_mt_distributor.avs -c:v rawvideo -an -f null NUL
28FPS

ffmpeg32 -i magicyuv_mt_distributor.avs -c:v rawvideo -an -f null NUL
32FPS

ffmpeg32 -i ut_mt_distributor.avs -c:v rawvideo -an -f null NUL
31FPS

ffmpeg32 -i huffyuv_mt_distributor.avs -c:v libx264 -an huffyuv.mp4
26FPS

ffmpeg32 -i magicyuv_mt_distributor.avs -c:v libx264 -an magicyuv.mp4
29FPS

ffmpeg32 -i ut_mt_distributor.avs -c:v libx264 -an ut.mp4
28FPS

[CZbwoi]

Hmmm...so in terms of just capturing lossless analog capture in VirtualDub like I've been doing with huffyuv and then adding avisynth filters and whatnot, which would you suggest, UT Video Codec or this MagicYUV? What more development does the developer have to do to make me question using it right now, like will there be capturing bugs or dropped frames or weird artifacts..? The captured file won't work in some media players or NLE's..?

[poisondeathray]

Anything you use huffyuv with, can be safely replaced with ut video codec. For sure. The benefit is slightly smaller filesize, faster decoding and encoding speed for the final encoder. UT is stable and battle tested in professional workflows. I have zero reservations recommending it. All the bases are covered. In fact more bases are covered because you can read UT on a mac more easily (because there are QT components), it also has more colorspace/colormodel coverage - 10bit variants, 4:2:0 variants, HD variants (709 vs 601). (The original huffyuv doesn't support 4:2:0)

Magic YUV is newer, but so far seems fine. Because it's newer, it hasn't been used as much or exposed to as many situations as the other codecs. Sometimes bugs come up over time (some situation or context might reveal a problem in the code). If you were of a conservative mindset I would go with ut video.

You can use huffyuv too. It's stable and has been used for many years. It's just slower, less compressed. You can see using UT was ~ 7.5% faster on the final encoding speed with ffmpeg libx264. And that falls within the 5-10% number commonly quoted

[CZbwoi]

I have Huffyuv v2.1.1. Quality stays the same throughout through all 3 codecs, right? Changing the codec through those 3 doesn't change the capture quality?

Magic YUV is newer, but so far seems fine. Because it's newer, it hasn't been used as much or exposed to as many situations as the other codecs. Sometimes bugs come up over time (some situation or context might reveal a problem in the code).

Now do these potential bugs relate to capturing RCA/S-Video like all I would be using it for, or for different things?


And to also get back to the topic at hand, for you, how long does it take to encode 60gb 2 hour long captured .avi's? Or half that amount and the math will fill in.

[poisondeathray]

Magic YUV is newer, but so far seems fine. Because it's newer, it hasn't been used as much or exposed to as many situations as the other codecs. Sometimes bugs come up over time (some situation or context might reveal a problem in the code).

Now do these potential bugs relate to capturing RCA/S-Video like all I would be using it for, or for different things?

It should be fine for analog capture, but it's hard to say, because sometimes things just come up, or there are isolated situations where something isn't ideal. That's why they are called "unrecognized" bugs. If you are paranoid,just use ut or huffyuv. For example , some bugs about chroma upsampling/converting to RGB were addressed in the last magic yuv build, where it used a point resample algorithm before. But, theoretically, you shouldn't be using the codec to do the RGB conversion in the first place. There might be quirks in the host program using the codec. For example, magic yuv doesn't show up as a "preview" file for sequences in PP, but ut video does. (A workflow some people use in RGB workflows is to use a lossless codec as the preview file and use "render previews". This makes exporting lossless RGB many times faster because it's just copying out the preview files.). Now none of those things will affect what you are doing right now, but still you have to wonder about other "things". Ut video went through the early bug period as well, but that was several years ago.

And to also get back to the topic at hand, for you, how long does it take to encode 60gb 2 hour long captured .avi's? Or half that amount and the math will fill in.

I don't know, it's going to depend on so many things, like the script, encoding settings, hardware etc... Your speed seems about right for your setup, but I would reduce the threads for setmtmode. You can make adjustments like faster QTGMC settings, faster x264 settings if it's bothering you that much. Do some mini tests and see if you can see the difference, or if the speed penalty is worth it to YOU .

Personally, for "critical" tasks, or if I have additional temporal filtering (not QTMGC alone) I don't use MT , I switch back to "vanilla" avisynth. Avisynth MT is not reliable enough in my experience.

I have Huffyuv v2.1.1. Quality stays the same throughout through all 3 codecs, right? Changing the codec through those 3 doesn't change the capture quality?

Yes - lossless codecs are mathematically lossless, IF they are handled properly in the same color space/ color model . Now you have some situations where a host application might be doing some conversions behind the scenes (an example is Sony Vegas, which will treat most YUV lossless codecs as computer RGB). In that case some lossless codecs might not be truly lossless. But for the avisynth/x264 workflow it is lossless and you can switch between any lossless codec

[CZbwoi]

but I would reduce the threads for setmtmode.

Lower than the 6 it's currently at?

You can make adjustments like faster QTGMC settings, faster x264 settings if it's bothering you that much.

With no loss at how the final product is? How can I do that? (unless you're just referring to changing to a different MT Mode and getting a bigger file size or less quality when encoding)

Personally, for "critical" tasks, or if I have additional temporal filtering (not QTMGC alone) I don't use MT , I switch back to "vanilla" avisynth. Avisynth MT is not reliable enough in my experience.

What do you mean by this? When you have more important things to work on you don't use MT..? All I'm doing here is super important family history stuff, the way I'm going to be doing it isn't reliable? I mean, I don't think I'm going to be using extensive filtering for these only than the basics unless it's really recommended to go all out unless they won't look any good, this is my first go around.

[poisondeathray]

but I would reduce the threads for setmtmode.

Lower than the 6 it's currently at?

I meant from the 16; you should be ok with 6 . I typically use 4 for SD QTMGC

You can make adjustments like faster QTGMC settings, faster x264 settings if it's bothering you that much.

With no loss at how the final product is? How can I do that? (unless you're just referring to changing to a different MT Mode and getting a bigger file size or less quality when encoding)

There will be differences, but there are diminishing returns for everything you do. Is 100x longer to encode but 1% higher "quality" at a given filesize worth it to you ? For some people, maybe. For most people probably not. We are just providing information - You have to decide where YOU personally want to make the trade offs. You might have to run some mini tests to see if it meets YOUR goals

Personally, for "critical" tasks, or if I have additional temporal filtering (not QTMGC alone) I don't use MT , I switch back to "vanilla" avisynth. Avisynth MT is not reliable enough in my experience.

What do you mean by this? When you have more important things to work on you don't use MT..? All I'm doing here is super important family history stuff, the way I'm going to be doing it isn't reliable? I mean, I don't think I'm going to be using extensive filtering for these only than the basics unless it's really recommended to go all out unless they won't look any good, this is my first go around.

Yes, for important things, or when I have more complex scripts, I don't use MT . There is a higher risk of mixed up frames. The more temporal filters, the higher the risk. Single threaded, vanilla avisynth is the safest . QTGMC is probably ok when run alone with MT, but for my peace of mind, I'd rather do parallel encodes in those situations. You will find some people run MT all the time, and others avoid it like the plague. For some people it's very unreliable, for others it seems reliable. YMMV

The other semi common problem with MT is crashing. It's usually memory related. So on HD material, QTGMC + a filter stack will have a much higher chance of crashing. Doesn't affect you here, but something to be aware of

Did you know more threads for encoding settings means lower x264 efficiency as well? Lower efficiency at a given bitrate means lower quality. The default --threads is logical cores *1.5. So for a 4C/8T desktop processor, it would be 12 threads . One of my old workstations has 2*6C/12T or 24 logical cores or 36 threads for default x264 settings. (and that's just a old low end workstation, more recent xeon workstations have many more cores, a typical setup would be 2*12C/24T 72 threads)

Here is a b-frame comparison at 1 and 36 threads for 1000kbps for 720x400. Open them in different tabs and flip back and forth

1 thread


36 threads


But it's unrealistic to run x264 with --threads 1

If you ran threads in between like 18 the compression/quality would be somewhere in between. Higher threads than 36 like on more modern workstations would be even lower quality. You can over come this by using higher bitrates (higher bitrates "fix" everything)

[_Al_]

CZbwoi - you are getting a top advices no question

I chose the other way dealing with CPU not close to 100% dealing with QTGMC and Avisynth script. I designed script that encodes AVI to MP4 (with H.264 or HEVC) or DVD or just streams (AAC, AC3, m2v, H.264, HEVC). Everything is selfcontained and you need only Avisynth (2.6.0 or higher for dealing with mpeg2 because HcEncoder needs that).

It is here: AVI to MP4-DVD.zip, it was designed frame serving but it deals with regular AVI, if loading a clip Avisource("path.avi") , I have not implemented yet special AVI types like 4:2:2, so only AVI with simple loading line without needing to specify other parameters. I use it for avi coming from frame server or DV.avi files.

WHY I am posting this - you encode for example two or three AVI's at the same time and CPU gets close to 100% even if using QTGMC in script. You just run more instances of the same script with different files. You can go easy on hardisks as well assigning different destinations or temp directories for different encodings by just creating other INI files and using them together etc.

Or You go for a vacation you make a batch that just executes one file after each other or two or three in groups, whatever group of batches you create. Those basic scripts can be started by simple command line, so it is limitless what you are going to do with this.
You just need to find all lines with "pause>nul & goto :eof" and change it to simple "goto :eof", if that would be a problem, I can fix that for you.

Choose you INI file, or make it to your liking and drop AVI and INI on particular BATCH and you get MP4 etc.
Or use command line it is all in info file with plenty of examples.

your INI file might look like that:

path_destination      :C:\Destination
path_temp             :C:\Temp
x264_variables        :--crf 16 --tune film --vbv-bufsize 20000 --vbv-maxrate 20000
nero_variables        :-lc -cbr 256000

Avisynth script between lines, no source loading:
----------------------------------------------------------------
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.5,1.5,10,1.0)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=1.2, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
Spline36Resize(640,480)
crop(0,0,-16,-8)
--------------------------------------------------------------------

years back i made DV avi helper that accepts only DV avi and no modern format like HEVC available to encode or to test, it is set hard, video parameteres so hard to change it, well it was a start ...

[CZbwoi]

I meant from the 16; you should be ok with 6 . I typically use 4 for SD QTMGC

I thought everything was perfect, it played great for a few minutes flawlessly...and then a pink half of a millisecond flash appeared. This is so aggravating, there's no way to have spotted that before encoding. It went minutes without happening for a brief nanosecond, if I had just clipped out a small clip it may have not been in there and I'd have thought I had it all figured out... Now I'm going to try to do it again on 4 like you, another 5-6 hours of waiting I guess. Scratch that, more, the fps when it's on 4 threads dropped to 16 from the 20ish it was with 6 threads :/

Single threaded, vanilla avisynth is the safest . QTGMC is probably ok when run alone with MT, but for my peace of mind, I'd rather do parallel encodes in those situations. You will find some people run MT all the time, and others avoid it like the plague. For some people it's very unreliable, for others it seems reliable. YMMV

The other semi common problem with MT is crashing. It's usually memory related. So on HD material, QTGMC + a filter stack will have a much higher chance of crashing. Doesn't affect you here, but something to be aware of

Mama mia. I was planning to frameserve some HD-esque material into AvsPmod to apply QTGMC when I'm done with the tapes (as I think you know from the other thread). Like DV camcorder footage, cameras from the last 10 years' footage, phone footage from a few years ago...I don't want it to screw up when dealing with those. Eff.

I would do away with it if I could get my script to at least be encoded with 16 fps. Before I tried this MT stuff, I was at like 6. If only there was a way to fix all this without MT...

Yes, for important things, or when I have more complex scripts, I don't use MT .

So what do you do, do you replace the .dll file in your system each time with the original Avisynth one and vise versa?

Here is a b-frame comparison at 1 and 36 threads for 1000kbps for 720x400. Open them in different tabs and flip back and forth

1 looks immensely better, thanks for the explanation. So in this case with MT threads, less = more and the downside is the encoding time? So MT with 4 threads is ultimately worse than just not using MT at all and straight shooting it into encoding?


_Al_ : Not too sure I understood all that as it may be too complicated for me, but I'll try to give it a shot after this encoding with 4 threads and WinFF is done in 6-8 hours.

Would this in any way speed up my encoding to MP4 compared to where I am now? And won't my files be way bigger encoding them to H264 than to x264? I'm encoding from 60gb files and I'm liking this 2-4gb average x264 it's giving me for 2 hour footage...

[poisondeathray]

I meant from the 16; you should be ok with 6 . I typically use 4 for SD QTMGC

I thought everything was perfect, it played great for a few minutes flawlessly...and then a pink half of a millisecond flash appeared. This is so aggravating, there's no way to have spotted that before encoding. It went minutes without happening for a brief nanosecond, if I had just clipped out a small clip it may have not been in there and I'd have thought I had it all figured out... Now I'm going to try to do it again on 4 like you, another 5-6 hours of waiting I guess. Scratch that, more, the fps when it's on 4 threads dropped to 16 from the 20ish it was with 6 threads :/

4 doesn't ensure success - it might be, for whatever reason, on your setup , you fall into the group that can't get it working properly. Recall I said some people just can't get it to work without glitches

You might also have better luck with x264 (without distributor) , or ffmpeg (with distributor) . So check if one fails consistently for you

I would do away with it if I could get my script to at least be encoded with 16 fps. Before I tried this MT stuff, I was at like 6. If only there was a way to fix all this without MT...

You do parallel encodes simultaneously with single threaded avisnyth. Encode more than 1 video at a time.

Yes, for important things, or when I have more complex scripts, I don't use MT .

So what do you do, do you replace the .dll file in your system each time with the original Avisynth one and vise versa?

You don't even need to swap out the .dll. If you don't use setmtmode, it "acts" like vanilla avisynth

However there are a few functions and rare scenarios that are buggy that still require the original .dll. So to be safe you should swap it out

Here is a b-frame comparison at 1 and 36 threads for 1000kbps for 720x400. Open them in different tabs and flip back and forth

1 looks immensely better, thanks for the explanation. So in this case with MT threads, less = more and the downside is the encoding time? So MT with 4 threads is ultimately worse than just not using MT at all and straight shooting it into encoding?

No, that comparison strictly deals with threads for x264, the encoder. It has nothing to do with threads for mt avisynth.

The point is if you limit x264 encoding threads, the quality will be higher at a given bitrate. But nobody does it because of tradeoffs. There is a balance in the choices you make. If you think 6 FPS was slow, it will be much slower with 1 thread encoding

[_Al_]

Would this in any way speed up my encoding to MP4 compared to where I am now?

Yes, it is a setup that alowes you to launch more instances at the same time, like poisondeathray explains, you encode more videos at the same time, well, you lauch first, then second perhaps even third and CPU get much closer to 100%. And it works with frame server avi as well, I saw in other threads to charge to that territory as well ...

[CZbwoi]

You do parallel encodes simultaneously with single threaded avisnyth. Encode more than 1 video at a time.

Well, this kills my plan of only working on/keeping 1 large captured file on the HD at a time. Can you encode more than 1 script simultaneously in MeGUI or something similar? Or would I have to work directly with x264 and command lines (which I know nothing of)?

You don't even need to swap out the .dll. If you don't use setmtmode, it "acts" like vanilla avisynth

However there are a few functions and rare scenarios that are buggy that still require the original .dll. So to be safe you should swap it out

Yeah, I don't think it works for me that way. I get ride of the MT Mode lines and QTGMC starts giving me errors:

Yes, it is a setup that alowes you to launch more instances at the same time, like poisondeathray explains, you encode more videos at the same time, well, you lauch first, then second perhaps even third and CPU get much closer to 100%. And it works with frame server avi as well, I saw in other threads to charge to that territory as well ...

How does the size compare to encoding to x264 though?

My encoding at 4 threads still isn't done (...I know) so I can't even attempt to try out what you're saying yet.


Edit: PDR - Back on the subject of UtVideo, exactly which compression codec do I use to capture analog tapes in the right colorspace like I did with huffyuv? It gave me 9 of them.

[jagabo]

Re the MT error, I get the same thing with some filters. Just at SetMtMode() with no arguments at the start.

In UT YUV 4:2:2 BT.601 is the same as huffyuv.

[poisondeathray]

I get ride of the MT Mode lines and QTGMC starts giving me errors:

I don't get that error with set's MT avisynth 2.6
http://forum.doom9.org/showthread.php?t=148782

Must be some differences in the .dll's we're using. It can be difficult tracing differences in .dll's, sometimes ones that aren't even called directly in the script are the culprit, because they are loaded at runtime. This is my least favorite part of avisynth....trying to debug ".dll hell"

But I have run across similar errors with other scripts, that are fixed when I swap the .dll back. So I'll keep that SetMTMode() workaround in mind , thanks jagabo




As an aside, when I use that script on "test elgato 4.avi", those values give "illegal" values Y<16 , Y>235 . I'm assuming it's the same one uploaded earlier

You do parallel encodes simultaneously with single threaded avisnyth. Encode more than 1 video at a time.

Well, this kills my plan of only working on/keeping 1 large captured file on the HD at a time. Can you encode more than 1 script simultaneously in MeGUI or something similar? Or would I have to work directly with x264 and command lines (which I know nothing of)?

I don't recall if megui can, I think it can, because it has "workers" that you can spawn. But you definitely can with commandline. If you setup batch files like _Al_ suggested, it's even easier than using megui. It might be as simple as double clicking and a whole folder can be processed automatically. With megui I think you might have to load each one, queue each one, one by one. Or there might some added functionality in the newer versions. But you can imagine it becomes tedious if you have many files to process 1 by 1. Some GUI's might have "folder" input, I think some ffmpeg GUI's like ffqueue might, I can't recall offhand. I highly recommend starting to learn basic commandline


RE: splitting / simultaneous encodes

You can encode segments and join, but like threads, the more "splits" the more quality loss. The reasoning is similar to threads - you reduce the amount of redundancies that might have have be used it increase compression efficiency. When you use more threads, the motion vectors get truncated. Same with multiple segments, but the loss also includes reducing GOP length . This is one of the reasons why large distributed encoding often produce lower quality video (e.g. Youtube) than what you would produce locally on 1 computer. They divide up the work (distributed encoding) and join. It's much faster, but at the "price" of lower quality

But is , say, 4 segments instead of 1 going to dramatically reduce the quality of a 2hour video. Definitely not. Same with "sane" number of x264 encoding threads - for a typical desktop 4C/8T, --threads 12 isn't that harmful at all. You only can see deterioration if you use low bitrates (relative to content complexity) . Everyone accepts those sorts of tradeoffs, it's not a problem. But if you were completely paranoid, you would run --threads 1, single threaded avisynth, "placebo" encoding settings. You'd get fractional FPS's

[CZbwoi]

Re the MT error, I get the same thing with some filters. Just at SetMtMode() with no arguments at the start.

So if I set my code like this, it's not using MT Mode anymore and it just acts like normal avisynth?

SetMTMode()
AviSource("k:\record Old Hi8 elgato.avi")
AssumeTFF()
ConvertToYV12(interlaced=true)
tweak(-2.7,1.5,9,1.0)
QTGMC(EZDenoise=4.0,Preset="Slow", Sharpness=2.3, SLMode=1 )
ColorYUV(gain_y=71, cont_u=48, cont_v=12)
crop(4,0,-8,-6)
Spline36Resize(640,480)
Distributor()

*also, is there any benefit to still having Distributor() at the end or does that do nothing anymore since MT Mode is not in use?

I don't get that error with set's MT avisynth 2.6
http://forum.doom9.org/showthread.php?t=148782

That's odd, I'm pretty sure that's the one I was told to dl and I get the error without the SetMTMode() at the start.

As an aside, when I use that script on "test elgato 4.avi", those values give "illegal" values Y<16 , Y>235 . I'm assuming it's the same one uploaded earlier.

What do you mean by this? You mean by inserting just SetMTMode() at the start and getting rid of the rest of the MT instances it gives you errors with the test elgato 4.avi and .avs? I did it on mine to see and it gives me no errors...

I don't recall if megui can, I think it can, because it has "workers" that you can spawn. But you definitely can with commandline. If you setup batch files like _Al_ suggested, it's even easier than using megui. It might be as simple as double clicking and a whole folder can be processed automatically. With megui I think you might have to load each one, queue each one, one by one. Or there might some added functionality in the newer versions. But you can imagine it becomes tedious if you have many files to process 1 by 1. Some GUI's might have "folder" input, I think some ffmpeg GUI's like ffqueue might, I can't recall offhand. I highly recommend starting to learn basic commandline

That's the thing, I'm ways away from learning how batch files and commandlines work, I will try though. I can't even figure out how to trim a video in avisynth so I can do shorter encodes for tests, I'm inserting trim(226614,230000) and all I get is a blue screen. I'm pretty sure I followed the avisynth wiki example correctly too...this stuff is insanely comprehensive.

[poisondeathray]

Re the MT error, I get the same thing with some filters. Just at SetMtMode() with no arguments at the start.

So if I set my code like this, it's not using MT Mode anymore and it just acts like normal avisynth?

Yes, you can check with avsmeter, and it will report the active MT mode as 0. But for some reason, I don't even need to use it, at least not on this script

*also, is there any benefit to still having Distributor() at the end or does that do nothing anymore since MT Mode is not in use?

You don't need it

As an aside, when I use that script on "test elgato 4.avi", those values give "illegal" values Y<16 , Y>235 . I'm assuming it's the same one uploaded earlier.

What do you mean by this? You mean by inserting just SetMTMode() at the start and getting rid of the rest of the MT instances it gives you errors with the test elgato 4.avi and .avs? I did it on mine to see and it gives me no errors...

Nothing to do with mt mode or threads. "test elgato 4.avi" has those levels to begin with. I'm talking about Y levels. Recall the waveform in your other thread, the missing details on the street and bright areas ? Add histogram() and it will show you the waveform, but turned on it's side. If you "fix" the levels in this shot you will "see" the details in the window and some in the shadows. But it might be a good idea to adjust the capture settings in the first place

I can't even figure out how to trim a video in avisynth so I can do shorter encodes for tests, I'm inserting trim(226614,230000) and all I get is a blue screen. I'm pretty sure I followed the avisynth wiki example correctly too...this stuff is insanely comprehensive.

Yes, that should return frames 226614 to 230000 . Blue screen suggests some larger problem like memory corruption or HW issue. Try commenting out all the other filters first . For commenting out just add a "#" to the beginning of the line and avisynth will ignore it. It's like a "rem" statement. When debugging avisynth scripts , start as simple as possible then start adding stuff back. Even for QTGMC you can start with a faster preset for preview purposes

e.g
AVISource("whatever.avi")
trim(226614,230000)

[CZbwoi]

Yes, that should return frames 226614 to 230000 . Blue screen suggests some larger problem like memory corruption or HW issue. Try commenting out all the other filters first . For commenting out just add a "#" to the beginning of the line and avisynth will ignore it. It's like a "rem" statement. When debugging avisynth scripts , start as simple as possible then start adding stuff back. Even for QTGMC you can start with a faster preset for preview purposes

e.g
AVISource("whatever.avi")
trim(226614,230000)

I fixed it, a darn number was off at the start of them two and the frames didn't exist -_- (also I didn't mean BSOD when I said that, I meant the preview window in AvsPmod turned blue)

Yes, you can check with avsmeter, and it will report the active MT mode as 0. But for some reason, I don't even need to use it, at least not on this script

Well, this is weird. I took out all the MT lines out of the script (apart from SetMTMode() at the start) and removed SetMemoryMax and Distributor() as well, saved the script and am running the trimmed portion I did through AVSMeter, it's showing me Active MT Mode: 2.

Add histogram() and it will show you the waveform, but turned on it's side. If you "fix" the levels in this shot you will "see" the details in the window and some in the shadows. But it might be a good idea to adjust the capture settings in the first place

I see the waveform appeared but I don't know how to decipher it atm and what you mean by Y<16 , Y>235, I'll have to look up guides or ask in another thread. Fix them with the tweak option though, right?

[Groucho2004]

Yes, that should return frames 226614 to 230000 . Blue screen suggests some larger problem like memory corruption or HW issue.

I don't think he's talking about a BSOD. Maybe the frame range specified is outside the script's range.

@CZbwoi
We were posting at the same time but it seems I was right.