VirtualDubMod crashes at certain output resolutions

[Kylenol]

I'm using StaxRip (the latest beta version) to encode on BootCamp (I have Windows Vista Ultimate SP1 32-bit installed, btw). And for some reason, my encoding will freeze on the at the start of 2nd pass IF the output resolution is for instance 512x384.However, if I encode the output at a higher output resolution such as 624x352 to encode HD, it will encode successfully. I've tried re-installing XviD, VirtualDubMod, AVISynth and installed older versions of StaxRip but nothing will help. I never had this problem on my older PC on Windows XP SP3. I also don't have a problem with this using a different XviD build such as xvid_encraw using MeGUI.

The only way I it SEEMS to work better is when I use an older version of VirtualDubMod, but even then at the end of 2nd pass it will freeze at 99%. If I use the current version of VirtualDubMod, and clear the stats file from another encode, I have found that it works in the same way as an older version of VirtualDubMod would.

I would like to encode successfully, without any crashes!

[Ai Haibara]

Which version of Xvid are you using?

(Anyone else reading this, I can't remember - does ffdshow's version of Xvid (with or without using Xvid's dlls) allow for two-pass encoding?)

[Kylenol]

Which version of Xvid are you using?

(Anyone else reading this, I can't remember - does ffdshow's version of Xvid (with or without using Xvid's dlls) allow for two-pass encoding?)

XviD 1.2.2 Final.

[Kylenol]

I used VirtualDub this time to encode, but it crashed at 100% on 2nd pass, here's the crash log:

VirtualDub crash report -- build 28346 (release)
--------------------------------------

Disassembly:
772566a0: c3              ret    
772566a1: 07              pop    es
772566a2: 0f8571950000    jnz    7725fc19
772566a8: 8d43f8          lea    eax, [ebx-08h]
772566ab: 80780705        cmp    byte ptr [eax+07h], 05h
772566af: 0f844d950000    jz     7725fc02
772566b5: f640073f        test   [eax+07h], 3fh
772566b9: 0f8451950000    jz     7725fc10
772566bf: 3bc7            cmp    eax, edi
772566c1: 894510          mov    [ebp+10h], eax
772566c4: 0f84b9950000    jz     7725fc83
772566ca: 807bff05        cmp    byte ptr [ebx-01h], 05h
772566ce: 0f8458950000    jz     7725fc2c
772566d4: 8b4510          mov    eax, [ebp+10h]
772566d7: f6400780        test   [eax+07h], 80h
772566db: 0f84530e0000    jz     77257534
772566e1: 8bd3            mov    edx, ebx
772566e3: 8bce            mov    ecx, esi
772566e5: e823000000      call   7725670d
772566ea: 84c0            test   al, al
772566ec: 0f84420e0000    jz     77257534
772566f2: 803d8003fe7f00  cmp    byte ptr [7ffe0380], 00h
772566f9: 0f85ad950000    jnz    7725fcac
772566ff: b001            mov    al, 01h
77256701: 5e              pop    esi
77256702: 5f              pop    edi
77256703: 5b              pop    ebx
77256704: 5d              pop    ebp
77256705: c20c00          ret    000c
77256708: 90              nop    
77256709: 90              nop    
7725670a: 90              nop    
7725670b: 90              nop    
7725670c: 90              nop    
7725670d: 8bff            mov    edi, edi
7725670f: 55              push   ebp
77256710: 8bec            mov    ebp, esp
77256712: 83ec28          sub    esp, 28h
77256715: 53              push   ebx
77256716: 56              push   esi
77256717: 57              push   edi
77256718: 8d7af8          lea    edi, [edx-08h]
7725671b: 807f0705        cmp    byte ptr [edi+07h], 05h
7725671f: 0f8415c20100    jz     7727293a
77256725: 8b07            mov    eax, [edi]
77256727: 8bf7            mov    esi, edi
77256729: c1ee03          shr    esi, 03h
7725672c: 33f0            xor    esi, eax
7725672e: 333544422b77    xor    esi, [772b4244]
77256734: 897dfc          mov    [ebp-04h], edi
77256737: 33f1            xor    esi, ecx
77256739: 8b4604          mov    eax, [esi+04h]      <-- FAULT
7725673c: 8945f4          mov    [ebp-0ch], eax
7725673f: c6470780        mov    byte ptr [edi+07h], 80h
77256743: c6470600        mov    byte ptr [edi+06h], 00h
77256747: 8b5e08          mov    ebx, [esi+08h]
7725674a: 8b4e0c          mov    ecx, [esi+0ch]
7725674d: 895de0          mov    [ebp-20h], ebx
77256750: 83c301          add    ebx, 01h
77256753: 894de4          mov    [ebp-1ch], ecx
77256756: 83d101          adc    ecx, 01h
77256759: 81e3ff7f0000    and    ebx, 00007fff
7725675f: 663b5e14        cmp    ebx, [esi+14h]
77256763: 0f84b3040000    jz     77256c1c
77256769: 8b45e0          mov    eax, [ebp-20h]
7725676c: 8b55e4          mov    edx, [ebp-1ch]
7725676f: 0facd010        shrd   eax, edx, 10h
77256773: 66894708        mov    [edi+08h], ax
77256777: 2b7df4          sub    edi, [ebp-0ch]
7725677a: 33c0            xor    eax, eax
7725677c: c1ef03          shr    edi, 03h
7725677f: 0fa4f810        shld   eax, edi, 10h
77256783: c1e710          shl    edi, 10h
77256786: c1ea10          shr    edx, 10h
77256789: 0bdf            or     ebx, edi
7725678b: 0bc8            or     ecx, eax
7725678d: 8b06            mov    eax, [esi]
7725678f: c745f801000000  mov    dword ptr [ebp-08h], 00000001
77256796: 8930            mov    [eax], esi
77256798: 8b45e0          mov    eax, [ebp-20h]
7725679b: 8b55e4          mov    edx, [ebp-1ch]
7725679e: 8d              db     8dh
7725679f: 7e              db     7eh

Built on KOS-MOS on Tue Feb 12 22:09:24 2008 using compiler version 1400

Windows 6.0 (Windows Vista x86 build 6002) [Service Pack 2]

EAX = 81818181
EBX = 086d4900
ECX = 07f50000
EDX = 086d4900
EBP = 0012deac
ESI = b9187969
EDI = 086d48f8
ESP = 0012de78
EIP = 77256739
EFLAGS = 00210286
FPUCW = ffff027f
FPUTW = ffffffff

Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'ntdll'...

...reading address B918796D.

Pointer dumps:

EBX   086d4900: 07f500c4 085b72d0 00000000 00000000 00000000 00000000 00000000 00000000
ECX   07f50000: b93dc2a0 0100b47c ffeeffee 00000000 07f80010 07f500a8 07f50000 07f50000
EDX   086d4900: 07f500c4 085b72d0 00000000 00000000 00000000 00000000 00000000 00000000
EDI   086d48f8: 81818181 82824563 07f500c4 085b72d0 00000000 00000000 00000000 00000000
ESP   0012de78: 00000000 07f50000 086d4900 07f56f08 01010118 0012ddc4 00000000 0012df0c
      0012de98: 771f99fa 00c1bbdd fffffffe 77257512 086d48f8 0012dec0 772566ea 086d4900
      0012deb8: 07f55e00 07f55d40 0012ded4 76529a26 07f50000 00000000 086d48f8 0012df1c
      0012ded8: 07e884ca 07f50000 00000000 086d4900 07f55e00 ffffd800 07f55d40 07f563f0
EBP   0012dea8: 086d48f8 0012dec0 772566ea 086d4900 07f55e00 07f55d40 0012ded4 76529a26
      0012dec8: 07f50000 00000000 086d48f8 0012df1c 07e884ca 07f50000 00000000 086d4900
      0012dee8: 07f55e00 ffffd800 07f55d40 07f563f0 ffffd800 00000002 07f55d40 07f55d40
      0012df08: 0012da9c 0012ee10 07e88c20 07eb70f8 ffffffff 07f56600 07e3c448 086d4900

Thread call stack:
77256739: ntdll!RtlFreeHeap [771f0000+66670+c9]
77257512: ntdll!RtlEnumerateGenericTableWithoutSplaying [771f0000+66dc2+750]
772566ea: ntdll!RtlFreeHeap [771f0000+66670+7a]
76529a26: kernel32!HeapFree [764e0000+49a12+14]
07e884ca: xvidcore!xvid_plugin_dump [07e10000+6bff4+c4d6]
07e3c448: xvidcore!xvid_encore [07e10000+2acc8+1780]
07e3b1d0: xvidcore!xvid_encore [07e10000+2acc8+508]
07e3af34: xvidcore!xvid_encore [07e10000+2acc8+26c]
07e3acf9: xvidcore!xvid_encore [07e10000+2acc8+31]
03b37134: xvidvfw!DriverProc [03b30000+6e04+330]
76c999cb: USER32!GetWindowLongA [76c80000+19994+37]
004ab09d: VDProjectUI::WndProc()
004baa3e: VDUIFrame::StaticWndProc()
7646d60f: USP10!ScriptGetCMap [76460000+b3e4+222b]
005a192c: _fputwc_nolock()
005a0b12: _mbtowc_l()
005a192c: _fputwc_nolock()
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fdf3: USER32!GetWindowLongW [76c80000+1f8bf+534]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
7646ed0b: USP10!ScriptTextOut [76460000+ea95+276]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c99d6a: USER32!ReleaseDC [76c80000+19ced+7d]
76c99d75: USER32!ReleaseDC [76c80000+19ced+88]
76c99d75: USER32!ReleaseDC [76c80000+19ced+88]
76c99f8d: USER32!WCSToMBEx [76c80000+19ec1+cc]
76c99f9d: USER32!WCSToMBEx [76c80000+19ec1+dc]
77255dae: ntdll!KiUserCallbackDispatcher [771f0000+65d80+2e]
76c98e7d: USER32!InflateRect [76c80000+18d4f+12e]
76ca06b8: USER32!IsWindow [76c80000+2067b+3d]
76ca03ab: USER32!DispatchMessageW [76c80000+2021c+18f]
7456af85: uxtheme!GetThemeBool [74560000+a886+6ff]
745625a7: uxtheme!000025a7
745625f4: uxtheme!000025f4
76ca0245: USER32!DispatchMessageW [76c80000+2021c+29]
76ca0454: USER32!DefWindowProcW [76c80000+203b4+a0]
76ca041f: USER32!DefWindowProcW [76c80000+203b4+6b]
76ca041f: USER32!DefWindowProcW [76c80000+203b4+6b]
004ba4c3: VDUIFrame::DefProc()
004abaca: VDProjectUI::DubWndProc()
76c999cb: USER32!GetWindowLongA [76c80000+19994+37]
004ab09d: VDProjectUI::WndProc()
004baa3e: VDUIFrame::StaticWndProc()
7723127c: ntdll!RtlImageNtHeader [771f0000+41261+1b]
7725661f: ntdll!RtlAllocateHeap [771f0000+66570+af]
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
772566ea: ntdll!RtlFreeHeap [771f0000+66670+7a]
76529a26: kernel32!HeapFree [764e0000+49a12+14]
0058bb72: free()
0058bb91: free()
0058bb91: free()
00483fb3: guiSetTitleW()
0044006c: VDFilterGraphControl::RequeryFormats()
76cb280d: USER32!GetClipboardOwner [76c80000+326ef+11e]
76c99560: USER32!DrawTextExW [76c80000+191ce+392]
76c93c42: USER32!GetWindowDC [76c80000+13ba7+9b]
76ca18ce: USER32!IsDialogMessage [76c80000+21847+87]
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fdf3: USER32!GetWindowLongW [76c80000+1f8bf+534]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76ca0943: USER32!CallMsgFilterW [76c80000+20801+142]
76c9f860: USER32!GetClipboardFormatNameA [76c80000+1a552+530e]
76c999bf: USER32!GetWindowLongA [76c80000+19994+2b]
76c999cb: USER32!GetWindowLongA [76c80000+19994+37]
768223a6: MSCTF!TF_GetAppCompatFlags [76820000+2361+45]
00476491: DubStatus::StatusTimerProc()
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c991ec: USER32!DrawTextExW [76c80000+191ce+1e]
76cb193e: USER32!DrawStateW [76c80000+31825+119]
76cb1955: USER32!DrawStateW [76c80000+31825+130]
76cb20ab: USER32!DrawStateW [76c80000+31825+886]
76cb20bd: USER32!DrawStateW [76c80000+31825+898]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c999bf: USER32!GetWindowLongA [76c80000+19994+2b]
00476575: DubStatus::StatusMainDlgProc()
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c94a0b: USER32!ActivateKeyboardLayout [76c80000+1478c+27f]
76c949c8: USER32!ActivateKeyboardLayout [76c80000+1478c+23c]
76c9495f: USER32!ActivateKeyboardLayout [76c80000+1478c+1d3]
76c94a0b: USER32!ActivateKeyboardLayout [76c80000+1478c+27f]
76ca0b5a: USER32!NotifyWinEvent [76c80000+20b4d+d]
728b6b9f: COMCTL32!_TrackMouseEvent [72870000+35a6e+11131]
76c9fd72: USER32!GetWindowLongW [76c80000+1f8bf+4b3]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76c9fdf3: USER32!GetWindowLongW [76c80000+1f8bf+534]
76c9fea1: USER32!GetWindowLongW [76c80000+1f8bf+5e2]
76c9fe61: USER32!GetWindowLongW [76c80000+1f8bf+5a2]
76ca0943: USER32!CallMsgFilterW [76c80000+20801+142]
0046d842: Dubber::UpdateFrames()
76c98e7d: USER32!InflateRect [76c80000+18d4f+12e]

-- End of report

[Ai Haibara]

Wonder if uninstalling Xvid and installing an older version would help. It doesn't crash (just for the sake of argument) if you use a different codec, does it?

Also, not sure about StaxRip, as I don't use it, but if you're just working with stand-alone VirtualDub, maybe AVIDemux would also work.

[Kylenol]

I tried the last two versions of XviD, but it still crashes. I know XviD is to blame so I don't experience crashing at all with different codecs, like x264.

[Ai Haibara]

Which versions? Yeah, I have been suspecting Xvid or how it's interacting, in this case. Did you do a complete uninstall and reboot before installing the other versions?

(Still would like to know if you can do two-pass encoding with ffdshow's version of Xvid... probably can if you tell it to use Xvid's DLLs, but that's not really the point. )

[Kylenol]

I tried the last two, 1.2.1 VAQ and 1.2.1, rebooting in between installs.

[Ai Haibara]

It probably wouldn't hurt to try the 1.1 and 1.0 versions, instead - I'm not sure about the 1.2.x versions.

Another good idea, if you have a crash log from VirtualDub, is to run it past Phaeron (VirtualDub's creator) at the VirtualDub forums. He'd probably be better able to tell what the problem is. (Not sure whether he'd allow sending it through email, or whether he'd be willing to take a look at crash logs from forked versions of VD, like VDubMod.)

[Kylenol]

Strangly enough, it didn't crash when I used XviD-1.1.2.

[Ai Haibara]

Hmm... I've been wary of the 1.2.x builds of Xvid, but I can't remember why.

Maybe 1.2 made some changes to the way two-pass encoding is handled, and there's something about it that VirtualDub doesn't like, I don't know.

[DarrellS]

Kylenol,

Phaeron replied to your post in the Virtualdub Forum. He said that you might try turning off "display encoding status" under XviD, Configuration Settings, Other Options.

Sometimes the status window causes problems. It's annoying having to manually turn it off all the time anyway.

[Kylenol]

Unfortuantely, turning off "display encoding status" didn't prevent it from crashing. In addition to what I said earlier about the older build; XviD-1.1.2, I have to say that the opposite is true in the case of a newer build of XviD. When using 1.1.2, it now only crashes if I encode a HD file at 624x352, but will not crash if I encode a 4:3 source at 512x384.