VirtualDub: Out of Bounds Memory Access Violation at about 2GB of RAM

[robjv1]

First off, I recently got a new system -- I've got a quad core machine with the x64 version of Windows 7 Professional and 8GB of RAM. I've having a problem with VDub 1.9.11 crashing that I've outlined below.

The issue is that whenever the memory usage of VirtualDub climbs over about 1,720,000k, it gives me the dreaded "Out Of Bounds Memory Access Violation".

It doesn't matter whether I am working in VDub alone (with filters, without them) or frameserving from Avisynth. If whatever VDub is doing crosses that memory threshold, it crashes.

In common practice, it often occurs an hour or so into processing a very long file using NeatVideo (typically one that is about three hours long) or after just a few seconds when trying to dejitter a video using QTGMC and DePan filters (the memory usage starts at about a gig of utilization with that particular script and just climbs from there quickly). All of the files being processed are captured using either huffyuv or lagarith. The crash always occurs in either the lagarith or huffyuv module (whatever it was captured in) according to the crash report, always in the same place (at memory utilization approaching 1.72GB). This occurs whether I am exporting the file to any codec, including as an uncompressed AVI, MPEG2, huffyuv, or lagarith.

I've had no other problems or errors aside from that.

I've messed around a bit with running it in various different compatibility modes but that doesn't seem to do the trick. One thing I've noticed about VirtualDub in Windows 7 is that the memory utilization just climbs constantly, it never seems to return memory back to the pool, even after it is done encoding. For example, if I used NeatVideo to process three one hour files, it'll eventually reach 1.72GB of utilization and crash on the third file -- if I close VDub and re-open it in between files, no problems.

Here is a crash report as an example, in this case a file compressed with Lagarith and using an avisynth script.

VirtualDub crash report -- build 32842 (release)
--------------------------------------

Disassembly:
02433420: 0f8c6fffffff jl 02433395
02433426: e9defeffff jmp 02433309
0243342b: 33c0 xor eax, eax
0243342d: 5f pop edi
0243342e: 5e pop esi
0243342f: 5b pop ebx
02433430: c9 leave
02433431: c21000 ret 0010
02433434: 55 push ebp
02433435: 8bec mov ebp, esp
02433437: 83ec4c sub esp, 4ch
0243343a: 33d2 xor edx, edx
0243343c: 2155f8 and [ebp-08h], edx
0243343f: 56 push esi
02433440: c745fc80000000 mov dword ptr [ebp-04h], 00000080
02433447: c745d401000000 mov dword ptr [ebp-2ch], 00000001
0243344e: c745d802000000 mov dword ptr [ebp-28h], 00000002
02433455: c745dc03000000 mov dword ptr [ebp-24h], 00000003
0243345c: c745e005000000 mov dword ptr [ebp-20h], 00000005
02433463: c745e408000000 mov dword ptr [ebp-1ch], 00000008
0243346a: c745e80d000000 mov dword ptr [ebp-18h], 0000000d
02433471: c745ec15000000 mov dword ptr [ebp-14h], 00000015
02433478: c745f022000000 mov dword ptr [ebp-10h], 00000022
0243347f: 57 push edi
02433480: 33c0 xor eax, eax
02433482: 2145f4 and [ebp-0ch], eax
02433485: 33f6 xor esi, esi
02433487: 8d4db4 lea ecx, [ebp-4ch]
0243348a: 8d7dd4 lea edi, [ebp-2ch]
0243348d: 837df400 cmp dword ptr [ebp-0ch], 00h
02433491: 7404 jz 02433497
02433493: 85c0 test eax, eax
02433495: 752c jnz 024334c3
02433497: 8945f4 mov [ebp-0ch], eax
0243349a: 8b4508 mov eax, [ebp+08h]
0243349d: 0fb60402 movzx eax, byte ptr [edx+eax]
024334a1: 2345fc and eax, [ebp-04h]
024334a4: d17dfc sar dword ptr [ebp-04h], 1
024334a7: 7508 jnz 024334b1
024334a9: 42 inc edx
024334aa: c745fc80000000 mov dword ptr [ebp-04h], 00000080
024334b1: 85c0 test eax, eax
024334b3: 7408 jz 024334bd
024334b5: 837df400 cmp dword ptr [ebp-0ch], 00h
024334b9: 7502 jnz 024334bd
024334bb: 0337 add esi, [edi] <-- FAULT
024334bd: 41 inc ecx
024334be: 83c704 add edi, 04h
024334c1: ebca jmp 0243348d
024334c3: 33c9 xor ecx, ecx
024334c5: 4e dec esi
024334c6: 41 inc ecx
024334c7: 85f6 test esi, esi
024334c9: 7e21 jle 024334ec
024334cb: 8b4508 mov eax, [ebp+08h]
024334ce: 0fb60402 movzx eax, byte ptr [edx+eax]
024334d2: 2345fc and eax, [ebp-04h]
024334d5: d17dfc sar dword ptr [ebp-04h], 1
024334d8: 7508 jnz 024334e2
024334da: 42 inc edx
024334db: c745fc80000000 mov dword ptr [ebp-04h], 00000080
024334e2: 03c9 add ecx, ecx
024334e4: 85c0 test eax, eax
024334e6: 7401 jz 024334e9
024334e8: 41 inc ecx
024334e9: 4e dec esi
024334ea: 75df jnz 024334cb
024334ec: 8b75f8 mov esi, [ebp-08h]
024334ef: 8b450c mov eax, [ebp+0ch]
024334f2: 49 dec ecx
024334f3: ff45f8 inc dword ptr [ebp-08h]
024334f6: 817df800010000 cmp dword ptr [ebp-08h], 00000100
024334fd: 890cb0 mov [eax+esi*4], ecx
02433500: 0f8c7affffff jl 02433480
02433506: 5f pop edi
02433507: 8d4201 lea eax, [edx+01h]
0243350a: 5e pop esi
0243350b: c9 leave
0243350c: c20800 ret 0008
0243350f: 55 push ebp
02433510: 8bec mov ebp, esp
02433512: 83ec10 sub esp, 10h
02433515: ff4508 inc dword ptr [ebp+08h]
02433518: 8b4508 mov eax, [ebp+08h]
0243351b: ff4508 inc dword ptr [ebp+08h]
0243351e: 53 push ebx
0243351f: 0f db 0fh

Built on Aegis on Fri Dec 24 13:18:44 2010 using compiler version 1400

Windows 6.1 (Windows Vista x64 build 7601) [Service Pack 1]

EAX = 00000001
EBX = 1a76e0e0
ECX = 1a76d92b
EDX = 0000019e
EBP = 1a76cc88
ESI = 00a74a2c
EDI = 1a770018
ESP = 1a76cc34
EIP = 024334bb
EFLAGS = 00010246
FPUCW = 027f
FPUTW = ffff

Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'lagarith'...

...reading address 1A770018.

Pointer dumps:

EBX 1a76e0e0: 00000000 46df0020 000002e0 000002d0 000001e0 46e5bd20 00000170 00000168
ECX 1a76d92b: 0024cf00 85642000 00017038 00016800 d883f000 00000106 d887f000 76d96006
ESI 00a74a28: 00000000 00000000 c07a5010 fffff900 00000f60 40059205 00000000 00000000
ESP 1a76cc30: 03778011 03778011 03860030 01010101 01010101 01010101 01010101 01010101
1a76cc50: 01010101 01010101 01010101 00000001 00000002 00000003 00000005 00000008
1a76cc70: 0000000d 00000015 00000022 00000000 00000000 00000080 1a76e0c4 0243380c
1a76cc90: 1a76cca8 1a76dca8 03778011 03860030 03993760 01010101 01010101 01010101
EBP 1a76cc88: 1a76e0c4 0243380c 1a76cca8 1a76dca8 03778011 03860030 03993760 01010101
1a76cca8: 01010101 01010101 01010101 01010101 01010101 01010101 01010101 01010101
1a76ccc8: 01010101 01010101 01010101 01010101 01010101 01010101 01010101 01010101
1a76cce8: 01010101 01010101 01010101 01010101 01010101 01010101 01010101 01010101

Thread call stack:
024334bb: lagarith!000234bb
0243380c: lagarith!0002380c
7760e38c: ntdll!RtlInitUnicodeString [775e0000+2e228+164]
06c81101: mt_masktools-25!000a1101
06c16ce8: mt_masktools-25!00036ce8
06c04482: mt_masktools-25!00024482
7760e023: ntdll!RtlFreeHeap [775e0000+2dfa5+7e]
76a914dd: kernel32!HeapFree [76a80000+114c9+14]
06c71f7f: mt_masktools-25!00091f7f
06c71f9e: mt_masktools-25!00091f9e
06c71f9e: mt_masktools-25!00091f9e
06c0a3e6: mt_masktools-25!0002a3e6
06c0f2a0: mt_masktools-25!0002f2a0
06c0611c: mt_masktools-25!0002611c
06c0c86f: mt_masktools-25!0002c86f
7760e38c: ntdll!RtlInitUnicodeString [775e0000+2e228+164]
06c81101: mt_masktools-25!000a1101
06c16ce8: mt_masktools-25!00036ce8
06bfecea: mt_masktools-25!0001ecea
7760e023: ntdll!RtlFreeHeap [775e0000+2dfa5+7e]
76a914dd: kernel32!HeapFree [76a80000+114c9+14]
06c71f7f: mt_masktools-25!00091f7f
06c71f9e: mt_masktools-25!00091f9e
7760e38c: ntdll!RtlInitUnicodeString [775e0000+2e228+164]
06c81101: mt_masktools-25!000a1101
06c16ce8: mt_masktools-25!00036ce8
06bfecea: mt_masktools-25!0001ecea
7760e023: ntdll!RtlFreeHeap [775e0000+2dfa5+7e]
76a914dd: kernel32!HeapFree [76a80000+114c9+14]
06c71f7f: mt_masktools-25!00091f7f
06c71f9e: mt_masktools-25!00091f9e
06c71f9e: mt_masktools-25!00091f9e
06c0a3e6: mt_masktools-25!0002a3e6
06c0f2a0: mt_masktools-25!0002f2a0
06c0611c: mt_masktools-25!0002611c
06c0ce6f: mt_masktools-25!0002ce6f
06c81101: mt_masktools-25!000a1101
06c16ce8: mt_masktools-25!00036ce8
06bfecea: mt_masktools-25!0001ecea
100045ae: AviSynth!000045ae
10004631: AviSynth!00004631
1000187d: AviSynth!0000187d
10005183: AviSynth!00005183
06c81101: mt_masktools-25!000a1101
06c16ce8: mt_masktools-25!00036ce8
06bee772: mt_masktools-25!0000e772
7760e023: ntdll!RtlFreeHeap [775e0000+2dfa5+7e]
76a914dd: kernel32!HeapFree [76a80000+114c9+14]
06c71f7f: mt_masktools-25!00091f7f
06c71f9e: mt_masktools-25!00091f9e
06c71f9e: mt_masktools-25!00091f9e
06c0a3e6: mt_masktools-25!0002a3e6
024337c6: lagarith!000237c6
0243397e: lagarith!0002397e
06c0f2a0: mt_masktools-25!0002f2a0
06c0611c: mt_masktools-25!0002611c
06c0c86f: mt_masktools-25!0002c86f
06c81101: mt_masktools-25!000a1101
06c16ce8: mt_masktools-25!00036ce8
06bfecea: mt_masktools-25!0001ecea
7760e023: ntdll!RtlFreeHeap [775e0000+2dfa5+7e]
76a914dd: kernel32!HeapFree [76a80000+114c9+14]
06c71f7f: mt_masktools-25!00091f7f
06c71f9e: mt_masktools-25!00091f9e
06c71f9e: mt_masktools-25!00091f9e
06c0a3e6: mt_masktools-25!0002a3e6
06c0f2a0: mt_masktools-25!0002f2a0
06c0611c: mt_masktools-25!0002611c
06c0ce6f: mt_masktools-25!0002ce6f
06c81101: mt_masktools-25!000a1101
06c16ce8: mt_masktools-25!00036ce8
775ff8e5: ntdll!NtReadFile [775e0000+1f8d0+15]
7540d348: KERNELBASE!ReadFile [75400000+d230+118]
7540d39a: KERNELBASE!ReadFile [75400000+d230+16a]
0241356c: lagarith!0000356c
024303c5: lagarith!000203c5
024306ad: lagarith!000206ad
024341a2: lagarith!DriverProc [02410000+24050+152]
736d1759: MSVFW32!ICSendMessage [736d0000+1728+31]
736d4e27: MSVFW32!ICDecompress [736d0000+4dea+3d]
1007aabb: AviSynth!DllGetClassObject [10000000+da50+6d06b]
1007b364: AviSynth!DllGetClassObject [10000000+da50+6d914]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1008357b: AviSynth!DllGetClassObject [10000000+da50+75b2b]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
100372e6: AviSynth!DllGetClassObject [10000000+da50+29896]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
1003a930: AviSynth!DllGetClassObject [10000000+da50+2cee0]
1005627b: AviSynth!DllGetClassObject [10000000+da50+4882b]
06c16ce8: mt_masktools-25!00036ce8
1003af1b: AviSynth!DllGetClassObject [10000000+da50+2d4cb]
100372e6: AviSynth!DllGetClassObject [10000000+da50+29896]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10040778: AviSynth!DllGetClassObject [10000000+da50+32d28]
10009559: AviSynth!avs_release_value [10000000+75d0+1f89]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]

-- End of report

[jman98]

Barring a better suggestion, maybe you should download and try the beta version for x64. I have seen software where the beta version fixed horrible bugs in the "stable" release when run under x64.

[robjv1]

^^ Yeah I would defenitely consider that as a last resort, although I'm not sure if all of the plugins I use have 64 bit versions available.

[jman98]

Sorry - I misread the info under the Tools section and thought based on the release number that you were using the x64 version but I see now that your version is also available in 32 bit and that is presumably what you are using. In that case you may wish to try using the beta 32 bit version.

[robjv1]

Yeah, I did try that as well. It does seem to go a little longer with the newest beta, but it still crashes eventually. Very frustrating!

Edit:

One interesting thing to note -- I tried using the "Background I/O" option. VirtualDub immediately drops from using over 1GB of memory to about 32MB and hovers in that range. Turn the option off and it shoots back up to over a gig and climbs fast. I'll let it process with Background I/O on and see if it crashes or not now.

[robjv1]

Nope, still crashing.

VirtualDub crash report -- build 34807 (release)
--------------------------------------

Disassembly:
02ab3420: 0f8c6fffffff jl 02ab3395
02ab3426: e9defeffff jmp 02ab3309
02ab342b: 33c0 xor eax, eax
02ab342d: 5f pop edi
02ab342e: 5e pop esi
02ab342f: 5b pop ebx
02ab3430: c9 leave
02ab3431: c21000 ret 0010
02ab3434: 55 push ebp
02ab3435: 8bec mov ebp, esp
02ab3437: 83ec4c sub esp, 4ch
02ab343a: 33d2 xor edx, edx
02ab343c: 2155f8 and [ebp-08h], edx
02ab343f: 56 push esi
02ab3440: c745fc80000000 mov dword ptr [ebp-04h], 00000080
02ab3447: c745d401000000 mov dword ptr [ebp-2ch], 00000001
02ab344e: c745d802000000 mov dword ptr [ebp-28h], 00000002
02ab3455: c745dc03000000 mov dword ptr [ebp-24h], 00000003
02ab345c: c745e005000000 mov dword ptr [ebp-20h], 00000005
02ab3463: c745e408000000 mov dword ptr [ebp-1ch], 00000008
02ab346a: c745e80d000000 mov dword ptr [ebp-18h], 0000000d
02ab3471: c745ec15000000 mov dword ptr [ebp-14h], 00000015
02ab3478: c745f022000000 mov dword ptr [ebp-10h], 00000022
02ab347f: 57 push edi
02ab3480: 33c0 xor eax, eax
02ab3482: 2145f4 and [ebp-0ch], eax
02ab3485: 33f6 xor esi, esi
02ab3487: 8d4db4 lea ecx, [ebp-4ch]
02ab348a: 8d7dd4 lea edi, [ebp-2ch]
02ab348d: 837df400 cmp dword ptr [ebp-0ch], 00h
02ab3491: 7404 jz 02ab3497
02ab3493: 85c0 test eax, eax
02ab3495: 752c jnz 02ab34c3
02ab3497: 8945f4 mov [ebp-0ch], eax
02ab349a: 8b4508 mov eax, [ebp+08h]
02ab349d: 0fb60402 movzx eax, byte ptr [edx+eax]
02ab34a1: 2345fc and eax, [ebp-04h]
02ab34a4: d17dfc sar dword ptr [ebp-04h], 1
02ab34a7: 7508 jnz 02ab34b1
02ab34a9: 42 inc edx
02ab34aa: c745fc80000000 mov dword ptr [ebp-04h], 00000080
02ab34b1: 85c0 test eax, eax
02ab34b3: 7408 jz 02ab34bd
02ab34b5: 837df400 cmp dword ptr [ebp-0ch], 00h
02ab34b9: 7502 jnz 02ab34bd
02ab34bb: 0337 add esi, [edi] <-- FAULT
02ab34bd: 41 inc ecx
02ab34be: 83c704 add edi, 04h
02ab34c1: ebca jmp 02ab348d
02ab34c3: 33c9 xor ecx, ecx
02ab34c5: 4e dec esi
02ab34c6: 41 inc ecx
02ab34c7: 85f6 test esi, esi
02ab34c9: 7e21 jle 02ab34ec
02ab34cb: 8b4508 mov eax, [ebp+08h]
02ab34ce: 0fb60402 movzx eax, byte ptr [edx+eax]
02ab34d2: 2345fc and eax, [ebp-04h]
02ab34d5: d17dfc sar dword ptr [ebp-04h], 1
02ab34d8: 7508 jnz 02ab34e2
02ab34da: 42 inc edx
02ab34db: c745fc80000000 mov dword ptr [ebp-04h], 00000080
02ab34e2: 03c9 add ecx, ecx
02ab34e4: 85c0 test eax, eax
02ab34e6: 7401 jz 02ab34e9
02ab34e8: 41 inc ecx
02ab34e9: 4e dec esi
02ab34ea: 75df jnz 02ab34cb
02ab34ec: 8b75f8 mov esi, [ebp-08h]
02ab34ef: 8b450c mov eax, [ebp+0ch]
02ab34f2: 49 dec ecx
02ab34f3: ff45f8 inc dword ptr [ebp-08h]
02ab34f6: 817df800010000 cmp dword ptr [ebp-08h], 00000100
02ab34fd: 890cb0 mov [eax+esi*4], ecx
02ab3500: 0f8c7affffff jl 02ab3480
02ab3506: 5f pop edi
02ab3507: 8d4201 lea eax, [edx+01h]
02ab350a: 5e pop esi
02ab350b: c9 leave
02ab350c: c20800 ret 0008
02ab350f: 55 push ebp
02ab3510: 8bec mov ebp, esp
02ab3512: 83ec10 sub esp, 10h
02ab3515: ff4508 inc dword ptr [ebp+08h]
02ab3518: 8b4508 mov eax, [ebp+08h]
02ab351b: ff4508 inc dword ptr [ebp+08h]
02ab351e: 53 push ebx
02ab351f: 0f db 0fh

Built on Althena on Sun Jun 03 14:28:23 2012 using compiler version 1400

Windows 6.1 (Windows 7 x64 build 7601) [Service Pack 1]
Memory status: virtual free 356M/2048M, commit limit 16365M, physical total 8183M

EAX = 00000001
EBX = 1e21dea0
ECX = 1e21d77b
EDX = 000001b0
EBP = 1e21ca48
ESI = 88884521
EDI = 1e220018
ESP = 1e21c9f4
EIP = 02ab34bb
EFLAGS = 00010246
FPUCW = 027f
FPUTW = ffff

Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'lagarith'...

...reading address 1E220018.

Pointer dumps:

EBX 1e21dea0: 00000000 00000000 0a161208 1e21deec 04061f7f 043e0000 00000000 04061f9e
ECX 1e21d77b: 01010101 01010101 01010101 01010101 01010101 01010101 01010101 01010101
ESP 1e21c9f0: 01010101 4301cc29 04ac0030 01010101 01010101 01010101 01010101 01010101
1e21ca10: 01010101 01010101 01010101 00000001 00000002 00000003 00000005 00000008
1e21ca30: 0000000d 00000015 00000022 00000000 00000000 00000080 1e21de84 02ab380c
1e21ca50: 1e21ca68 1e21da68 4301cc29 04ac0030 02d95958 01010101 01010101 01010101
EBP 1e21ca48: 1e21de84 02ab380c 1e21ca68 1e21da68 4301cc29 04ac0030 02d95958 01010101
1e21ca68: 01010101 01010101 01010101 01010101 01010101 01010101 01010101 01010101
1e21ca88: 01010101 01010101 01010101 01010101 01010101 01010101 01010101 01010101
1e21caa8: 01010101 01010101 01010101 01010101 01010101 01010101 01010101 01010101

Thread call stack:
02ab34bb: lagarith!000234bb
02ab380c: lagarith!0002380c
774ce38c: ntdll!RtlInitUnicodeString [774a0000+2e228+164]
04071101: mt_masktools-25!000a1101
04006ce8: mt_masktools-25!00036ce8
03feecea: mt_masktools-25!0001ecea
774ce023: ntdll!RtlFreeHeap [774a0000+2dfa5+7e]
763414dd: kernel32!HeapFree [76330000+114c9+14]
04061f7f: mt_masktools-25!00091f7f
04061f9e: mt_masktools-25!00091f9e
774ce38c: ntdll!RtlInitUnicodeString [774a0000+2e228+164]
04071101: mt_masktools-25!000a1101
774ce38c: ntdll!RtlInitUnicodeString [774a0000+2e228+164]
04071101: mt_masktools-25!000a1101
04006ce8: mt_masktools-25!00036ce8
03feecea: mt_masktools-25!0001ecea
774ce023: ntdll!RtlFreeHeap [774a0000+2dfa5+7e]
763414dd: kernel32!HeapFree [76330000+114c9+14]
04061f7f: mt_masktools-25!00091f7f
04061f9e: mt_masktools-25!00091f9e
774ce38c: ntdll!RtlInitUnicodeString [774a0000+2e228+164]
04071101: mt_masktools-25!000a1101
04006ce8: mt_masktools-25!00036ce8
03feecea: mt_masktools-25!0001ecea
774ce023: ntdll!RtlFreeHeap [774a0000+2dfa5+7e]
763414dd: kernel32!HeapFree [76330000+114c9+14]
04061f7f: mt_masktools-25!00091f7f
04061f9e: mt_masktools-25!00091f9e
04061f9e: mt_masktools-25!00091f9e
03ffa3e6: mt_masktools-25!0002a3e6
03fff2a0: mt_masktools-25!0002f2a0
03ff611c: mt_masktools-25!0002611c
03ffce6f: mt_masktools-25!0002ce6f
04071101: mt_masktools-25!000a1101
04006ce8: mt_masktools-25!00036ce8
03feecea: mt_masktools-25!0001ecea
1000190b: AviSynth!0000190b
100045ae: AviSynth!000045ae
10004631: AviSynth!00004631
1000187d: AviSynth!0000187d
10005183: AviSynth!00005183
10005337: AviSynth!00005337
10005407: AviSynth!00005407
04071101: mt_masktools-25!000a1101
04006ce8: mt_masktools-25!00036ce8
03fde772: mt_masktools-25!0000e772
02ab37c6: lagarith!000237c6
02ab3948: lagarith!00023948
04061f7f: mt_masktools-25!00091f7f
04061f9e: mt_masktools-25!00091f9e
04061f9e: mt_masktools-25!00091f9e
03ffa3e6: mt_masktools-25!0002a3e6
03fff2a0: mt_masktools-25!0002f2a0
03ff611c: mt_masktools-25!0002611c
03ffc86f: mt_masktools-25!0002c86f
04071101: mt_masktools-25!000a1101
04006ce8: mt_masktools-25!00036ce8
03feecea: mt_masktools-25!0001ecea
774ce023: ntdll!RtlFreeHeap [774a0000+2dfa5+7e]
763414dd: kernel32!HeapFree [76330000+114c9+14]
04061f7f: mt_masktools-25!00091f7f
04061f9e: mt_masktools-25!00091f9e
04061f9e: mt_masktools-25!00091f9e
03ffa3e6: mt_masktools-25!0002a3e6
03fff2a0: mt_masktools-25!0002f2a0
774bf8e5: ntdll!NtReadFile [774a0000+1f8d0+15]
76f8d348: KERNELBASE!ReadFile [76f80000+d230+118]
76f8d39a: KERNELBASE!ReadFile [76f80000+d230+16a]
02a9356c: lagarith!0000356c
02ab03c5: lagarith!000203c5
02ab06ad: lagarith!000206ad
02ab41a2: lagarith!DriverProc [02a90000+24050+152]
71011759: MSVFW32!ICSendMessage [71010000+1728+31]
71014e27: MSVFW32!ICDecompress [71010000+4dea+3d]
1007aabb: AviSynth!DllGetClassObject [10000000+da50+6d06b]
1007b364: AviSynth!DllGetClassObject [10000000+da50+6d914]
10009846: AviSynth!avs_release_value [10000000+75d0+2276]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10008cd1: AviSynth!avs_release_value [10000000+75d0+1701]
100839be: AviSynth!DllGetClassObject [10000000+da50+75f6e]
1008357b: AviSynth!DllGetClassObject [10000000+da50+75b2b]
10009846: AviSynth!avs_release_value [10000000+75d0+2276]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10008cd1: AviSynth!avs_release_value [10000000+75d0+1701]
100372e6: AviSynth!DllGetClassObject [10000000+da50+29896]
10009846: AviSynth!avs_release_value [10000000+75d0+2276]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10008cd1: AviSynth!avs_release_value [10000000+75d0+1701]
1003a930: AviSynth!DllGetClassObject [10000000+da50+2cee0]
1005627b: AviSynth!DllGetClassObject [10000000+da50+4882b]
1003af1b: AviSynth!DllGetClassObject [10000000+da50+2d4cb]
100372e6: AviSynth!DllGetClassObject [10000000+da50+29896]
10009846: AviSynth!avs_release_value [10000000+75d0+2276]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10008cd1: AviSynth!avs_release_value [10000000+75d0+1701]
10040778: AviSynth!DllGetClassObject [10000000+da50+32d28]
10009846: AviSynth!avs_release_value [10000000+75d0+2276]
1000adb7: AviSynth!avs_release_value [10000000+75d0+37e7]
10008cd1: AviSynth!avs_release_value [10000000+75d0+1701]
100539d3: AviSynth!DllGetClassObject [10000000+da50+45f83]

-- End of report

[robjv1]

It is interesting to note that it seems to be having some sort of error on the same instruction each time:

024334bb: 0337 add esi, [edi] <-- FAULT

02ab34bb: 0337 add esi, [edi] <-- FAULT

[Selur]

iirc 32bit Avisynth can only address 2GB or memory,...

[jman98]

Others have reported this error with VirtualDub with vary differing "solutions" offered.
What that means is that nobody has any real idea what causes it.

One suggestion on Doom9 is:
disable xvid/divx support in ffdshow

One member here had a solution 2 years that worked for him:
https://forum.videohelp.com/threads/320556-Virtualdub-An-out-of-bounds-memory-access-(a...tion)-occurred

Another website suggested demuxing and remuxing the video source file.

[Selur]

...can result in crashes because of 2GB address space limit.

http://avisynth.org/mediawiki/Internal_functions/Control_functions
+
unless a 32bit application is compiled with IMAGE_FILE_LARGE_ADDRESS_AWARE enabled, it is limited to 2GB RAM, see:
http://msdn.microsoft.com/en-us/library/windows/desktop/aa366778%28v=vs.85%29.aspx

-> if your 32bit Avisynth script needs 2GB RAM, which can easily happen with "QTGMC and DePan filters"depending on your settings and resolution, it will crash.

It is interesting to note that it seems to be having some sort of error on the same instruction each time

that's because the same subroutine of one of the avisynth filters is crashing into the memory limitation,...

-> probably the best solution to this is to switch to 64bit Avisynth, which might lead to another downfall since last time I checked, not all filters are available for Avisynth 64bit and it (or the filters) tended to crash without a good reason,...
ugly solution would be to:
1st run qtgmc and create a temporary file
2nd run DePan on the temporary file

Cu Selur

[robjv1]

Thanks for the tip and the links guys, I'll give it a try! Sadly I think QTGMC is enough alone to cause the error, even without DePan run as a separate process on an intermediary file, but I'll give it a shot.

Yeah the memory thing was something I wondered about, but I guess I have always been confused about how that works when comparing say a 32-bit version of Windows XP and a 32-bit application with a 64-bit version of Windows and a 32 bit application.

Just as a note, as I stated in my overly long explanation, this is not an AviSynth issue exclusively -- it is an issue with VirtualDub. It occurs when running VDub in standalone mode with the NeatVideo plugin as well (also 32-bit though).