Virtualdub 'out-of-bounds memory access(access violation) when hardsubbing'

[luisppk]

I'm trying to insert an ass sub into an mkv. I've done it before (last time about 4 months) and had no problems.
Today I tried 4 times, and already having the same issue:
--------------
An out-of-bounds memory access (access violation) occurred in module 'Veedub64'...
...reading address FFFFFFFFB0907D48...
...while running filter "TextSub" (FilterInstance.cpp:2198)...
...while running thread "Scheduler thread" (thread.cpp:197).
---------------

I have: win 7 64 ultimate, with vdub 1.10.4-AMD64, 8gb ram + AMD A8 7600 Radeon R7

As I said before, I did many times the same operation (last time was almost 4 -5 months ago).
I detail here what I'm trying to do:
- Insert ass subtitle in x264 mkv video (with no sound - extracted previously)
- video is x264 with 2 pass... audio is 'no audio'... subtitle is textsub

At 2% of 1st pass, shows message above.

Here is the dumpfile:

--------------------------------------

VirtualDub crash report -- build 35491 (release-AMD64)
--------------------------------------

Disassembly:
7fee7027ec0: 8b8390000000 mov eax, [rbx+90]
7fee7027ec6: 488b8b88000000 mov rcx, [rbx+88]
7fee7027ecd: 03c0 add eax, eax
7fee7027ecf: 4863d0 movsxd rdx, eax
7fee7027ed2: 48899390000000 mov [rbx+90], rdx
7fee7027ed9: 48c1e203 shl rdx, 03h
7fee7027edd: e8eea51700 call e71a24d0
7fee7027ee2: 488bce mov rcx, esi
7fee7027ee5: 48898388000000 mov [rbx+88], rax
7fee7027eec: 488b8398000000 mov rax, [rbx+98]
7fee7027ef3: 482bcf sub ecx, edi
7fee7027ef6: 488d4c0101 lea rcx, [rcx+rax+01h]
7fee7027efb: 483b8b90000000 cmp ecx, [rbx+90]
7fee7027f02: 77bc ja e7027ec0
7fee7027f04: 448b442478 mov r8d, [rsp+78h]
7fee7027f09: 4c8b4c2460 mov r9, [rsp+60h]
7fee7027f0e: 4c8b5c2430 mov r11, [rsp+30h]
7fee7027f13: 442bc5 sub eax, ebp
7fee7027f16: 4963c0 movsxd rax, eax
7fee7027f19: 490fafc3 imul eax, ebx
7fee7027f1d: 48c1f803 sar rax, 03h
7fee7027f21: 483bfe cmp edi, esi
7fee7027f24: 7f65 jg e7027f8b
7fee7027f26: 4d8d8401008000 lea r8, [r9+rax+8000]
00
7fee7027f2e: 6690 nop
7fee7027f30: 488b83a0000000 mov rax, [rbx+a0]
7fee7027f37: 488b9398000000 mov rdx, [rbx+98]
7fee7027f3e: 488b8b88000000 mov rcx, [rbx+88]
7fee7027f45: 8b04f8 mov eax, [rax+rdi*8] <-- FAULT
7fee7027f48: 48ffc7 inc rdi
7fee7027f4b: 8904d1 mov [rcx+rdx*8], eax
7fee7027f4e: 488b8b98000000 mov rcx, [rbx+98]
7fee7027f55: 488b8388000000 mov rax, [rbx+88]
7fee7027f5c: 498bd0 mov rdx, eax
7fee7027f5f: 4d03c3 add eax, ebx
7fee7027f62: 48c1fa10 sar rdx, 10h
7fee7027f66: 03d2 add edx, edx
7fee7027f68: 8954c804 mov [rax+rcx*8+04h], edx
7fee7027f6c: 488b8398000000 mov rax, [rbx+98]
7fee7027f73: 488b8ba0000000 mov rcx, [rbx+a0]
7fee7027f7a: 488944f9f8 mov [rcx+rdi*8-08h], rax
7fee7027f7f: 48ff8398000000 inc qword ptr [rbx+98]
7fee7027f86: 483bfe cmp edi, esi
7fee7027f89: 7ea5 jle e7027f30
7fee7027f8b: 488b5c2468 mov rbx, [rsp+68h]
7fee7027f90: 4883c440 add rsp, 40h
7fee7027f94: 5f pop edi
7fee7027f95: 5e pop esi
7fee7027f96: 5d pop ebp
7fee7027f97: c3 ret
7fee7027f98: cc int 3
7fee7027f99: cc int 3
7fee7027f9a: cc int 3
7fee7027f9b: cc int 3
7fee7027f9c: cc int 3
7fee7027f9d: cc int 3
7fee7027f9e: cc int 3
7fee7027f9f: cc int 3
7fee7027fa0: 48896c2410 mov [rsp+10h], rbp
7fee7027fa5: 4889742418 mov [rsp+18h], rsi
7fee7027faa: 57 push edi
7fee7027fab: 4883ec20 sub rsp, 20h
7fee7027faf: 488b3d4ab22800 mov rdi, [e72b3200]
7fee7027fb6: 488be9 mov rbp, ecx
7fee7027fb9: 488bcf mov rcx, edi
7fee7027fbc: ff15 call dword ptr [rbp]
7fee7027fbe: 6e outsb
7fee7027fbf: 23 db 23h

Built on Althena on Sun Oct 27 16:00:02 2013 using compiler version 1400

Windows 6.1 (Windows 7 x64 build 7601) [Service Pack 1]
Memory status: virtual free 8385811M/8388608M, commit limit 14225M, physical total 7114M

RAX = 25b4c410
RBX = 109be400
RCX = 180040040
RDX = 1
RSI = ffffffffffffffff
RDI = fffffffff2001881
RBP = 9000c409
R8 = 40087c5648e
R9 = 40087c50000
R10 = 0
R11 = ffffffffffffb6d2
R12 = 109be400
R13 = 0
R14 = 41b
R15 = 15
RSP = 1049f260
RIP = 7fee7027f45
EFLAGS = 00010297


Crash reason: Access Violation

Crash context:
An out-of-bounds memory access (access violation) occurred in module 'Veedub64'...

...reading address FFFFFFFFB5B58818...

...while running filter "TextSub" (FilterInstance.cpp:2198)...

...while running thread "Scheduler thread" (thread.cpp:197).

Pointer dumps:

RAX 25b4c410: 00000000 00000000 00000000 00000000 00000000 00000000 00000000 00000000
RBX 109be400: e7234138 000007fe 40000001 00000000 00000000 20043e28 9000c409 40000072
RCX 180040040: 24117160 00000000 240b37c0 00000000 00000000 00000001 00000000 00000001
RSP 1049f260: 109be400 00000000 ffffffff ffffffff 00000060 00000060 00000060 00000060
1049f280: 00000015 00000000 0000041b 00000000 ffffb6d2 ffffffff 109be400 00000000
1049f2a0: 20043e28 00000000 9000c409 00000000 1049f510 00000000 e7027cbb 000007fe
1049f2c0: 87c50000 00000400 109be400 00000000 00004000 00000000 9000c40c 00000000
R12 109be400: e7234138 000007fe 40000001 00000000 00000000 20043e28 9000c409 40000072

Thread call stack:
7fee7027f45: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+308f5]
7fee7027cbb: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+3066b]
7fee705c4f4: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+64ea4]
7fee70285a6: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+30f56]
7fee6ffb33e: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+3cee]
7fee6ffebc6: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+7576]
7fee7008a0e: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+113be]
7fefce7a06d: KERNELBASE!RaiseException [7fefce60000+1a030+3d]
7fee7040ad9: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+49489]
7fee7041d79: VSFilter64!DllCanUnloadNow [7fee6fd0000+27650+4a729]
7fee6fe8235: VSFilter64!csri_renderer_default [7fee6fd0000+3100+15135]
1401ef814: VDPixmapGenResampleRow_x2_p0_lin_u8::Compute()
7fee6fe8fa1: VSFilter64!csri_renderer_default [7fee6fd0000+3100+15ea1]
1401efc7a: VDPixmapGenResampleCol_d2_pnqrtr_lin_u8::Compute()
1400553ac: FilterInstance::RunFilterInner()
140059180: FilterInstance::RunFilter()
1401e200d: VDPixmapUberBlitter::Blit()
7fefce610ac: KERNELBASE!WaitForSingleObjectEx [7fefce60000+1010+9c]
14005a473: FilterInstance::RunProcess()
14017dcae: VDScheduler::Run()
14017de6e: VDSchedulerThread::ThreadRun()
14017c00a: VDThread::StaticThreadStart()
140298b7f: threadstartex()
76bd59cd: kernel32!BaseThreadInitThunk [76bc0000+159c0+d]
76e0a2e1: ntdll!RtlUserThreadStart [76de0000+2a2c0+21]

-- End of report


Please, tell me what's going wrong!

[Bjs]

What appears to be happening is your trying to encode the file, not insert

Use mkvtoolnix

Filter has issues in 64bit vdub

Poorly formatted subtitle can also trigger faults.

[luisppk]

What appears to be happening is your trying to encode the file, not insert

Use mkvtoolnix

Filter has issues in 64bit vdub

Poorly formatted subtitle can also trigger faults.

Aegisub accepts the subtitle and shows no problems with it.
Also, I'm TRYING to encode a hardsub video. That's the reason I'm using vdub.
Mkvtoolnix I used to extract the audio (aac), 'cause vdub can't support it.

If filter is a known issue... do you know which one I can use to solve it once and for all?

[sneaker]

I don't think there is any other ASS filter for VirtualDub. How old is the last vsfilter version that's compatible with it? 15 years?

Others use newer xy-vsfilter through AviSynth (which can in turn be opened by VirtualDub) or switched to solutions like HandBrake, ffmpeg or mpv for ASS hardsubbing.

[luisppk]

I don't think there is any other ASS filter for VirtualDub. How old is the last vsfilter version that's compatible with it? 15 years?

Others use newer xy-vsfilter through AviSynth (which can in turn be opened by VirtualDub) or switched to solutions like HandBrake, ffmpeg or mpv for ASS hardsubbing.

All VSfilter in vdub 64 are from 2010.
PS: I'm updating now with VSFilter 2014.

I tried some other software like handbrake, but I got better results with vdub. I'm not planning to change it just for one problem I have in years. Just wanna know WHY... not recommendations to change to another encoding software.

I tried avidemux and everything worked fine. Encoding video with hardsubs was successful.
Still wanna know why vdub shows error.

[sneaker]

Just wanna know WHY... not recommendations to change to another encoding software.

Well, there's seems to be an error in the software. Nothing you did wrong. You can only try to work around the problem, i.e. try different binaries (like you are with the 2014 version, apparently) or vdub 32. Or try to isolate the part of the script or the font that provokes the crash. In the end: the software is no longer developed. Unless you are a programmer there's little you can do yourself to fix that. (You can try to report it at x-vsfilter or MPC-HC on the off-chance that development picks up again in the future)

You are right that newer versions exist. Even xy-vsfilter still works with VirtualDub. I got that wrong. In case you want to test more binaries I'd test xy-vsfilter and the latest version from the MPC-HC standalone filter package (maybe also from MPC-BE team, haven't checked). Good luck.

[luisppk]

Just wanna know WHY... not recommendations to change to another encoding software.

Well, there's seems to be an error in the software. Nothing you did wrong. You can only try to work around the problem, i.e. try different binaries (like you are with the 2014 version, apparently) or vdub 32. Or try to isolate the part of the script or the font that provokes the crash. In the end: the software is no longer developed. Unless you are a programmer there's little you can do yourself to fix that. (You can try to report it at x-vsfilter or MPC-HC on the off-chance that development picks up again in the future)

You are right that newer versions exist. Even xy-vsfilter still works with VirtualDub. I got that wrong. In case you want to test more binaries I'd test xy-vsfilter and the latest version from the MPC-HC standalone filter package (maybe also from MPC-BE team, haven't checked). Good luck.

I just installed xy-vsfilter 64 in my computer.
Not sure if I have to copy it to the plugins64 folder (the vsfilter.dll file) or if I have to rename it as vsfilter.vdf but I'll try everything.


Edit:
Well... tried many things, until somehow I changed my encoding from '2pass' to 'single pass - quantizer based (19)', and it worked.
After watching the new video, I tried again the 1pass... and crashed.
So... I officially don't know what happened. Neither if I solved with the xy-vsfilter or not.

I'm trying now the CRF just to know if it also works.