"Protection error - Debugger Detected"

[DaveMidgley]

I hope this is not considered off-topic. I am a s/w developer, but by no means a video expert.
I have just started using VS2010 and have written the tiniest possible WPF program using a MediaElement control (used to diaply images, video and sound).
It works fine with a .WMV file, but any other sort of file (.MPG, .AVI, even .JPG) causes a message to come up "Debugger detected - please close it down and restart".
My research so far indicates that it is some sort of clash between video codecs, the most common theory being something to do with Nero 6/7 and the DivX codec. I do have Nero 6 installed, but I've checked in Control Panel | Sounds and Audio Devices | Video Codecs, and although I have a whole list Video Compression Codecs, none is called DivX or anything with Nero in the name.
I could uninstall Nero 6, but I do use it, and it seems a bit drastic.
I have also downloaded GraphEdit, which seems to be the diagnostic tool of choice for this problem, but not being an expert in video codecs I haven't the faintest idea what to do with it now that I've got it.
As I say, I know very little about video technology, and I would really appreciate some help on how to use GraphEdit (or anything else - I see elsewhere on this forum that there are some newer tools) to diagnose this problem.

[kmorger]

There are a number of codecs that freak out and hurl exceptions when they detect a debugger. It's a real pain, because some of them do it even when you are not using their codec, but instead simply enumerating codecs in general.

WinDVD, DiVX, etc. all have this annoying behavior.

Fortunately, their method of detection is rather lame, and easy to work around in C++. You just need their call to IsDebuggerPresent() to return false.

I'm uncertain how you would accomplish this directly in WPF, but you can do it indirectly by creating a dependency to a C++ DLL. This DLL should be static bound, but you could do dynamic load as long as you do it before you create your graph.

Anyway, the DLL can get IsDebuggerPresent() to return false a number of ways. If you are lazy, you can use detours, set a trampoline to a routine that returns 0. Because of where IsDebuggerPresent is located, it is safe to inject even in process attach.

A better way involves a few lines of code to modify the PEB, which is how I always do it. It's the first few lines of any program I write that enumerates codecs, simply because I like being able to debug my code.

For 32 bit, the code is this, make a routine called by process attach that does the following:

	__asm
	{		mov		eax, fs:[0x18]		// Address of Thread Env Block (TEB)
			mov		eax, [eax+0x30]		// Address of Proc Env Block (PEB)
			xor		cl,cl				// 0 == No Debugger
			mov		[eax+2],cl			// IsDebuggerPressent()? NOT!
	}

If you load said DLL into your process before you build a filtergraph, all will be happy.
If your process is 64 bit, you will need to create your DLL in ML64, as VS doesn't allow inline asm in 64 bit, and the offsets are slightly different. Google x64 TEB PEB.

Anyway, the above is trivial, and I've never had that fail. BUT, if you get one that's using query information for debug port, or other nasty, your best bet is to modify their codec to just not do that any more. While not hard, it is beyond the scope of this website.

[DaveMidgley]

Well, thanks kmorger. You've gone to a lot of trouble to help me. Unfortunately while that may be all trivial to you, I'm afraid it's complete gibberish to me. I'm not building a filtergraph, or emumerating codecs, and I've no idea what a PEB is. I've never written code that accesses codecs directly, and wasn't planning to now. I don't even have any code worth talking about at the moment, just a basic Windows application with a MediaBox control in it.
As I understand it form what I've read on the web, this only happens when debugging code, it wouldn't be a problem when the app is released. So I beloeve what I need to do is find out which codec is causing the problem and uninstall/disable it. I think I use GraphEdit to do that, but I've no idea how. If you could tell me how ro do that, that would be great.

[kmorger]

Correct, it only happens when debugging code, and will not effect an eventual release.

It's worth noting that the "debugger detected" error is a part of copy protection (albeit a worthless part), so you can rule out any codecs that are free. DiVX, WinDVD, Nero, etc., are all codecs with some form of copy protection embedded into them. You may have gotten them free with your computer, or as part of some other application, but they are still protected.

GraphEdit, by itself, won't give you that info, as it's not "debugging" anything.

To get GraphEdit to hurl, launch GraphEdit, use visual studio to attach to its process, run, and then from GraphEdit, load a video or whatever. Unfortunately, this won't give you a lot more info than before, unless you enable Win32 exceptions with the little checkbox (to halt process when it first fires the error). Use the callstack to determine what is firing it.

Anyway, the reason I mentioned the code in prior post was because the culprit is probably Nero and you didn't want to uninstall it just so you can debug your program.
By using that code, or similar code if you are building 64 bit, you can debug your code and Nero (or any other) codec won't complain. And, it's safe even for a release mode.

If you decide you'd rather just uninstall and then re-install, a cute trick you can do is just rename the offending codec's file. Windows won't be able to enumerate it, so it can't run. But, when you rename it back, it'll all be back to normal. It's a lot faster than actually having to uninstall / re-install. You can get the codec's name from GraphEdit in the filters list (CTRL-F).

And, to be complete, PEB is the Process Environment Block. It is a structure that all processes have, and one of its members is called "BeingDebugged".
I'm including the structure from microsofts site here: http://msdn.microsoft.com/en-us/library/aa813706(v=vs.85).aspx

typedef struct _PEB {
  BYTE                          Reserved1[2];
  BYTE                          BeingDebugged;
  BYTE                          Reserved2[1];
  PVOID                         Reserved3[2];
  PPEB_LDR_DATA                 Ldr;
  PRTL_USER_PROCESS_PARAMETERS  ProcessParameters;
  BYTE                          Reserved4[104];
  PVOID                         Reserved5[52];
  PPS_POST_PROCESS_INIT_ROUTINE PostProcessInitRoutine;
  BYTE                          Reserved6[128];
  PVOID                         Reserved7[1];
  ULONG                         SessionId;
} PEB, *PPEB;

If BeingDebugged is non-zero, anyone calling "IsDebuggerPresent()" will see that a debugger is attached, and in the case of WinDVD or DiVX or Nero, they do crappy things to make programmers miserable.

PEB is not exposed to programmers, but it's easy to get to by the TEB (Thread Environment Block). This normally requires a couple lines of ASM (as illustrated in prior post). There are other ways, but not as cleanly. In x64, you can use a stand alone ASM file and create a 2KB DLL.

If you have further questions, it would be helpful to know if you are debugging in 32 bit or 64 bit.

[DaveMidgley]

Well, huge thanks once again for your amazingly thorough replies. I'm afraid it's been many a long year since I wrote a line of assembler, I'm strictly a high-level language man these days. Nevertheless I might just have a go at putting some assembler into my C# and see what comes out, but I'll try the simpler solutions first.
It certainly seems that everything ponts to Nero (all roads lead to Rome!), and I like your idea of temporarily disabling the codec by renaming the file. All I have to do now is identify the offending codec. I did Crtl-F in GraphEdit, went to Video Compressors, and it does list a Nero Digital AVC Video Enc whose filename is c:\Program Files\Common Files\Ahead\DSFilter\NeNDVid.ax, so I renamed that and rebooted, but sadly it hasn't had any effect. There is a whole bunch of .ax files in that folder, so maybe it's one of the others. I'll keep hunting. I'll also uninstall Nero temporarily just to prove that it is the culprit.
Oh, I'm 32 bit.

[kmorger]

Just rename directory "c:\Program Files\Common Files\Ahead\DSFilter\" to "c:\Program Files\Common Files\Ahead\DSFilter_\" to catch them all.

You won't be able to use in-line in C#, but you can force load of a DLL. If you create a native DLL whos dllmain calls' that asm code on process attach, you could work around. There is also some clever tricks you can do with AppCompat to force a DLL to load just for a particular process against its will.

But, I think if I were doing C# development for DShow, I'd create a dummy filter (in C++ native) with high marit that simply nuke the debug flag. The moment DShow went out enumerating things, it'd wipe the flag unconditionally (setup w/ null media type, and have check media type return false so it doesn't actually try to USE it as a filter, but would still always load it to check. )

[DaveMidgley]

Before I could try renaming the folder I found this post, which suggested renaming NeVideo.ax. When I did that it fixed the problem, just like that! (Didn't even need a reboot). Not exactly ideal, but Nero 6 is a bit old, maybe they've fixed it in the meantime, who knows!

[JCHERN]

I have the same issue when I use MediaElement in a WinForm/WPF mixed project. I wrote a short C function with inline assembly code that kmorger provided, wrapped it with a unmanaged C++ dll, then call it from C# code via dllimport. It works just fine.
A question about the assembly code: What if I just do this:
__asm
{
mov eax, fs:[0x30] // address of current process
xor cl, cl
mov [eax+2], cl
}
without accessing fs:[0x18]? It seems working as well but is is safe to do it this way?
Thanks!