A program can communicate with a debugger by sending strings to it through the OutputDebugString API.
What happens then is that a ring3 debugger using the Windows Debug API will receive an OUTPUT_DEBUG_STRING_EVENT. Along with it, it receives relevant information about the debug string inside an OUTPUT_DEBUG_STRING_INFO struct.
First of all, some trivia:
Unicode? Yes? No?
OUTPUT_DEBUG_STRING_INFO contains a member called fUnicode:
The format of the debugging string. If this member is zero, the debugging string is ANSI; if it is nonzero, the string is Unicode.
Let’s look at what MSDN says about OutputDebugString:
OutputDebugStringW converts the specified string based on the current system locale information and passes it to OutputDebugStringA to be displayed. As a result, some Unicode characters may not be displayed correctly.
So we can’t actually receive any unicode debug strings. I assume this is a remainder from pre NT times as it’s officially unused according to the quote above.
Now back to the Debug API:
Exception? Yes? No?
To continue debugging, the debugger calls ContinueDebugEvent for that event. You additionally have to specify the continue status which can be either DBG_CONTINUE or DBG_EXCEPTION_NOT_HANDLED.
According to MSDN there is no difference between the two for events other than EXCEPTION_DEBUG_EVENT. Unfortunately that’s not right. There is another debug event for which it does make a difference. Yes you guessed correctly, a debug string event :D
OUTPUT_DEBUG_STRING_EVENT behaves just like an exception event, so that if you continue with DBG_EXCEPTION_NOT_HANDLED, it sends the event a second time, just like a second chance exception.
Recently I’ve been wondering why I got 2 debug string events for each call to OutputDebugString from the debuggee. I was looking for all sorts of weird solutions until I found out that returning DBG_CONTINUE did the trick.
Reason? Yes? No?
Internally, a debug string is handled as an exception. OutputDebugString calls RaiseException with DBG_PRINTEXCEPTION_C (defined as 0x40010006) and the string address and size as the exception parameters. To keep this transparent to the debugger, Windows effectively swallows the exception and notifies the debugger of it through the debug string notification.
All this is done inside DbgUiConvertStateChangeStructure inside the debugger which sets up and translates all the debug notification stuff.
This is even used as anti-debug code. You just set up an exception handler and call RaiseException with DBG_PRINTEXCEPTION_C. If a debugger is present the exception is swallowed, ie. if your exception handler does not get called, you’re being debugged.
As to why this is not affected by the continue status, I have no clue. It can, however, be fixed by patching DbgUiConvertStateChangeStructure to handle it like any other exception.
For pseudo source code for DbgUiConvertStateChangeStructure, you can take a look at OpenRCE. You can also see how the fUnicode member is set to false by default.
Bottom line, always continue with DBG_CONTINUE and ignore the fUnicode member :)