Programming for DEP environment in Windows

This article represents the results of the small interesting research conducted in our Research Team. The Leader of this team is sharing the results of his work.

Data Execution Prevention

Data Execution Prevention (DEP) technology was introduced in Windows XP SP2/Windows 2003 SP1. The aim of DEP was to disable execution of the code placed into memory regions marked as data: heap, stack. It was necessary to decrease the number of security vulnerabilities caused by a lot of viruses, rootkits and exploits that used the techniques based on running of code in Stack.

DEP technology is based on NX bit CPU technology which allows to mark memory pages as not executable and prevent executing of the code from such areas.

Hardware DEP prevents:

Code execution in Heap. If Heap was created without HEAP_CREATE_ENABLE_EXECUTE flag you won’t be able to run code in this Heap.

Code execution in Stack.

Software DEP checks:

If the exception handler is registered in the application exception table.

Since Windows Vista it checks the address of the exception handler. The address of the exception handler must belong to the memory region with MEM_IMAGE attribute.

The virtual machine software doesn’t emulate hardware DEP so there’s only software DEP enabled.

Problems

What problems can such checks cause for the people, who are not interested in writing malware?

First of all it became a headache for the software protections. The Software Protectors are decrypting the code into memory and running it there. Also they are using exception handlers for anti-debugging tricks.

The second problem is met by the Simulator Software, the software which actually loads the executable binaries as data file and simulates processes and threads running within it. It also needs to run the code from “data memory” and simulate exception handlers.

Can DEP be disabled? Of course!

Disabling DEP

Manual way

The first way is the easiest – disable DEP for you process. It can be done manually via “System Properties” – “Advanced” – “Performance Settings”. In the “Performance Settings” window select the “Data Execution Prevention” tab. There you can disable DEP for the executable file by your choice:

This is the manual, official way. But there is a problem: you won’t be able to add the .NET executables to the exclusions list in Windows Vista. Instead you’ll have got the error message:

This message is caused by IMAGE_DLLCHARACTERISTICS_NX_COMPAT bit, which is set in the PE header by Visual Studio compiler by default. There’s no way to disable such bit via project settings in Visual Studio for .NET applications (though there’s a setting for native applications in VS 2008). It can be done via edit bin utility, which is the part of Visual Studio:

editbin.exe /NXCOMPAT:NO

For the native executables you can disable it in the project properties in VS 2008:

Manual way 2

In Windows Vista DEP can be disabled for the whole system. It can be done via bcdedit tool.

bcdedit /set {current} nx AlwaysOff

In Windows XP you can edit boot.ini file and change noexecute option:

/noexecute=alwaysoff

Programmatically

There are actually undocumented API in sysdm.cpl which allows to control DEP settings:

int __stdcall EnableExecuteProtectionSupportW();

int __stdcall ModifyExecuteProtectionSupportW(int, int, wchar_t *OptionName, int);

int __stdcall NoExecuteAddFileOptOutList(LPCWSTR lpSrc);

NoExecuteAddFileOptOutList() function allows to add the executable file into DEP exclusions list.