Logo
blank Skip to main content

Programming for DEP environment in Windows

This article represents the results of the small interesting research conducted in our Research Team. The Leader of this team is sharing the results of his work.

Data Execution Prevention

Data Execution Prevention (DEP) technology was introduced in Windows XP SP2/Windows 2003 SP1. The aim of DEP was to disable execution of the code placed into memory regions marked as data: heap, stack. It was necessary to decrease the number of security vulnerabilities caused by a lot of viruses, rootkits and exploits that used the techniques based on running of code in Stack.

DEP technology is based on NX bit CPU technology which allows to mark memory pages as not executable and prevent executing of the code from such areas.

Hardware DEP prevents:

Code execution in Heap. If Heap was created without HEAP_CREATE_ENABLE_EXECUTE flag you wonโ€™t be able to run code in this Heap.

Code execution in Stack.

Software DEP checks:

If the exception handler is registered in the application exception table.

Since Windows Vista it checks the address of the exception handler. The address of the exception handler must belong to the memory region with MEM_IMAGE attribute.

The virtual machine software doesnโ€™t emulate hardware DEP so thereโ€™s only software DEP enabled.

Problems

What problems can such checks cause for the people, who are not interested in writing malware?

First of all it became a headache for the software protections. The Software Protectors are decrypting the code into memory and running it there. Also they are using exception handlers for anti-debugging tricks.

The second problem is met by the Simulator Software, the software which actually loads the executable binaries as data file and simulates processes and threads running within it. It also needs to run the code from โ€œdata memoryโ€ and simulate exception handlers.

Can DEP be disabled? Of course!

Disabling DEP

Manual way

The first way is the easiest โ€“ disable DEP for you process. It can be done manually via โ€œSystem Propertiesโ€ โ€“ โ€œAdvancedโ€ โ€“ โ€œPerformance Settingsโ€. In the โ€œPerformance Settingsโ€ window select the โ€œData Execution Preventionโ€ tab. There you can disable DEP for the executable file by your choice:

This is the manual, official way. But there is a problem: you wonโ€™t be able to add the .NET executables to the exclusions list in Windows Vista. Instead youโ€™ll have got the error message:

This message is caused by IMAGE_DLLCHARACTERISTICS_NX_COMPAT bit, which is set in the PE header by Visual Studio compiler by default. Thereโ€™s no way to disable such bit via project settings in Visual Studio for .NET applications (though thereโ€™s a setting for native applications in VS 2008). It can be done via edit bin utility, which is the part of Visual Studio:

PowerShell
editbin.exe /NXCOMPAT:NO

For the native executables you can disable it in the project properties in VS 2008:

Manual way 2

In Windows Vista DEP can be disabled for the whole system. It can be done via bcdedit tool.

PowerShell
bcdedit /set {current} nx AlwaysOff

In Windows XP you can edit boot.ini file and change noexecute option:

PowerShell
/noexecute=alwaysoff

Programmatically

There are actually undocumented API in sysdm.cpl which allows to control DEP settings:

int __stdcall EnableExecuteProtectionSupportW();

int __stdcall ModifyExecuteProtectionSupportW(int, int, wchar_t *OptionName, int);

int __stdcall NoExecuteAddFileOptOutList(LPCWSTR lpSrc);

NoExecuteAddFileOptOutList() function allows to add the executable file into DEP exclusions list.

Have a question?

Ask our expert!

Tell us about
your project

...And our team will:

  • Process your request within 1-2 business days.
  • Get back to you with an offer based on your project's scope and requirements.
  • Set a call to discuss your future project in detail and finalize the offer.
  • Sign a contract with you to start working on your project.

Do not have any specific task for us in mind but our skills seem interesting? Get a quick Apriorit intro to better understand our team capabilities.