windows - How to make my program DEP-compatible?

Question

I have a windows forms (.net 3.0) project that won't run on my customer's vista computer due to a DEP error. It runs on my vista machine, and in a clean version of vista sp1 in a virtual machine. I am having trouble tracking down ways to make my program DEP, Data Execution Prevention compatible. I really can't do anything to end user machines, it just has to run. Is there any way out of this latest vista development nightmare? My program uses devexpress controls, sql express, and the .net ie web browser control. I've already jumpered out the ie control, but to no avail. I have other program that use devexpress and sql express on that same machine and they run ok. I am at a loss to debug this on the user's computer.

Answer 1

DEP runs in one of two modes:

1) Hardware DEP is for CPUs that can mark memory pages as non-executable. This helps to prevent certain exploits such as buffer overflows.

2) Software DEP is for CPUs that do not have hardware DEP support. It doesn't prevent execution of code in data pages, but instead stops SEH overwrite (another type of attack).

On Windows XP with CPUs that support it, hardware DEP is enabled by default only for certain Windows system binaries, and also for programs that choose to "opt-in".

On Vista with CPUs that support it, hardware DEP is enabled by default for nearly all processes. This can occasionally be problematic, usually for older programs and drivers, and for ISVs that haven't done any Vista testing.

So I suspect that the first step is to discover whether you're dealing with software or hardware DEP. Also, are you using C#/VB or Managed C++? And are you using any native code or components? If your application uses a native component or an ActiveX control that was built using the old ATL framework, then it's quite possible that your application will fail with hardware DEP.

Since .NET Framework 2.0 SP1, I believe that the C# compiler emits managed code which is DEP-compatible. But if your application is generating DEP exceptions, then you can try clearing the IMAGE_DLLCHARACTERISTICS_NX_COMPAT flag for your executable. To do this you use EDITBIN.EXE from the VC toolset like so:

editbin.exe /NXCOMPAT:NO <your binary>

If you're using Visual Studio, you can add a post-build step to your executable's project. You'll need to setup the environment so that EDITBIN's dependencies can be resolved. When I'm using native code as part of my app, the post-build step looks like this:

call $(DevEnvDir)..oolsvsvars32.bat
editbin.exe /NXCOMPAT:NO $(TargetPath)