We just migrated our project to Visual Studio 2008 today and… it stopped working.  Our application is a VB.NET/C# hybrid that makes use of a number of legacy ActiveX controls.  When we ran the freshly migrated project the ActiveX controls would fail with “Invalid Access to Memory Location” errors.

This is a side effect of the C# compiler now generating executables with the NXCOMPAT bit set (with no option in the IDE to turn that off).  This setting on the EXE enabled DEP (Data Execution Protection) on our program which prevents the legacy DLLs from doing what they need to do.

We’ve encountered this before here with our build machine and were able to resolve it by tweaking the build script.  Unfortunately the IDE was now failing and that’s more complicated.

On of our developers came up with a fix:

Turn of NXCOMPAT bit on build output

There is a utility called “editbin” that lets you modify the EXE and turn the NXCOMPAT bit off.  You can add it to the post-build events and that fixes the generated EXEs.  Unfortunately, the application still will not run under the IDE because it’s inheriting DEP from the vshost process.

Turn off NXCOMPAT bit on vshost32.exe

To make the application run under the IDE we can turn of NXCOMPAT on the vshost32 executable using “editbin”.

At this point the application seems to run like it always did under the IDE.

<grumble>

My first garlic cheese bread.  It’s a basic white bread with big chunks of garlic and cheese.  The whole house smells but this bread is super tasty.