API hooking with Detours on Windows
Ever wondered how tools track when certain WinAPI functions are called? As an example, an EDR solution will immeditately flag CreateRemoteThread attempts. One of the ways is “API hooking” — hijacking calls to WinAPI functions and redirecting them to custom functions. It is then up to these custom functions to decide whether to pass execution to the actual WinAPI functions, or to simply halt.
Detours is a library from Microsoft Research that aids in exactly this, abstracting away the function overwrites and adding jumps.
This article is meant to introduce the concept of API hooking, then use Detours to hook a WinAPI function.
Primer to API hooking
In Windows, DLLs are library files that contain executable code meant to be shared. WinAPIs come in the form of DLLs such as kernel32.dll and user32.dll.
When your code needs to invoke functions in these DLLs, function calls are made to the exported functions of these DLLs. The export table of a DLL contain function names and offsets that help determine where exactly in a DLL’s loaded memory image is a function located. Once found, execution is passed to the located address.
API hooking works by overwriting the first few bytes of the target function with opcodes that pass the execution to another chosen address. What this means is that any function call jump will be immediately followed by another jump, with the latter being due to the overwritten opcodes.
This latter jump is controlled, because the jump destination address can be chosen. With this jump, execution can be passed on to a custom hookfunction. This custom function would then do what it’s supposed to do, then decide whether to pass execution to the actual intended API function or not.
What API hooking essentially achieves is the addition of extra functionality to a WinAPI function without modifying the actual function (except for the first few opcodes; we’ll get into this soon).
The process of unhooking is the exact opposite of hooking. The jump opcodes in the target function are overwritten with the original opcodes that were replaced in the first place. This restores original functionality, because subsequent API calls would not jump to the custom function.
Unhooking requires you to have saved the replaced original opcodes of the target function somewhere, so they can be placed back to restore original functionality.
Introduction to Detours
Detours is a library from Microsoft Research that does the above-mentioned job for you, plus a whole lot more. The methods offered allow to attach and detach hooks into/from target WinAPI functions, while saving the overwritten opcodes for later unhooking.
How Detours works
To better understand how Detours works, first let’s see how a normal call to MessageBoxA would look like.
There’s two jumps that happen in a normal call to MessageBoxA. The first jump (marked “1”) jumps to the beginning of MessageBoxA in memory-loaded user32.dll. The second jump (marked “2”) jumps back to the calling function, right after where the call took place. Pretty simple.
This is how the whole arrangement would look after Detours is used to hook into MessageBoxA .
The first call (“1”) is identical to the previous one. However, now the first few opcodes of MessageBoxA has been replaced with jump instructions that jump to a hook function. These first few opcodes form the “trampoline function”, named because it redirects execution right away. The first call (“1”) is therefore immediately followed by a second jump (“2”) to the hook function.
The hook function does what it’s supposed to do. This forms a “pre-hook” functionality, since it executes before an optional call to the actual MessageBoxA function. At its end, it can choose to pass execution to the original target function MessageBoxA . However, observe that such a call would cause an infinite loop, since MessageBoxA directly jumps again to the hook function.
To solve this, Detours saves a copy of the first few opcodes of MessageBoxA which it is supposed to overwrite during hooking. The hook function can therefore jump to this to execute the actual MessageBoxA . The third jump (“3”) is exactly that.
This copy of first few opcodes of MessageBoxA is executed. At the end of this copy Detours has already added a jump instruction to the address right after the replaced opcodes in MessageBoxA . The fourth jump (“4”) happens right at the end of the original opcodes copy, and takes execution right after where the original opcodes were overwritten. This whole jump bridges the first opcodes with the later opcodes of MessageBoxA , thus completing the full execution of this function.
From there, MessageBoxA returns to the hook function in the fifth jump (“5”). This is because it is the hook function which called the original MessageBoxA , and so execution must return back to the hook function.
The hook function could implement some more functionality here if it wants. This forms a “post-hook” functionality, since this is executed after the original MessageBoxA executes.
From here, since the hook function is now complete, execution now jumps (“6”) back to the original caller.
Using Detours
Compiling and linking Detours
To use Detours, you need to compile the library. Clone the Detours repository, then use nmake to compile.
If you have Visual Studio installed, you already have nmake installed. Plus you also have handy shortcuts to the C++ toolset command line environments needed for each architecture you want to compile for.
If you’re looking to hook in x64 processes, use the “x64 Native Tools Command Prompt for VS 20XX” shortcut. Similar naming for x86. This command prompt shortcut initialise the toolsets and environment you need to correctly call nmake.
Once done, go to detours/src directory and run nmake to compile the library. A lib.<ARCH> directory should now exist, containing the Detours static library, where <ARCH> is the target architecture you are compiling for. The include directory will also be generated during the build, it contains the headers for the library.
Once done, take the detours.h and detours.lib files to your project repository. Then include the header and link the library. Make sure to build in Release mode, because Detours does not correctly hook in Debug builds.
Hooking MessageBoxA
// Target pointer
int (WINAPI* MessageBoxAActual)(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType) = MessageBoxA;
// Hook function
int hookFunction(HWND hWnd, LPCSTR lpText, LPCSTR lpCaption, UINT uType) {
// Custom functionality
printf("From hooked function; MessageBoxA about to be called with body-text: \"%s\"\n", lpText);
// Call actual target function
return MessageBoxAActual(hWnd,lpText, lpCaption, uType);
}
// Installs the hook
BOOL installHook() {
DWORD error = NO_ERROR;
error = DetourTransactionBegin();
if (error != NO_ERROR) {
printf("Failed to begin Detour transaction; error: %d\n", error);
return FALSE;
}
error = DetourUpdateThread(GetCurrentThread());
if (error != NO_ERROR) {
printf("Failed to update thread for Detour transaction; error: %d\n", error);
return FALSE;
}
error = DetourAttach(&(PVOID)MessageBoxAActual, hookFunction);
if (error != NO_ERROR) {
printf("Failed to attach Detour hook function; error: %d\n", error);
return FALSE;
}
error = DetourTransactionCommit();
if (error != NO_ERROR) {
printf("Failed to commit Detour transaction; error: %d\n", error);
return FALSE;
}
return TRUE;
}Detours works in a transactional manner. This means that every hooking you perform is cached till you commit it.
And this is exactly what happens in the installHook() function above. A transaction is created with DetourTransactionBegin() and commited with DetourTransactionCommit() . After beginning the transaction, DetourUpdateThread() is called to specify the thread to be used to perform the hooking. I used the current main thread.
Following this is the actual call to hook MessageBoxA . DetourAttach() performs this. It takes two arguments — a double pointer to the target function (MessageBoxA) and the hook function address.
As for the first argument, there are two things Detours would do with it:
- It will use this address to locate
MessageBoxA. This address would be first used to take a copy of the first few opcodes (labelled as “Replaced opcodes of MessageBoxA” in the diagram in the “How Detours Works” section here), then overwrite at the address with jump instructions to the hook function. - Following the above, this double pointer would then be appropriately incremented to point to the above copy of the first few original opcodes. This is done so that if the hook function wants to call the original
MessageBoxAfunction, it has a reference to the original opcodes. Else we would get into the infinite loop I mentioned.
As for the second argument, Detours needs it as the operand of the jump instruction that it overwrites at MessageBoxA .
Once the hooking is done, any calls to MessageBoxAActual (which now points at the saved original opcodes) will execute everything like a normal MessageBoxA would (jump “3” for example), but with an additional jump following it (“4”).
This is used by the hook function hookFunction() to call the actual MessageBoxA and not itself (thus preventing the infinite loop).
Unhooking MessageBoxA
The process of unhooking happens in the exact opposite manner of hooking. If you understood the code above, the below unhooking code should make sense.
BOOL removeHook() {
DWORD error = NO_ERROR;
error = DetourTransactionBegin();
if (error != NO_ERROR) {
printf("Failed to begin Detour transaction; error: %d\n", error);
return FALSE;
}
error = DetourUpdateThread(GetCurrentThread());
if (error != NO_ERROR) {
printf("Failed to update thread for Detour transaction; error: %d\n", error);
return FALSE;
}
error = DetourDetach(&(PVOID)MessageBoxAActual, hookFunction);
if (error != NO_ERROR) {
printf("Failed to detach Detour hook function; error: %d\n", error);
return FALSE;
}
error = DetourTransactionCommit();
if (error != NO_ERROR) {
printf("Failed to commit Detour transaction; error: %d\n", error);
return FALSE;
}
return TRUE;
}As expected, calls to MessageBoxA no longer execute the hookFunction() after unhooking, because the original opcodes (labelled as “Replaced opcodes of MessageBoxA” in the diagram in the “How Detours Works” section here) have been placed back.
The whole code
I’ve made my entire code for this public. You’ll find it below.
Applications of API hooking
API hooking would be immensely useful both for offsec and defensive operations.
From a defensive standpoint, API hooking can detect and generate alerts for malicious use of WinAPIs. Very useful for EDR solutions, event monitoring softwares, and so on.
From an offsec standpoint, API hooking can be used to backdoor processes, implement spywares, corrupt data, and such.
