Practical Comparison of the Most Popular API Hooking Libraries: Microsoft Detours, EasyHook, Nektra Deviare, and Mhook

Hooking API function calls is an efficient way to change or augment the behavior of an operating system (OS). Developers rely on API hooking in projects where they need to carry out tracing and debugging tasks, build sandboxes, enhance browser security, intercept operating system calls, analyze malicious code, and so on.

There are many powerful libraries that make it easy to implement API call hooking. In this article, we provide a practical comparison of the four top API hooking libraries:

• Microsoft Detours
• EasyHook
• Nektra Deviare
• Mhook

We discuss the key pros and cons of each library and show how to use it to intercept API calls. We also provide a well-structured comparison table at the end of this overview so that you can determine exactly which library would suit your project the best.

The article will be useful for development teams working on projects requiring the implementation of API hooks and who want to know what the best API hooking library is to achieve their goals.

Building a sample project

To make our comparison fair, we implemented the same functionality — a project that can hide files with a specified pattern in their names from a user — by intercepting API functions using different libraries. Thus, we can demonstrate how each of the four most popular API hooking libraries performs in practice.

These are the libraries we’ll be working with:

In general, applications using API hooking techniques have two main components: 

• A user-mode dynamic link library (DLL) that intercepts target functions 
• An executable program that injects this DLL into the target process 

Both of these components are written in C++ with the use of API hooking libraries. 

The only exception is the Mhook library, for which it’s impossible to implement an executable program for DLL injection. Instead, we inject the DLL into the target process using the AppInit_DLLs infrastructure that loads a predefined set of DLLs to all user-mode processes. You can explore key aspects of working with this infrastructure in one of our previous articles.

In our example, the target process is File Explorer. The NtQueryDirectoryFile NTAPI function lists all files in a folder. Our goal is to hide all files with the “+*.txt” filename pattern so that a user can’t see them in a folder.

We created several files, both with and without the “+*.txt” pattern in their names, and placed them in the same folder. To achieve the result we want, we need to inject a DLL to hook the NtQueryDirectoryFile NTAPI function and intercept all NtQueryDirectoryFile API function calls in the File Explorer process.

Before DLL injection, we can see all the files in the folder.

Then, we inject a DLL that intercepts all NtQueryDirectoryFile API function calls in the File Explorer process (Screenshot 2).

After successful DLL injection, files with the “+*.txt” filename pattern are no longer visible in our sample folder (Screenshot 3).

To achieve this result, we need to set hooks in the NtQueryDirectoryFile NTAPI function and change the behavior of API function calls in it.

Below, we demonstrate how to hook the source function and what changes to apply to a hooked function to make it hide certain files from a user. This code is the same for each of the four most used API hooking libraries we examine in this article.

Let’s start with the source function.

Working with the NtQueryDirectoryFile NTAPI function

We can use the winternl.h header to get the information we need to manage the NtQueryDirectoryFile function, which is responsible for listing files in a folder. The data types, however, must be defined manually:

// Special data structures and typedefs for managing NtQueryDirectoryFile.
#define STATUS_SUCCESS   ((NTSTATUS)0x00000000L)
#define FileIdBothDirectoryInformation 37

typedef struct _FILE_ID_BOTH_DIR_INFORMATION
{
    ULONG         NextEntryOffset;
    ULONG         FileIndex;
    LARGE_INTEGER CreationTime;
    LARGE_INTEGER LastAccessTime;
    LARGE_INTEGER LastWriteTime;
    LARGE_INTEGER ChangeTime;
    LARGE_INTEGER EndOfFile;
    LARGE_INTEGER AllocationSize;
    ULONG         FileAttributes;
    ULONG         FileNameLength;
    ULONG         EaSize;
    CCHAR         ShortNameLength;
    WCHAR         ShortName[12];
    LARGE_INTEGER FileId;
    WCHAR         FileName[1];
} FILE_ID_BOTH_DIR_INFORMATION, * PFILE_ID_BOTH_DIR_INFORMATION;

typedef NTSTATUS(WINAPI* _NtQueryDirectoryFile)(
    IN           HANDLE                 FileHandle,
    IN OUT       HANDLE                 Event,
    IN OPTIONAL  PIO_APC_ROUTINE        ApcRoutine,
    IN OPTIONAL  PVOID                  ApcContext,
    OUT          PIO_STATUS_BLOCK       IoStatusBlock,
    OUT          PVOID                  FileInformation,
    IN           ULONG                  Length,
    IN           FILE_INFORMATION_CLASS FileInformationClass,
    IN           BOOLEAN                ReturnSingleEntry,
    IN OPTIONAL  PUNICODE_STRING        FileName,
    IN           BOOLEAN                RestartScan
    );

// Original function
const _NtQueryDirectoryFile TrueNtQueryDirectoryFile = (_NtQueryDirectoryFile)GetProcAddress(GetModuleHandleW(L"ntdll"), "NtQueryDirectoryFile");

Next, let’s take a look at the already hooked function.

Working with the NtQueryDirectoryFile NTAPI function after hooking

After the NtQueryDirectoryFile NTAPI function is hooked, it calls the original function. Then we examine the FileInformationClass class, and if it’s the FileIdBothDirectoryInformation class, we need to find and remove from the listing all files with the specified filename pattern — in our case, all files with the “+*.txt” pattern in their names. And this is it!

Here’s what the code of the hooked NtQueryDirectoryFile NTAPI function looks like:

// Hooked function
NTSTATUS WINAPI HookNtQueryDirectoryFile(
    IN           HANDLE                 FileHandle,
    IN OUT       HANDLE                 Event,
    IN OPTIONAL  PIO_APC_ROUTINE        ApcRoutine,
    IN OPTIONAL  PVOID                  ApcContext,
    OUT          PIO_STATUS_BLOCK       IoStatusBlock,
    OUT          PVOID                  FileInformation,
    IN           ULONG                  Length,
    IN           FILE_INFORMATION_CLASS FileInformationClass,
    IN           BOOLEAN                ReturnSingleEntry,
    IN OPTIONAL  PUNICODE_STRING        FileName,
    IN           BOOLEAN                RestartScan)
{
    NTSTATUS status = TrueNtQueryDirectoryFile(FileHandle, Event, ApcRoutine, ApcContext, IoStatusBlock, FileInformation,
        Length, FileInformationClass, ReturnSingleEntry, FileName, RestartScan);

    if (FileIdBothDirectoryInformation == FileInformationClass && STATUS_SUCCESS == status)
    {
        PFILE_ID_BOTH_DIR_INFORMATION pCurrent = nullptr;
        PFILE_ID_BOTH_DIR_INFORMATION pNext = reinterpret_cast<PFILE_ID_BOTH_DIR_INFORMATION>(FileInformation);

        do
        {
            pCurrent = pNext;
            pNext = reinterpret_cast<PFILE_ID_BOTH_DIR_INFORMATION>((PUCHAR)pCurrent + pCurrent->NextEntryOffset);
            const std::wstring fileName = std::wstring(pNext->FileName, pNext->FileNameLength / sizeof(wchar_t));

            // Hiding all files in the directory that match the pattern "+*.txt".
            if (!fileName.empty() && PathMatchSpecW(fileName.c_str(), L"+*.txt"))
            {
                if (0 == pNext->NextEntryOffset)
                {
                    pCurrent->NextEntryOffset = 0;
                }
                else
                {
                    pCurrent->NextEntryOffset += pNext->NextEntryOffset;
                }
                pNext = pCurrent;
            }

        } while (pCurrent->NextEntryOffset != 0);
    }
    return status;
}

Using this code, we create the DLL that intercepts the NtQueryDirectoryFile NTAPI function. As this functionality hides files with a specified pattern, this code is the same for every API hooking library we examine in this article.

However, since every library uses its own APIs to hook and unhook functions for the DLL and different APIs to create a process with an injected DLL, there are code pieces unique to each of the libraries we work with.

Let’s take a closer look at each library, their pros and cons, and the code needed to hook and unhook a function using that library, starting with Microsoft Detours.

Microsoft Detours

Microsoft Detours is a library for intercepting, monitoring, and instrumenting arbitrary Win32 functions in Microsoft Windows. Released in 2002 by Microsoft, this library is widely used to intercept Win32 API calls within Windows applications.

Detours intercepts Win32 functions by re-writing the in-memory code for target functions. The Detours package also contains utilities to add debugging instrumentation and attach arbitrary DLLs and data segments to any Win32 binary.

The library is available under the MIT license, where only a part of the functionality is available for free, and under a commercial license. It’s widely used by many independent software vendors as well as product teams at Microsoft. In this article, we’ll be using the publicly available version of the library.

The Microsoft Detours library has its pros and cons:

Pros: 

• Microsoft Detours is a well-documented, lightweight library with a distinct API you can work with even if you only have a superficial understanding of Windows internals. 

• Detours’ instruments allow for efficient coding, with benchmarks placing interception overhead at less than 400 nanoseconds (ns) on 200 MHz processors (on which the library was initially tested). The library performs a minimal amount of static manipulation on the target binary’s import section to have it load the DLL.

• Detours has efficient memory management, allocating memory blocks as needed and using the resulting data areas to store as many trampolines as possible. A trampoline is a piece of code that replicates the functionality of the first few bytes of the original API and then jumps into the API after the changed bytes.

• The library supports transactional hooking and unhooking, allowing you to set a bunch of hooks simultaneously with an all-or-nothing approach. The library will only set hooks if all of them can be set and will roll back any changes made if at least one hook can’t be installed. 

Cons: 

• In contrast to other libraries we discuss in this article, Microsoft Detours can’t set and remove hooks in running applications and can cause random crashes if you try doing so.

• The library is relatively costly for commercial use and requires a commercial license for working with the х64 architecture.

Using Microsoft Detours for API hooking

Detours preserves the un-instrumented target function (callable through a trampoline) as a subroutine to be used by instrumentation. The trampoline design enables the use of a large class of innovative extensions for existing binary software.

To start working with Detours, we need to build the library itself, add library samples, and set up our project to use this library. 

Here’s the code we need to add to a newly created DLL when working with Detours:

// allow calls to Mhook functions
#include <detours.h>

BOOL APIENTRY DllMain(
    HINSTANCE hinstDLL,
    DWORD fdwReason,
    LPVOID lpvReserved)
{
    if (DetourIsHelperProcess())
    {
        return TRUE;
    }
    
    switch (fdwReason)
    {
    case DLL_PROCESS_ATTACH:
…

To set hooks, we use the following code: 

DetourRestoreAfterWith();

    printf("MSDetoursHideFile.dll:Starting.\n");
    fflush(stdout);

    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourAttach(&(PVOID&)TrueNtQueryDirectoryFile, HookNtQueryDirectoryFile);
    const LONG error = DetourTransactionCommit();

    if (error == NO_ERROR) 
    {
        printf("MSDetoursHideFile.dll: Detoured NtQueryDirectoryFile().\n");
    }

To unhook a function, we need the following code:

    DetourTransactionBegin();
    DetourUpdateThread(GetCurrentThread());
    DetourDetach(&(PVOID&)TrueNtQueryDirectoryFile, HookNtQueryDirectoryFile);
    const LONG error = DetourTransactionCommit();

    printf("MSDetoursHideFile.dll: Removed NtQueryDirectoryFile() (result=%ld)\n", error);

    fflush(stdout);

Next, let’s move to the second library on our list — EasyHook.

EasyHook

EasyHook is a library that allows you to hook unmanaged code with pure managed functions from within a fully managed environment on 32-bit and 64-bit Windows versions. EasyHook supports injecting assemblies built for .NET Framework 3.5 and 4.0 and can also inject native DLLs. The library is distributed under the MIT license.

Let’s go over the key pros and cons of the EasyHook library:

Pros: 

• EasyHook has an open-source base, so you can modify, extract, and compile the library. 

• With this library, you can write managed hook handlers for unmanaged APIs and use all the convenience provided by managed code, such as .NET Remoting, WPF, and WCF.

• This library can write injection libraries and host processes compiled for AnyCPU, allowing you to inject your assembly into 32-bit processes from 64-bit processes, and vice versa.

• EasyHook provides support for hooking COM interfaces. For example, we can hook such functions using a hook with the EasyHook.COMClassInfo class that simplifies the retrieval of method addresses from COM interfaces. 

• The library also has an experimental stealth injection mechanism that won’t catch the attention of antivirus software. This can be achieved by hijacking an existing thread and letting it execute an invocation stub. Using the CreateThread() function, this stub creates a thread that will run the given remote thread process again. After such a stealth remote thread has been created, the hijacked thread continues its usual execution. 

• You can use the RhInstallDriver function of this library to install drivers in the system.

Cons: 

• The EasyHook API isn’t user-friendly and has poor documentation. 

• The performance of this library isn’t stable. For example, we had several incidents where our application would crash after calling the unhook function.

• The library seems to have poor support, with the last commit made in 2019. Finding an answer to your questions regarding the use of this library may become a challenge. 

Using EasyHook for API hooking

To set hooks using the EasyHook library, we need to build the EasyHook binary and set up our project to use this library.

To start using the library, we must create the NativeInjectionEntryPoint function in our DLL. For remote native/unmanaged hooks, EasyHook is expecting to find the NativeInjectionEntryPoint export method within our DLL. EasyHook will call this method once the DLL has been successfully injected.

Here’s the code for setting an API hook with EasyHook:

extern "C" void __declspec(dllexport) WINAPI NativeInjectionEntryPoint(REMOTE_ENTRY_INFO * inRemoteInfo);

void WINAPI NativeInjectionEntryPoint(REMOTE_ENTRY_INFO* inRemoteInfo)
{
    OutputDebugStringW(L"EasyHookHideFile.dll:NativeInjectionEntryPoint - Entered to the main function");

    // Install the hook
    HOOK_TRACE_INFO hHook = { NULL };
    NTSTATUS result = LhInstallHook(
        GetProcAddress(GetModuleHandle(TEXT("ntdll")), "NtQueryDirectoryFile"),
        (void*)HookNtQueryDirectoryFile,
        nullptr,
        &hHook);

After hook installation, we need to activate the hook in the thread:

ULONG ACLEntries[1] = { 0 }; // If the threadId in ACL is set to 0, then internally EasyHook uses GetCurrentThreadId()
LhSetInclusiveACL(ACLEntries, 1, &hHook); // Activates hook for the thread

For now, all the NtQueryDirectoryFile calls will be intercepted by EasyHook. But since we need to inject the written DLL into the target process, we call the RhInjectLibrary function in our executable program: 

RhInjectLibrary(
        dwPid,   // The process to inject the DLL into
        0,       // ThreadId to wake up upon injection
        EASYHOOK_INJECT_DEFAULT,
        NULL,    // 32-bit DLL
        const_cast<WCHAR*>(dllToInjectNameW), // 64-bit DLL
        NULL,    // data to send to injected DLL entry point
        0        // size of data to send
    );

After DLL injection, the hook is set and works as needed.

Next, let’s see how to work with hooks using the Nektra Deviare library.

Nektra Deviare

Nektra Deviare is an open-source library that can intercept unmanaged Win32 functions, COM objects, and functions in 32-bit and 64-bit applications. This library can control pre-calls and post-calls of hooked functions.

Deviare has a free version, but if you want to use it in a commercial product, you must purchase a commercial license.

Let’s take a look at the key pros and cons of the Deviare library:

Pros: 

• The library comes with a comprehensive API and can help you deal with code injection, inter-process communication, and parameter marshaling.

• The Deviare library is implemented as a COM component, so it can be integrated with all programming languages that support COM, such as C/C++, Visual Basic, C#, Delphi, and Python. 

• Due to its open-source nature, the library is highly customizable.

• Similarly to Detours, Deviare supports transactional hooking and unhooking.

Cons: 

• This library comes with poor documentation and has limited efficiency.

• In contrast to EasyHook, Deviare can’t hook terminal sessions. Also, this library can’t be executed as a service.

Now, let’s see how to hook API function calls with the help of Deviare.

Using Nektra Deviare for API hooking

To use the library, we first need to build binaries for it and set up our project to use these binaries. 

We start with adding the following code to our DLL:

// Allow calls to Deviare functions
#include <NktHookLib.h>

static struct {
    SIZE_T nHookId;
    _NtQueryDirectoryFile trueNtQueryDirectoryFile;
} sNtQueryDirectoryFile_Hook = { 0, NULL };

static CNktHookLib cHookMgr;

The code below allows us to install the hook:

cHookMgr.SetEnableDebugOutput(TRUE);

    HINSTANCE ntdllDLL = NktHookLibHelpers::GetModuleBaseAddress(L"ntdll.dll");
    if (ntdllDLL == NULL)
    {
        OutputDebugStringW(L"DeviarePlugin.dll:HookFunction - Error: Cannot get handle of ntdll.dll");
        return 1;
    }

    LPVOID NtQueryDirectoryFile = NktHookLibHelpers::GetProcedureAddress(ntdllDLL, "NtQueryDirectoryFile");
    if (NtQueryDirectoryFile == NULL)
    {
        OutputDebugStringW(L"DeviarePlugin.dll:HookFunction - Error: Cannot get address of NtQueryDirectoryFile");
        return 1;
    }

    HRESULT dwOsErr = cHookMgr.Hook(&(sNtQueryDirectoryFile_Hook.nHookId), (LPVOID*)&(sNtQueryDirectoryFile_Hook.trueNtQueryDirectoryFile),
        NtQueryDirectoryFile, HookNtQueryDirectoryFile, NKTHOOKLIB_DisallowReentrancy);

To uninstall the hook, we can use the following function:

cHookMgr.Unhook(sNtQueryDirectoryFile_Hook.nHookId);

We can use this function for the executable program to inject DLL into the target process:

DWORD dwOsErr = NktHookLibHelpers::CreateProcessWithDllW(exeNameW, NULL, NULL, NULL, FALSE,
        0, NULL, NULL, &sSiW, &sPi, dllToInjectNameW.c_str(), NULL, NULL);

Finally, let’s move to the last library we’ll be overviewing in this article — Mhook.

Mhook

Mhook is an open-source API hooking library you can use for injecting code into an application’s flow. Originally created by Marton Anka as a free alternative to Microsoft Detours, Mhook also uses trampoline hooking to hook API function calls. 

There is a popular fork of the Mhook library created by Apriorit specialists. In this article, we describe how to work with our forked version.

Let’s go over the key pros and cons of this library:

Pros: Mhook is freely distributed under the MIT license and supports both x86 and x64 architectures.

It’s an efficient, open-source library that offers high customization and can be used for non-trivial tasks.

In contrast to Microsoft Detours, Mhook can set and remove hooks in running applications.

The forked version of Mhook is continuously improved and maintained by Apriorit specialists.

Cons: Mhook isn’t good with memory management for its trampolines. The library uses one call to VirtualAlloc per set hook. And since every hook needs fewer than 100 bytes of storage but the VirtualAlloc function allocates 64 kilobytes from the process virtual address space every time Mhook calls it, this is very inefficient. 

Using Mhook for API hooking

To use the Mhook library, we clone the project from the Apriorit GitHub repository, build it, and set up our project to use this library.

To start working with Mhook, we add the following code to our DLL:

#include <mhook.h>

BOOL APIENTRY DllMain(
    HINSTANCE hinstDLL,
    DWORD fdwReason,
    LPVOID lpvReserved)
{
    BOOL result = FALSE;
    
    switch (fdwReason)
    {
    case DLL_PROCESS_ATTACH:
        result = HookFunction();
        if (result)
        {
            OutputDebugStringW(L"MhookHideFile.dll:DllMain - DLL_PROCESS_ATTACH: Successfully set hook");
        }
        else
        {
            OutputDebugStringW(L"MhookHideFile.dll:DllMain - DLL_PROCESS_ATTACH: Error: Failed to set hook.");
        }
        break;
    case DLL_PROCESS_DETACH:
        UnhookFunction();
…

To install the hook, we use the BOOL HookFunction function:

BOOL HookFunction()
{
    // Set the hook for the NtQueryDirectoryFile function.
    return Mhook_SetHook((PVOID*)&TrueNtQueryDirectoryFile, HookNtQueryDirectoryFile);
}

BOOL UnhookFunction()
{
    // Unhook the NtQueryDirectoryFile function.
    return Mhook_Unhook((PVOID*)&TrueNtQueryDirectoryFile);
}

And that’s it. You can download and examine the source code of the projects using each of the described libraries on the Apriorit GitHub page.

To sum up, let’s compare these four libraries across key factors that might be important for projects requiring API hooking.

Summary of API hooking libraries comparison

Comparing API hooking libraries is a challenging task, as they were created for different purposes and rely on different technologies. However, we’ve composed a comprehensive comparison table that can help you determine the most fitting library for your project. 

For instance, to compare the speed, we measured the time it takes each library to set and remove hooks depending on the number of threads in the process. As a result, we can conclude that in heavy applications with 10,000 threads, Detours performed the best.

See the full API hooking libraries comparison summary below:

Based on our experience, we can recommend using Detours and EasyHook for small API hooking projects. These libraries allow for quick installation of hooks and don’t require extensive experience from the developers using them.

For large projects that require high stability, efficiency, and customization, using Deviare or Mhook would be more appropriate.

Conclusion

By setting API hooks, developers can change system behavior and add the functionality they want to implement. API hooking libraries help make it easier to set and remove hooks from API function calls.

In this article, we described how to install and uninstall hooks with some of the best API hooking libraries: Microsoft Detours, EasyHook, Nektra Deviare, and Mhook. We compared them based on essential criteria and showed how to intercept the Ntdll.dll!NtQueryDirectoryFile that NTAPI calls using each of these API hooking libraries.

At Apriorit, we have highly experienced, competent system programming experts who can assist you with improving the security and performance of your software solutions. 

Get in touch with us to discuss your product and the possibilities for its enhancement!