How we have increased mhook’s performance, enhanced its capabilities and eliminated certain bugs.
Mhook is an open source API hooking library for intercepting function calls (setting hooks). In other words, it's a library for embedding code into application threads. This library comes in handy when you need to monitor and log system function calls or run your own code before, after, or instead of a system call. To intercept a function call, mhook replaces five bytes at the address to be intercepted with the unconditional jump code (jmp #addr) for the interception function. Then mhook transfers those removed five bytes to a special allocated place called a trampoline. When the interception function becomes inactive, it can make an unconditional jump to the springboard that has those 5 stored bytes running. Finally, a jump to the intercepted code happens. To learn more about how to use mhook for API hooking, you can read our mhook tutorial.
Detours is another API hooking library that's similar to mhook. In the mhook vs detours contest, it's hard to declare a certain winner, though we at Apriorit find mhook more convenient. Mhook supports both x64 and x86 platforms for free while Detours supports only x86 with a noncommercial license; to get official x64 support, you have to pay. The main advantage of Detours is that it supports transactional hooking and unhooking.
We often use mhook to solve tasks within projects related to cybersecurity and reverse engineering. When using mhook, we've faced the following issues:
- Poor performance with a large number of system threads and when setting multiple hooks in a row;
- The necessity to manually create projects for all integrated development environments (IDEs);
- The impossibility to hook functions that don't have a suitable first five bytes for recording the jump to the hook;
- Infinite recursion (bug);
- Deadlock (bug);
- Hooks leading to the wrong function (bug).
To overcome these issues, we've improved the original version of mhook and made our updated version public. In this article, we'll describe the problems we faced during our work with the original mhook, and how we solved them with our own mhook enhancements.
We increased the performance of the mhook library using the NtQuerySystemInformation function.
Mhook starts working very slowly with a large number of system threads.
When setting a hook,information about processes and threads is used to suspend all threads of the current process except its own thread and to change the function address to one specified by the developer. As a result, despite the fast speed of getting a thread status snapshot using CreateToolhelp32Snapshot, the Thread32Next function starts working very slowly with an increasing number of system threads. Microsoft doesn't open its source code, but you can find similar methods in the ReactOS project. It seems that each Thread32Next call triggers the NtMapViewOfSection which performs a rather resource-intensive operation.
Our tests showed that when using CreateToolhelp32Snapshot, calling Thread32Next took about 10 times resources than getting a snapshot. While using NtQuerySystemInformation, getting the snapshot was cheap enough (cheaper than the initial implementation), and the thread iterations were almost free (about 10 times cheaper than the snapshot), basically coming down to calculating pointers. In general, the NtQuerySystemInformation-based approach is about 10 times faster than the CreateToolhelp32Snapshot-based one. In a system with about 3000 threads, setting one hook takes about 0.02 seconds, while the original method could take as long as 0.14 seconds per hook.
Here's the code we measured:
We increased the performance of the mhook library using the Mhook_SetHookEx method.
Setting multiple hooks in a row is slow.
As we've already mentioned, you have to suspend all threads of the current process to set a hook. If you set 100 hooks in a row, then you have to suspend threads 100 times and restart them 100 times, which is obviously inefficient.
We added the Mhook_SetHookEx method to set several hooks during a single thread suspension. The input retrieves an array of HOOK_INFO structures containing the same information that used to be transmitted to Mhook_SetHook.
How to use the Mhook_SetHookEx method in mhook: example:
This modification provides a substantial performance increase, as compared to setting the same hooks consecutively.
For example, here's a comparison of the performance when setting three hooks with both methods:
On average, it's about 2.8 times faster to set three hooks using the Mhook_SetHookEx method than it is to set just one hook using the traditional setHook method.
The code for this test is basically the same as for the previous one. You need to do to test a hook is set several hooks in the testPerformance function using the Mhook_SetHook method and the Mhook_SetHookEx method.
Below, we'll describe how we managed to get project files for different IDEs instead of operating with a single .sln file.
The need for manual creation of different projects for each integrated development environments (IDE) other than Visual Studio. In addition, it's difficult to work with different versions of Visual Studio.
Mhook only has a .sln file for Visual Studio 2010. Furthermore, there's no project auto-generation system.
We decided to implement CMake which is a popular cross-platform build automation solution. CMake allows us to easily get project files for different IDEs without using Visual Studio.
We needed to be able to hook functions that contain no suitable first five bytes for a hook.
Some functions don't have a suitable first five bytes for recording the jump to the hook. For example, when assembling with msvs 2015 in x64 release with the /MT switch, the free function doesn't contain a suitable first five bytes:
This situation occurs when the function code assembler contains a conditional or unconditional jump or call to another Windows API function in the first five bytes. In this case, mhook cannot transfer this code to its layer, because the jump addressing will be incorrect and these jumps will be invalid. Mhook can handle unconditional jumps but not conditional ones.
We solved this issue using the free function, which has the je operator at the start. A conditional jump should be transferred to the mhook layer and then the instruction and the jump address should be changed so that it points to the same location as before the transfer.
The free function uses near je jump which sets a one-byte offset from the current position. The mhook layer can be located farther than the path that can be stored in one byte. That’s why we replaced the jump instruction for je with the rel32 argument (a 32-bit offset from the current position).
The system compiles a new jump address by subtracting the target address in the layer from the address where the jump used to lead.
This solution is suitable for near je and near jne since their opcodes and the opcodes of the corresponding long jumps are almost the same.
We eliminated an issue with infinite recursion.
When trying to set hooks for certain system functions, various issues occurred such as a call stack overflow.
Functions are called directly inside mhook after setting the jump leading to the hook from the system function and before modifying the layer that leads back to the system function.
We transferred the layer recording higher in the code, so that between the jump setting and the layer modification there are no calls to system functions.
We eliminated deadlocks.
After migrating to NtQuerySystemInformation, deadlocks appear in mhook.
When migrating to NtQuerySystemInformation, we allocated a dynamic buffer in the heap where thread information is stored. CreateToolhelp32Snapshot handles this itself and returns only HANDLE.
Here's how the whole process works:
- A buffer is allocated to get information about the threads
- All threads are suspended all threads
- The hook is set
- The buffer with information about threads is cleared
- The thread is executed.
This sequence contains a hard-to-detect bug. If any thread manages to grab the free lock, then our attempt to clear the buffer results in a deadlock because the thread that has captured the free lock isn't active.
To reproduce this bug, you can create several threads that allocate and free memory in the heap while another separate thread sets and removes hooks:
We found several different solutions for this problem. First, we simply moved the buffer clear until after all threads have resumed. But then we decided to use VirtualAlloc/VirtualFree instead of malloc/free. Since memory allocation when installing the hook occurs only a few times (and out of the loop), it doesn’t lead to any measurable performance losses.
We eliminated a bug whereby different hooks lead to the same handler.
When you install hooks for functions from different modules and the distance in memory between these modules exceeds 2GB, the addresses of hook handlers are recorded incorrectly. For example, let’s set the hook for a function from module 1, then for a function from module 2 located in memory at a distance of more than 2GB from the first hook). Let's then install two more hooks for the functions from the first module. As a result, the last two hooks will lead to the same handler, which they shoudn't.
In the BlockAlloc function, while adding a new memory block for the module, the allocated memory moves to the cycled list. In the original code, you should not set the pointer to the previous element for the list head. It remains zero.
After adding a new memory block, the following happens:
- When searching for a free memory block to set the hook, since the pointer to the previous element is zero, the pointer to the previous element isn’t overwritten by the current element.
- The code adds the current item to the list of memory blocks used for hooks. However, this element remains in the list of free memory blocks. The next pointer still points to this element from the previous element.
- Every time you try to set a hook in the current module, the first memory block you'll find is the one from point 1, although, it's assigned to the hook.
Thus, all other hooks in this module will lead to the handler of the last hook set in this module.
We replaced the pointer to the previous element of the list head with a pointer to the last element of this list, as it should be in the cyclical list.
With our improvements, we managed to increase mhook’s performance tenfold and the speed of the hook setting process nearly threefold. In addition, we easily got project files for different IDEs without using Visual Studio and managed to hook functions that don’t contain a suitable first five bytes for recording a jump to the necessary hook. Furthermore, we eliminated some bugs that led to deadlock, infinite recursion, and hooks leading to the wrong function.
You can download our improved version of mhook here: https://github.com/apriorit/mhook
OurApriorit System Programming team will continue supporting our version of mhook, so feel free to create an issue or send your own pull requests to participate in the development.