LSP (Layered Service Provider) based HTTP sniffer

This article describes how to create a simple sniffer for monitoring HTTP traffic on Windows OS. This tool is based on open source technology provided by Microsoft – LSP (Layered Service Provider). This technology is widely used by different software. The majority of these software products are antivirus, firewalls and traffic filtering applications.

To create this tool I`ve taken Microsoft Platform SDK sample (Program Files\Microsoft Platform SDK\Samples\NetDS\WinSock\LSP\) and implemented some additional logic for filtering HTTP traffic and collecting results in one separate storage.

1. Concept
2. Basic scheme
3. Getting started
4. Conclusion and Advices

Concept

The main idea of LSP technology is creation of a new provider that will be embedded into the chain of existing providers.

During the installation of your provider, you can choose its place in the new chain. So, the new chain will be rebuilt according to your new settings. In my case, the provider is installed only over the [TCP/IP] standard provider. Be careful with installation of new providers on your physical machine. If installation fails, it may produce a lot of troubles – loses of network and internet activity, crashes of some network applications. To avoid this problem it is strongly recommended to use Virtual machines to test and debug a provider.

In LSP provider, you have to override all methods of winsock library. Actually, Platform SDK sample is already defined with overridden logic so we only have to add the logic for intercepting, filtering and saving HTTP protocol traffic to the separate storage.

Basic scheme

This tool can be described by means of such scheme:

After installation of the tool, we have a lot of applications that have loaded our Test Provider (actually Test Provider is dll file, which is loaded into each application that uses winsock library). But we have to detect only HTTP traffic. So, there is a line with “hardcoded” port number, which will be inspected (this number is 80 – default HTTP protocol port), in the provider source code.

if((namelen >= sizeof(sockaddr_in)) && (((sockaddr_in*)name)->sin_port == htons(HTTPPort)))
    {
        SocketContext->intercept = TRUE;
    }

You can improve this tool and implement more elegant logic for saving its settings instead of “hardcoded” constants.

Also we have to detect HTTP requests. HTTP GET requests are detected by simple comparison with the “GET” string token. You can add detection of POST requests in the same way.

Also there is a Service component that collects all filtered data – it was created to avoid data corruption, which may occur if more than one application is inspected. All intercepted data is transmitted from browsers to this service. The Service application contains socket server (that is listening on 4004 port) so no trouble in synchronization of collecting data appears. Data storage is a simple text file, but it can be easily replaced with some more convenient and stable storage (e.g. any kind of database).

Getting Started

Test Solution contains such projects:

LSP component project. This project contains re-definition of the basic winsock methods. So, exactly at this place, you have to add your changes. In my case, this project contains Platform SDK sample, where i`ve added the logic for detecting 80 port connection and marking it as intercepted in the Connect method:

if((namelen >= sizeof(sockaddr_in)) && (((sockaddr_in*)name)->sin_port == htons(HTTPPort)))
    {
        SocketContext->intercept = TRUE;
    }

So, in future calls of the Send method, I can discover that this socket is used with HTTP protocol. I also establish connection to the Traffic Collector service in the Connect method. In the Send method, I`ve implemented the logic of detection HTTP requests and redirecting them to the service:

if (IsHTTPRequest(lpBuffers->buf) && SocketContext->intercept)
    {
        SetBlockingProvider(SocketContext->Provider);
        ret = SocketContext->Provider->NextProcTable.lpWSPSend(
            serviceConnection.GetSocket(),
            lpBuffers,
            dwBufferCount,
           lpNumberOfBytesSent,
           dwFlags,
            lpOverlapped,
            lpCompletionRoutine,
            lpThreadId,
            lpErrno
            );
        SetBlockingProvider(NULL);
    }

I`ve created specialized class that keeps one connection per loaded DLL file:

{code}class ServiceConnectionKeeper; {/code}

Its role is to keep connected socket. So, only one connection is established to the collector service.

Common – this project contains all utilities that were provided with Platform SDK sample. Some manipulations with GUID of LSP were made.
Installer – this is the installer of LSP. I`ve changed its main method – removed parsing of command line parameters and added search of TCP provider. Now, during the installation we search ID of the TCP provider and rebuild the provider chain. We layer our provider over TCP.

if (IsHTTPRequest(lpBuffers->buf) && SocketContext->intercept)
    {
        SetBlockingProvider(SocketContext->Provider);
        ret = SocketContext->Provider->NextProcTable.lpWSPSend(
            serviceConnection.GetSocket(),
            lpBuffers,
            dwBufferCount,
           lpNumberOfBytesSent,
           dwFlags,
            lpOverlapped,
            lpCompletionRoutine,
            lpThreadId,
            lpErrno
            );
        SetBlockingProvider(NULL);
    }

Service – this is traffic collector. It contains implementation of a simple windows service and also methods for installing and removing it. In the “main” function of this service, the listening socket server is implemented. All logic of manipulation with service was taken from MSDN. It accepts all incoming connections and starts a separate thread per each connected application. Started thread receives messages separated with “\r\n\r\n” and saves them to the storage:

do
    {
        result = recv(clientSocket, buffer, PACKSIZE, 0);
        if (result > 0)
        {
            response += std::string(buffer, result);
            do
            {
                position = response.find(messageTerminator);
                if (std::string::npos != position)
                {
                    if (!CollectorServer::Instance()->SaveData(std::string(response.begin(), response.begin() + position)))
                    {
                        return -1;
                    }
                    response = response.substr(position + messageTerminator.size());
                }
            } while (std::string::npos != position);
        }
        else
        {
            break;
        }
    } while (SOCKET_ERROR != result);

To start working with this tool you have to build the whole solution. After that, put the NSI script file to the solution output directory and compile it (the script file). As a result, you`ll get the setup.exe file.

During the installation process of setup.exe, all necessary components will be unpacked to their working folders (LSP.dll file will be stored in the system directory – %SYSTEMROOT%\\system32\\LSP.dll; Service executable and provider installer files will be stored in the Program Files directory). Also uninstall shortcut will be created on the desktop. History file is located at C://.

Conclusion and Advices

This article describes how to create your own provider and monitor all network traffic. But it is not the only way you can use this technology. You can also easily implement the logic for:

blocking HTTP request and answers;
modifying and replacing traffic;
blocking connection (like a firewall does);
intercept SSL encrypted data (it is even possible to implement man-in-the-middle interception).

Looking back to the experience of LSP software development, the best way to avoid pain of crashes and looses of internet connectivity during the development process is using Virtual machines. It is much more convenient to roll back to the previous snapshot than restore the previous state of your Windows OS.

The great part of existing antivirus software also uses this technology, so you can find them in the chain of the registered providers. This also can produce troubles during testing of your provider, because almost all antivirus software filters network traffic too. You may implement a logic for ignoring the selected applications. So Test provider will do nothing but transmitting the data if a parent application is in the ignore list.

This tool was designed only for 32-bit applications. But it can be easily ported for 64-bit applications. You just have to rebuild the provider in 64-bit configuration and set the LspCatalog64Only flag during the installation process.

To discover the changes in the Platform SDK sample that were made to create this tool you may use any merger. Also I have marked all code blocks that were added, with the “// ADDED” comment.

Also notice that you have to generate the new GUID for your provider to avoid collisions with the other providers.

Download Project Source (ZIP, 107 KB)