This article is aimed at helping you test drivers for Windows. Since there are many different types of drivers, we cover the specifics of each type and explain how the Windows device driver testing process differs. We also cover supplementary tools and – another quite important topic – driver signatures. 

Contents:

Definition and Types of Windows Drivers

 Driver Location in Windows

 Driver types

 The Relationship between a Driver and a Device

The Main Aspects of Windows Driver Testing

 Operating Systems

 Updates

 Hardware Dependency

How To Test Windows Drivers

Utilities for Driver Testing and Analysis

Error Localization

Driver Testing Report Sample

 

Written by:
Denys Rudov, Senior Tester at Driver Testing Team
Dmitriy Yurko, Test Designer at Driver Testing Team

Definition and Types of Windows Drivers

A driver is a software component that provides an interface for a physical or virtual device. A driver interprets high-level requests coming from user software and the operating system into low-level commands recognized by a device.

Driver Location in Windows

In Windows, drivers are stored as binary files with the .sys extension along with optional supplementary files with .inf and .cat extensions.

.inf Files

The .inf file is a driver installation file that describes the type of device that a driver is designed for, the location of driver files, and any dependencies. During driver installation, data is entered in the system registry from the .inf file. This registry serves as a database for a driver’s configuration.

.cat Files

A .cat catalogue file contains a list of cryptographic hash sums of all driver files. In Windows, installed drivers are usually stored in the %SystemRoot%\System32\drivers system catalogue, but they can also be stored in any other place. After installation, a driver is loaded in the system and is ready to work. Some driver types require a reboot after installation.

Driver Types

Drivers can be divided into user-mode and kernel-mode types depending on their mode of execution.

User-Mode Drivers

User-mode drivers provide an interface between user applications and other operating system components such as kernel-mode drivers. A printer driver is an example of a user-mode driver.

Kernel-Mode Drivers

Kernel-mode drivers are executed in the privileged kernel mode. Kernel-mode drivers are usually in the form of a chain formed during driver execution. The location of each driver in this chain is defined by its task.

Queries from user applications to a device pass through several drivers. Each of these drivers processes and filters the query, which is called an IRP packet (I/O request packet) according to Windows driver terminology. If a driver is loaded in the wrong place in the chain, then the query will be processed incorrectly and, as a result, the system may crash.

Figure 1 below represents a simplified model of such a query for disk-based input-output (I/O). A user app creates a query to read a file on the disk. This query passes through a chain of drivers, each of which processes incoming and outgoing packets.

Simplified model of a driver query from a user application
Figure 1: Simplified model of a driver query from a user application

In this article, we’ll focus on kernel-mode drivers.

Kernel-mode drivers can be divided into the following types:

Plug and play device drivers. These drivers ensure access to physical plug and play (PnP) devices and manage device power.
Non plug- and play drivers. These drivers enhance user application functionality and provide access to kernel-mode features that are unavailable via standard API calls. These drivers don’t work with physical devices.
File system drivers. These drivers provide access to the file system. They transform high-level queries for reading/recording files to low-level commands for a disk driver (reading/recording a sector on a physical disk).

There is a driver development model called the Windows Driver Model (WDM) as well as a Windows Driver Framework (WDF), which consists of the Kernel-Mode Driver Framework (KMDF) and User-Mode Driver Framework (UMDF). Both the WDM and WDF simplify the process of making driver code compatible across Windows versions.

Within the WDM are the following driver types:

Bus drivers. These drivers support a specific PCI, SCSI, USB, or other port, controlling the connection of new devices to the bus.
Functional drivers. These drivers ensure the functionality of a specific device. They usually support reading/recording operations and device power management.
Filter drivers. These drivers modify queries to a device. They can be situated either above or below a functional driver in the chain of drivers.

The Relationship between a Driver and a Device

Each kernel-mode driver works with a specific device, represented in Windows as a device object. This means that the final destination for an I/O query coming through a driver is always a physical or virtual device. This applies to both drivers of physical PnP devices as well as non-PnP software drivers. While testing drivers, it’s important to understand that more than one driver exists between a user application and a device. In the chain, each driver can influence the final result of the query to the device.

The Main Aspects of Windows Driver Testing

There are certain tests that are required for Windows driver testing in Linux, and Windows regardless of driver type. So before covering the nuances of testing different types of drivers, we’ll consider their common aspects.

Operating Systems

First, you always have to keep in mind that a particular driver can behave differently on different operating systems. Furthermore, you need to take different kernel versions into account because they can differ even within the same operating system. For instance, Windows 7 and Windows 7 sp1 have different kernels. Therefore, you must test as many systems as possible. It’s worth mentioning that Microsoft supports Windows versions starting from Windows 7/2008. You also have to take into account that the most popular Windows versions now are Windows 7 and 10.

Updates

It’s necessary to check critical situations for a driver such as shutdown, reboot, and reset. You should also keep a system’s security systems in mind: firewalls, data execution prevention (DEP), user account control (UAC), and antivirus software. Operating system updates can also influence driver functionality. Therefore, it’s crucial to perform testing with the latest updates. In addition, you also need to test driver updates.

Hardware Dependency

Besides software dependency, there’s also hardware dependency. That’s why you have to check how a driver works with various processor and kernel configurations with an enabled and disabled page file. While testing a driver, you have to enable a driver verifier, which will create an additional load for the driver. During a testing process, check the correctness of driver installation and uninstallation, system reset, and hibernation.

How To Test Windows Drivers

Testing a driver using a real machine isn’t always secure since incorrect operation can lead to serious consequences. That’s why you should test a driver in a virtual machine until it’s stable.

File System Filter Drivers

As the name suggests, system filter drivers work with file systems. Therefore, while testing such drivers, you should use file systems such as NTFS, FAT32, exFAT, and ReFS.

To test a Windows driver correctly, you should take into account that various file managers can be used besides Explorer, such as FAR or Windows Total Commander. And don’t forget about complex file system changes in addition to simple operations such as copying, deleting, and renaming.

Complex file system changes include:

  • Mounting/unmounting new disks:
    • ISO images;
    • Network disks
    • Virtual hard disks
    • USB flash drives
  • Making sector configuration changes (changing drive letter or name)
  • Actions performed on a disk:
    • Formatting sectors
    • Shrinking sectors
    • Defragmenting sectors
    • Checking for errors
    • Compressing sectors
    • Deleting sectors
    • Making a disk dynamic
    • Converting a disk in GPT/MBR
    • Creating a new sector.

You also have to check:

  • Various hardware configurations (SSDs and HDDs with different capacities)
  • Driver behavior when stopping and starting services or installing/uninstalling an app
  • How a driver works with an encrypted disk using Windows tools
  • Driver compatibility with antivirus software, as such software is also a filter driver.

Virtual Storage Driver

Before testing a Windows driver created for virtual storage, you should understand that your file system will be stable. For virtual storage drivers, you should check the following things:

  • How the driver works with files and folders when
    • opening
    • creating
    • editing
    • saving
    • copying
    • removing
    • renaming
    • deleting
  • How the driver works with a search in files and folders.
  • How the driver works with files that have names containing
    • lots of characters
    • digits
    • special symbols
    • spaces
    • hieroglyphs
    • non-unicode symbols
    • cyrillic characters
  • How the driver works with files of different formats:
    • text
    • images
    • archives
    • Microsoft Office files
  • How the driver works with files with various attributes:
    • read-only
    • hidden
    • system
    • archive
  • How the driver handles changing file permissions and using various NTFS functions:
    • compression
    • encryption
  • Whether shortcuts (symlink and hard link) and hidden copies are correct.
  • How the driver handles files of different sizes:
    • very small
    • many very small files
  • How the driver works with folders that contain a large number of subfolders (more than five).
  • How the driver handles conflicts, for instance copying a file with the name of a file that already exists in the destination or cancelling copying or deleting.
  • How the driver handles saving a file downloaded from the internet or a shared network disk.
  • Disk mounting/unmounting in both standard situations and edge cases. For instance, try unmounting while copying to storage, then check whether the disk has been successfully mounted after rebooting the system.
  • The disk’s read/write speed.

USB Device Driver

With USB driver testing, you should try to cover as many USB devices as possible. You can start with the most popular, such as flash drives, printers, scanners, mice, keyboards, portable hard drives, smartphones, and card readers. But you should also test less popular devices such as Bluetooth devices, Ethernet devices, USB hubs, microphones and headsets, webcams, and CD-ROM drives.

You should take various USB interfaces into account: USB 1.0, 2.0, 3.0, and 3.1. In addition, don’t forget about unplugging/plugging devices, disabling safe and unsafe devices, and deleting devices in the device manager. Furthermore, check the device driver installation and uninstallation.

Utilities for Driver Testing and Analysis

There are numerous tools for testing Windows drivers that allow you to monitor the status of the driver in the system, verify its functionality, and perform testing.

Built-in Windows utilities are enough to get basic information regarding a driver’s status (e.g. whether it’s loaded in the system).

Built-in Windows utilities:

  • Msinfo32
  • Driverquery
  • Sc Driver Verifier

Sc Driver Verifier is a built-in utility that allows you to verify driver functionality. To deeply analyze test drivers, you’ll need additional tools available in the Windows Driver Kit (WDK).

Built-in Windows Utilities

Windows System Information (msinfo32)

Msinfo32 allows you to get a list of all registered drivers in the system, the type of each driver, its current status (loaded/not loaded), and start mode (System/Manual).

To call the System Information console, call the Run dialog using Win+R and launch msinfo32. On the left sidebar of the System Information console, choose the following tabs: Software Environments > System Drivers.


System Information console

This utility allows you to review and store information about registered drivers. It also lets you view a list of drivers from a remote computer if you have access to its Windows Management Instrumentation (WMI). This option is located in View > Remote Computer.

Driverquery, a Command Line Utility

Driverquery provides information similar to that found in msinfo32. It can be launched through cmd using the driverquery command:

cmd with the driverquery command

Additional parameters allow you to modify the output to the console:
/V is the command for detailed output. It allows you to get driver status information similar to that shown by msinfo32.
/SI provides information about signed drivers.
/S system allows you to get the information about a driver on a remote system.

The sc Command for Communicating with the Service Control Manager

The sc, command allows you to review a driver’s status and launch or stop the driver. To see a list of drivers, run the following command:

sc query type= driver
sc, command in cmd

Windows Driver Kit Utilities

The Windows Driver Kit (WDK) provides a wide set of tools for driver testing. WDK is integrated with MS Visual Studio, but it can also be used as an independent set of utilities.

The WDK contains a set of testing modules called Device Fundamentals Tests as well as other particular utilities that allow you to manage devices and drivers, monitor resource usage, and has utilities for verification, and so on.

Device Fundamentals Test

A Device Fundamentals Test set consists of the following tests:

  • Concurrent Hardware and Operating System (CHAOS) test;
  • Coverage test
  • CPU stress test
  • Driver installation test
  • I/O test
  • Penetration test
  • Plug and play test;
  • Reboot test
  • Sleep test

To perform testing, WDTF Simple I/O plugins must support your tested device. Follow this link to learn more about WDTF Simple I/O plugins.

Device Fundamental Tests are organized in the form of dll libraries and are situated in the %ProgramFiles%\Windows Kits\10\Testing\Tests\Additional Test directory (in Windows 10). These tests can be launched by the TE.exe utility that’s part of the Text Authoring and Execution Framework (TAEF), and they have to be installed with the WDK. You can find TE.exe in the %ProgramFiles%\Windows Kits\10\Testing\Runtimes\TAEF directory.

Here’s an example of how we might launch a test:

TE.exe Devfund_Device_IO.dll /P:”DQ=DriverBinaryNames=testdriver.sys”

At this stage, we launch a device I/O test with a test driver called testdriver.sys as a parameter. Device Fundamentals Tests and TAEF are both perfectly suitable for automated driver testing.

Windows Device Console (devcon.exe)

The Windows Device Console is a command line utility that gives information about plug and play devices and their system drivers and manage devices and filter drivers for specific device classes. Using devcon, you can install, remove, connect, disconnect, and configure devices. Devcon allows you to set a template while searching for a specific device. An asterisk (*) can replace one or several symbols in queries.

Examples of commands:

  • devcon.exe hwids * displays a list of names and IDs of all devices
  • devcon.exe classes displays a list of all device classes
  • devcon.exe driverfiles * displays a list of driver files for all system devices
  • devcon.exe classfilter USBDevice upper displays filter drivers for the DiskDrive device class
  • devcon.exe /r classfilter DiskDrive upper !OldFilter +NewFilter replaces a filter driver for the DiskDrive device class

PoolMon (poolmon.exe)

The Memory Pool Monitor displays information about the allocation of downloadable and non-downloadable core memory. This utility is used for detecting memory leaks while testing.
Memory Pool Monitor (PoolMon)
The memory allocation statistics for a driver are sorted by tag rather than by driver name. This tag should be set in the driver code using the ExAllocatePoolWithTag and ExAllocatePoolWithQuotaTag procedures. If a tag is not set in the code, then the system sets a None tag. Therefore, localization of memory issues can be complicated.

Windows Hardware Lab Kit

The Windows Hardware Lab Kit (HLK) is a framework for testing Windows 10-based devices. For Windows device testing on Windows 7, Windows 8, and Windows 8.1, you should use Windows HLK’s predecessor, the Windows Hardware Certification Kit (HCK).

While testing with Windows HLK, you should use an environment consisting of two components: an HLK test server (controller) and a test system (client). An HLK controller manages a set of tests, connects them with a test system, and defines an execution schedule. The controller allows you to manage testing on a set of client machines.

In the client system, devices and drivers are configured for further testing and test scenarios are performed.

The preparation and testing process can be completed with the following steps:

  1. Install the HLK controller on a dedicated machine.
  2. Install the agent on one or several test machines.
  3. Create a set of test machines that connects one or several machines in a logical manner.
  4. Create a controller-based project that defines the elements to be tested.
  5. Choose a test target, such as external devices of a test machine or software components such as filter drivers.
  6. Choose and launch tests. You can use playlists to perform a specific set of scenarios.
  7. Review and analyze test results.

The Windows HLK allows you to test many types of devices. You can learn more about Windows HLK by following this link.

Driver Verifier

The Driver Verifier is a built-in Windows utility created to verify kernel-mode drivers. The Driver Verifier allows you to detect driver bugs that can damage the operating system.

The Driver Verifier is most effective with manual or automated testing using WDK tools. The Driver Verifier is stored as a binary Verifier.exe file in the %WinDir%\system32 directory.

This utility can be launched in two modes: via command line and via the Driver Verifier Manager. To launch the command line version, run the Command Prompt as administrator and enter the verifier command with minimum one parameter, for instance help - verifier /?). To open the Driver Verifier Manager, run verifier without parameters.

Let’s look at a driver verification procedure using the Driver Verifier Manager as an example:

  1. Run the Driver Verifier Manager: Win + R > verifier
  2. Choose a set of standard tests or create custom tests. The manager also can display and delete current settings as well as display information about verified drivers:Driver Verifier Manager
  3. Choose one or several drivers to verify.
  4. Reboot the computer. The driver will be tested according to the chosen settings until it is removed from the list of verified drivers.

Driver Verifier Standard Settings

Below, we’ll describe the standard settings of the Driver Verifier for Windows 10. The list of standard and supplementary settings may be different in different Windows versions.

Here are the standard options for the Driver Verifier in Windows 10:

  • Special Pool
  • Force IRQ Checking
  • Pool Tracking
  • I/O Verification
  • Deadlock Detection
  • DMA Verification
  • Security Checks
  • Miscellaneous Checks
  • DDI Compliance Checking

Let’s look into each setting in more detail.

Special Pool

The Special Pool option allows the Driver Verifier to allocate memory for a driver in a special place that’s monitored for memory damage, in other words access to the released memory.

Force IRQ Checking

In Windows, a driver cannot access unloaded memory with high IRQL if the spin lock option is enabled. The Force IRQ Checking option detects such issues.

Pool Tracking

Pool Tracking monitors a driver’s memory allocation. The Driver Verifier checks that the allocated memory for a driver eventually gets released. This helps detect memory leaks.

I/O Verification

The I/O Verification option detects a driver’s incorrect use of input/output procedures. In Windows 7 and higher, this option also contains an Enhanced I/O Verification function that performs stress testing for the following elements: PnP IRPs, power IRPs, and WMI IRPs.

With Deadlock Detection, the Driver Verifier monitors synchronization objects such as mutexes and spin locks used by a driver. This is how potential deadlocks can be detected.

DMA Verification

Using DMA Verification, the Driver Verifier monitors the use of Direct Memory Access procedures. The DMA allows devices to work directly with memory without the CPU.

Security Checks

The Driver Verifier detects security issues such as calls of kernel-mode procedures with user-mode memory addresses or incorrect parameters.

Miscellaneous Checks

With Miscellaneous Checks enabled, a driver is tested for potential errors that can lead to a driver or system crash – for example, if a driver releases memory that still contains working driver structures.

DDI Compliance Checking

A driver is checked for potential errors in the communication (Device Driver Interface) with a kernel interface of the operating system.
To efficiently detect bugs within the Driver Verifier, you should follow the recommendations below:

  1. Don’t verify several drivers at the same time except if that’s precisely your goal.
  2. Enable memory dump collection for cases where the operating system crashes (BSOD).
  3. If needed, enable debug mode and connect to the test system with a debugger using the network or COM/USB port.

Digital Driver Signature

In Windows XP, Windows Vista, and Windows 7, there is no strict requirement for a package signature in order to install a driver package. Therefore, you can easily install a driver without a signature. However, if a package isn’t signed, you’ll see this warning:
Windows Security warning
For a driver to be recognized as coming from a trusted publisher, the driver package has to be signed with a Windows Hardware Quality Labs (WHQL) signature in Window XP. In Windows Vista and Windows 7, a driver package has to be signed with a Trusted Root CA certificate. In Windows 8, Windows 8.1, and Windows 10 a driver package signature is required, as you cannot install a driver package without it. It used to be required to have the certificate encrypted with an SHA-1 algorithm. Now, SHA-1 is outdated and SHA-2 algorithms are usually applied to certificates. You can learn more about SHA-2 algorithms by following this link.

Driver .sys File Signature

Before running a driver in kernel mode, Windows checks the digital signature of the driver’s binary .sys file. It’s worth noting that Windows XP and Windows Vista 32-bit don’t require a digital driver signature. Windows Vista 64-bit, Windows 7, Windows 8, and Windows 8.1 require a signature with a certificate containing the Microsoft Code Verification Root in its root or with another certificate trusted by the kernel. Windows 10 version 1607 and newer requires a driver to be signed with the Windows Hardware Developer Center Dashboard portal.

To start a Windows driver test, you can temporarily disable the digital driver signature check. In Windows 10, you can do this the following way:

  1. Hold the Shift button and choose the Restart option in the main Windows menu.
  2. Select Troubleshoot -> Advanced Options -> Startup Settings -> Restart
  3. In Startup Settings, push F7 to choose the Disable driver signature enforcement option.

Startup Settings in main Windows menu

To test drivers in Windows with the Secure Boot option enabled, you must ensure that your drivers have valid signatures.

Error Localization

An existing bug in a driver can lead to a system crash. That’s why, besides defining specific steps, localizing a bug means understanding whether your driver has caused the BSOD or not. In order to determine that, you have to review the system memory dump. This is collected automatically after the BSOD and you can find it in the C:\Windows\Memory.dmp directory. To review the full kernel dump, you should tell the system to collect it. The kernel dump also contains necessary information regarding free memory on the disk. To get this information, you should open Advanced system settings > Startup and Recovery and click Settings.
System Properties in the Control Panel Home
Check whether the Complete memory dump or Kernel memory dump option is enabled.
Startup and Recovery Settings
In these settings, you can change the storage directory for the memory dump.

Once you have the full dump, you should analyze it. This is where WinDbg will be useful. Before using this tool, you have to download the specific Microsoft symbols. You can read this guide on how to do it.

Open the dump and run the !analyze –v command. Pay attention to the stack and you’ll see the reason for the BSOD. In some cases, you won’t be able to get the dump file from the system because it’s continually crashing. In this case, you should use the Windows advanced startup options. You can then run the system in several special modes. The simplest and most reliable one is afe mode. In safe mode, your options within the system will be limited. But the only thing you need to get is the dump file from the C:\Windows folder, and this mode allows you to get it.

You can also try the Disable automatic restart on system failure option, which may help prevent continual restarts. In case of a system crash, you should check whether a driver verifier was involved. This information can be very helpful for developers when they try to reproduce the bug. Besides a system crash, you can also face other issues such as those related to functionality or performance.

To localize functionality issues, you should take the following actions:

Try to recreate the same issue in another environment (in a different operating system, without antivirus software, on another file system, using a real machine, etc.).
You also can try other conditions, such as different types of files and different file sizes.
If the issue is related to your USB device, check other devices of different types.
If the network is a potential cause of an error with a network device driver, check various network settings (latency, bandwidth, enable or disable the firewall).
When the issue appears solely for users with specific permissions, use different permissions (administrator or standard) to localize the issue.

If the issue is related to performance, your actions will depend on the element whose performance you are trying to improve:

  • If the issue is in the network device driver, emulate network latencies or try recreating this issue in a high-speed network.
  • If the issue is related to file operations, then check it using a different number of files of different sizes.
  • If the issue is related to the USB camera, check its performance with different applications.

Driver Testing Report Sample

Testing type

Regression testing

Driver

DriverFsfilter.sys (file system filter driver)

Version

1.0.0.1

Windows Versions

Windows 10 x64, Windows 8.1 x64, Windows 7 x64

Tests performed

  • Installation/Uninstallation
  • Processing of read/write operations:
    • Sectors with the following file systems: NTFS, FAT32, exFAT, ReFS
    • Different file sizes (1Kb to 10 Gb).
    • On physical mounted and network disks
  • A stress test with unsafe disk disconnection
  • Compatibility with Windows 10 updates (from 1607 to 1703)
  • Driver Verifier: standard settings plus the Randomized low resources simulation option

Errors

Periodic BSOD emerging with Randomized low resources simulation enabled in Driver Verifier

Result

With the Randomized low resources simulation option enabled in Driver Verifier, the driver periodically causes the BSOD in Windows 10 x64.
In other situations, the driver is stable.

Application

Windows 10 x64 Full memory dump attached: MEMORY.dmp

 

Conclusion

In this article, we have described the main types of drivers and approaches and utilities for testing them. Windows kernel driver testing differs significantly from testing desktop apps. If a driver contains a bug, it usually influences the stability of the whole system and eventually leads to the BSOD.

Detecting, localizing, and eliminating driver errors significantly decreases the risk of unstable system behavior for end-users.

This article covers virus detection system testing and is written for quality assurance specialists with no experience in testing malware detection systems. We provide recommendations for organizing your testing process to ensure software quality and talk about key concepts and principles related to vulnerabilities and exploits.

Written by:

Vyacheslav Skoroda,

Tester at Driver Testing Team

 

Contents:

Operating system security issues

Exploit protection systems

Potential vulnerabilities

    Defining malware

    Types of malware

        Network-based malware

    General methods of malware protection

Preparing documentation

Environment configuration and testing

    Analysing detected problems

Example of testing ROP UAF exploit protection

   Examples of testing results

Conclusion

   Useful links

Operating System Security Issues

As software grows and improves with time, it also accumulates vulnerabilities, prompting developers to release security updates. Moreover, security issues in operating systems are becoming more prominent. With each new operating system release, older versions get fewer and fewer updates until vendors fully stop supporting them.

However, operating systems with known vulnerabilities continue to be used by companies that are either unwilling to upgrade or want to ensure compatibility with legacy software. So the issue of legacy operating systems is still very important.

Exploit Protection Systems

The best way to protect legacy systems is to use third-party software (Sophos, RevBits, McAfee) to mitigate vulnerabilities. Microsoft’s Enhanced Mitigation Experience Toolkit (EMET) is one of the most popular solutions for protecting against various types of exploits, but Microsoft terminated its support in 2017.

Potential Vulnerabilities

When we talk about a cybersecurity vulnerability, we mean a vulnerability that has been found in an operating system or a piece of software. A vulnerability is a weakness in an operating system or piece of software that can be exploited by an attacker to intentionally violate the integrity of that operating system or software and make it exhibit unintended behavior. Of course, in practice it’s almost impossible to eliminate all vulnerabilities, but we can significantly reduce their number.

Basic security solutions don’t affect users’ workflows, and by using such solutions you can substantially reduce the risk of external attacks. You can use either security tools built into your operating system or use specialized security software. In this article, we’ll discuss the latter.

Defining Malware

First of all, what is malware? Malicious software, or malware, is any code that is added to, changed in, or removed from an operating system or application in order to gain unauthorized access to a computer or to information stored on it. The main source of operating system malware infections is the transfer of infected files from an affected system to an unaffected one over a local network or the internet.

Types of Malware

As of today, there are several classifications of malware depending on their specifications. In this article, we’ll focus on malware that uses the internet and local networks to execute or spread malicious code.

Certain types of malicious software require local networks and the internet. Intrusion detection systems (IDSs) are capable of detecting worms that spread over LAN, preventing SQL attacks, performing real-time malware classification, and more.

Network-Based Malware

Malware that spreads across networks can be classified into the following types:

  • Adware automatically plays advertisements on a user’s computer.
  • Backdoor (or backdoor-based) malware is a malicious code included in infected software or an operating system with the intention of gaining unauthorized access to the target system. Backdoors can be used by hackers.
  • Sniffers are computer applications specially developed to capture network traffic and decode packets. Sniffers can be used to intercept confidential information.
  • Spyware collects information about users without their consent.
  • A trojan horse is malicious code disguised as a useful application in order to steal information or perform other actions.
  • A worm is self-replicating malware that exploits a software vulnerability on the target system to spread itself to other computers.
  • A botnet is a network of devices that have been infected by hackers with the intention of performing malicious activities such as DDoS attacks without the need to log into the target devices.

General Methods of Malware Protection

Exploiting software vulnerabilities allows attackers to infect a system with malware or gain unauthorized access. There’s a range of virus detection methods that can help you prevent malicious code from being executed. Windows operating system has built-in services that include the following security tools:

  • Executable-Space Protection is a set of technologies that prevent data execution by marking certain memory regions as non-executable. This technology protects against attacks that are based on inserting and running code from non-executable memory locations.
  • Structured Exception Handler Overwrite Protection (SEHOP) prevents attackers from exploiting vulnerable software on a user’s computer. The tool first verifies whether a thread’s exception handler list is intact and only allows any of the registered exception handlers to be called if it is.

However, these tools aren’t always enough. Only a combination of various malware detection techniques can provide adequate protection and prevent vulnerabilities from being exploited. The most common techniques are the following:

  • NullPage Allocation prevents attacks that exploit null pointer dereferencing. This technique sets special flags at the 0x00000000 address in virtual memory thereby controlling access to this memory location.
  • HeapSpray Protection prevents exploits from facilitating the execution of arbitrary code by pre-allocating common memory spaces.
  • Export Address Table Access Filtering (EAF) prevents operations from being performed on a memory page with an export address. When malicious code tries to access a memory page with an export address, EAF blocks access to an unknown loaded module.
  • Address Space Layout Randomization (ASLR) is a combination of techniques that make attacks based on the known address layout of target software ineffective.
  • Attack Surface Reduction (ASR) protects against attacks that rely on vulnerabilities in software. This technique blocks the vulnerable parts of software so any attacks on them are rejected.
  • Return-Oriented Programming (ROP) mitigations:
    • Loaded libraries confirm the libraries loaded by an application.
    • Memory protection checks mark memory regions as non-executable, so many attacks using shellcodes and ROP gadgets are rejected.
    • Caller check verifies the instruction that was used to make a call. For instance, many ROP gadgets induce calls using the RET instruction rather than the CALL instruction.
    • Execution flow simulation detects unwanted behavior that can be caused by ROP gadgets.
    • Stack verification allows you to check whether a stack has pivoted from the real stack to a fake one, thereby prevent malicious code from being executed.
    • You can find more information on how to defend against ROP attacks in our article about ROP exploit protection.

Preparing Documentation

It’s necessary to define what to test and how to test antivirus software. Knowing what to test and how to test it is important if you want to ensure good coverage and avoid spending your time on unnecessary details.

What. Create a checklist where you will define a series of procedures for verifying that your product has been configured properly. It’s reasonable to divide your comprehensive checklist into several that cover different things: normal software operation, attack prevention, possible impact on the operating system and other functions, compatibility with other applications.

It’s important to fully test behaviors that might cause antivirus software to show false positives. Before you test a malware protection solution, make sure it is compatible with the security software you use in order to avoid conflicts. You can use additional applications to monitor system logs for signs of the impact that tested software has on other services/applications and the operating system as a whole. Additional applications can also help you monitor low-level logs that aren’t in the main application log (DebugView, etc.).

How. Testing methods will vary based on your checklist.

Test protection software for false positives in order to ensure that the additional cybersecurity solution has no impact on the regular user workflow or on other applications.

Using a list of known exploits, attack the protected software. It's important to choose exploits for testing whose mechanisms correspond to the protection methods implemented in the tested software. At this stage, it’s reasonable to use an exploit database for testing.

You can refer to Special Publication 800-94 from the National Institute of Standards and Technologies (Guide to Intrusion Detection and Prevention Systems) when writing checklists.

After making your checklist, you can evaluate the scope of work and resources required. Using decomposition, you can draw up a detailed plan covering all procedures from the beginning of work to a report describing the current system security status.

Environment Configuration and Testing

It’s necessary to configure your environment for testing. In this case, you need two environments: an attacker and a target.

We’ll be using Kali Linux, a Linux distribution for penetration testing that comes with all the software you need, including for proof-of-concept (PoC) exploits that are used to demonstrate our method of protection. You just need to properly configure the PoC exploit for your attack (we describe how to do this in the testing example).

You can configure your target environment with the following steps:

  • Disable additional security software (it’s important to evaluate the operation of the tested system as is).
  • Install additional software to monitor the operating system’s state.
  • Install specific versions of certain software that is necessary for the PoC exploit to work properly (some exploits require Microsoft Office or Flash Player to be installed, for instance).

When your documentation software and testing environments are ready, the testing comes down to simply completing your checklist and analyzing the results. If some items take more time than you initially expected, you can skip them to make the testing less comprehensive.

All vulnerabilities and all actions taken by the tested system to protect against vulnerability exploits should be documented to make sure that the necessary security components are activated for certain exploits. You can skip logging some information over time but at the beginning you should log everything to make sure that each security component works properly.

As a rule of thumb, a PoC exploit needs to be written for a particular execution environment, so it’s important to thoroughly configure the target and check the PoC execution. Otherwise, it may be the case that it’s not your tested system that deflects the attack but rather the PoC that executes improperly.

Analyzing Detected Problems

Analyzing detected problems includes evaluating risks. There are two parameters for evaluating a problem’s risk: impact and probability. Each problem’s impact and probability are rated on a five-point scale, where 5 is a critical risk and 1 is a minor risk. Problems that have an impact and probability of 4 or greater should be analyzed.

The main considerations when analyzing problems are the magnitude of risk involved, the cost to fix the problem, and the resources required to do so. These considerations should be reflected in a task order.

You should pay attention to the vulnerabilities that are the quickest and easiest to fix. We recommend fixing the problems that are easiest to deal with immediately. Difficult vulnerabilities may take more than one day to mitigate. Don’t let small yet dangerous problems wait their turn to be fixed.

Besides, keep in mind that updates to protected software may impact its compatibility with security software. Therefore, regularly test your antivirus software with new alpha/beta versions of protected software to make sure that there won’t be any conflicts after release.

Example of Testing ROP UAF Exploit Protection

We used Metasploit to create our proof-of-concept exploit. This set of tools was specially developed for designing and debugging exploits. In addition, it includes a database of operation codes (opcodes), a shellcodes archive, and data on cybersecurity research. Metasploit includes a relatively large database of PoCs for various types of exploits that you can use for attacking a target machine.

The principle of executing return-oriented programming use-after-free exploits is clearly explained in the example of the classic Flash exploit by Guanxing Wen, a member of the Pangu Team:

“For instance, a heap overflow exploit sprays Vectors and creates memory holes by freeing some Vectors. Vulnerable buffer is created to occupy one of the memory holes, corrupting the length field of a Vector object by triggering an overflow. A Vector with a large length is then utilized to search the process memory for ROP gadgets. Then ROP chain is triggered by a fake virtual function table. For “use-after-free” bugs, the exploitation process is largely similar to that of heap overflow exploits. With a proper heap memory layout, vulnerable objects falling in one of the memory holes are released because of the “free” in “use-after-free” and occupied with controlled Vector data. After the member function of a dangling pointer is invoked, a fully controllable memory write will corrupt one of the length fields. Other aspects are similar to that of heap overflow exploits.”

Here’s how we tested ROP UAF exploit protection:

  1. Using Metasploit, we created a web server to host our web page that will execute the PoC exploit.
  2. We set up the environment on the testing machine so we could successfully execute our PoC exploit. When setting up your own environment, you may need the correct versions of various software applications such as Flash player, Microsoft Office, and so on, as well as programs to monitor the system’s state during the attack: DebugView (for reviewing logs), Process Explorer (for monitoring the state of processes including resource consumption, CPU load status, external dll injection), netstat (for reviewing active network connections), and Fiddler (for monitoring network traffic).
  3. We configured exploit settings to attack specific software. We set up a payload (code that will be implemented if the exploit is successful), adjusted runtime settings (the operating system address and port), and specified the type of system being targeted. Configuration requirements will vary depending on the exploit used.
  4. During testing, we disabled DEP (Data Execution Prevention) and other security software except for the software being tested.
  5. Finally, we launched the attack: the client followed the link to our infected page while we monitored the operating system behavior and logs. In some cases, you can observe unpredicted software behavior, crashes, hangs, and other impacts. In other cases, the impact isn’t obvious, resulting in memory leaks, high CPU utilization, external dll injection, establishment of network connections, and so on.

You should collect all results of testing, as this information may be important to ensure the workflow of the user whose operating system is under the control of your security tool.

You should also take into account that exploits with similar attack methods can be executed differently, so it’s not enough to check your product using only one exploit.

Examples of Testing Results

Report on testing of an ROP UAF exploit protection system

Configuration:

Target:         Windows 7 SP1, IE8, Flash Player 16

Attacker:      Kali Linux, Metasploit, CVE-2015-0313, CVE-2015-5119

 

Summary:

During testing, we configured exploits CVE-2015-0313 and CVE-2015-5119. We tested computer virus detection of our security system running on the protected system. Examples CVE-2015-0313 and CVE-2015-5119 demonstrate that techniques implemented in our product protect against PoC UAF exploits. We found corresponding notifications in the logs about attack detection and protection:

 
   2017-08-11 14:53:59.368 INFO  [1336] Trying to protect process 
   2017-08-11 14:53:59.377 FATAL [1336] Potentially insecure action was detected! Exiting...
   2017-08-11 14:53:59.378 WARN  [1336] Process 0x3116. Potential attack detected. Origin URL: 192.168.23.145 
   2017-08-11 14:53:59.379 INFO  [1336] DLL Unloaded - process is terminating 

We intentionally omit the name of the tested product, as it makes no difference for this article.

Conclusion

We’ve discussed one possible method for testing attack protection systems. We’ve also answered important questions that allowed us to define our scope of testing depending on our needs and budget. It’s crucial to thoroughly develop your testing strategy to fully test a malware detection system. If you miss something important in the beginning, it may lead to significant consequences such as flaws in the final product or vulnerabilities in the protected system.

Useful links

https://www.usenix.org/legacy/event/sec09/tech/full_papers/kolbitsch.pdf

http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.403.9299&rep=rep1&type=pdf

http://csrc.nist.gov/publications/drafts/800-94-rev1/draft_sp800-94-rev1.pdf

https://www.blackhat.com/docs/eu-16/materials/eu-16-Wen-Use-After-Use-After-Free-Exploit-UAF-By-Generating-Your-Own-wp.pdf

https://en.wikipedia.org/wiki/Intrusion_detection_system

https://en.wikipedia.org/wiki/Return-oriented_programming

https://en.wikipedia.org/wiki/Computer_worm

 

 

 

 

This article covers the topic of email server security. Security measures covered here will allow you to greatly increase the level of protection of your email server and prevent any successful attacks.

Reducing a test set without losing coverage of major test cases is one of the main goals of any tester working on test design. In this article, we will discuss how to reduce the number of tests without losing major ones using the analysis of interconnections and the inner logic of a product. This article will be mainly useful for testers who often test different windows with a number of interconnected options and parameters.

So let’s see to which products and how we can apply the analysis of interconnections…

The most of testers at least once performed testing for compatibility of the developed application with the antiviruses. It’s widely known that antivirus software influences significantly the system functioning and also the work of the particular programs, and nowadays, the antivirus is almost must-have part of the computer system. Moreover, there are a lot of diverse malware protection products, and sometimes they can be unpredictable.

So, to adequately check the product for the compatibility with the antivirus software, you should apply some creative approaches. It is natural that you’ll need some knowledge for that – this is what we aim in this article.

In the first part of this article, we’ll consider the main types of antivirus programs, their features and specialties, and also the current statistics of antivirus product popularity. This is the theoretical minimum that a tester should know to start planning and performing antivirus compatibility testing. In the second part, we’ll discuss the role of such testing and its place in the software product life cycle, and also consider the problem situations that can appear during compatibility testing. This information is especially useful when planning and performing tests.

 

This article is for those who write or are going to write functional tests for any programs. The goal of this article is to give the flavor of the main techniques of test design and also to answer the question: What method is better to choose for testing your project?

When you first meet the concept of the exploratory testing you can think that its name fully exposes the sense of this sort of activity. But if you get a little bit deeper – this confidence weakens and disappears. Especially clear it can be seen when balancing between the concepts of the exploratory and wild testing, which in one hand are synonyms in QA industry, and in another (from the good sense point of view) – contrary one to another. It’s because what can have in common the exploration – serious, responsible and well-organized process – and something wild and chaotic, that is wild testing?

 

In this article, we will tell you our own way of Impact Analysis introduction and work with it. Why did we decide to introduce Impact Analysis in software testing in our teams? What practical benefit from this innovation have we got and are getting today? You will get answers to these and many other questions concerning this topic while reading this material.

 

This article is intended to provide full and detailed description of deployment of automated testing system consisting of Team Foundation Server + Microsoft Test Manager + System Center Virtual Machine Manager.

This article may be useful for Automated Testing Specialists and System Administrators.

Subscribe to updates