Testing Malware Detection Systems: ROP Chain Exploit Example

This article covers virus detection system testing and is written for quality assurance specialists with no experience in testing malware detection systems. We provide recommendations for organizing your testing process to ensure software quality and talk about key concepts and principles related to vulnerabilities and exploits.

Operating system security issues
Exploit protection systems
Potential vulnerabilities
Preparing documentation
Environment configuration and testing
Example of testing ROP UAF exploit protection
Conclusion

Operating System Security Issues

As software grows and improves with time, it also accumulates vulnerabilities, prompting developers to release security updates. Moreover, security issues in operating systems are becoming more prominent. With each new operating system release, older versions get fewer and fewer updates until vendors fully stop supporting them.

However, operating systems with known vulnerabilities continue to be used by companies that are either unwilling to upgrade or want to ensure compatibility with legacy software. So the issue of legacy operating systems is still very important.

Exploit Protection Systems

The best way to protect legacy systems is to use third-party software (Sophos, RevBits, McAfee) to mitigate vulnerabilities. Microsoft’s Enhanced Mitigation Experience Toolkit (EMET) is one of the most popular solutions for protecting against various types of exploits, but Microsoft terminated its support in 2017.

Potential Vulnerabilities

When we talk about a cybersecurity vulnerability, we mean a vulnerability that has been found in an operating system or a piece of software. A vulnerability is a weakness in an operating system or piece of software that can be exploited by an attacker to intentionally violate the integrity of that operating system or software and make it exhibit unintended behavior. Of course, in practice it’s almost impossible to eliminate all vulnerabilities, but we can significantly reduce their number.

Basic security solutions don’t affect users’ workflows, and by using such solutions you can substantially reduce the risk of external attacks. You can use either security tools built into your operating system or use specialized security software. In this article, we’ll discuss the latter.

Defining Malware

First of all, what is malware? Malicious software, or malware, is any code that is added to, changed in, or removed from an operating system or application in order to gain unauthorized access to a computer or to information stored on it. The main source of operating system malware infections is the transfer of infected files from an affected system to an unaffected one over a local network or the internet.

Types of Malware

As of today, there are several classifications of malware depending on their specifications. In this article, we’ll focus on malware that uses the internet and local networks to execute or spread malicious code.

Certain types of malicious software require local networks and the internet. Intrusion detection systems (IDSs) are capable of detecting worms that spread over LAN, preventing SQL attacks, performing real-time malware classification, and more.

Related services

Engineering for Cybersecurity Projects

Network-Based Malware

Malware that spreads across networks can be classified into the following types:

Adware automatically plays advertisements on a user’s computer.
Backdoor (or backdoor-based) malware is a malicious code included in infected software or an operating system with the intention of gaining unauthorized access to the target system. Backdoors can be used by hackers.
Sniffers are computer applications specially developed to capture network traffic and decode packets. Sniffers can be used to intercept confidential information.
Spyware collects information about users without their consent.
A trojan horse is malicious code disguised as a useful application in order to steal information or perform other actions.
A worm is self-replicating malware that exploits a software vulnerability on the target system to spread itself to other computers.
A botnet is a network of devices that have been infected by hackers with the intention of performing malicious activities such as DDoS attacks without the need to log into the target devices.

General Methods of Malware Protection

Exploiting software vulnerabilities allows attackers to infect a system with malware or gain unauthorized access. There’s a range of virus detection methods that can help you prevent malicious code from being executed. Windows operating system has built-in services that include the following security tools:

Executable-Space Protection is a set of technologies that prevent data execution by marking certain memory regions as non-executable. This technology protects against attacks that are based on inserting and running code from non-executable memory locations.
Structured Exception Handler Overwrite Protection (SEHOP) prevents attackers from exploiting vulnerable software on a user’s computer. The tool first verifies whether a thread’s exception handler list is intact and only allows any of the registered exception handlers to be called if it is.

However, these tools aren’t always enough. Only a combination of various malware detection techniques can provide adequate protection and prevent vulnerabilities from being exploited. The most common techniques are the following:

NullPage Allocation prevents attacks that exploit null pointer dereferencing. This technique sets special flags at the 0x00000000 address in virtual memory thereby controlling access to this memory location.
HeapSpray Protection prevents exploits from facilitating the execution of arbitrary code by pre-allocating common memory spaces.
Export Address Table Access Filtering (EAF) prevents operations from being performed on a memory page with an export address. When malicious code tries to access a memory page with an export address, EAF blocks access to an unknown loaded module.
Address Space Layout Randomization (ASLR) is a combination of techniques that make attacks based on the known address layout of target software ineffective.
Attack Surface Reduction (ASR) protects against attacks that rely on vulnerabilities in software. This technique blocks the vulnerable parts of software so any attacks on them are rejected.
Return-Oriented Programming (ROP) mitigations:
- Loaded libraries confirm the libraries loaded by an application.
- Memory protection checks mark memory regions as non-executable, so many attacks using shellcodes and ROP gadgets are rejected.
- Caller check verifies the instruction that was used to make a call. For instance, many ROP gadgets induce calls using the RET instruction rather than the CALL instruction.
- Execution flow simulation detects unwanted behavior that can be caused by ROP gadgets.
- Stack verification allows you to check whether a stack has pivoted from the real stack to a fake one, thereby prevent malicious code from being executed.
In our previous article, you can find more information on defending against ROP attacks and code examples.

Preparing Documentation

It’s necessary to define what to test and how to test antivirus software. Knowing what to test and how to test it is important if you want to ensure good coverage and avoid spending your time on unnecessary details.

What. Create a checklist where you will define a series of procedures for verifying that your product has been configured properly. It’s reasonable to divide your comprehensive checklist into several that cover different things: normal software operation, attack prevention, possible impact on the operating system and other functions, compatibility with other applications.

It’s important to fully test behaviors that might cause antivirus software to show false positives. Before you test a malware protection solution, make sure it is compatible with the security software you use in order to avoid conflicts. You can use additional applications to monitor system logs for signs of the impact that tested software has on other services/applications and the operating system as a whole. Additional applications can also help you monitor low-level logs that aren’t in the main application log (DebugView, etc.).

How. Testing methods will vary based on your checklist.

Test protection software for false positives in order to ensure that the additional cybersecurity solution has no impact on the regular user workflow or on other applications.

Using a list of known exploits, attack the protected software. It’s important to choose exploits for testing whose mechanisms correspond to the protection methods implemented in the tested software. At this stage, it’s reasonable to use an exploit database for testing.

You can refer to Special Publication 800-94 from the National Institute of Standards and Technologies (Guide to Intrusion Detection and Prevention Systems) when writing checklists.

After making your checklist, you can evaluate the scope of work and resources required. Using decomposition, you can draw up a detailed plan covering all procedures from the beginning of work to a report describing the current system security status.

Environment Configuration and Testing

It’s necessary to configure your environment for testing. In this case, you need two environments: an attacker and a target.

We’ll be using Kali Linux, a Linux distribution for penetration testing that comes with all the software you need, including for proof-of-concept (PoC) exploits that are used to demonstrate our method of protection. You just need to properly configure the PoC exploit for your attack (we describe how to do this in the testing example).

You can configure your target environment with the following steps:

Disable additional security software (it’s important to evaluate the operation of the tested system as is).
Install additional software to monitor the operating system’s state.
Install specific versions of certain software that is necessary for the PoC exploit to work properly (some exploits require Microsoft Office or Flash Player to be installed, for instance).

When your documentation software and testing environments are ready, the testing comes down to simply completing your checklist and analyzing the results. If some items take more time than you initially expected, you can skip them to make the testing less comprehensive.

All vulnerabilities and all actions taken by the tested system to protect against vulnerability exploits should be documented to make sure that the necessary security components are activated for certain exploits. You can skip logging some information over time but at the beginning you should log everything to make sure that each security component works properly.

As a rule of thumb, a PoC exploit needs to be written for a particular execution environment, so it’s important to thoroughly configure the target and check the PoC execution. Otherwise, it may be the case that it’s not your tested system that deflects the attack but rather the PoC that executes improperly.

Related services

Specialized Quality Assurance & Testing Solutions

Analyzing Detected Problems

Analyzing detected problems includes evaluating risks. There are two parameters for evaluating a problem’s risk: impact and probability. Each problem’s impact and probability are rated on a five-point scale, where 5 is a critical risk and 1 is a minor risk. Problems that have an impact and probability of 4 or greater should be analyzed.

The main considerations when analyzing problems are the magnitude of risk involved, the cost to fix the problem, and the resources required to do so. These considerations should be reflected in a task order.

You should pay attention to the vulnerabilities that are the quickest and easiest to fix. We recommend fixing the problems that are easiest to deal with immediately. Difficult vulnerabilities may take more than one day to mitigate. Don’t let small yet dangerous problems wait their turn to be fixed.

Besides, keep in mind that updates to protected software may impact its compatibility with security software. Therefore, regularly test your antivirus software with new alpha/beta versions of protected software to make sure that there won’t be any conflicts after release.

Example of Testing ROP UAF Exploit Protection

We used Metasploit to create our proof-of-concept exploit. This set of tools was specially developed for designing and debugging exploits. In addition, it includes a database of operation codes (opcodes), a shellcodes archive, and data on cybersecurity research. Metasploit includes a relatively large database of PoCs for various types of exploits that you can use for attacking a target machine.

The principle of executing return-oriented programming use-after-free exploits is clearly explained in the example of the classic Flash exploit by Guanxing Wen, a member of the Pangu Team:

“For instance, a heap overflow exploit sprays Vectors and creates memory holes by freeing some Vectors. Vulnerable buffer is created to occupy one of the memory holes, corrupting the length field of a Vector object by triggering an overflow. A Vector with a large length is then utilized to search the process memory for ROP gadgets. Then ROP chain is triggered by a fake virtual function table. For “use-after-free” bugs, the exploitation process is largely similar to that of heap overflow exploits. With a proper heap memory layout, vulnerable objects falling in one of the memory holes are released because of the “free” in “use-after-free” and occupied with controlled Vector data. After the member function of a dangling pointer is invoked, a fully controllable memory write will corrupt one of the length fields. Other aspects are similar to that of heap overflow exploits.”

Here’s how we tested ROP UAF exploit protection:

Using Metasploit, we created a web server to host our web page that will execute the PoC exploit.
We set up the environment on the testing machine so we could successfully execute our PoC exploit. When setting up your own environment, you may need the correct versions of various software applications such as Flash player, Microsoft Office, and so on, as well as programs to monitor the system’s state during the attack: DebugView (for reviewing logs), Process Explorer (for monitoring the state of processes including resource consumption, CPU load status, external dll injection), netstat (for reviewing active network connections), and Fiddler (for monitoring network traffic).
We configured exploit settings to attack specific software. We set up a payload (code that will be implemented if the exploit is successful), adjusted runtime settings (the operating system address and port), and specified the type of system being targeted. Configuration requirements will vary depending on the exploit used.
During testing, we disabled DEP (Data Execution Prevention) and other security software except for the software being tested.
Finally, we launched the attack: the client followed the link to our infected page while we monitored the operating system behavior and logs. In some cases, you can observe unpredicted software behavior, crashes, hangs, and other impacts. In other cases, the impact isn’t obvious, resulting in memory leaks, high CPU utilization, external dll injection, establishment of network connections, and so on.

You should collect all results of testing, as this information may be important to ensure the workflow of the user whose operating system is under the control of your security tool.

You should also take into account that exploits with similar attack methods can be executed differently, so it’s not enough to check your product using only one exploit.

Examples of Testing Results

Report on testing of an ROP UAF exploit protection system

Configuration:

Target: Windows 7 SP1, IE8, Flash Player 16

Attacker: Kali Linux, Metasploit, CVE-2015-0313, CVE-2015-5119

Summary:

During testing, we configured exploits CVE-2015-0313 and CVE-2015-5119. We tested computer virus detection of our security system running on the protected system. Examples CVE-2015-0313 and CVE-2015-5119 demonstrate that techniques implemented in our product protect against PoC UAF exploits. We found corresponding notifications in the logs about attack detection and protection:

     2017-08-11 14:53:59.368 INFO  [1336] Trying to protect process     2017-08-11 14:53:59.377 FATAL [1336] Potentially insecure action was detected! Exiting...    2017-08-11 14:53:59.378 WARN  [1336] Process 0x3116. Potential attack detected. Origin URL: 192.168.23.145     2017-08-11 14:53:59.379 INFO  [1336] DLL Unloaded - process is terminating

We intentionally omit the name of the tested product, as it makes no difference for this article.

Conclusion

We’ve discussed one possible method for testing attack protection systems. We’ve also answered important questions that allowed us to define our scope of testing depending on our needs and budget. It’s crucial to thoroughly develop your testing strategy to fully test a malware detection system. If you miss something important in the beginning, like fileless malware detection practices, it may lead to significant consequences such as flaws in the final product or vulnerabilities in the protected system.