An increasing number of internet attacks is forcing enterprises to take control of their network activities as part of their security policies.
A wide range of intrusion detection systems (IDSs) have been designed to help enterprises protect their network infrastructures. But as a commercial IDS usually costs thousands of dollars, this software may be financially burdensome for small businesses. Fortunately, open source IDSs can also be efficiently used for these purposes. They offer modifiable plugins that can dynamically scan your network and ensure detection of intrusions from the internet.
In this article, we provide a detailed report of our study on analyzing network activities with the free Bro IDS and Intel Critical Stack. Our study proved the effectiveness of these systems for detecting network intruders in real-time. We also used the ELK Stack to visualize only collected data that’s useful for network intrusion analysis. This article may be useful for security administrators and DevSecOps who are looking for alternative ways of detecting network intrusions and suspicious activities.
Test Engineer on the Network Testing Team
and Liliia Bidniak,
QA Lead on the Network Testing Team
The goal of this study was to set up an effective network intrusion analysis environment using a combination of open source tools.
As the main monitoring tool, we chose Bro IDS, an effective open source solution that collects information about all network activities within an enterprise. At the same time, it generates an enormous number of logs that are difficult for system administrators to analyze without any separation or visualization. Thus, we needed to define what logs of network activities could potentially indicate suspicious activity and how to visualize this data in a form convenient for further network intrusion analysis.
For our study, we wanted to build a monitoring system with a minimal number of components.
After we ran the system in a testing environment and got results, we would investigate those results and provide recommendations for the system’s deployment in a real network environment.
The monitoring system we tested consists of the following components:
- Host – a monitoring object that generates network activities
- Network intrusion detection system (IDS) – software for distributed analysis of the host’s traffic
- System for analyzing and visualizing data
- An operating system (OS) that runs on physical hardware, an operating system used for running the IDS, and R data analysis and visualization tools
We conducted all experiments in a testing laboratory that included several virtual machines replicating a corporate environment:
To implement our monitoring system, we chose the following software:
- IDS: Bro Network Security Monitor with Intel Critical Stack
- Data visualization: ELK Stack, which consists of Elasticsearch, Logstash, and Kibana
- OS: Ubuntu 16.04 virtual machine configured as an internet gateway
What is Bro IDS?
Bro Network Security Monitor is a Unix-style intrusion detection system that monitors network traffic and detects intrusions and abnormal activities.
Bro parses network traffic by extracting its application-layer semantics. After that, it detects intrusions by executing event-oriented Bro IDS protocol analyzers that compare the current traffic to potentially harmful patterns. As a result of this analysis, Bro can detect network attacks either by finding particular signatures or defining attacks in terms of events and specific conditions.
The system is also useful for detecting unusual activities like numerous host connections to certain services or patterns of failed connection attempts.
Keep in mind that Bro is not an in-line IDS that intervenes in network activities, though. Bro conducts analysis in parallel with network activities and sends alerts in case of attacks or unauthorized access if properly configured.
Why did we use additional software?
Intel Critical Stack is an addition to Bro IDS that has signatures for detecting malware websites. We installed Intel Critical Stack along with Bro IDS and then collected data on internet use and sent that data to the Intel Critical Stack database. Thus, we configured Bro and Critical Stack Agent in order to understand what malicious websites were visited.
The ELK Stack consists of three products – Elasticsearch, Logstash, and Kibana – that are necessary to collect, normalize, store, visualize, and analyze log data generated by Bro IDS. The results of network monitoring are written in different logs which aren’t always understandable by administrators. Thus, we used the ELK Stack to visualize data in charts that are convenient for conducting analysis and making decisions.
All the software we used is publicly available.
We conducted our testing of the monitoring system in the following way:
- A client sends a request to the internet. Thus, the host generates network activities.
- Bro uses tcpdump to analyze traffic from the enp0s8 (eth1) interface and distributes records in logs using its plugins (including Intel Critical Stack).
- Elasticsearch uses Logstash to analyze Bro logs and collect them into a local database.
- Kibana extracts data from the database and builds patterns.
In order to visualize our network data in the most convenient way for analysis, we chose Kibana, which can clearly reveal suspicious activity in the network.
We chose the following charts for visualizing our data:
Connections Count per Minute Chart
The connections count per minute chart shows the total number of connections per minute. An increasing number of connections during non-working hours can be a sign of abnormal activity.
Top Protocols Chart
The top protocols chart shows how much and what type of traffic is going over the network.
Top 10 Talkers Chart
The top 10 talkers chart indicates the most “talkative” computers, which can be potentially infected.
Top 10 HTTP Requests Chart
The top 10 HTTP requests chart shows requests that don’t have encryption, so these websites may be infected with malware.
Top 10 Remote Ports Chart
The top 10 remote ports chart shows ports with the largest number of requests. An increasing number of connections and requests per minute to some ports may indicate suspicious activity.
Bro Log Files
Bro log files shows the number of records in Bro files and the overall state of the Bro IDS.
Top 10 Malware Domains Chart
The top 10 malware domains chart contains feed data from the malware database provided by Intel Critical Stack.
As mentioned above, during the course of normal operation Bro produces a large volume of log files.
However, if your database storage is limited, you can clean them up over a specified period of time. In order to delete unnecessary data, we used Curator for Logstash and added a daily task to Crontab to delete old ELK data.
Here’s the contents of actionfile.yml, according to which Curator chooses data for cleanup.
After analyzing the results of testing in the laboratory environment, we concluded that deploying the system on a real network would require the following:
- A switch with port mirroring
- A server with 32+ GB RAM and 6–10 TB HDD
Note that the system configuration you’ll need depends on the bandwidth of your network service provider. If you have several channels at 1 Gbps, you’ll need to install high-performance network equipment from Arista, Cisco, Myricom, or similar.
1. Installing Bro IDS
Here’s a step-by-step guide to how we installed the components of our monitoring system and configured them for testing.
Configuring the virtual machine
For our study, we used a virtual machine with two network adapters: the first is for the internet connection while the second is for the intranet connection.
For performing all the following commands, you need
root (superuser) privileges. Begin with installing the DHCP server:
Do the following to configure it:
to the file
Open the file
/etc/sysctl.conf and uncomment
Execute following command
In the file
/etc/network/interfaces you need to specify the following:
You need to configure routing so that computers connected to the intranet get access to the internet. Begin with turning on the firewall:
rc-local. This is necessary for restoring iptables rules after the machine reboots.
Open the file
/etc/rc.local and add
Then run the following:
Next, configure the DHCP server to automatically receive IP addresses. Open the file
/etc/dhcp/dhcpd.conf and add the following:
Installing dependencies for Bro
For Bro to operate properly, it needs certain applications to be installed. So run the following commands:
Installing Bro IDS
To install Bro IDS, run the following commands:
Download Bro IDS:
Extract the archive:
Configure installation and install Bro IDS:
Configuring Bro IDS
To configure Bro IDS, specify what interface will monitor network traffic in the node.cfg file:
Specify which subnetwork should be monitored in
Starting Bro IDS
To start Bro, run the following commands:
by adding the following
and then restart the virtual machine
WatchDog for Bro IDS
WatchDog automatically starts Bro after a specified period of time in case it crashes:
Intel Critical Stack
In order to add Intel Critical Stack to Bro IDS, you need to visit https://intel.criticalstack.com/, create a sensor, and subscribe to feeds. After that, run the following commands on a virtual machine with Bro IDS installed:
Configure the sensor with the following commands:
Check and install updates for receiving feeds:
2. Installing ELK
Install the Java Development Kit:
Install and configure Kibana:
Use the following commands to configure Logstash:
NOTE: In the conf files, you should change the path to the log files.
Then install the Filter Translate plugin for Logstash:
3. Configuring Kibana Visualizations
4. Configuring the Kibana Dashboard
The following JSON file describes in what order charts will be displayed on the dashboard:
Everything is set up! You’re ready to monitor network activities.
In this article, we described a method for analyzing network activities with open source tools, in particular by integrating BRO IDS with Intel Critical Stack.
This method is effective for detecting infected computers and doesn’t require any investments from small enterprises except labor costs to set it up.
We also provided recommendations on how to configure Bro and Critical Stack Agent for network monitoring and data collection. Finally, we explained how we used the ELK Stack for visualizing data and interpreting charts.
Our Network Testing Team has vast experience interpreting network processes. We would be glad to assist you in testing your own network management solutions.
Download examples of JSON files: