This article covers the topic of email server security. Security measures covered here will allow you to greatly increase the level of protection of your email server and prevent any successful attacks.

Written by:
Apriorit security testing team
(special thanks to Stas Ignatenko, Denys Rudov, and Dmitriy Yurko)



1. Introduction

1.1 Challenges of server software security

1.2 Email server security

1.3 Standard methods of software protection

2. Potential vulnerabilities

2.1 Unauthorized access to data

2.2 Threat of data leakage

2.3 Spam mail

2.4 Threat of malware sent via email

2.5 DoS threat

2.6 Server performance and stability

2.7 Ignoring best practices, and other issues

3. Detecting and analyzing vulnerabilities

3.1 Preparing the necessary documentation

3.2 Testing (searching for vulnerabilities)

3.3 Analysis of detected problems

3.4 Fixing vulnerabilities

4. Example of cyber security audit of MS Exchange server for Windows OS

4.1 Software and operating system specifics

4.2 Testing scenarios and software to use

4.3 Example of results

5. Example of Zimbra server security monitoring on Linux OS

5.1 Software and operating system specifics

5.2 Testing scenarios and software to use

Example of results

6. Conclusion

6.1 Final recommendations



Challenges of server software security

Nowadays, with the constant development of informational technology, the role of cyber security gets ever bigger. While it is impossible to imagine modern world without constant communications over the network, almost all valuable data that often acts as a target for attacks, is stored in various forms on the servers. Not to mention that the stability of the whole system depends on the servers. This is why the servers are very attractive targets for malicious attacks.

Email server security

Security of mail servers, among others, is a particularly important question. This question is important because email is one of the most popular means of communication and doing business. And for businesses in particular, loss of confidential information can result in large financial loses. It is also important for the server to run stable, so that users are able to access it at any time. When the server runs unstable, it can lead to the loss of customers.

Standard methods of software protection

In order to avoid data loss, problems with stability and other troubles, we need to adhere to the general recommendations on how to setup an email server and monitor its security in order to detect and immediately fix any vulnerabilities. This is the main topic of this article. We will show examples of security measures and various problems that can arise if you don’t adhere to them, as well as some features of certain email servers and example of their vulnerabilities together with the general recommendations with regards to testing.


Potential vulnerabilities

When we talk about cyber security breach, it means there are some vulnerabilities out there. Of course, dealing with all of them is impossible, but we can greatly reduce their numbers. In order to do this we need to stick to the best practices when setting up the security of email servers. Such best practices and problems that can arise when you don’t follow them will be the focus of the next part of this article.

Unauthorized access to data

One very widespread type of attack is when a perpetrator tries to bypass the authentication procedure in order to get access to data.

The first thing you need to do to avoid this is to establish strong requirements for a password, used to access the server. This prevents the password from being cracked via brute force, which is one of the universal ways to bypass authentication. Everything else depends on the server type. Different types of servers use different operating systems, interfaces, etc.

Another way to protect the server from cracking is an SMTP authentication option. We will look at it in more detail in a section that covers various problems with performance and stability.

Threat of data leakage

Getting personal data is one of the key targets for hackers. When email is sent via the internet, it goes through unprotected communication channels. Passwords, user names and messages themselves can be intercepted. In order to prevent this, you need to encrypt both incoming and ongoing mail. Thus, SMTP, POP3, and IMAP protocols should be encrypted with SSL/TLS.

Spam mail

The problem of spam mail is one of the most relevant when it comes to emails.

From the server security standpoint, we can divide the threat of spam mail in two categories:

  • Sending external spam messages to your own clients
  • Sending external spam messages to other clients. In this case, server acts as an Open Reply.

To prevent this threat, you need to use content filters. They are installed either on the Mail server, or on a proxy application, installed to protect the access to the server (such as firewall, proxy-component of the Mail server, etc.). In addition to the content filters you can also use blacklists of known spam servers. For example, DNS-based black lists (DNSBL), Spam URI RBL SURBL, as well as local blacklists of IP addresses of spam senders.

To prevent Open Relay, you need to properly configure Mail Relay parameter of the email server.

When testing anti-spam protection, we analyze the effectiveness of the applied content filters.

Threat of malware

Both server and email client are susceptible to malware. When email server is infected, stability of the whole system gets compromised. Integrity and privacy of personal data falls under threat. Malware spreads among email clients mostly thanks to the infected attachments.

Protection from malware involves both built-in tools and third-party anti-viruses.

DoS threat

The damage that DoS attack can do to a mail server is hard to overstate. It is both unreceived and unsent emails, not to mention time spent on trying to restore the service. Ultimately, reputation of the whole company suffers.

To prevent such a threat, you need to at least limit the amount of possible connections to SMTP server. You need to look into limiting the general amount of connections over time, as well as simultaneous connections.

Server performance and stability

When we see the words “server” and “performance”, we immediately think about load balancing.

If your server was attacked and stopped working, you need to have a plan B. In such cases, we often use a reserve server. For email servers in particular it is done via two MX records for each domain.

Email servers also have an option to use SMTP authentication. If it’s enabled, then in order to send an email to the server you need to provide additional user name and password.

It is very important to enable that option. It allows to protect the server from attack via numerous sent requests, thus ensuring continuous operation.

It is also very important to configure the Mail Relay. You can specify from which IP addresses the server can send mail. It can also prevent the large quantity of messages that can be send to destabilize the server. Another filter for email sent from clients is Reverse DNS. It is used to compare IP address with domain and host name. It also serves to protect the server from malicious mail.

Ignoring best practices, and other issues

One of the most relevant best practices is to not store anything unnecessary on the server. You need to carefully consider whether an additional software, installed on a server, can be used by a perpetrator. Check each of your open network ports to make sure that they are necessary (and if not, you need to immediately close them), and protected (for example, check whether an authorization is required to send data via this port).

You also need to update all the components of the server. Despite all of the efforts from developers and testers, no software is 100% error free. There are whole lists of various software vulnerabilities that are easily accessible for malicious hackers. When new vulnerability or exploit is detected, software vendors will usually issue a fix over the course of several days. If server doesn’t receive this update on time, then perpetrators will be able to use this vulnerability to their advantage.

You also shouldn’t dismiss the human factor. Uptime of the server shouldn’t rely on only one person. You also should trust a server only to the people who will be responsible for it.


Detecting and analyzing vulnerabilities

At the start, it is necessary to design the approach that will be used to monitor server security. In many cases it is easier to look at the available solutions – there are numerous companies out there that provide cyber security audit services. But in certain cases it can be necessary to solve the problem by yourself. How tough and formal this process should be depends on the task at hand.

Next, we will look at the compromise, where we use formal, yet flexible approach.

Preparing the necessary documentation

First, you need to decide what, why, and how you need to check. All three questions are necessary, because you need to cover as much ground as you can with your audit, and yet not waste your time on unnecessary details.

  • What. Create a list of all the data (user names, contact lists, attachments, etc.) and parameters (performance, uptime, etc.) that you think you need to track and check for vulnerabilities. This list can be divided into several checklists with different areas of responsibility (server, network, operating system). Each entry in the list should be weighted according to the magnitude of impact that potential problems with this entry may cause.
  • How. We need to apply two approaches:
    1. By using previously created list of components we search for tools and utilities that can be used to check, whether a particular component is vulnerable or not. Each entry should be assigned a corresponding way to check, and as a result, all those ways will be our tests, or objects for monitoring.
    2. Using the list of potential vulnerabilities we expand our monitoring object list with additional controls, allowing to check for such vulnerabilities. In case some entries are repeating, we need to mark them, but don’t remove them.
  • Why. We need to check the target, weight and how much each individual check covers in order to assign a priority to it. At this stage, we can remove repeating entries from the list of our components, but only if the deleted component is fully covered by some other entry or their combination. Otherwise, we keep the repeated entry, but assign lower priority to it.

After that, we need to make a decision whether each entry is worth the resources necessary to run it. We need to estimate the time and money needed to buy software, train employees and execute checks. Some components with middle and low priority can be excluded from the list if you can’t justify the expense. But they shouldn’t be fully removed from the list – priorities can change any minute.

Planning security testing for email server

When creating the list of controls it’s best to use checklists from NIST SP 800-45.

After forming the list, you can establish the scope of work and necessary resources. Depending on the breakdown, you can also create detailed plan that will fully cover every procedure from start to finish, when you create a report on current state of server security.

Testing (searching for vulnerabilities)

With the checklist the process boils down to conducting checks, covered in sections Products and Utilities of this article. The order of the list will depend on the priority of the task. If there is limited time, than you need to check high-priority components. If there are no time limit, than organizing checks based on what is more convenient can save both time and money.

Some checks can take more time than planned – in this case you should skip them and move them to a separate pool, as well as try to find a way to optimize the process.

All incidents where data and settings from the server were compromised should be logged. Some of them can be later ignored, but at a current stage it is important to not spend time on trying to investigate details, but rather make sure that the check covered as much as possible over the corresponding time.

Analysis of detected problems

Analysis boils down to assessing risks based on the formula – impact*likelihood* exposure. Each control can be scored 1 to 5, where 5 is an indication that this problem can’t be ignored, while 1 is a good indication that solving the problem most likely is too inefficient to consider.

  • Impact of the incident depends on the component that was compromised. In some cases it can be extremely small (for example, server stops working on 100 ms), and sometimes it can be extremely large (data base loss). If checklists are correctly constructed, than an impact can be filled in asthmatically by referencing the checklist.
  • Likelihood depends on how repeatable the problem is and how easily it can be repeated. Changes from very rare (running out of memory once a year while constantly sending letters) to stable (always when receiving 2²⁵⁶ udp packets through an open tcp port).
  • Exposure depends on whether it is hard to detect the problem and whether the problem can occur during the course of normal operation of the server. Exposure can vary from impossible (several unlikely problems happening at the same time) to unavoidable (the word “password” used as an admin password, or server down due to receiving 1000 emails).

Estimating security risk as impact * likelihood * exposure

After assessment all the problems are sorted by risk in a descending order (Impact * Likelihood * Exposure). Next, each incident with the risk value greater than 8 (at least) needs to be discussed. Discussion should help to categorize incidents into vulnerabilities (cyber security threat, for example, loss of data), flaws (a threat of losing loyalty, because of possession of excessive data, or spam), and ignored problems. Priorities then need to be assigned to each vulnerability and flaw.

Fixing vulnerabilities

Usually, there are three ways to fix vulnerabilities:

  • Switch to version or other product that doesn’t have the problem
  • Install additional software that allows to eliminate the problem
  • Disable functionality that exhibits the problem

Main things that need to be considered are the level of associated risks, costs and budget for fixes, as well as necessary resources. All of these things can have large impact on creation of a schedule for fixes and designation of order of tasks.

It is best to immediately take care of the problems that can be solved easy and fast, without putting them off for later. Complex vulnerabilities may require several days to fix, and you’d be better not to leave small, but dangerous problems to simply wait their turn.

In some cases, you can group vulnerabilities that can be solved with a single fix. This can save you money and time, particularly if this approach is considered the most reliable (which means that it wouldn’t require any changes in the future). However, it’s better to not look for silver bullets, because they can prove more complex and expensive than several simple solutions.


Example of cyber security audit of MS Exchange server for Windows OS

Software and operating system specifics

MS Exchange Server is a popular email server provided by Microsoft. Working only on Windows Server OS, Exchange Server supports both standard (SMTP, POP3, IMAP) and proprietary (MAPI, EAS) mail protocols.

Let’s look at specifics of Exchange Server with regard to cyber security. Here we will cover all newer versions starting with Exchange 2010 (official support for Exchange 2007 expired on April 11, 2017).

  • Edge transport server – this is a server for incoming and outgoing external mail. This server works great for companies with network infrastructure divided into protected internal network and protected perimeter, or demilitarized zone (DMZ). Edge Transport Server is usually located inside the DMZ, while Mailbox is located inside the private network. Edge Transport Serve provides additional layer of defense for any messages. This way, mail server can experience less external attacks. Edge Transport is an optional role during Exhcnage Server 2010 and 2016 installation, but mission from Exchange 2013.
  • Database availability group (DAG) – is a component that provides high availability and recovery for data on the server. It was first introduced in Exchange 2010. DAG is a base component of Mailbox server that ensures availability and data recovery after various incidents.
  • Spam protection is achieved via internal antispam agents. They are available by default on Edge Transport server starting with Exchange 2010, and can be enabled directly on Mailbox server (in Exchnage 2016)
  • Anti-malware protection is achieved with Malware agent on the Mailbox server. This agent was first introduced in Exchange 2013. In Exchange 2016 it is on by default.
  • Outlook Web Access (OWA) – is a mail web-client. It doesn’t require the installation of a full-fledged desktop mail client.
  • Proprietary protocols and services:
    • Exchange ActiveSync – protocol for email correspondence with mobile devices
    • Exchange Web Services (EWS) cross-platform API that provides access to the email, contacts and other data to client applications
    • RPC over HTTP, MAPI over HTTP – proprietary protocols allowing mail clients to communicate with Exchange server

Since MS Exchange server supports solely MS Windows Server operating systems, security testing should look into potential malware threats, such as viruses and Trojans. In this case, the threat of malware infection exists not only for the client, but also for the server.

Testing scenarios and software to use

Security testing for MS Exchange Server if done with consideration for infrastructure configuration and environment at hand. To illustrate, let’s look into two examples of infrastructure:


Example 1. Company has internal network with Internet access and without network perimeter. MS Exchange server is installed with base settings without Edge Transport server.


Example 2. Company has internal corporate network. Internet access is implemented via DMZ. Two MS Exchange servers are installed with DAG replication, as well as on Edge Transport server, located inside the DMZ.


First example is much more vulnerable from network attacks standpoint, but it needs to be considered as there are companies out there that use similar infrastructure. Before starting testing, we configure the infrastructure corresponding to the give use case. We test in ways that correspond to certain types of threats:


Testing protection from personal data leaks

In this test we will intercept communications between Exchange Server and email clients.

Testing goes according with the following scenario:

  1. The MAPI, SMTP, IMAP, and EAS mail protocols needs to be set up on a mail server
  2. Mail client is installed. Desktop (Outlook, Thunderbird), mobile, and OWA web clients are considered
  3. Man-in-the-middle (MITM) hardware is configured, located between the server and the client. Hardware has its own interceptor, for example, Wireshark, tcpdump or Fiddler.
  4. Transfer data between the client and the server via one of the protocols (MAPi, SMTP, IMAP, EAS). Review authorizations via the protocol and further communications
  5. Intercept network packets via MITM, and search them for unencrypted data

 Man-in-the-middle attack

Testing spam protection

Example of scenario for testing Exchange Server spam protection:

  1. Configure several local email servers, capable of sending spam to tested Exchange Server
  2. Send a round of emails to Exchanged Server. Use additional scripts to generate spam mail
  3. Search target inboxes for spam

Test cases with enabled and disabled anti-spam filters for Mailbox and Edge Transport server. This will allow you to assess the effectiveness of anti-spam protection.


Testing protection from emails infected with malware

To test protection from emails infected with malware we use a number of test malware attachments. First, it’s a EICAR test file called Eicar test virus Standard Anti-Virus Test File. This file actually is not a virus and doesn’t include any parts of a virus, but most anti-virus software detects it as malicious.

Also in such test we use specially created files. For example, files containing reflective dll loader code. Essentially, this code isn’t harmful, but most anti-viruses will detect it.

An example of scenario for a test:

  1. Disable Malware agent for Exchange Server
  2. Send several emails infected with malware to various clients
  3. On receiving clients, search emails for malware.
  4. Repeat scenario with enabled Malware agent for Mail server, and with enabled Malware agent for Edge Transport server
  5. Compare results at hand


Testing user passwords

To test reliability of user passwords we can use a software built into Kali Linux called Hydra. This software allows to identify weak passwords via an attempt to crack them with brute force. Attacks are conducted through SMTP, IMAP, and POP3 protocols.


Testing DoS attack protection

To test DoS attack protection we need to emulate particular network traffic, aimed at destabilizing services of tested Exchange Server. We also emulate various failures in the network. We can use additional software, such as WANem to do this.

In this test we detect how resistant Exchange Server to this type of attacks and how quickly service can be restored.

While testing high availability we especially consider whether Database availability group component is enabled for an Exchange Server or not, and whether there is additional backup server.

 DoS attack for email server

Example of results

Testing report for Exchange Server

Exchange Server 2010 SP3 with enabled Edge Transport server role.

OS Windows Server 2008 R2


Exchange Server located in the local network.

Local network is connected to the internet via DMZ.

General Vision:

Exchange Server security was tested for the following areas:

  • Personal data protection
  • Reliability of user passwords
  • Protection from spam

During testing a single user account with weak password was discovered.

In local testes of protection from spam, enabling the anti-spam agent prevented spam emails.

Personal data protection testing did not discover any vulnerabilities. Communications via MAPI, SMTP, IMAP, and EAS protocols where tested.


Security policy with regards to user passwords needs to be strengthened, because compromising even a single compromised password can allow perpetrators to compromise private data from other email accounts.

Anti-spam agent needs to be always enabled, and spam server back lists need to be constantly updated.


The example of Zimbra server security monitoring on Linux OS

Software and operating system specifics

Zimbra Collaboration Server is a famous product by Sinacor that provides not only enterprise-level email service, but also calendar, and tools for cooperation, used both by large and small companies. The product is free and supports Linux OS. There are also paid version with additional features that can be found here:

Product can be managed and configured via web-interface.

There are several specifics with regards to Zimbra mail server and cyber security:

  1. When updating OS, Zimba server isn’t updated, but rather removed and installed from scratch. This is why it is necessary to check all settings after an update. The problem should be solved in the new version 8.7.
  2. Zimbra has an open source code, allowing anybody to use it and increasing the possibility that vulnerabilities will be discovered.
  3. Two-factor Authentication: the first part of an authentication procedure involves regular credentials, the second one involves some kind of physical device, such as USB-token or a smartphone.
  4. SSL SNI: server has several certificates for a single IP address and TSP port, which is why it can work with several domains without the need to multiply IP addresses for each domain.
  5. Email Security with Postscreen: additional security check that protects from spam bots, limiting the load on the server
  6. S/MIME Digital Signatures and Encryption: you can sign your message and encrypt it before sending.
  7. HSM: Zimbra allows to store old message in temporary storage, i.e., cheaper and safer.

Testing scenarios and software to use

When talking about security testing scenarios for Zimbra server, a tool such as Kali Linux can be extensively used.

  • Kali Linux Metasploit even has a special modules for Zimbra, such as, for example, zimbra_lfi, which can be used to plant your own malicious file on the server.

Kali Linux Metasploit tools for Zimbra

  • Man in the middle. You can also intercept information, transferred over the internet between the client and the server. To do this, you ca use, for example, arpspoof, which is a part of already mentioned Kali Linux.
  • Because server is controlled via web interface, you also need to consider the following:
  1. First, you can scan for vulnerabilities. Use w3af and sqlmap to do this. With this tools you can, for example, find out credentials for certain email account.
  2. You can also use XSS injection to get credentials of server administrator
  3. It is also important to check the reliability of the admin password with brute force cracking
  • You should definitely check anti-spam and anti-virus. To do this, send spam or malicious emails to the server. Zimbra has special settings for these security measures, which is why during testing you can try different combinations.

Mail server security testing - Anti-spam and anti-virus

  • Zimbra also allows to work with MS Exchnage Server. This feature is available only in paid versions. While testing, you need to both try to intercept information, as well as get Zimbra data, stored on MS Exchange Server.

Zimbra works with MS Exchange Server

  • An finally, you shouldn’t forget about DoS attacks. Emulate network traffic to try and destabilize the server.

Example of results

Bug report:  CVE-2013-70-91
Directory traversal vulnerability in /res/I18nMsg,AjxMsg,ZMsg,ZmMsg,AjxKeys,ZmKeys,ZdMsg,Ajx%20TemplateMsg.js.zgz in Zimbra 7.2.2 and 8.0.2 allows remote attackers to read arbitrary files via a .. (dot dot) in the skin parameter. NOTE: this can be leveraged to execute arbitrary code by obtaining LDAP credentials and accessing the service/admin/soap API.
Global report:

Many file operations are intended to take place within a restricted directory. By using special elements such as ".." and "/" separators, attackers can escape outside of the restricted location to access files or directories that are elsewhere on the system. One of the most common special elements is the "../" sequence, which in most modern operating systems is interpreted as the parent directory of the current location. This is referred to as relative path traversal. Path traversal also covers the use of absolute pathnames such as "/usr/local/bin", which may also be useful in accessing unexpected files. This is referred to as absolute path traversal.

In many programming languages, the injection of a null byte (the 0 or NUL) may allow an attacker to truncate a generated filename to widen the scope of attack. For example, the software may add ".txt" to any pathname, thus limiting the attacker to text files, but a null injection may effectively remove this restriction.


Use an "accept known good" input validation strategy, i.e., use a whitelist of acceptable inputs that strictly conform to specifications. Reject any input that does not strictly conform to specifications, or transform it into something that does.

For any security checks that are performed on the client side, ensure that these checks are duplicated on the server side

Use an application firewall that can detect attacks against this weakness.



In this article we covered the topic of email server security and various approaches to test it. We looked into typical threats faced by email servers, and also described how vulnerability detection and analysis is conducted.

As an example of email severs we took Exchange Server and Zimbra. This way we covered popular email servers for both Windows and Linux.

Cyber security technologies for email servers are constantly evolving. New versions of popular servers with increased security and less errors are constantly released. However, there are certain basic recommendations that you should follow if you want to secure your email server.

Final recommendations

The question of email server security should be raised at earliest stages when you only planning to install the server, because it is much more cost effective to plan ahead and prevent potential threats.

When you planning to install the email server, you need to consider the following:

  1. The purpose of email server. What services will it support and what kind of data will go through this server
  2. Security requirements for this email server
  3. Decide on the types of users, their privileges and authentication methods that will be used on this server
  4. Where inside the network infrastructure the server will be located
  5. What additional software you need to install beside the server itself
  6. How the email server will be managed.

By answering these questions you need to consider the expected level of cyber security the server will have, as well as potential vectors of attacks. Email server security is also determined by the security of operating system it will be installed on.

Below you will find some basic recommendations on email server security:

  • Create network infrastructure in a way that makes the attack surface of your email server as small as possible. For example, create a network perimeter that will shield your private corporate network from the internet. Install proxy applications inside the network perimeter, that will serve external emails and will have access to your email server located inside the private network. An example of such an application for Exchange Server is Edge Transport server.
  • Always use an option to encrypt data transfer for every component of your email server, whenever possible. Carefully choose SSL certificates for each component and don’t use self-written certificates.
  • Use reliable third-party anti-viruses and other anti-malware solutions to provide email server security this will support any built-in anti-malware protection that your server may have.
  • Regularly apply updates to your email server. For example, Microsoft releases Security Bulletins with patches for latest vulnerabilities
  • Create email backup server and set at least two MX DNS records to ensure stable operation.

We develop security and test security - check out details.

Subscribe to updates