Protecting communication channels is extremely important for both business and private communications, since any breach can lead to a loss of sensitive information. Most popular messaging apps, including WhatsApp, Viber, and Telegram, use encryption to secure conversations. There are a variety of cryptographic algorithms available, however, and it can be challenging to choose the best way to protect your own messaging service.
This article provides recommendations on how to secure your chat or other message exchange protocol using popular and reliable cryptographic algorithms of the OpenSSL library. The article is aimed at people who are already familiar with basic encryption concepts and terminology but who don’t have a lot of experience with practical implementation.
Cryptographic algorithms are mathematical procedures or formulas used to encode information. Information is considered encrypted when it can be accessed only by the sender and the receiver. Even with cloud storage, properly encrypted data should be accessible only to the owner and should be inaccessible to the cloud service provider. This type of encryption is called zero-knowledge encryption, or private-key encryption.
The following types of algorithms are used for zero-knowledge encryption:
- Hashes and PBKDF2
- Asymmetric algorithms
- Symmetric algorithms
There are a variety of Secure Hash Algorithms (SHAs) available. Old SHA1 algorithms are too simple for modern computers. SHA256 or SHA512 (varieties of SHA2) are currently the most popular hash functions used for cybersecurity. However, even SHA2 algorithms are too simple if they only use one iteration and don’t add a random sequence of bits called salt. The PBKDF2 function is more complex and, as a result, is widely used today for password hashing. PBKDF2 typically uses an SHA algorithm and salt to calculate a hash. A hash calculated with this method cannot be used in Rainbow or other attacks that rely on hacking hashes.
Asymmetric algorithms are very slow and cannot be used for encryption of large amounts of data. So in most cases, they’re used to encrypt the keys of symmetric algorithms. The most popular asymmetric algorithms are Rivest–Shamir–Adleman (RSA) and Elliptic-curve Diffie–Hellman (ECDH).
The Advanced Encryption Standard (AES) is the absolute winner in the world of symmetric algorithms. It’s used almost everywhere. It’s fast, has lots of implementations in different languages, and modern CPUs hardware even have built-in support for it.
Certificates are another important part of secure data exchange between the server and the client. Without certificate validation, there’s no way to know that a connection is actually secure. Certificates are essentially a system of asymmetric algorithms that validate the server.
To provide a secure connection between two users, it’s important to take into account all steps in using a messaging app, from registration to actually sending messages. There are four steps to exchanging messages:
- Registration (Figure 2)
- Login (Figures 3 and 4)
- Adding a user to a contact list (Figure 5)
- Exchanging messages (Figure 6)
For the registration procedure, your application should request the server’s keys so that user data can be handled securely:
Figure 1 shows what keys the application requests from the server to use for registration, login, and other interactions with the server. These keys are required to verify and encrypt data exchanged with the server. After this request, all user data must be encrypted with the server’s keys.
The server generates the following keys:
SPbK – server public key (RSA or ECDH)
SPrK – server private key (RSA or ECDH)
SPbSK – server public signing key (RSA or ECDSA)
SPrSK – server private signing key (RSA or ECDSA)
The SPbK should be used to encrypt session keys or any other small amounts of secure information that should be understood only by the server.
During registration, the program generates all necessary keys and then sends them to the server in an encrypted state. The private key should be sent to the server in case the user logs in from a different device. Keys are encrypted with a master key that never leaves the user’s device, which means that they cannot be decrypted on the server side.
Here’s a diagram of this process:
As you can see in Figure 2, all private data that’s sent to the server is encrypted with the master key based on the user’s credentials, so it cannot be decrypted by the server.
After this point, all other data can be exchanged more securely with the user keys. The server can use the PbK to encrypt a session key.
Once registration has been completed successfully, the user has the following keys:
- Authentication key(AK). This key is used for authentication, and the client application must send it to the server. The server authenticates the user with this key. The AK is generated with the PBKDF2 function, and it must be different than the master key. Here’s an example of how the AK can be generated: AK=PBKDF2(password, user’s email as salt, 10000)
- Master key (MK). This key is used to encrypt and decrypt a user’s key pack. It must not be sent to the server or anywhere else. The MK must be kept locally. Here’s an example of how the MK can be generated: MK=PBKDF3(password, username + “@” + domain + username as salt, 20000)
- User’s private key (UPrK)
- User’s private key for signing (UPrSK)
- User’s public key (UPbK)
- User’s public key to check signatures (UPbSK)
- Server’s public key (SPbK)
- Server’s public key to check signatures (SPbSK)
- Session key to encrypt all data that will go to server (SK)
The session key is required to speed up data decryption between the server and the device. Asymmetric algorithms aren’t suitable in this situation due to their slowness.
Let’s say that the user wants to log in from a new device. Here’s how that works:
Now the user has all of their keys and is able to start chatting with other users. As you can see, the user’s password in memory is required for generating the AK and MK. If the user logs in from a new device, the application will request the server’s keys and use them to encrypt the AK (Figure 1).
The original user (User 1) now has information about the public keys of the person they initiated communication with (User 2). These keys are used for secure chatting. All data encrypted with these keys will be available only for these two users.
And now these users can send messages to each other:
The same logic for encrypting messages applies to User 2: encrypt a message with the SK and send it to User 1. If it’s necessary to send some large binary data, then this data can contain its own data key and this data key can be encrypted with the users’ public keys.
Applied OpenSSL: CTR mode in file encryption
If you want to build a secure chat application, you can secure your programming with OpenSSL. All functions in OpenSSL are well-documented, and it’s usually easy to find examples and descriptions. Let's look at our example of secure chat application development.
Here’s a list of some operations that were used in the encryption process above:
1. Prepare master key
2. Generate user’s public and private keys for encryption
3. Encrypt and decrypt using AES
4. Encrypt key pairs
5. Sign message with RSA
6. Verify signature information
All functions below use the OpenSSL API.
The image below shows how to generate a master key based on a password and a user’s email.
This next image shows how to generate RSA keys. These will be used as the user’s public and private keys for encryption and signing. RSA_generate_key is the OpenSSL function that actually generates keys.
The code below shows a simple class that hides OpenSSL AES encryption and provides a simple interface to use it:
Here’s an example of AES decryption:
The following shows encryption of key pairs that were generated using the function above:
The image below shows how to sign outgoing data using RSA:
And here’s how to verify data on the other side:
When somebody says that an application can’t be hashed, that’s actually not true. We all remember the issue in OpenSSL that affected all applications that use this library as their main cryptographic module. Only the public availability of the OpenSSL programming examples makes it possible to find such security holes.
Sometimes, people try to implement their own AES algorithms or other algorithms. However, it’s very dangerous to use such libraries. And if authors of a library don’t provide the source code, then it’s best not to touch it at all because nobody can verify the correctness of the algorithm inside. Unknown libraries may contain a lot of vulnerabilities that allow perpetrators access to your application.
The same goes for applications that claim to be secure. If no one can see the code, then such claims can’t be verified. Consumers may not trust your security because the source code isn’t publicly available.
It’s a good idea to prepare a separate library that contains all security logic and make it available to everyone.
In this article, we’ve covered the security principles behind encrypting instant message communications between two users. OpenSSL is the library of choice for most use cases, and we’ve provided detailed example of how to use it. The encryption scheme we’ve described proves effective in most cases and can be easily modified according to software requirements.
Communication security is a pretty wide issue, and it can take a lot of time to cover all the nuances of implementation. In this article, we’ve covered only how to securely transport messages from user to user, but you should also keep the following in mind when designing your own chat app:
- Server-side security and how to securely store passwords in a database (not as plain text, of course)
- Client-side caching of security data
- Generating salt
- Choosing cryptographic algorithms
You can learn more about our experience using OpenSSL programming for data encryption here.
The Apriorit team has also developed our own data encryption technologies. We would be glad to assist you with data encryption for your communication software. Get in touch!