In this article, I’m going to describe the cURL program and the LibcURL library, from the general aspects of the using of cURL for file downloading to the asynchronous methods provided by the LibcURL library.
At first, we’ll explore an LibcURL example of simple file download. After that, there will be a more complicated example: LibcURL asynchronous downloading of files. Both LibcURL examples are written in C++.
1. What is cURL
1.1 General aspects
2.1 What is asynchrony
2.2 Asynchronous methods provided by cURL for file downloading
3. cURL interfaces
3.1 Easy interface
3.2 Multi interface
4.1 One file download
4.2 Asynchronous files download
What Is cURL
cURL is a cross-browser command line utility that enables interaction with a great number of various servers using various protocols with URL syntax.
Initially, cURL was developed for point-to-point file transferring. But now it is also a library that supports integration for more than 30 languages. Due to this, instead of using cURL from command line, it is possible to create programs which include all the necessary functions.
cURL was developed by Danial Stenberg and it is distributed under the MIT license. In a few words, the MIT license allows developers to use the code in proprietary software under the condition of including a copy of the MIT license Terms into this software.
LibcURL is a multiprotocol library for files transferring. It is free and it can be used by programmers for integration into applications.
cURL supports a wide range of protocols, including, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet, and TFTP. In addition, it supports SSL certificates, cookies, various authentication methods (Basic, Digest, NTLM), and many other useful features.
There are integration modules for the LibcURL library for work with more than 30 programming languages (Ada95, Basic, C, C++, Ch, Cocoa, D, Dylan, Eiffel, Euphoria, Falcon, etc.).
What Is Asynchrony
We’ll find out what is asynchrony on the data transferring example. Asynchronous Transfer Mode is a kind of data transfer when intervals between sent blocks of data are not constant. The Server and the Client are working independently during asynchronous data transfer.
Asynchronous Methods Provided by cURL for File Downloading
cURL provides the Easy Interface and the Multi Interface for work.
The Easy Interface is a synchronous interface, which is able to download only one file at a time, and the process will not stop until it’s completed. For instance, if you have a list of five files for download, as long as the first file is not downloaded, the rest of files will be waiting.
And on the contrary, the Multi Interface allows a user to download several files at the same time, without threads using. It can be used in a single-threaded application, giving a user an advantage of not worrying about the complexity of management and synchronization of several threads.
The Easy Interface consists of the following functions:
curl_easy_init– starts an easy session
curl_easy_cleanup– ends an easy session
curl_easy_setopt– sets options for an easy session
curl_easy_perform– performs files transfer in an easy session
- And others
I’ve listed only the main functions, without which it is impossible to transfer files in an easy session.
Files transfer (
curl_easy_perform) has to be performed after the start of the session (
curl_easy_init) and after setup of the required options (
curl_easy_setopt). The session has to be ended (
curl_easy_cleanup) after the completion of the files transfer.
curl_easy_perform function is a synchronous one (a blocking function). That’s why an application freezes during the work of such function, and returns control only after the function execution is completed.
The Multi Interface consists of the following functions:
curl_multi_init– starts a multi session
curl_multi_cleanup– ends a multi session
curl_multi_add_handle– adds an easy session to a multi session stack
curl_multi_perform– performs files transfer in a multi session
- And others
I’ve listed only the main functions, without which it is impossible to transfer files in a multi session.
Files transfer (
curl_multi_perform) has to be performed after the start of the session (
curl_multi_init) and easy interfaces addition. For this, a user must initialize a list of easy sessions (
curl_easy_init), set the required options (
curl_easy_setopt) and add them to a multi session stack (
On the completion of files transfer, a multi session (
curl_multi_cleanup) and a list of easy sessions (
curl_easy_cleanup) have to be ended.
curl_easy_perform function, the
curl_multi_perform function is an asynchronous one (a non-blocking function). And it’s a great advantage as an application won’t freeze during the work of such function.
One File Download
In order to start using the LibcURL function, a global initialization has to be performed:
After the LibcURL function usage, a deinitialization has to be performed:
Our application will receive two parameters: a file link and a path for saving a downloaded file. For example:
EI.exe http://curl.haxx.se/pix/curl-refined.jpg C:\cURL.jpg
Now, let’s move to the DownloadFile function, which receives a file link, and a path for saving a downloaded file as parameters.
In this function, an easy interface object is created and initialized. With the help of it, we are going to implement the download of a given file:
curl_easy_init initializes an object, and
curl_easy_cleanup deinitializes it.
After that we’ll create a file, which a data buffer will be recorded to.
Then we’ll have to prepare parameters for an easy interface object. It is made by the
The function receives a file pointer, an easy interface object, a file link, and a directory, which a downloaded file has to be saved to. Let’s set the required options for an easy interface.
Setting of the link to the file:
Setting of the function that is responsible for data buffer recording to the file:
The function of data buffer recording is as follows:
Setting of an object to the file for recording:
Setting of an indicator to display the progress of file download:
After all the required parameters have been set, we must call the main function:
This function starts the process of file downloading. You are able to monitor the progress of it in the console window.
LibcURL Asynchronous Example: Asynchronous File Download
This example is much alike with the previous one. But here, we will use functions of the multi interface.
Before start using the multi interface, we must create and initialize it with the corresponding functions:
As you can see, functions of initialization (
curl_multi_init) and deinitialization (
curl_multi_cleanup) are very much alike.
After that, we need to create an easy interface and add it to the multi interface stack. The function
cURLUtils::PrepareMultiInterface is responsible for this. It receives a pointer to the multi interface, a pointer to a pair of an easy interface and corresponding file object, and a file path where the list of files for downloading is located:
Opening of the file is performed in this function:
After that, each line from the file is looked through in the cycle and placed in the specified variables:
Let’s create an easy interface and a file, where the data buffer will be recorded:
The known function
cURLUtils::PrepareEasyInterface is executed and after its completion, an easy interface and a file object will be placed to a pair:
The Easy Interface pointer is being placed into the multi interface stack:
After all preparations we need only to execute the
curl_multi_perform function , which performs the same actions as
It is a typical example of
curl_multi_perform usage in C++. After the call of the
curl_multi_perform function, the number of files, which have not been downloaded yet, are placed to sizeDownload. Also the result of this function is checked. If the result is wrong, the download will stop. The cycle will work until the sizeDownload is equal to zero, meaning all files have been downloaded.
cURLUtils::CloseDownloadedFiles checks the results of files downloading. If the file is downloaded, the file object will be closed. If we did not use this function, the program would have to be closed or we would have to wait until its completing, in order to view all the downloaded files.
In this article, we got acquainted with cURL and learned how to use the LibcURL library. The listed above LibcURLn examples show how to use interfaces for downloading files both synchronously and asynchronously. Provided asynchronous methods are one of the most important advantages for the related software development.
The LibcURL library is extremely useful and popular due to its rich features, a proper documentation, examples, and usability.
Basically, I used the official cURL website. It contains pretty much information and many examples:
Continue reading our Dev Blog with this article about File system filter driver development.