Subscribe to receive all latest blog updates

In this article, I’m going to describe the cURL program and the LibcURL library, from the general aspects of the using of cURL for file downloading to the asynchronous methods provided by the LibcURL library.

At first, we’ll explore an LibcURL example of simple file download. After that, there will be a more complicated example: LibcURL asynchronous downloading of files. Both LibcURL examples are written in C++.

Written by:
Dmitriy Tverdokhleb,
Junior Developer

1. What is cURL

  1.1 General aspects

  1.2 Features

2. Asynchrony

  2.1 What is asynchrony

  2.2 Asynchronous methods provided by cURL for file downloading

3. cURL interfaces

  3.1 Easy interface

  3.2 Multi interface

4. Development

  4.1 One file download

  4.2 Asynchronous files download

5. Summary

  5.1 Conclusion

  5.2 References

What Is cURL

General Aspects

cURL is a cross-browser command line utility that enables interaction with a great number of various servers using various protocols with URL syntax.

Initially, cURL was developed for point-to-point file transferring. But now it is also a library that supports integration for more than 30 languages. Due to this, instead of using cURL from command line, it is possible to create programs which include all the necessary functions.

cURL was developed by Danial Stenberg and it is distributed under the MIT license. In a few words, the MIT license allows developers to use the code in proprietary software under the condition of including a copy of the MIT license Terms into this software.

LibcURL is a multiprotocol library for files transferring. It is free and it can be used by programmers for integration into applications.

Features

cURL supports a wide range of protocols, including, DICT, FILE, FTP, FTPS, Gopher, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, Telnet, and TFTP. In addition, it supports SSL certificates, cookies, various authentication methods (Basic, Digest, NTLM), and many other useful features.

There are integration modules for the LibcURL library for work with more than 30 programming languages (Ada95, Basic, C, C++, Ch, Cocoa, D, Dylan, Eiffel, Euphoria, Falcon, etc.).

Asynchrony

What Is Asynchrony

We’ll find out what is asynchrony on the data transferring example. Asynchronous Transfer Mode is a kind of data transfer when intervals between sent blocks of data are not constant. The Server and the Client are working independently during asynchronous data transfer.

Asynchronous Methods Provided by cURL for File Downloading

cURL provides the Easy Interface and the Multi Interface for work.

The Easy Interface is a synchronous interface, which is able to download only one file at a time, and the process will not stop until it’s completed. For instance, if you have a list of five files for download, as long as the first file is not downloaded, the rest of files will be waiting.

And on the contrary, the Multi Interface allows a user to download several files at the same time, without threads using. It can be used in a single-threaded application, giving a user an advantage of not worrying about the complexity of management and synchronization of several threads.

cURL Interfaces

Easy interface

The Easy Interface consists of the following functions:

  • curl_easy_init – starts an easy session
  • curl_easy_cleanup – ends an easy session
  • curl_easy_setopt – sets options for an easy session
  • curl_easy_perform – performs files transfer in an easy session
  • And others

I’ve listed only the main functions, without which it is impossible to transfer files in an easy session.

Files transfer (curl_easy_perform) has to be performed after the start of the session (curl_easy_init) and after setup of the required options (curl_easy_setopt). The session has to be ended (curl_easy_cleanup) after the completion of the files transfer.

The curl_easy_perform function is a synchronous one (a blocking function). That’s why an application freezes during the work of such function, and returns control only after the function execution is completed.

Multi Interface

The Multi Interface consists of the following functions:

  • curl_multi_init – starts a multi session
  • curl_multi_cleanup – ends a multi session
  • curl_multi_add_handle – adds an easy session to a multi session stack
  • curl_multi_perform – performs files transfer in a multi session
  • And others

I’ve listed only the main functions, without which it is impossible to transfer files in a multi session.

Files transfer (curl_multi_perform) has to be performed after the start of the session (curl_multi_init) and easy interfaces addition. For this, a user must initialize a list of easy sessions (curl_easy_init), set the required options (curl_easy_setopt) and add them to a multi session stack (curl_multi_add_handle).

On the completion of files transfer, a multi session (curl_multi_cleanup) and a list of easy sessions (curl_easy_cleanup) have to be ended.

Unlike the curl_easy_perform function, the curl_multi_perform function is an asynchronous one (a non-blocking function). And it’s a great advantage as an application won’t freeze during the work of such function.

Development

One File Download

In order to start using the LibcURL function, a global initialization has to be performed:

 
 curl_global_init(CURL_GLOBAL_ALL); 

After the LibcURL function usage, a deinitialization has to be performed:

  
curl_global_cleanup(); 

Our application will receive two parameters: a file link and a path for saving a downloaded file. For example:

  EI.exe http://curl.haxx.se/pix/curl-refined.jpg  C:\cURL.jpg

Now, let’s move to the DownloadFile function, which receives a file link, and a path for saving a downloaded file as parameters.

 
 DownloadFile(argv[1],  argv[2]); 

In this function, an easy interface object is created and initialized. With the help of it, we are going to implement the download of a given file:

 
 std::unique_ptr<CURL,  void(*)(CURL*)> easyHandle(curl_easy_init(),   curl_easy_cleanup); 

Accordingly, curl_easy_init initializes an object, and curl_easy_cleanup deinitializes it.

After that we’ll create a file, which a data buffer will be recorded to.

 
 FILE* fileSavePtr = nullptr;
  const errno_t errorFileSave = fopen_s(&fileSavePtr, PathToSave.c_str(), "wb");
  ... 
  auto fileDeleter = [] (FILE* file) { std::fclose(file); };
  std::unique_ptr<FILE,  void(*)(FILE*)> fileSave(fileSavePtr,  fileDeleter); 

Then we’ll have to prepare parameters for an easy interface object. It is made by the cURLUtils::PrepareEasyInterface function:

  cURLUtils::PrepareEasyInterface(fileSave.get(), easyHandle.get(), urlToFile,  PathToSave); 

The function receives a file pointer, an easy interface object, a file link, and a directory,  which a downloaded file has to be saved to. Let’s set the required options for an easy interface.

Setting of the link to the file:

  CURLcode resultSetOpt = curl_easy_setopt(easyHandle, CURLOPT_URL, urlToFile.c_str()); 

Setting of the function that is responsible for data buffer recording to the file:

  resultSetOpt = curl_easy_setopt(easyHandle, CURLOPT_WRITEFUNCTION, WriteCallback); 

The function of data buffer recording is as follows:

  size_t cURLUtils::WriteCallback(void* ptr, size_t  size, size_t  nmemb, FILE* stream)
  {
       return fwrite(ptr, size, nmemb, stream);
  } 

Setting of an object to the file for recording:

  resultSetOpt = curl_easy_setopt(easyHandle, CURLOPT_WRITEDATA, fileToSave); 

Setting of an indicator to display the progress of file download:

  resultSetOpt = curl_easy_setopt(easyHandle, CURLOPT_NOPROGRESS, 0); 

After all the required parameters have been set, we must call the main function:

 const CURLcode resource = curl_easy_perform(easyHandle.get()); 

This function starts the process of file downloading. You are able to monitor the progress of it in the console window.

LibcURL Asynchronous Example: Asynchronous File Download

This example is much alike with the previous one. But here, we will use functions of the multi interface.

Before start using the multi interface, we must create and initialize it with the corresponding functions:

  auto multiHandleDeleter = [] (CURLM* multiHandle) { curl_multi_cleanup(multiHandle); };
  std::unique_ptr<CURLM,  void(*)(CURLM*)> multiHandle(curl_multi_init(), multiHandleDeleter); 

As you can see, functions of initialization (curl_multi_init) and deinitialization (curl_multi_cleanup) are very much alike.

After that, we need to create an easy interface and add it to the multi interface stack. The function cURLUtils::PrepareMultiInterface is responsible for this. It receives a pointer to the multi interface, a pointer to a pair of an easy interface and corresponding file object, and a file path where the list of files for downloading is located:

  typedef  std::map<std::shared_ptr<CURL>, std::shared_ptr<FILE>>  EasyInterface; 
  ... 
  EasyInterface  arrayEasyInterfaces; 
  cURLUtils::PrepareMultiInterface(multiHandle.get(), arrayEasyInterfaces,  PathToListFiles); 

Opening of the file is performed in this function:

  std::ifstream file(PathToListFiles.c_str()); 

After that, each line from the file is looked through in the cycle and placed in the specified variables:

 while (!file.eof())
{
    std::string urlToFile;
    std::string pathToSave;
    file >> urlToFile;
    file >> pathToSave;

Let’s create an easy interface and a file, where the data buffer will be recorded:

  std::shared_ptr<CURL> easyHandle(curl_easy_init(), curl_easy_cleanup); 
  ... 
  FILE* fileSavePtr = nullptr;
  const errno_t errorFileSave = fopen_s(&fileSavePtr, pathToSave.c_str(), "wb"); 
  ... 
  auto fileDeleter = [] (FILE* file) { std::fclose(file); };
  std::shared_ptr<FILE>  fileSave(fileSavePtr,  fileDeleter); 

The known function cURLUtils::PrepareEasyInterface is executed and after its completion, an easy interface and a file object will be placed to a pair:

  cURLUtils::PrepareEasyInterface(fileSave.get(), easyHandle.get(), urlToFile, pathToSave);
  arrayEasyInterfaces.insert(std::make_pair(easyHandle,  fileSave)); 

The Easy Interface pointer is being placed into the multi interface stack:

 CURLMcode code = curl_multi_add_handle(multiHandle, easyHandle.get());
... } // End of while

After all preparations we need only to execute the curl_multi_perform function , which performs the same actions as curl_easy_perform:

  for (;;)
{
    CURLMcode resource = curl_multi_perform(multiHandle.get(), &sizeDownload);
    if (resource != CURLM_OK &&
        resource != CURLM_CALL_MULTI_PERFORM)
    {
        std::cout << "curl_multi_perform is failed! Error " << curl_multi_strerror(resource) << std::endl;
        break;
    }
    if (sizeDownload == 0)
    {
        break;
    }
    cURLUtils::CloseDownloadedFiles(multiHandle.get(), arrayEasyInterfaces);
    Sleep(1);
}

It is a typical example of curl_multi_perform usage in C++. After the call of the curl_multi_perform function, the number of files, which have not been downloaded yet, are placed to sizeDownload. Also the result of this function is checked. If the result is wrong, the download will stop. The cycle will work until the sizeDownload is equal to zero, meaning all files have been downloaded.

The function cURLUtils::CloseDownloadedFiles checks the results of files downloading. If the file is downloaded, the file object will be closed. If we did not use this function, the program would have to be closed or we would have to wait until its completing, in order to view all the downloaded files.

Summary

Conclusion

In this article, we got acquainted with cURL and learned how to use the LibcURL library. The listed above LibcURLn examples show how to use interfaces for downloading files both synchronously and asynchronously. Provided asynchronous methods are one of the most important advantages for the related software development.

The LibcURL library is extremely useful and popular due to its rich features, a proper documentation, examples, and usability.

References

Basically, I used the official cURL website. It contains pretty much information and many examples:

 

Continue reading our Dev Blog with this article about File system filter driver development.