curl

Beyond Simple Web Page Downloads: Unveiling curl’s Power

While many know curl for retrieving web pages (like curl www.example.com), its functionality extends far beyond this basic use case. It supports a wide array of protocols including HTTP, HTTPS, FTP, FTPS, SCP, SFTP, TFTP, and more. This makes it a tool for a range of tasks, from downloading files to interacting with APIs.

1. Downloading Files: The Basics

The most common use of curl is downloading files. This is achieved simply by specifying the URL of the file:

curl https://www.example.com/myfile.txt > myfile.txt

This command downloads myfile.txt from www.example.com and saves it to a local file with the same name. The > operator redirects the output to the specified file.

2. Specifying Output Filename

You can easily specify a different filename for the downloaded file:

curl https://www.example.com/myfile.txt -o my_downloaded_file.txt

The -o (or --output) option allows you to control the output filename.

3. Downloading with Progress Bar

For larger files, a progress bar is helpful. curl provides this functionality with the -s and -# options:

curl -s -# https://www.example.com/largefile.zip

-s (or --silent) suppresses progress output, -# enables a progress meter. Combining them shows only the progress bar.

4. Handling Authentication

Many resources require authentication. curl supports basic authentication using the -u option:

curl -u username:password https://private.example.com/data.json

Caution: Hardcoding passwords directly in the command line is generally discouraged for security reasons. Consider using environment variables or more secure methods for production systems.

5. POST Requests and Data Submission

curl is not limited to GET requests. It can also perform POST requests, sending data to a server:

curl -X POST -H "Content-Type: application/json" -d '{"key1":"value1", "key2":"value2"}' https://api.example.com/submit

This example sends a JSON payload to an API endpoint. -X POST specifies the POST method, -H sets the content type header, and -d provides the data.

6. HTTP Headers

Fine-grained control over HTTP headers is possible:

curl -H "User-Agent: My Custom Agent" -H "Accept: application/json" https://api.example.com/data

This sets a custom User-Agent and specifies that the client accepts JSON responses.

7. Handling Cookies

curl can manage cookies, allowing you to maintain session state:

curl -c cookies.txt "https://example.com/login"
curl -b cookies.txt "https://example.com/profile"

The first command saves cookies to cookies.txt. The second command uses those cookies for subsequent requests.

8. Verifying SSL Certificates

For HTTPS connections, verifying SSL certificates is crucial:

curl --insecure https://example.com  #Insecure - Use with caution!
curl https://example.com --cert client.crt --key client.key # Client certificate authentication

The --insecure option disables SSL certificate verification (use with extreme caution!). The second example demonstrates client certificate authentication.

9. Timeouts

Setting timeouts prevents curl from hanging indefinitely:

curl --connect-timeout 10 --max-time 30 https://example.com

This sets a 10-second connection timeout and a 30-second maximum time for the entire operation.

These examples showcase a fraction of curl’s capabilities. Its extensive options and support for various protocols make it an essential command-line tool for any Linux user involved in network administration or development. Exploring the curl man page (man curl) will reveal even more advanced features and possibilities.