�ɲɾ�����ӯ�����һ��ˣ��������С���˴��ͣ�������P���ҹ��ñ˽��ά�Բ��������˸߸ԣ�������ơ��ҹ��ñ�����ά�Բ���ˡ���˳^�ӣ������ӡ� ���ͯj�ӣ��ƺ���ӣ� ? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!? PNG ?%k25u25%fgd5n!PK0]^ҋ11BUGSnu[ _ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| BUGS 1. Bugs 1.1 There are still bugs 1.2 Where to report 1.3 Security bugs 1.4 What to report 1.5 libcurl problems 1.6 Who will fix the problems 1.7 How to get a stack trace 1.8 Bugs in libcurl bindings 1.9 Bugs in old versions 2. Bug fixing procedure 2.1 What happens on first filing 2.2 First response 2.3 Not reproducible 2.4 Unresponsive 2.5 Lack of time/interest 2.6 KNOWN_BUGS 2.7 TODO 2.8 Closing off stalled bugs ============================================================================== 1.1 There are still bugs Curl and libcurl keep being developed. Adding features and changing code means that bugs will sneak in, no matter how hard we try not to. Of course there are lots of bugs left. And lots of misfeatures. To help us make curl the stable and solid product we want it to be, we need bug reports and bug fixes. 1.2 Where to report If you can't fix a bug yourself and submit a fix for it, try to report an as detailed report as possible to a curl mailing list to allow one of us to have a go at a solution. You can optionally also post your bug/problem at curl's bug tracking system over at https://github.com/curl/curl/issues Please read the rest of this document below first before doing that! If you feel you need to ask around first, find a suitable mailing list and post there. The lists are available on https://curl.haxx.se/mail/ 1.3 Security bugs If you find a bug or problem in curl or libcurl that you think has a security impact, for example a bug that can put users in danger or make them vulnerable if the bug becomes public knowledge, then please report that bug using our security development process. Security related bugs or bugs that are suspected to have a security impact, should be reported by email to curl-security@haxx.se so that they first can be dealt with away from the public to minimize the harm and impact it will have on existing users out there who might be using the vulnerable versions. The curl project's process for handling security related issues is documented here: https://curl.haxx.se/dev/secprocess.html 1.4 What to report When reporting a bug, you should include all information that will help us understand what's wrong, what you expected to happen and how to repeat the bad behavior. You therefore need to tell us: - your operating system's name and version number - what version of curl you're using (curl -V is fine) - versions of the used libraries that libcurl is built to use - what URL you were working with (if possible), at least which protocol and anything and everything else you think matters. Tell us what you expected to happen, tell use what did happen, tell us how you could make it work another way. Dig around, try out, test. Then include all the tiny bits and pieces in your report. You will benefit from this yourself, as it will enable us to help you quicker and more accurately. Since curl deals with networks, it often helps us if you include a protocol debug dump with your bug report. The output you get by using the -v or --trace options. If curl crashed, causing a core dump (in unix), there is hardly any use to send that huge file to anyone of us. Unless we have an exact same system setup as you, we can't do much with it. Instead we ask you to get a stack trace and send that (much smaller) output to us instead! The address and how to subscribe to the mailing lists are detailed in the MANUAL file. 1.5 libcurl problems When you've written your own application with libcurl to perform transfers, it is even more important to be specific and detailed when reporting bugs. Tell us the libcurl version and your operating system. Tell us the name and version of all relevant sub-components like for example the SSL library you're using and what name resolving your libcurl uses. If you use SFTP or SCP, the libssh2 version is relevant etc. Showing us a real source code example repeating your problem is the best way to get our attention and it will greatly increase our chances to understand your problem and to work on a fix (if we agree it truly is a problem). Lots of problems that appear to be libcurl problems are actually just abuses of the libcurl API or other malfunctions in your applications. It is advised that you run your problematic program using a memory debug tool like valgrind or similar before you post memory-related or "crashing" problems to us. 1.6 Who will fix the problems If the problems or bugs you describe are considered to be bugs, we want to have the problems fixed. There are no developers in the curl project that are paid to work on bugs. All developers that take on reported bugs do this on a voluntary basis. We do it out of an ambition to keep curl and libcurl excellent products and out of pride. But please do not assume that you can just lump over something to us and it will then magically be fixed after some given time. Most often we need feedback and help to understand what you've experienced and how to repeat a problem. Then we may only be able to assist YOU to debug the problem and to track down the proper fix. We get reports from many people every month and each report can take a considerable amount of time to really go to the bottom with. 1.7 How to get a stack trace First, you must make sure that you compile all sources with -g and that you don't 'strip' the final executable. Try to avoid optimizing the code as well, remove -O, -O2 etc from the compiler options. Run the program until it cores. Run your debugger on the core file, like ' curl core'. should be replaced with the name of your debugger, in most cases that will be 'gdb', but 'dbx' and others also occur. When the debugger has finished loading the core file and presents you a prompt, enter 'where' (without the quotes) and press return. The list that is presented is the stack trace. If everything worked, it is supposed to contain the chain of functions that were called when curl crashed. Include the stack trace with your detailed bug report. It'll help a lot. 1.8 Bugs in libcurl bindings There will of course pop up bugs in libcurl bindings. You should then primarily approach the team that works on that particular binding and see what you can do to help them fix the problem. If you suspect that the problem exists in the underlying libcurl, then please convert your program over to plain C and follow the steps outlined above. 1.9 Bugs in old versions The curl project typically releases new versions every other month, and we fix several hundred bugs per year. For a huge table of releases, number of bug fixes and more, see: https://curl.haxx.se/docs/releases.html The developers in the curl project do not have bandwidth or energy enough to maintain several branches or to spend much time on hunting down problems in old versions when chances are we already fixed them or at least that they've changed nature and appearance in later versions. When you experience a problem and want to report it, you really SHOULD include the version number of the curl you're using when you experience the issue. If that version number shows us that you're using an out-of-date curl, you should also try out a modern curl version to see if the problem persists or how/if it has changed in appearance. Even if you cannot immediately upgrade your application/system to run the latest curl version, you can most often at least run a test version or experimental build or similar, to get this confirmed or not. At times people insist that they cannot upgrade to a modern curl version, but instead they "just want the bug fixed". That's fine, just don't count on us spending many cycles on trying to identify which single commit, if that's even possible, that at some point in the past fixed the problem you're now experiencing. Security wise, it is almost always a bad idea to lag behind the current curl versions by a lot. We keeping discovering and reporting security problems over time see you can see in this table: https://curl.haxx.se/docs/vulnerabilities.html 2. Bug fixing procedure 2.1 What happens on first filing When a new issue is posted in the issue tracker or on the mailing list, the team of developers first need to see the report. Maybe they took the day off, maybe they're off in the woods hunting. Have patience. Allow at least a few days before expecting someone to have responded. In the issue tracker you can expect that some labels will be set on the issue to help categorize it. 2.2 First response If your issue/bug report wasn't perfect at once (and few are), chances are that someone will ask follow-up questions. Which version did you use? Which options did you use? How often does the problem occur? How can we reproduce this problem? Which protocols does it involve? Or perhaps much more specific and deep diving questions. It all depends on your specific issue. You should then respond to these follow-up questions and provide more info about the problem, so that we can help you figure it out. Or maybe you can help us figure it out. An active back-and-forth communication is important and the key for finding a cure and landing a fix. 2.3 Not reproducible For problems that we can't reproduce and can't understand even after having gotten all the info we need and having studied the source code over again, are really hard to solve so then we may require further work from you who actually see or experience the problem. 2.4 Unresponsive If the problem haven't been understood or reproduced, and there's nobody responding to follow-up questions or questions asking for clarifications or for discussing possible ways to move forward with the task, we take that as a strong suggestion that the bug is not important. Unimportant issues will be closed as inactive sooner or later as they can't be fixed. The inactivity period (waiting for responses) should not be shorter than two weeks but may extend months. 2.5 Lack of time/interest Bugs that are filed and are understood can unfortunately end up in the "nobody cares enough about it to work on it" category. Such bugs are perfectly valid problems that *should* get fixed but apparently aren't. We try to mark such bugs as "KNOWN_BUGS material" after a time of inactivity and if no activity is noticed after yet some time those bugs are added to KNOWN_BUGS and are closed in the issue tracker. 2.6 KNOWN_BUGS This is a list of known bugs. Bugs we know exist and that have been pointed out but that haven't yet been fixed. The reasons for why they haven't been fixed can involve anything really, but the primary reason is that nobody has considered these problems to be important enough to spend the necessary time and effort to have them fixed. The KNOWN_BUGS are always up for grabs and we will always love the ones who bring one of them back to live and offers solutions to them. The KNOWN_BUGS document has a sibling document known as TODO. 2.7 TODO Issues that are filed or reported that aren't really bugs but more missing features or ideas for future improvements and so on are marked as 'enhancement' or 'feature-request' and will be added to the TODO document instead and the issue is closed. We don't keep TODO items in the issue tracker. The TODO document is full of ideas and suggestions of what we can add or fix one day. You're always encouraged and free to grab one of those items and take up a discussion with the curl development team on how that could be implemented or provided in the project so that you can work on ticking it odd that document. If the issue is rather a bug and not a missing feature or functionality, it is listed in KNOWN_BUGS instead. 2.8 Closing off stalled bugs The issue and pull request trackers on https://github.com/curl/curl will only hold "active" entries (using a non-precise definition of what active actually is, but they're at least not completely dead). Those that are abandoned or in other ways dormant will be closed and sometimes added to TODO and KNOWN_BUGS instead. This way, we only have "active" issues open on github. Irrelevant issues and pull requests will not distract developers or casual visitors. PK0]+rrTheArtOfHttpScriptingnu[ _ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| The Art Of Scripting HTTP Requests Using Curl 1. HTTP Scripting 1.1 Background 1.2 The HTTP Protocol 1.3 See the Protocol 1.4 See the Timing 1.5 See the Response 2. URL 2.1 Spec 2.2 Host 2.3 Port number 2.4 User name and password 2.5 Path part 3. Fetch a page 3.1 GET 3.2 HEAD 3.3 Multiple URLs in a single command line 3.4 Multiple HTTP methods in a single command line 4. HTML forms 4.1 Forms explained 4.2 GET 4.3 POST 4.4 File Upload POST 4.5 Hidden Fields 4.6 Figure Out What A POST Looks Like 5. HTTP upload 5.1 PUT 6. HTTP Authentication 6.1 Basic Authentication 6.2 Other Authentication 6.3 Proxy Authentication 6.4 Hiding credentials 7. More HTTP Headers 7.1 Referer 7.2 User Agent 8. Redirects 8.1 Location header 8.2 Other redirects 9. Cookies 9.1 Cookie Basics 9.2 Cookie options 10. HTTPS 10.1 HTTPS is HTTP secure 10.2 Certificates 11. Custom Request Elements 11.1 Modify method and headers 11.2 More on changed methods 12. Web Login 12.1 Some login tricks 13. Debug 13.1 Some debug tricks 14. References 14.1 Standards 14.2 Sites ============================================================================== 1. HTTP Scripting 1.1 Background This document assumes that you're familiar with HTML and general networking. The increasing amount of applications moving to the web has made "HTTP Scripting" more frequently requested and wanted. To be able to automatically extract information from the web, to fake users, to post or upload data to web servers are all important tasks today. Curl is a command line tool for doing all sorts of URL manipulations and transfers, but this particular document will focus on how to use it when doing HTTP requests for fun and profit. I'll assume that you know how to invoke 'curl --help' or 'curl --manual' to get basic information about it. Curl is not written to do everything for you. It makes the requests, it gets the data, it sends data and it retrieves the information. You probably need to glue everything together using some kind of script language or repeated manual invokes. 1.2 The HTTP Protocol HTTP is the protocol used to fetch data from web servers. It is a very simple protocol that is built upon TCP/IP. The protocol also allows information to get sent to the server from the client using a few different methods, as will be shown here. HTTP is plain ASCII text lines being sent by the client to a server to request a particular action, and then the server replies a few text lines before the actual requested content is sent to the client. The client, curl, sends a HTTP request. The request contains a method (like GET, POST, HEAD etc), a number of request headers and sometimes a request body. The HTTP server responds with a status line (indicating if things went well), response headers and most often also a response body. The "body" part is the plain data you requested, like the actual HTML or the image etc. 1.3 See the Protocol Using curl's option --verbose (-v as a short option) will display what kind of commands curl sends to the server, as well as a few other informational texts. --verbose is the single most useful option when it comes to debug or even understand the curl<->server interaction. Sometimes even --verbose is not enough. Then --trace and --trace-ascii offer even more details as they show EVERYTHING curl sends and receives. Use it like this: curl --trace-ascii debugdump.txt http://www.example.com/ 1.4 See the Timing Many times you may wonder what exactly is taking all the time, or you just want to know the amount of milliseconds between two points in a transfer. For those, and other similar situations, the --trace-time option is what you need. It'll prepend the time to each trace output line: curl --trace-ascii d.txt --trace-time http://example.com/ 1.5 See the Response By default curl sends the response to stdout. You need to redirect it somewhere to avoid that, most often that is done with -o or -O. 2. URL 2.1 Spec The Uniform Resource Locator format is how you specify the address of a particular resource on the Internet. You know these, you've seen URLs like https://curl.haxx.se or https://yourbank.com a million times. RFC 3986 is the canonical spec. And yeah, the formal name is not URL, it is URI. 2.2 Host The host name is usually resolved using DNS or your /etc/hosts file to an IP address and that's what curl will communicate with. Alternatively you specify the IP address directly in the URL instead of a name. For development and other trying out situations, you can point to a different IP address for a host name than what would otherwise be used, by using curl's --resolve option: curl --resolve www.example.org:80:127.0.0.1 http://www.example.org/ 2.3 Port number Each protocol curl supports operates on a default port number, be it over TCP or in some cases UDP. Normally you don't have to take that into consideration, but at times you run test servers on other ports or similar. Then you can specify the port number in the URL with a colon and a number immediately following the host name. Like when doing HTTP to port 1234: curl http://www.example.org:1234/ The port number you specify in the URL is the number that the server uses to offer its services. Sometimes you may use a local proxy, and then you may need to specify that proxy's port number separately for what curl needs to connect to locally. Like when using a HTTP proxy on port 4321: curl --proxy http://proxy.example.org:4321 http://remote.example.org/ 2.4 User name and password Some services are setup to require HTTP authentication and then you need to provide name and password which is then transferred to the remote site in various ways depending on the exact authentication protocol used. You can opt to either insert the user and password in the URL or you can provide them separately: curl http://user:password@example.org/ or curl -u user:password http://example.org/ You need to pay attention that this kind of HTTP authentication is not what is usually done and requested by user-oriented web sites these days. They tend to use forms and cookies instead. 2.5 Path part The path part is just sent off to the server to request that it sends back the associated response. The path is what is to the right side of the slash that follows the host name and possibly port number. 3. Fetch a page 3.1 GET The simplest and most common request/operation made using HTTP is to GET a URL. The URL could itself refer to a web page, an image or a file. The client issues a GET request to the server and receives the document it asked for. If you issue the command line curl https://curl.haxx.se you get a web page returned in your terminal window. The entire HTML document that that URL holds. All HTTP replies contain a set of response headers that are normally hidden, use curl's --include (-i) option to display them as well as the rest of the document. 3.2 HEAD You can ask the remote server for ONLY the headers by using the --head (-I) option which will make curl issue a HEAD request. In some special cases servers deny the HEAD method while others still work, which is a particular kind of annoyance. The HEAD method is defined and made so that the server returns the headers exactly the way it would do for a GET, but without a body. It means that you may see a Content-Length: in the response headers, but there must not be an actual body in the HEAD response. 3.3 Multiple URLs in a single command line A single curl command line may involve one or many URLs. The most common case is probably to just use one, but you can specify any amount of URLs. Yes any. No limits. You'll then get requests repeated over and over for all the given URLs. Example, send two GETs: curl http://url1.example.com http://url2.example.com If you use --data to POST to the URL, using multiple URLs means that you send that same POST to all the given URLs. Example, send two POSTs: curl --data name=curl http://url1.example.com http://url2.example.com 3.4 Multiple HTTP methods in a single command line Sometimes you need to operate on several URLs in a single command line and do different HTTP methods on each. For this, you'll enjoy the --next option. It is basically a separator that separates a bunch of options from the next. All the URLs before --next will get the same method and will get all the POST data merged into one. When curl reaches the --next on the command line, it'll sort of reset the method and the POST data and allow a new set. Perhaps this is best shown with a few examples. To send first a HEAD and then a GET: curl -I http://example.com --next http://example.com To first send a POST and then a GET: curl -d score=10 http://example.com/post.cgi --next http://example.com/results.html 4. HTML forms 4.1 Forms explained Forms are the general way a web site can present a HTML page with fields for the user to enter data in, and then press some kind of 'OK' or 'Submit' button to get that data sent to the server. The server then typically uses the posted data to decide how to act. Like using the entered words to search in a database, or to add the info in a bug tracking system, display the entered address on a map or using the info as a login-prompt verifying that the user is allowed to see what it is about to see. Of course there has to be some kind of program on the server end to receive the data you send. You cannot just invent something out of the air. 4.2 GET A GET-form uses the method GET, as specified in HTML like:
In your favorite browser, this form will appear with a text box to fill in and a press-button labeled "OK". If you fill in '1905' and press the OK button, your browser will then create a new URL to get for you. The URL will get "junk.cgi?birthyear=1905&press=OK" appended to the path part of the previous URL. If the original form was seen on the page "www.hotmail.com/when/birth.html", the second page you'll get will become "www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK". Most search engines work this way. To make curl do the GET form post for you, just enter the expected created URL: curl "http://www.hotmail.com/when/junk.cgi?birthyear=1905&press=OK" 4.3 POST The GET method makes all input field names get displayed in the URL field of your browser. That's generally a good thing when you want to be able to bookmark that page with your given data, but it is an obvious disadvantage if you entered secret information in one of the fields or if there are a large amount of fields creating a very long and unreadable URL. The HTTP protocol then offers the POST method. This way the client sends the data separated from the URL and thus you won't see any of it in the URL address field. The form would look very similar to the previous one:
And to use curl to post this form with the same data filled in as before, we could do it like: curl --data "birthyear=1905&press=%20OK%20" \ http://www.example.com/when.cgi This kind of POST will use the Content-Type application/x-www-form-urlencoded and is the most widely used POST kind. The data you send to the server MUST already be properly encoded, curl will not do that for you. For example, if you want the data to contain a space, you need to replace that space with %20 etc. Failing to comply with this will most likely cause your data to be received wrongly and messed up. Recent curl versions can in fact url-encode POST data for you, like this: curl --data-urlencode "name=I am Daniel" http://www.example.com If you repeat --data several times on the command line, curl will concatenate all the given data pieces - and put a '&' symbol between each data segment. 4.4 File Upload POST Back in late 1995 they defined an additional way to post data over HTTP. It is documented in the RFC 1867, why this method sometimes is referred to as RFC1867-posting. This method is mainly designed to better support file uploads. A form that allows a user to upload a file could be written like this in HTML:
This clearly shows that the Content-Type about to be sent is multipart/form-data. To post to a form like this with curl, you enter a command line like: curl --form upload=@localfilename --form press=OK [URL] 4.5 Hidden Fields A very common way for HTML based applications to pass state information between pages is to add hidden fields to the forms. Hidden fields are already filled in, they aren't displayed to the user and they get passed along just as all the other fields. A similar example form with one visible field, one hidden field and one submit button could look like:
To POST this with curl, you won't have to think about if the fields are hidden or not. To curl they're all the same: curl --data "birthyear=1905&press=OK&person=daniel" [URL] 4.6 Figure Out What A POST Looks Like When you're about fill in a form and send to a server by using curl instead of a browser, you're of course very interested in sending a POST exactly the way your browser does. An easy way to get to see this, is to save the HTML page with the form on your local disk, modify the 'method' to a GET, and press the submit button (you could also change the action URL if you want to). You will then clearly see the data get appended to the URL, separated with a '?'-letter as GET forms are supposed to. 5. HTTP upload 5.1 PUT Perhaps the best way to upload data to a HTTP server is to use PUT. Then again, this of course requires that someone put a program or script on the server end that knows how to receive a HTTP PUT stream. Put a file to a HTTP server with curl: curl --upload-file uploadfile http://www.example.com/receive.cgi 6. HTTP Authentication 6.1 Basic Authentication HTTP Authentication is the ability to tell the server your username and password so that it can verify that you're allowed to do the request you're doing. The Basic authentication used in HTTP (which is the type curl uses by default) is *plain* *text* based, which means it sends username and password only slightly obfuscated, but still fully readable by anyone that sniffs on the network between you and the remote server. To tell curl to use a user and password for authentication: curl --user name:password http://www.example.com 6.2 Other Authentication The site might require a different authentication method (check the headers returned by the server), and then --ntlm, --digest, --negotiate or even --anyauth might be options that suit you. 6.3 Proxy Authentication Sometimes your HTTP access is only available through the use of a HTTP proxy. This seems to be especially common at various companies. A HTTP proxy may require its own user and password to allow the client to get through to the Internet. To specify those with curl, run something like: curl --proxy-user proxyuser:proxypassword curl.haxx.se If your proxy requires the authentication to be done using the NTLM method, use --proxy-ntlm, if it requires Digest use --proxy-digest. If you use any one of these user+password options but leave out the password part, curl will prompt for the password interactively. 6.4 Hiding credentials Do note that when a program is run, its parameters might be possible to see when listing the running processes of the system. Thus, other users may be able to watch your passwords if you pass them as plain command line options. There are ways to circumvent this. It is worth noting that while this is how HTTP Authentication works, very many web sites will not use this concept when they provide logins etc. See the Web Login chapter further below for more details on that. 7. More HTTP Headers 7.1 Referer A HTTP request may include a 'referer' field (yes it is misspelled), which can be used to tell from which URL the client got to this particular resource. Some programs/scripts check the referer field of requests to verify that this wasn't arriving from an external site or an unknown page. While this is a stupid way to check something so easily forged, many scripts still do it. Using curl, you can put anything you want in the referer-field and thus more easily be able to fool the server into serving your request. Use curl to set the referer field with: curl --referer http://www.example.come http://www.example.com 7.2 User Agent Very similar to the referer field, all HTTP requests may set the User-Agent field. It names what user agent (client) that is being used. Many applications use this information to decide how to display pages. Silly web programmers try to make different pages for users of different browsers to make them look the best possible for their particular browsers. They usually also do different kinds of javascript, vbscript etc. At times, you will see that getting a page with curl will not return the same page that you see when getting the page with your browser. Then you know it is time to set the User Agent field to fool the server into thinking you're one of those browsers. To make curl look like Internet Explorer 5 on a Windows 2000 box: curl --user-agent "Mozilla/4.0 (compatible; MSIE 5.01; Windows NT 5.0)" [URL] Or why not look like you're using Netscape 4.73 on an old Linux box: curl --user-agent "Mozilla/4.73 [en] (X11; U; Linux 2.2.15 i686)" [URL] 8. Redirects 8.1 Location header When a resource is requested from a server, the reply from the server may include a hint about where the browser should go next to find this page, or a new page keeping newly generated output. The header that tells the browser to redirect is Location:. Curl does not follow Location: headers by default, but will simply display such pages in the same manner it displays all HTTP replies. It does however feature an option that will make it attempt to follow the Location: pointers. To tell curl to follow a Location: curl --location http://www.example.com If you use curl to POST to a site that immediately redirects you to another page, you can safely use --location (-L) and --data/--form together. Curl will only use POST in the first request, and then revert to GET in the following operations. 8.2 Other redirects Browser typically support at least two other ways of redirects that curl doesn't: first the html may contain a meta refresh tag that asks the browser to load a specific URL after a set number of seconds, or it may use javascript to do it. 9. Cookies 9.1 Cookie Basics The way the web browsers do "client side state control" is by using cookies. Cookies are just names with associated contents. The cookies are sent to the client by the server. The server tells the client for what path and host name it wants the cookie sent back, and it also sends an expiration date and a few more properties. When a client communicates with a server with a name and path as previously specified in a received cookie, the client sends back the cookies and their contents to the server, unless of course they are expired. Many applications and servers use this method to connect a series of requests into a single logical session. To be able to use curl in such occasions, we must be able to record and send back cookies the way the web application expects them. The same way browsers deal with them. 9.2 Cookie options The simplest way to send a few cookies to the server when getting a page with curl is to add them on the command line like: curl --cookie "name=Daniel" http://www.example.com Cookies are sent as common HTTP headers. This is practical as it allows curl to record cookies simply by recording headers. Record cookies with curl by using the --dump-header (-D) option like: curl --dump-header headers_and_cookies http://www.example.com (Take note that the --cookie-jar option described below is a better way to store cookies.) Curl has a full blown cookie parsing engine built-in that comes in use if you want to reconnect to a server and use cookies that were stored from a previous connection (or hand-crafted manually to fool the server into believing you had a previous connection). To use previously stored cookies, you run curl like: curl --cookie stored_cookies_in_file http://www.example.com Curl's "cookie engine" gets enabled when you use the --cookie option. If you only want curl to understand received cookies, use --cookie with a file that doesn't exist. Example, if you want to let curl understand cookies from a page and follow a location (and thus possibly send back cookies it received), you can invoke it like: curl --cookie nada --location http://www.example.com Curl has the ability to read and write cookie files that use the same file format that Netscape and Mozilla once used. It is a convenient way to share cookies between scripts or invokes. The --cookie (-b) switch automatically detects if a given file is such a cookie file and parses it, and by using the --cookie-jar (-c) option you'll make curl write a new cookie file at the end of an operation: curl --cookie cookies.txt --cookie-jar newcookies.txt \ http://www.example.com 10. HTTPS 10.1 HTTPS is HTTP secure There are a few ways to do secure HTTP transfers. By far the most common protocol for doing this is what is generally known as HTTPS, HTTP over SSL. SSL encrypts all the data that is sent and received over the network and thus makes it harder for attackers to spy on sensitive information. SSL (or TLS as the latest version of the standard is called) offers a truckload of advanced features to allow all those encryptions and key infrastructure mechanisms encrypted HTTP requires. Curl supports encrypted fetches when built to use a TLS library and it can be built to use one out of a fairly large set of libraries - "curl -V" will show which one your curl was built to use (if any!). To get a page from a HTTPS server, simply run curl like: curl https://secure.example.com 10.2 Certificates In the HTTPS world, you use certificates to validate that you are the one you claim to be, as an addition to normal passwords. Curl supports client- side certificates. All certificates are locked with a pass phrase, which you need to enter before the certificate can be used by curl. The pass phrase can be specified on the command line or if not, entered interactively when curl queries for it. Use a certificate with curl on a HTTPS server like: curl --cert mycert.pem https://secure.example.com curl also tries to verify that the server is who it claims to be, by verifying the server's certificate against a locally stored CA cert bundle. Failing the verification will cause curl to deny the connection. You must then use --insecure (-k) in case you want to tell curl to ignore that the server can't be verified. More about server certificate verification and ca cert bundles can be read in the SSLCERTS document, available online here: https://curl.haxx.se/docs/sslcerts.html At times you may end up with your own CA cert store and then you can tell curl to use that to verify the server's certificate: curl --cacert ca-bundle.pem https://example.com/ 11. Custom Request Elements 11.1 Modify method and headers Doing fancy stuff, you may need to add or change elements of a single curl request. For example, you can change the POST request to a PROPFIND and send the data as "Content-Type: text/xml" (instead of the default Content-Type) like this: curl --data "" --header "Content-Type: text/xml" \ --request PROPFIND url.com You can delete a default header by providing one without content. Like you can ruin the request by chopping off the Host: header: curl --header "Host:" http://www.example.com You can add headers the same way. Your server may want a "Destination:" header, and you can add it: curl --header "Destination: http://nowhere" http://example.com 11.2 More on changed methods It should be noted that curl selects which methods to use on its own depending on what action to ask for. -d will do POST, -I will do HEAD and so on. If you use the --request / -X option you can change the method keyword curl selects, but you will not modify curl's behavior. This means that if you for example use -d "data" to do a POST, you can modify the method to a PROPFIND with -X and curl will still think it sends a POST. You can change the normal GET to a POST method by simply adding -X POST in a command line like: curl -X POST http://example.org/ ... but curl will still think and act as if it sent a GET so it won't send any request body etc. 12. Web Login 12.1 Some login tricks While not strictly just HTTP related, it still causes a lot of people problems so here's the executive run-down of how the vast majority of all login forms work and how to login to them using curl. It can also be noted that to do this properly in an automated fashion, you will most certainly need to script things and do multiple curl invokes etc. First, servers mostly use cookies to track the logged-in status of the client, so you will need to capture the cookies you receive in the responses. Then, many sites also set a special cookie on the login page (to make sure you got there through their login page) so you should make a habit of first getting the login-form page to capture the cookies set there. Some web-based login systems feature various amounts of javascript, and sometimes they use such code to set or modify cookie contents. Possibly they do that to prevent programmed logins, like this manual describes how to... Anyway, if reading the code isn't enough to let you repeat the behavior manually, capturing the HTTP requests done by your browsers and analyzing the sent cookies is usually a working method to work out how to shortcut the javascript need. In the actual
tag for the login, lots of sites fill-in random/session or otherwise secretly generated hidden tags and you may need to first capture the HTML code for the login form and extract all the hidden fields to be able to do a proper login POST. Remember that the contents need to be URL encoded when sent in a normal POST. 13. Debug 13.1 Some debug tricks Many times when you run curl on a site, you'll notice that the site doesn't seem to respond the same way to your curl requests as it does to your browser's. Then you need to start making your curl requests more similar to your browser's requests: * Use the --trace-ascii option to store fully detailed logs of the requests for easier analyzing and better understanding * Make sure you check for and use cookies when needed (both reading with --cookie and writing with --cookie-jar) * Set user-agent to one like a recent popular browser does * Set referer like it is set by the browser * If you use POST, make sure you send all the fields and in the same order as the browser does it. A very good helper to make sure you do this right, is the LiveHTTPHeader tool that lets you view all headers you send and receive with Mozilla/Firefox (even when using HTTPS). Chrome features similar functionality out of the box among the developer's tools. A more raw approach is to capture the HTTP traffic on the network with tools such as ethereal or tcpdump and check what headers that were sent and received by the browser. (HTTPS makes this technique inefficient.) 14. References 14.1 Standards RFC 7230 is a must to read if you want in-depth understanding of the HTTP protocol RFC 3986 explains the URL syntax RFC 1867 defines the HTTP post upload format RFC 6525 defines how HTTP cookies work 14.2 Sites https://curl.haxx.se is the home of the curl project PK0]$tJJREADMEnu[ _ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| README Curl is a command line tool for transferring data specified with URL syntax. Find out how to use curl by reading the curl.1 man page or the MANUAL document. Find out how to install Curl by reading the INSTALL document. libcurl is the library curl is using to do its job. It is readily available to be used by your software. Read the libcurl.3 man page to learn how! You find answers to the most frequent questions we get in the FAQ document. Study the COPYING file for distribution terms and similar. If you distribute curl binaries or other binaries that involve libcurl, you might enjoy the LICENSE-MIXING document. CONTACT If you have problems, questions, ideas or suggestions, please contact us by posting to a suitable mailing list. See https://curl.haxx.se/mail/ All contributors to the project are listed in the THANKS document. WEB SITE Visit the curl web site for the latest news and downloads: https://curl.haxx.se/ GIT To download the very latest source off the GIT server do this: git clone https://github.com/curl/curl.git (you'll get a directory named curl created, filled with the source code) NOTICE Curl contains pieces of source code that is Copyright (c) 1998, 1999 Kungliga Tekniska Högskolan. This notice is included here to comply with the distribution terms. PK0]ش00MANUALnu[LATEST VERSION You always find news about what's going on as well as the latest versions from the curl web pages, located at: https://curl.haxx.se SIMPLE USAGE Get the main page from Netscape's web-server: curl http://www.netscape.com/ Get the README file the user's home directory at funet's ftp-server: curl ftp://ftp.funet.fi/README Get a web page from a server using port 8000: curl http://www.weirdserver.com:8000/ Get a directory listing of an FTP site: curl ftp://cool.haxx.se/ Get the definition of curl from a dictionary: curl dict://dict.org/m:curl Fetch two documents at once: curl ftp://cool.haxx.se/ http://www.weirdserver.com:8000/ Get a file off an FTPS server: curl ftps://files.are.secure.com/secrets.txt or use the more appropriate FTPS way to get the same file: curl --ftp-ssl ftp://files.are.secure.com/secrets.txt Get a file from an SSH server using SFTP: curl -u username sftp://example.com/etc/issue Get a file from an SSH server using SCP using a private key (not password-protected) to authenticate: curl -u username: --key ~/.ssh/id_rsa \ scp://example.com/~/file.txt Get a file from an SSH server using SCP using a private key (password-protected) to authenticate: curl -u username: --key ~/.ssh/id_rsa --pass private_key_password \ scp://example.com/~/file.txt Get the main page from an IPv6 web server: curl "http://[2001:1890:1112:1::20]/" Get a file from an SMB server: curl -u "domain\username:passwd" smb://server.example.com/share/file.txt DOWNLOAD TO A FILE Get a web page and store in a local file with a specific name: curl -o thatpage.html http://www.netscape.com/ Get a web page and store in a local file, make the local file get the name of the remote document (if no file name part is specified in the URL, this will fail): curl -O http://www.netscape.com/index.html Fetch two files and store them with their remote names: curl -O www.haxx.se/index.html -O curl.haxx.se/download.html USING PASSWORDS FTP To ftp files using name+passwd, include them in the URL like: curl ftp://name:passwd@machine.domain:port/full/path/to/file or specify them with the -u flag like curl -u name:passwd ftp://machine.domain:port/full/path/to/file FTPS It is just like for FTP, but you may also want to specify and use SSL-specific options for certificates etc. Note that using FTPS:// as prefix is the "implicit" way as described in the standards while the recommended "explicit" way is done by using FTP:// and the --ftp-ssl option. SFTP / SCP This is similar to FTP, but you can use the --key option to specify a private key to use instead of a password. Note that the private key may itself be protected by a password that is unrelated to the login password of the remote system; this password is specified using the --pass option. Typically, curl will automatically extract the public key from the private key file, but in cases where curl does not have the proper library support, a matching public key file must be specified using the --pubkey option. HTTP Curl also supports user and password in HTTP URLs, thus you can pick a file like: curl http://name:passwd@machine.domain/full/path/to/file or specify user and password separately like in curl -u name:passwd http://machine.domain/full/path/to/file HTTP offers many different methods of authentication and curl supports several: Basic, Digest, NTLM and Negotiate (SPNEGO). Without telling which method to use, curl defaults to Basic. You can also ask curl to pick the most secure ones out of the ones that the server accepts for the given URL, by using --anyauth. NOTE! According to the URL specification, HTTP URLs can not contain a user and password, so that style will not work when using curl via a proxy, even though curl allows it at other times. When using a proxy, you _must_ use the -u style for user and password. HTTPS Probably most commonly used with private certificates, as explained below. PROXY curl supports both HTTP and SOCKS proxy servers, with optional authentication. It does not have special support for FTP proxy servers since there are no standards for those, but it can still be made to work with many of them. You can also use both HTTP and SOCKS proxies to transfer files to and from FTP servers. Get an ftp file using an HTTP proxy named my-proxy that uses port 888: curl -x my-proxy:888 ftp://ftp.leachsite.com/README Get a file from an HTTP server that requires user and password, using the same proxy as above: curl -u user:passwd -x my-proxy:888 http://www.get.this/ Some proxies require special authentication. Specify by using -U as above: curl -U user:passwd -x my-proxy:888 http://www.get.this/ A comma-separated list of hosts and domains which do not use the proxy can be specified as: curl --noproxy localhost,get.this -x my-proxy:888 http://www.get.this/ If the proxy is specified with --proxy1.0 instead of --proxy or -x, then curl will use HTTP/1.0 instead of HTTP/1.1 for any CONNECT attempts. curl also supports SOCKS4 and SOCKS5 proxies with --socks4 and --socks5. See also the environment variables Curl supports that offer further proxy control. Most FTP proxy servers are set up to appear as a normal FTP server from the client's perspective, with special commands to select the remote FTP server. curl supports the -u, -Q and --ftp-account options that can be used to set up transfers through many FTP proxies. For example, a file can be uploaded to a remote FTP server using a Blue Coat FTP proxy with the options: curl -u "Remote-FTP-Username@remote.ftp.server Proxy-Username:Remote-Pass" \ --ftp-account Proxy-Password --upload-file local-file \ ftp://my-ftp.proxy.server:21/remote/upload/path/ See the manual for your FTP proxy to determine the form it expects to set up transfers, and curl's -v option to see exactly what curl is sending. RANGES HTTP 1.1 introduced byte-ranges. Using this, a client can request to get only one or more subparts of a specified document. Curl supports this with the -r flag. Get the first 100 bytes of a document: curl -r 0-99 http://www.get.this/ Get the last 500 bytes of a document: curl -r -500 http://www.get.this/ Curl also supports simple ranges for FTP files as well. Then you can only specify start and stop position. Get the first 100 bytes of a document using FTP: curl -r 0-99 ftp://www.get.this/README UPLOADING FTP / FTPS / SFTP / SCP Upload all data on stdin to a specified server: curl -T - ftp://ftp.upload.com/myfile Upload data from a specified file, login with user and password: curl -T uploadfile -u user:passwd ftp://ftp.upload.com/myfile Upload a local file to the remote site, and use the local file name at the remote site too: curl -T uploadfile -u user:passwd ftp://ftp.upload.com/ Upload a local file to get appended to the remote file: curl -T localfile -a ftp://ftp.upload.com/remotefile Curl also supports ftp upload through a proxy, but only if the proxy is configured to allow that kind of tunneling. If it does, you can run curl in a fashion similar to: curl --proxytunnel -x proxy:port -T localfile ftp.upload.com SMB / SMBS curl -T file.txt -u "domain\username:passwd" smb://server.example.com/share/ HTTP Upload all data on stdin to a specified HTTP site: curl -T - http://www.upload.com/myfile Note that the HTTP server must have been configured to accept PUT before this can be done successfully. For other ways to do HTTP data upload, see the POST section below. VERBOSE / DEBUG If curl fails where it isn't supposed to, if the servers don't let you in, if you can't understand the responses: use the -v flag to get verbose fetching. Curl will output lots of info and what it sends and receives in order to let the user see all client-server interaction (but it won't show you the actual data). curl -v ftp://ftp.upload.com/ To get even more details and information on what curl does, try using the --trace or --trace-ascii options with a given file name to log to, like this: curl --trace trace.txt www.haxx.se DETAILED INFORMATION Different protocols provide different ways of getting detailed information about specific files/documents. To get curl to show detailed information about a single file, you should use -I/--head option. It displays all available info on a single file for HTTP and FTP. The HTTP information is a lot more extensive. For HTTP, you can get the header information (the same as -I would show) shown before the data by using -i/--include. Curl understands the -D/--dump-header option when getting files from both FTP and HTTP, and it will then store the headers in the specified file. Store the HTTP headers in a separate file (headers.txt in the example): curl --dump-header headers.txt curl.haxx.se Note that headers stored in a separate file can be very useful at a later time if you want curl to use cookies sent by the server. More about that in the cookies section. POST (HTTP) It's easy to post data using curl. This is done using the -d option. The post data must be urlencoded. Post a simple "name" and "phone" guestbook. curl -d "name=Rafael%20Sagula&phone=3320780" \ http://www.where.com/guest.cgi How to post a form with curl, lesson #1: Dig out all the tags in the form that you want to fill in. If there's a "normal" post, you use -d to post. -d takes a full "post string", which is in the format =&=&... The 'variable' names are the names set with "name=" in the tags, and the data is the contents you want to fill in for the inputs. The data *must* be properly URL encoded. That means you replace space with + and that you replace weird letters with %XX where XX is the hexadecimal representation of the letter's ASCII code. Example: (page located at http://www.formpost.com/getthis/ We want to enter user 'foobar' with password '12345'. To post to this, you enter a curl command line like: curl -d "user=foobar&pass=12345&id=blablabla&ding=submit" (continues) http://www.formpost.com/getthis/post.cgi While -d uses the application/x-www-form-urlencoded mime-type, generally understood by CGI's and similar, curl also supports the more capable multipart/form-data type. This latter type supports things like file upload. -F accepts parameters like -F "name=contents". If you want the contents to be read from a file, use <@filename> as contents. When specifying a file, you can also specify the file content type by appending ';type=' to the file name. You can also post the contents of several files in one field. For example, the field name 'coolfiles' is used to send three files, with different content types using the following syntax: curl -F "coolfiles=@fil1.gif;type=image/gif,fil2.txt,fil3.html" \ http://www.post.com/postit.cgi If the content-type is not specified, curl will try to guess from the file extension (it only knows a few), or use the previously specified type (from an earlier file if several files are specified in a list) or else it will use the default type 'application/octet-stream'. Emulate a fill-in form with -F. Let's say you fill in three fields in a form. One field is a file name which to post, one field is your name and one field is a file description. We want to post the file we have written named "cooltext.txt". To let curl do the posting of this data instead of your favourite browser, you have to read the HTML source of the form page and find the names of the input fields. In our example, the input field names are 'file', 'yourname' and 'filedescription'. curl -F "file=@cooltext.txt" -F "yourname=Daniel" \ -F "filedescription=Cool text file with cool text inside" \ http://www.post.com/postit.cgi To send two files in one post you can do it in two ways: 1. Send multiple files in a single "field" with a single field name: curl -F "pictures=@dog.gif,cat.gif" 2. Send two fields with two field names: curl -F "docpicture=@dog.gif" -F "catpicture=@cat.gif" To send a field value literally without interpreting a leading '@' or '<', or an embedded ';type=', use --form-string instead of -F. This is recommended when the value is obtained from a user or some other unpredictable source. Under these circumstances, using -F instead of --form-string would allow a user to trick curl into uploading a file. REFERRER An HTTP request has the option to include information about which address referred it to the actual page. Curl allows you to specify the referrer to be used on the command line. It is especially useful to fool or trick stupid servers or CGI scripts that rely on that information being available or contain certain data. curl -e www.coolsite.com http://www.showme.com/ NOTE: The Referer: [sic] field is defined in the HTTP spec to be a full URL. USER AGENT An HTTP request has the option to include information about the browser that generated the request. Curl allows it to be specified on the command line. It is especially useful to fool or trick stupid servers or CGI scripts that only accept certain browsers. Example: curl -A 'Mozilla/3.0 (Win95; I)' http://www.nationsbank.com/ Other common strings: 'Mozilla/3.0 (Win95; I)' Netscape Version 3 for Windows 95 'Mozilla/3.04 (Win95; U)' Netscape Version 3 for Windows 95 'Mozilla/2.02 (OS/2; U)' Netscape Version 2 for OS/2 'Mozilla/4.04 [en] (X11; U; AIX 4.2; Nav)' NS for AIX 'Mozilla/4.05 [en] (X11; U; Linux 2.0.32 i586)' NS for Linux Note that Internet Explorer tries hard to be compatible in every way: 'Mozilla/4.0 (compatible; MSIE 4.01; Windows 95)' MSIE for W95 Mozilla is not the only possible User-Agent name: 'Konqueror/1.0' KDE File Manager desktop client 'Lynx/2.7.1 libwww-FM/2.14' Lynx command line browser COOKIES Cookies are generally used by web servers to keep state information at the client's side. The server sets cookies by sending a response line in the headers that looks like 'Set-Cookie: ' where the data part then typically contains a set of NAME=VALUE pairs (separated by semicolons ';' like "NAME1=VALUE1; NAME2=VALUE2;"). The server can also specify for what path the "cookie" should be used for (by specifying "path=value"), when the cookie should expire ("expire=DATE"), for what domain to use it ("domain=NAME") and if it should be used on secure connections only ("secure"). If you've received a page from a server that contains a header like: Set-Cookie: sessionid=boo123; path="/foo"; it means the server wants that first pair passed on when we get anything in a path beginning with "/foo". Example, get a page that wants my name passed in a cookie: curl -b "name=Daniel" www.sillypage.com Curl also has the ability to use previously received cookies in following sessions. If you get cookies from a server and store them in a file in a manner similar to: curl --dump-header headers www.example.com ... you can then in a second connect to that (or another) site, use the cookies from the 'headers' file like: curl -b headers www.example.com While saving headers to a file is a working way to store cookies, it is however error-prone and not the preferred way to do this. Instead, make curl save the incoming cookies using the well-known netscape cookie format like this: curl -c cookies.txt www.example.com Note that by specifying -b you enable the "cookie awareness" and with -L you can make curl follow a location: (which often is used in combination with cookies). So that if a site sends cookies and a location, you can use a non-existing file to trigger the cookie awareness like: curl -L -b empty.txt www.example.com The file to read cookies from must be formatted using plain HTTP headers OR as netscape's cookie file. Curl will determine what kind it is based on the file contents. In the above command, curl will parse the header and store the cookies received from www.example.com. curl will send to the server the stored cookies which match the request as it follows the location. The file "empty.txt" may be a nonexistent file. To read and write cookies from a netscape cookie file, you can set both -b and -c to use the same file: curl -b cookies.txt -c cookies.txt www.example.com PROGRESS METER The progress meter exists to show a user that something actually is happening. The different fields in the output have the following meaning: % Total % Received % Xferd Average Speed Time Curr. Dload Upload Total Current Left Speed 0 151M 0 38608 0 0 9406 0 4:41:43 0:00:04 4:41:39 9287 From left-to-right: % - percentage completed of the whole transfer Total - total size of the whole expected transfer % - percentage completed of the download Received - currently downloaded amount of bytes % - percentage completed of the upload Xferd - currently uploaded amount of bytes Average Speed Dload - the average transfer speed of the download Average Speed Upload - the average transfer speed of the upload Time Total - expected time to complete the operation Time Current - time passed since the invoke Time Left - expected time left to completion Curr.Speed - the average transfer speed the last 5 seconds (the first 5 seconds of a transfer is based on less time of course.) The -# option will display a totally different progress bar that doesn't need much explanation! SPEED LIMIT Curl allows the user to set the transfer speed conditions that must be met to let the transfer keep going. By using the switch -y and -Y you can make curl abort transfers if the transfer speed is below the specified lowest limit for a specified time. To have curl abort the download if the speed is slower than 3000 bytes per second for 1 minute, run: curl -Y 3000 -y 60 www.far-away-site.com This can very well be used in combination with the overall time limit, so that the above operation must be completed in whole within 30 minutes: curl -m 1800 -Y 3000 -y 60 www.far-away-site.com Forcing curl not to transfer data faster than a given rate is also possible, which might be useful if you're using a limited bandwidth connection and you don't want your transfer to use all of it (sometimes referred to as "bandwidth throttle"). Make curl transfer data no faster than 10 kilobytes per second: curl --limit-rate 10K www.far-away-site.com or curl --limit-rate 10240 www.far-away-site.com Or prevent curl from uploading data faster than 1 megabyte per second: curl -T upload --limit-rate 1M ftp://uploadshereplease.com When using the --limit-rate option, the transfer rate is regulated on a per-second basis, which will cause the total transfer speed to become lower than the given number. Sometimes of course substantially lower, if your transfer stalls during periods. CONFIG FILE Curl automatically tries to read the .curlrc file (or _curlrc file on win32 systems) from the user's home dir on startup. The config file could be made up with normal command line switches, but you can also specify the long options without the dashes to make it more readable. You can separate the options and the parameter with spaces, or with = or :. Comments can be used within the file. If the first letter on a line is a '#'-symbol the rest of the line is treated as a comment. If you want the parameter to contain spaces, you must enclose the entire parameter within double quotes ("). Within those quotes, you specify a quote as \". NOTE: You must specify options and their arguments on the same line. Example, set default time out and proxy in a config file: # We want a 30 minute timeout: -m 1800 # ... and we use a proxy for all accesses: proxy = proxy.our.domain.com:8080 White spaces ARE significant at the end of lines, but all white spaces leading up to the first characters of each line are ignored. Prevent curl from reading the default file by using -q as the first command line parameter, like: curl -q www.thatsite.com Force curl to get and display a local help page in case it is invoked without URL by making a config file similar to: # default url to get url = "http://help.with.curl.com/curlhelp.html" You can specify another config file to be read by using the -K/--config flag. If you set config file name to "-" it'll read the config from stdin, which can be handy if you want to hide options from being visible in process tables etc: echo "user = user:passwd" | curl -K - http://that.secret.site.com EXTRA HEADERS When using curl in your own very special programs, you may end up needing to pass on your own custom headers when getting a web page. You can do this by using the -H flag. Example, send the header "X-you-and-me: yes" to the server when getting a page: curl -H "X-you-and-me: yes" www.love.com This can also be useful in case you want curl to send a different text in a header than it normally does. The -H header you specify then replaces the header curl would normally send. If you replace an internal header with an empty one, you prevent that header from being sent. To prevent the Host: header from being used: curl -H "Host:" www.server.com FTP and PATH NAMES Do note that when getting files with the ftp:// URL, the given path is relative the directory you enter. To get the file 'README' from your home directory at your ftp site, do: curl ftp://user:passwd@my.site.com/README But if you want the README file from the root directory of that very same site, you need to specify the absolute file name: curl ftp://user:passwd@my.site.com//README (I.e with an extra slash in front of the file name.) SFTP and SCP and PATH NAMES With sftp: and scp: URLs, the path name given is the absolute name on the server. To access a file relative to the remote user's home directory, prefix the file with /~/ , such as: curl -u $USER sftp://home.example.com/~/.bashrc FTP and firewalls The FTP protocol requires one of the involved parties to open a second connection as soon as data is about to get transferred. There are two ways to do this. The default way for curl is to issue the PASV command which causes the server to open another port and await another connection performed by the client. This is good if the client is behind a firewall that doesn't allow incoming connections. curl ftp.download.com If the server, for example, is behind a firewall that doesn't allow connections on ports other than 21 (or if it just doesn't support the PASV command), the other way to do it is to use the PORT command and instruct the server to connect to the client on the given IP number and port (as parameters to the PORT command). The -P flag to curl supports a few different options. Your machine may have several IP-addresses and/or network interfaces and curl allows you to select which of them to use. Default address can also be used: curl -P - ftp.download.com Download with PORT but use the IP address of our 'le0' interface (this does not work on windows): curl -P le0 ftp.download.com Download with PORT but use 192.168.0.10 as our IP address to use: curl -P 192.168.0.10 ftp.download.com NETWORK INTERFACE Get a web page from a server using a specified port for the interface: curl --interface eth0:1 http://www.netscape.com/ or curl --interface 192.168.1.10 http://www.netscape.com/ HTTPS Secure HTTP requires SSL libraries to be installed and used when curl is built. If that is done, curl is capable of retrieving and posting documents using the HTTPS protocol. Example: curl https://www.secure-site.com Curl is also capable of using your personal certificates to get/post files from sites that require valid certificates. The only drawback is that the certificate needs to be in PEM-format. PEM is a standard and open format to store certificates with, but it is not used by the most commonly used browsers (Netscape and MSIE both use the so called PKCS#12 format). If you want curl to use the certificates you use with your (favourite) browser, you may need to download/compile a converter that can convert your browser's formatted certificates to PEM formatted ones. This kind of converter is included in recent versions of OpenSSL, and for older versions Dr Stephen N. Henson has written a patch for SSLeay that adds this functionality. You can get his patch (that requires an SSLeay installation) from his site at: http://www.drh-consultancy.demon.co.uk/ Example on how to automatically retrieve a document using a certificate with a personal password: curl -E /path/to/cert.pem:password https://secure.site.com/ If you neglect to specify the password on the command line, you will be prompted for the correct password before any data can be received. Many older SSL-servers have problems with SSLv3 or TLS, which newer versions of OpenSSL etc use, therefore it is sometimes useful to specify what SSL-version curl should use. Use -3, -2 or -1 to specify that exact SSL version to use (for SSLv3, SSLv2 or TLSv1 respectively): curl -2 https://secure.site.com/ Otherwise, curl will first attempt to use v3 and then v2. To use OpenSSL to convert your favourite browser's certificate into a PEM formatted one that curl can use, do something like this: In Netscape, you start with hitting the 'Security' menu button. Select 'certificates->yours' and then pick a certificate in the list Press the 'Export' button enter your PIN code for the certs select a proper place to save it Run the 'openssl' application to convert the certificate. If you cd to the openssl installation, you can do it like: # ./apps/openssl pkcs12 -in [file you saved] -clcerts -out [PEMfile] In Firefox, select Options, then Advanced, then the Encryption tab, View Certificates. This opens the Certificate Manager, where you can Export. Be sure to select PEM for the Save as type. In Internet Explorer, select Internet Options, then the Content tab, then Certificates. Then you can Export, and depending on the format you may need to convert to PEM. In Chrome, select Settings, then Show Advanced Settings. Under HTTPS/SSL select Manage Certificates. RESUMING FILE TRANSFERS To continue a file transfer where it was previously aborted, curl supports resume on HTTP(S) downloads as well as FTP uploads and downloads. Continue downloading a document: curl -C - -o file ftp://ftp.server.com/path/file Continue uploading a document(*1): curl -C - -T file ftp://ftp.server.com/path/file Continue downloading a document from a web server(*2): curl -C - -o file http://www.server.com/ (*1) = This requires that the FTP server supports the non-standard command SIZE. If it doesn't, curl will say so. (*2) = This requires that the web server supports at least HTTP/1.1. If it doesn't, curl will say so. TIME CONDITIONS HTTP allows a client to specify a time condition for the document it requests. It is If-Modified-Since or If-Unmodified-Since. Curl allows you to specify them with the -z/--time-cond flag. For example, you can easily make a download that only gets performed if the remote file is newer than a local copy. It would be made like: curl -z local.html http://remote.server.com/remote.html Or you can download a file only if the local file is newer than the remote one. Do this by prepending the date string with a '-', as in: curl -z -local.html http://remote.server.com/remote.html You can specify a "free text" date as condition. Tell curl to only download the file if it was updated since January 12, 2012: curl -z "Jan 12 2012" http://remote.server.com/remote.html Curl will then accept a wide range of date formats. You always make the date check the other way around by prepending it with a dash '-'. DICT For fun try curl dict://dict.org/m:curl curl dict://dict.org/d:heisenbug:jargon curl dict://dict.org/d:daniel:web1913 Aliases for 'm' are 'match' and 'find', and aliases for 'd' are 'define' and 'lookup'. For example, curl dict://dict.org/find:curl Commands that break the URL description of the RFC (but not the DICT protocol) are curl dict://dict.org/show:db curl dict://dict.org/show:strat Authentication is still missing (but this is not required by the RFC) LDAP If you have installed the OpenLDAP library, curl can take advantage of it and offer ldap:// support. On Windows, curl will use WinLDAP from Platform SDK by default. Default protocol version used by curl is LDAPv3. LDAPv2 will be used as fallback mechanism in case if LDAPv3 will fail to connect. LDAP is a complex thing and writing an LDAP query is not an easy task. I do advise you to dig up the syntax description for that elsewhere. One such place might be: RFC 2255, "The LDAP URL Format" https://curl.haxx.se/rfc/rfc2255.txt To show you an example, this is how I can get all people from my local LDAP server that has a certain sub-domain in their email address: curl -B "ldap://ldap.frontec.se/o=frontec??sub?mail=*sth.frontec.se" If I want the same info in HTML format, I can get it by not using the -B (enforce ASCII) flag. You also can use authentication when accessing LDAP catalog: curl -u user:passwd "ldap://ldap.frontec.se/o=frontec??sub?mail=*" curl "ldap://user:passwd@ldap.frontec.se/o=frontec??sub?mail=*" By default, if user and password provided, OpenLDAP/WinLDAP will use basic authentication. On Windows you can control this behavior by providing one of --basic, --ntlm or --digest option in curl command line curl --ntlm "ldap://user:passwd@ldap.frontec.se/o=frontec??sub?mail=*" On Windows, if no user/password specified, auto-negotiation mechanism will be used with current logon credentials (SSPI/SPNEGO). ENVIRONMENT VARIABLES Curl reads and understands the following environment variables: http_proxy, HTTPS_PROXY, FTP_PROXY They should be set for protocol-specific proxies. General proxy should be set with ALL_PROXY A comma-separated list of host names that shouldn't go through any proxy is set in (only an asterisk, '*' matches all hosts) NO_PROXY If the host name matches one of these strings, or the host is within the domain of one of these strings, transactions with that node will not be proxied. When a domain is used, it needs to start with a period. A user can specify that both www.example.com and foo.example.com should not uses a proxy by setting NO_PROXY to ".example.com". By including the full name you can exclude specific host names, so to make www.example.com not use a proxy but still have foo.example.com do it, set NO_PROXY to "www.example.com" The usage of the -x/--proxy flag overrides the environment variables. NETRC Unix introduced the .netrc concept a long time ago. It is a way for a user to specify name and password for commonly visited FTP sites in a file so that you don't have to type them in each time you visit those sites. You realize this is a big security risk if someone else gets hold of your passwords, so therefore most unix programs won't read this file unless it is only readable by yourself (curl doesn't care though). Curl supports .netrc files if told to (using the -n/--netrc and --netrc-optional options). This is not restricted to just FTP, so curl can use it for all protocols where authentication is used. A very simple .netrc file could look something like: machine curl.haxx.se login iamdaniel password mysecret CUSTOM OUTPUT To better allow script programmers to get to know about the progress of curl, the -w/--write-out option was introduced. Using this, you can specify what information from the previous transfer you want to extract. To display the amount of bytes downloaded together with some text and an ending newline: curl -w 'We downloaded %{size_download} bytes\n' www.download.com KERBEROS FTP TRANSFER Curl supports kerberos4 and kerberos5/GSSAPI for FTP transfers. You need the kerberos package installed and used at curl build time for it to be available. First, get the krb-ticket the normal way, like with the kinit/kauth tool. Then use curl in way similar to: curl --krb private ftp://krb4site.com -u username:fakepwd There's no use for a password on the -u switch, but a blank one will make curl ask for one and you already entered the real password to kinit/kauth. TELNET The curl telnet support is basic and very easy to use. Curl passes all data passed to it on stdin to the remote server. Connect to a remote telnet server using a command line similar to: curl telnet://remote.server.com And enter the data to pass to the server on stdin. The result will be sent to stdout or to the file you specify with -o. You might want the -N/--no-buffer option to switch off the buffered output for slow connections or similar. Pass options to the telnet protocol negotiation, by using the -t option. To tell the server we use a vt100 terminal, try something like: curl -tTTYPE=vt100 telnet://remote.server.com Other interesting options for it -t include: - XDISPLOC= Sets the X display location. - NEW_ENV= Sets an environment variable. NOTE: The telnet protocol does not specify any way to login with a specified user and password so curl can't do that automatically. To do that, you need to track when the login prompt is received and send the username and password accordingly. PERSISTENT CONNECTIONS Specifying multiple files on a single command line will make curl transfer all of them, one after the other in the specified order. libcurl will attempt to use persistent connections for the transfers so that the second transfer to the same host can use the same connection that was already initiated and was left open in the previous transfer. This greatly decreases connection time for all but the first transfer and it makes a far better use of the network. Note that curl cannot use persistent connections for transfers that are used in subsequence curl invokes. Try to stuff as many URLs as possible on the same command line if they are using the same host, as that'll make the transfers faster. If you use an HTTP proxy for file transfers, practically all transfers will be persistent. MULTIPLE TRANSFERS WITH A SINGLE COMMAND LINE As is mentioned above, you can download multiple files with one command line by simply adding more URLs. If you want those to get saved to a local file instead of just printed to stdout, you need to add one save option for each URL you specify. Note that this also goes for the -O option (but not --remote-name-all). For example: get two files and use -O for the first and a custom file name for the second: curl -O http://url.com/file.txt ftp://ftp.com/moo.exe -o moo.jpg You can also upload multiple files in a similar fashion: curl -T local1 ftp://ftp.com/moo.exe -T local2 ftp://ftp.com/moo2.txt IPv6 curl will connect to a server with IPv6 when a host lookup returns an IPv6 address and fall back to IPv4 if the connection fails. The --ipv4 and --ipv6 options can specify which address to use when both are available. IPv6 addresses can also be specified directly in URLs using the syntax: http://[2001:1890:1112:1::20]/overview.html When this style is used, the -g option must be given to stop curl from interpreting the square brackets as special globbing characters. Link local and site local addresses including a scope identifier, such as fe80::1234%1, may also be used, but the scope portion must be numeric or match an existing network interface on Linux and the percent character must be URL escaped. The previous example in an SFTP URL might look like: sftp://[fe80::1234%251]/ IPv6 addresses provided other than in URLs (e.g. to the --proxy, --interface or --ftp-port options) should not be URL encoded. METALINK Curl supports Metalink (both version 3 and 4 (RFC 5854) are supported), a way to list multiple URIs and hashes for a file. Curl will make use of the mirrors listed within for failover if there are errors (such as the file or server not being available). It will also verify the hash of the file after the download completes. The Metalink file itself is downloaded and processed in memory and not stored in the local file system. Example to use a remote Metalink file: curl --metalink http://www.example.com/example.metalink To use a Metalink file in the local file system, use FILE protocol (file://): curl --metalink file://example.metalink Please note that if FILE protocol is disabled, there is no way to use a local Metalink file at the time of this writing. Also note that if --metalink and --include are used together, --include will be ignored. This is because including headers in the response will break Metalink parser and if the headers are included in the file described in Metalink file, hash check will fail. MAILING LISTS For your convenience, we have several open mailing lists to discuss curl, its development and things relevant to this. Get all info at https://curl.haxx.se/mail/. Some of the lists available are: curl-users Users of the command line tool. How to use it, what doesn't work, new features, related tools, questions, news, installations, compilations, running, porting etc. curl-library Developers using or developing libcurl. Bugs, extensions, improvements. curl-announce Low-traffic. Only receives announcements of new public versions. At worst, that makes something like one or two mails per month, but usually only one mail every second month. curl-and-php Using the curl functions in PHP. Everything curl with a PHP angle. Or PHP with a curl angle. curl-and-python Python hackers using curl with or without the python binding pycurl. Please direct curl questions, feature requests and trouble reports to one of these mailing lists instead of mailing any individual. PK0]@/YYTODOnu[ _ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| Things that could be nice to do in the future Things to do in project curl. Please tell us what you think, contribute and send us patches that improve things! Be aware that these are things that we could do, or have once been considered things we could do. If you want to work on any of these areas, please consider bringing it up for discussions first on the mailing list so that we all agree it is still a good idea for the project! All bugs documented in the KNOWN_BUGS document are subject for fixing! 1. libcurl 1.2 More data sharing 1.3 struct lifreq 1.4 signal-based resolver timeouts 1.5 get rid of PATH_MAX 1.6 Modified buffer size approach 1.7 Support HTTP/2 for HTTP(S) proxies 1.8 CURLOPT_RESOLVE for any port number 1.9 Cache negative name resolves 1.10 auto-detect proxy 1.11 minimize dependencies with dynamically loaded modules 1.12 updated DNS server while running 1.13 DNS-over-HTTPS 1.14 Typesafe curl_easy_setopt() 1.15 Monitor connections in the connection pool 1.16 Try to URL encode given URL 1.17 Add support for IRIs 1.18 try next proxy if one doesn't work 1.19 Timeout idle connections from the pool 1.20 SRV and URI DNS records 1.21 API for URL parsing/splitting 1.22 CURLINFO_PAUSE_STATE 1.23 Offer API to flush the connection pool 1.24 TCP Fast Open for windows 1.25 Expose tried IP addresses that failed 1.26 CURL_REFUSE_CLEARTEXT 1.27 hardcode the "localhost" addresses 1.28 FD_CLOEXEC 2. libcurl - multi interface 2.1 More non-blocking 2.2 Better support for same name resolves 2.3 Non-blocking curl_multi_remove_handle() 2.4 Split connect and authentication process 2.5 Edge-triggered sockets should work 3. Documentation 3.2 Provide cmake config-file 4. FTP 4.1 HOST 4.2 Alter passive/active on failure and retry 4.3 Earlier bad letter detection 4.4 REST for large files 4.5 ASCII support 4.6 GSSAPI via Windows SSPI 4.7 STAT for LIST without data connection 4.8 Option to ignore private IP addresses in PASV response 5. HTTP 5.1 Better persistency for HTTP 1.0 5.2 support FF3 sqlite cookie files 5.3 Rearrange request header order 5.5 auth= in URLs 5.6 Refuse "downgrade" redirects 5.7 QUIC 5.8 Leave secure cookies alone 6. TELNET 6.1 ditch stdin 6.2 ditch telnet-specific select 6.3 feature negotiation debug data 7. SMTP 7.1 Pipelining 7.2 Enhanced capability support 7.3 Add CURLOPT_MAIL_CLIENT option 8. POP3 8.1 Pipelining 8.2 Enhanced capability support 9. IMAP 9.1 Enhanced capability support 10. LDAP 10.1 SASL based authentication mechanisms 11. SMB 11.1 File listing support 11.2 Honor file timestamps 11.3 Use NTLMv2 11.4 Create remote directories 12. New protocols 12.1 RSYNC 13. SSL 13.1 Disable specific versions 13.2 Provide mutex locking API 13.3 Support in-memory certs/ca certs/keys 13.4 Cache/share OpenSSL contexts 13.5 Export session ids 13.6 Provide callback for cert verification 13.7 improve configure --with-ssl 13.8 Support DANE 13.9 Configurable loading of OpenSSL configuration file 13.10 Support Authority Information Access certificate extension (AIA) 13.11 Support intermediate & root pinning for PINNEDPUBLICKEY 13.12 Support HSTS 13.13 Support HPKP 13.14 Support the clienthello extension 14. GnuTLS 14.1 SSL engine stuff 14.2 check connection 15. WinSSL/SChannel 15.1 Add support for client certificate authentication 15.3 Add support for the --ciphers option 16. SASL 16.1 Other authentication mechanisms 16.2 Add QOP support to GSSAPI authentication 16.3 Support binary messages (i.e.: non-base64) 17. SSH protocols 17.1 Multiplexing 17.2 SFTP performance 17.3 Support better than MD5 hostkey hash 17.4 Support CURLOPT_PREQUOTE 18. Command line tool 18.1 sync 18.2 glob posts 18.3 prevent file overwriting 18.4 simultaneous parallel transfers 18.5 UTF-8 filenames in Content-Disposition 18.6 warning when setting an option 18.7 warning if curl version is not in sync with libcurl version 18.8 offer color-coded HTTP header output 18.9 Choose the name of file in braces for complex URLs 18.10 improve how curl works in a windows console window 18.11 -w output to stderr 18.12 keep running, read instructions from pipe/socket 18.13 support metalink in http headers 18.14 --fail without --location should treat 3xx as a failure 18.15 --retry should resume 18.16 send only part of --data 18.17 consider file name from the redirected URL with -O ? 18.18 retry on network is unreachable 18.19 expand ~/ in config files 18.20 host name sections in config files 19. Build 19.1 roffit 19.2 Enable PIE and RELRO by default 20. Test suite 20.1 SSL tunnel 20.2 nicer lacking perl message 20.3 more protocols supported 20.4 more platforms supported 20.5 Add support for concurrent connections 20.6 Use the RFC6265 test suite 21. Next SONAME bump 21.1 http-style HEAD output for FTP 21.2 combine error codes 21.3 extend CURLOPT_SOCKOPTFUNCTION prototype 22. Next major release 22.1 cleanup return codes 22.2 remove obsolete defines 22.3 size_t 22.4 remove several functions 22.5 remove CURLOPT_FAILONERROR 22.6 remove CURLOPT_DNS_USE_GLOBAL_CACHE 22.7 remove progress meter from libcurl 22.8 remove 'curl_httppost' from public ============================================================================== 1. libcurl 1.2 More data sharing curl_share_* functions already exist and work, and they can be extended to share more. For example, enable sharing of the ares channel. 1.3 struct lifreq Use 'struct lifreq' and SIOCGLIFADDR instead of 'struct ifreq' and SIOCGIFADDR on newer Solaris versions as they claim the latter is obsolete. To support IPv6 interface addresses for network interfaces properly. 1.4 signal-based resolver timeouts libcurl built without an asynchronous resolver library uses alarm() to time out DNS lookups. When a timeout occurs, this causes libcurl to jump from the signal handler back into the library with a sigsetjmp, which effectively causes libcurl to continue running within the signal handler. This is non-portable and could cause problems on some platforms. A discussion on the problem is available at https://curl.haxx.se/mail/lib-2008-09/0197.html Also, alarm() provides timeout resolution only to the nearest second. alarm ought to be replaced by setitimer on systems that support it. 1.5 get rid of PATH_MAX Having code use and rely on PATH_MAX is not nice: https://insanecoding.blogspot.com/2007/11/pathmax-simply-isnt.html Currently the SSH based code uses it a bit, but to remove PATH_MAX from there we need libssh2 to properly tell us when we pass in a too small buffer and its current API (as of libssh2 1.2.7) doesn't. 1.6 Modified buffer size approach Current libcurl allocates a fixed 16K size buffer for download and an additional 16K for upload. They are always unconditionally part of the easy handle. If CRLF translations are requested, an additional 32K "scratch buffer" is allocated. A total of 64K transfer buffers in the worst case. First, while the handles are not actually in use these buffers could be freed so that lingering handles just kept in queues or whatever waste less memory. Secondly, SFTP is a protocol that needs to handle many ~30K blocks at once since each need to be individually acked and therefore libssh2 must be allowed to send (or receive) many separate ones in parallel to achieve high transfer speeds. A current libcurl build with a 16K buffer makes that impossible, but one with a 512K buffer will reach MUCH faster transfers. But allocating 512K unconditionally for all buffers just in case they would like to do fast SFTP transfers at some point is not a good solution either. Dynamically allocate buffer size depending on protocol in use in combination with freeing it after each individual transfer? Other suggestions? 1.7 Support HTTP/2 for HTTP(S) proxies Support for doing HTTP/2 to HTTP and HTTPS proxies is still missing. 1.8 CURLOPT_RESOLVE for any port number This option allows applications to set a replacement IP address for a given host + port pair. Consider making support for providing a replacement address for the host name on all port numbers. See https://github.com/curl/curl/issues/1264 1.9 Cache negative name resolves A name resolve that has failed is likely to fail when made again within a short period of time. Currently we only cache positive responses. 1.10 auto-detect proxy libcurl could be made to detect the system proxy setup automatically and use that. On Windows, macOS and Linux desktops for example. The pull-request to use libproxy for this was deferred due to doubts on the reliability of the dependency and how to use it: https://github.com/curl/curl/pull/977 libdetectproxy is a (C++) library for detecting the proxy on Windows https://github.com/paulharris/libdetectproxy 1.11 minimize dependencies with dynamically loaded modules We can create a system with loadable modules/plug-ins, where these modules would be the ones that link to 3rd party libs. That would allow us to avoid having to load ALL dependencies since only the necessary ones for this app/invoke/used protocols would be necessary to load. See https://github.com/curl/curl/issues/349 1.12 updated DNS server while running If /etc/resolv.conf gets updated while a program using libcurl is running, it is may cause name resolves to fail unless res_init() is called. We should consider calling res_init() + retry once unconditionally on all name resolve failures to mitigate against this. Firefox works like that. Note that Windows doesn't have res_init() or an alternative. https://github.com/curl/curl/issues/2251 1.13 DNS-over-HTTPS By adding support for DNS-over-HTTPS curl could resolve host names using a totally separate name server than the standard system resolver, while at the same time doing so over a communication channel that enhances privacy and security. https://github.com/curl/curl/wiki/DNS-over-HTTPS 1.14 Typesafe curl_easy_setopt() One of the most common problems in libcurl using applications is the lack of type checks for curl_easy_setopt() which happens because it accepts varargs and thus can take any type. One possible solution to this is to introduce a few different versions of the setopt version for the different kinds of data you can set. curl_easy_set_num() - sets a long value curl_easy_set_large() - sets a curl_off_t value curl_easy_set_ptr() - sets a pointer curl_easy_set_cb() - sets a callback PLUS its callback data 1.15 Monitor connections in the connection pool libcurl's connection cache or pool holds a number of open connections for the purpose of possible subsequent connection reuse. It may contain a few up to a significant amount of connections. Currently, libcurl leaves all connections as they are and first when a connection is iterated over for matching or reuse purpose it is verified that it is still alive. Those connections may get closed by the server side for idleness or they may get a HTTP/2 ping from the peer to verify that they're still alive. By adding monitoring of the connections while in the pool, libcurl can detect dead connections (and close them) better and earlier, and it can handle HTTP/2 pings to keep such ones alive even when not actively doing transfers on them. 1.16 Try to URL encode given URL Given a URL that for example contains spaces, libcurl could have an option that would try somewhat harder than it does now and convert spaces to %20 and perhaps URL encoded byte values over 128 etc (basically do what the redirect following code already does). https://github.com/curl/curl/issues/514 1.17 Add support for IRIs IRIs (RFC 3987) allow localized, non-ascii, names in the URL. To properly support this, curl/libcurl would need to translate/encode the given input from the input string encoding into percent encoded output "over the wire". To make that work smoothly for curl users even on Windows, curl would probably need to be able to convert from several input encodings. 1.18 try next proxy if one doesn't work Allow an application to specify a list of proxies to try, and failing to connect to the first go on and try the next instead until the list is exhausted. Browsers support this feature at least when they specify proxies using PACs. https://github.com/curl/curl/issues/896 1.19 Timeout idle connections from the pool libcurl currently keeps connections in its connection pool for an indefinite period of time, until it either gets reused, gets noticed that it has been closed by the server or gets pruned to make room for a new connection. To reduce overhead (especially for when we add monitoring of the connections in the pool), we should introduce a timeout so that connections that have been idle for N seconds get closed. 1.20 SRV and URI DNS records Offer support for resolving SRV and URI DNS records for libcurl to know which server to connect to for various protocols (including HTTP!). 1.21 API for URL parsing/splitting libcurl has always parsed URLs internally and never exposed any API or features to allow applications to do it. Still most or many applications using libcurl need that ability. In polls to users, we've learned that many libcurl users would like to see and use such an API. 1.22 CURLINFO_PAUSE_STATE Return information about the transfer's current pause state, in both directions. https://github.com/curl/curl/issues/2588 1.23 Offer API to flush the connection pool Sometimes applications want to flush all the existing connections kept alive. An API could allow a forced flush or just a forced loop that would properly close all connections that have been closed by the server already. 1.24 TCP Fast Open for windows libcurl supports the CURLOPT_TCP_FASTOPEN option since 7.49.0 for Linux and Mac OS. Windows supports TCP Fast Open starting with Windows 10, version 1607 and we should add support for it. 1.25 Expose tried IP addresses that failed When libcurl fails to connect to a host, it should be able to offer the application the list of IP addresses that were used in the attempt. https://github.com/curl/curl/issues/2126 1.26 CURL_REFUSE_CLEARTEXT An environment variable that when set will make libcurl refuse to use any cleartext network protocol. That's all non-encrypted ones (FTP, HTTP, Gopher, etc). By adding the check to libcurl and not just curl, this environment variable can then help users to block all libcurl-using programs from accessing the network using unsafe protocols. The variable could be given some sort of syntax or different levels and be used to also allow for example users to refuse libcurl to do transfers with HTTPS certificate checks disabled. It could also offer to refuse usernames in URLs (see TODO 1.1) 1.27 hardcode the "localhost" addresses There's this new spec getting adopted that says "localhost" should always and unconditionally be a local address and not get resolved by a DNS server. A fine way for curl to fix this would be to simply hard-code the response to 127.0.0.1 and/or ::1 (depending on what IP versions that are requested). This is what the browsers probably will do with this hostname. https://bugzilla.mozilla.org/show_bug.cgi?id=1220810 https://tools.ietf.org/html/draft-ietf-dnsop-let-localhost-be-localhost-02 1.28 FD_CLOEXEC It sets the close-on-exec flag for the file descriptor, which causes the file descriptor to be automatically (and atomically) closed when any of the exec-family functions succeed. Should probably be set by default? https://github.com/curl/curl/issues/2252 2. libcurl - multi interface 2.1 More non-blocking Make sure we don't ever loop because of non-blocking sockets returning EWOULDBLOCK or similar. Blocking cases include: - Name resolves on non-windows unless c-ares or the threaded resolver is used - SOCKS proxy handshakes - file:// transfers - TELNET transfers - The "DONE" operation (post transfer protocol-specific actions) for the protocols SFTP, SMTP, FTP. Fixing Curl_done() for this is a worthy task. 2.2 Better support for same name resolves If a name resolve has been initiated for name NN and a second easy handle wants to resolve that name as well, make it wait for the first resolve to end up in the cache instead of doing a second separate resolve. This is especially needed when adding many simultaneous handles using the same host name when the DNS resolver can get flooded. 2.3 Non-blocking curl_multi_remove_handle() The multi interface has a few API calls that assume a blocking behavior, like add_handle() and remove_handle() which limits what we can do internally. The multi API need to be moved even more into a single function that "drives" everything in a non-blocking manner and signals when something is done. A remove or add would then only ask for the action to get started and then multi_perform() etc still be called until the add/remove is completed. 2.4 Split connect and authentication process The multi interface treats the authentication process as part of the connect phase. As such any failures during authentication won't trigger the relevant QUIT or LOGOFF for protocols such as IMAP, POP3 and SMTP. 2.5 Edge-triggered sockets should work The multi_socket API should work with edge-triggered socket events. One of the internal actions that need to be improved for this to work perfectly is the 'maxloops' handling in transfer.c:readwrite_data(). 3. Documentation 3.2 Provide cmake config-file A config-file package is a set of files provided by us to allow applications to write cmake scripts to find and use libcurl easier. See https://github.com/curl/curl/issues/885 4. FTP 4.1 HOST HOST is a command for a client to tell which host name to use, to offer FTP servers named-based virtual hosting: https://tools.ietf.org/html/rfc7151 4.2 Alter passive/active on failure and retry When trying to connect passively to a server which only supports active connections, libcurl returns CURLE_FTP_WEIRD_PASV_REPLY and closes the connection. There could be a way to fallback to an active connection (and vice versa). https://curl.haxx.se/bug/feature.cgi?id=1754793 4.3 Earlier bad letter detection Make the detection of (bad) %0d and %0a codes in FTP URL parts earlier in the process to avoid doing a resolve and connect in vain. 4.4 REST for large files REST fix for servers not behaving well on >2GB requests. This should fail if the server doesn't set the pointer to the requested index. The tricky (impossible?) part is to figure out if the server did the right thing or not. 4.5 ASCII support FTP ASCII transfers do not follow RFC959. They don't convert the data accordingly. 4.6 GSSAPI via Windows SSPI In addition to currently supporting the SASL GSSAPI mechanism (Kerberos V5) via third-party GSS-API libraries, such as Heimdal or MIT Kerberos, also add support for GSSAPI authentication via Windows SSPI. 4.7 STAT for LIST without data connection Some FTP servers allow STAT for listing directories instead of using LIST, and the response is then sent over the control connection instead of as the otherwise usedw data connection: http://www.nsftools.com/tips/RawFTP.htm#STAT This is not detailed in any FTP specification. 4.8 Option to ignore private IP addresses in PASV response Some servers respond with and some other FTP client implementations can ignore private (RFC 1918 style) IP addresses when received in PASV responses. To consider for libcurl as well. See https://github.com/curl/curl/issues/1455 5. HTTP 5.1 Better persistency for HTTP 1.0 "Better" support for persistent connections over HTTP 1.0 https://curl.haxx.se/bug/feature.cgi?id=1089001 5.2 support FF3 sqlite cookie files Firefox 3 is changing from its former format to a a sqlite database instead. We should consider how (lib)curl can/should support this. https://curl.haxx.se/bug/feature.cgi?id=1871388 5.3 Rearrange request header order Server implementors often make an effort to detect browser and to reject clients it can detect to not match. One of the last details we cannot yet control in libcurl's HTTP requests, which also can be exploited to detect that libcurl is in fact used even when it tries to impersonate a browser, is the order of the request headers. I propose that we introduce a new option in which you give headers a value, and then when the HTTP request is built it sorts the headers based on that number. We could then have internally created headers use a default value so only headers that need to be moved have to be specified. 5.5 auth= in URLs Add the ability to specify the preferred authentication mechanism to use by using ;auth= in the login part of the URL. For example: http://test:pass;auth=NTLM@example.com would be equivalent to specifying --user test:pass;auth=NTLM or --user test:pass --ntlm from the command line. Additionally this should be implemented for proxy base URLs as well. 5.6 Refuse "downgrade" redirects See https://github.com/curl/curl/issues/226 Consider a way to tell curl to refuse to "downgrade" protocol with a redirect and/or possibly a bit that refuses redirect to change protocol completely. 5.7 QUIC The standardization process of QUIC has been taken to the IETF and can be followed on the [IETF QUIC Mailing list](https://www.ietf.org/mailman/listinfo/quic). I'd like us to get on the bandwagon. Ideally, this would be done with a separate library/project to handle the binary/framing layer in a similar fashion to how HTTP/2 is implemented. This, to allow other projects to benefit from the work and to thus broaden the interest and chance of others to participate. 5.8 Leave secure cookies alone Non-secure origins (HTTP sites) should not be allowed to set or modify cookies with the 'secure' property: https://tools.ietf.org/html/draft-ietf-httpbis-cookie-alone-01 6. TELNET 6.1 ditch stdin Reading input (to send to the remote server) on stdin is a crappy solution for library purposes. We need to invent a good way for the application to be able to provide the data to send. 6.2 ditch telnet-specific select Move the telnet support's network select() loop go away and merge the code into the main transfer loop. Until this is done, the multi interface won't work for telnet. 6.3 feature negotiation debug data Add telnet feature negotiation data to the debug callback as header data. 7. SMTP 7.1 Pipelining Add support for pipelining emails. 7.2 Enhanced capability support Add the ability, for an application that uses libcurl, to obtain the list of capabilities returned from the EHLO command. 7.3 Add CURLOPT_MAIL_CLIENT option Rather than use the URL to specify the mail client string to present in the HELO and EHLO commands, libcurl should support a new CURLOPT specifically for specifying this data as the URL is non-standard and to be honest a bit of a hack ;-) Please see the following thread for more information: https://curl.haxx.se/mail/lib-2012-05/0178.html 8. POP3 8.1 Pipelining Add support for pipelining commands. 8.2 Enhanced capability support Add the ability, for an application that uses libcurl, to obtain the list of capabilities returned from the CAPA command. 9. IMAP 9.1 Enhanced capability support Add the ability, for an application that uses libcurl, to obtain the list of capabilities returned from the CAPABILITY command. 10. LDAP 10.1 SASL based authentication mechanisms Currently the LDAP module only supports ldap_simple_bind_s() in order to bind to an LDAP server. However, this function sends username and password details using the simple authentication mechanism (as clear text). However, it should be possible to use ldap_bind_s() instead specifying the security context information ourselves. 11. SMB 11.1 File listing support Add support for listing the contents of a SMB share. The output should probably be the same as/similar to FTP. 11.2 Honor file timestamps The timestamp of the transferred file should reflect that of the original file. 11.3 Use NTLMv2 Currently the SMB authentication uses NTLMv1. 11.4 Create remote directories Support for creating remote directories when uploading a file to a directory that doesn't exist on the server, just like --ftp-create-dirs. 12. New protocols 12.1 RSYNC There's no RFC for the protocol or an URI/URL format. An implementation should most probably use an existing rsync library, such as librsync. 13. SSL 13.1 Disable specific versions Provide an option that allows for disabling specific SSL versions, such as SSLv2 https://curl.haxx.se/bug/feature.cgi?id=1767276 13.2 Provide mutex locking API Provide a libcurl API for setting mutex callbacks in the underlying SSL library, so that the same application code can use mutex-locking independently of OpenSSL or GnutTLS being used. 13.3 Support in-memory certs/ca certs/keys You can specify the private and public keys for SSH/SSL as file paths. Some programs want to avoid using files and instead just pass them as in-memory data blobs. There's probably a challenge to make this work across the plethory of different TLS and SSH backends that curl supports. https://github.com/curl/curl/issues/2310 13.4 Cache/share OpenSSL contexts "Look at SSL cafile - quick traces look to me like these are done on every request as well, when they should only be necessary once per SSL context (or once per handle)". The major improvement we can rather easily do is to make sure we don't create and kill a new SSL "context" for every request, but instead make one for every connection and re-use that SSL context in the same style connections are re-used. It will make us use slightly more memory but it will libcurl do less creations and deletions of SSL contexts. Technically, the "caching" is probably best implemented by getting added to the share interface so that easy handles who want to and can reuse the context specify that by sharing with the right properties set. https://github.com/curl/curl/issues/1110 13.5 Export session ids Add an interface to libcurl that enables "session IDs" to get exported/imported. Cris Bailiff said: "OpenSSL has functions which can serialise the current SSL state to a buffer of your choice, and recover/reset the state from such a buffer at a later date - this is used by mod_ssl for apache to implement and SSL session ID cache". 13.6 Provide callback for cert verification OpenSSL supports a callback for customised verification of the peer certificate, but this doesn't seem to be exposed in the libcurl APIs. Could it be? There's so much that could be done if it were! 13.7 improve configure --with-ssl make the configure --with-ssl option first check for OpenSSL, then GnuTLS, then NSS... 13.8 Support DANE DNS-Based Authentication of Named Entities (DANE) is a way to provide SSL keys and certs over DNS using DNSSEC as an alternative to the CA model. https://www.rfc-editor.org/rfc/rfc6698.txt An initial patch was posted by Suresh Krishnaswamy on March 7th 2013 (https://curl.haxx.se/mail/lib-2013-03/0075.html) but it was a too simple approach. See Daniel's comments: https://curl.haxx.se/mail/lib-2013-03/0103.html . libunbound may be the correct library to base this development on. Björn Stenberg wrote a separate initial take on DANE that was never completed. 13.9 Configurable loading of OpenSSL configuration file libcurl calls the OpenSSL function CONF_modules_load_file() in openssl.c, Curl_ossl_init(). "We regard any changes in the OpenSSL configuration as a security risk or at least as unnecessary." Please add a configuration switch or something similar to disable the CONF_modules_load_file() call. See https://github.com/curl/curl/issues/2724 13.10 Support Authority Information Access certificate extension (AIA) AIA can provide various things like CRLs but more importantly information about intermediate CA certificates that can allow validation path to be fullfilled when the HTTPS server doesn't itself provide them. Since AIA is about downloading certs on demand to complete a TLS handshake, it is probably a bit tricky to get done right. See https://github.com/curl/curl/issues/2793 13.11 Support intermediate & root pinning for PINNEDPUBLICKEY CURLOPT_PINNEDPUBLICKEY does not consider the hashes of intermediate & root certificates when comparing the pinned keys. Therefore it is not compatible with "HTTP Public Key Pinning" as there also intermediate and root certificates can be pinned. This is very useful as it prevents webadmins from "locking themself out of their servers". Adding this feature would make curls pinning 100% compatible to HPKP and allow more flexible pinning. 13.12 Support HSTS "HTTP Strict Transport Security" is TOFU (trust on first use), time-based features indicated by a HTTP header send by the webserver. It is widely used in browsers and it's purpose is to prevent insecure HTTP connections after a previous HTTPS connection. It protects against SSLStripping attacks. Doc: https://developer.mozilla.org/en-US/docs/Web/Security/HTTP_strict_transport_security RFC 6797: https://tools.ietf.org/html/rfc6797 13.13 Support HPKP "HTTP Public Key Pinning" is TOFU (trust on first use), time-based features indicated by a HTTP header send by the webserver. It's purpose is to prevent Man-in-the-middle attacks by trusted CAs by allowing webadmins to specify which CAs/certificates/public keys to trust when connection to their websites. It can be build based on PINNEDPUBLICKEY. Wikipedia: https://en.wikipedia.org/wiki/HTTP_Public_Key_Pinning OWASP: https://www.owasp.org/index.php/Certificate_and_Public_Key_Pinning Doc: https://developer.mozilla.org/de/docs/Web/Security/Public_Key_Pinning RFC: https://tools.ietf.org/html/draft-ietf-websec-key-pinning-21 13.14 Support the clienthello extension Certain stupid networks and middle boxes have a problem with SSL handshake pakets that are within a certain size range because how that sets some bits that previously (in older TLS version) were not set. The clienthello extension adds padding to avoid that size range. https://tools.ietf.org/html/rfc7685 https://github.com/curl/curl/issues/2299 14. GnuTLS 14.1 SSL engine stuff Is this even possible? 14.2 check connection Add a way to check if the connection seems to be alive, to correspond to the SSL_peak() way we use with OpenSSL. 15. WinSSL/SChannel 15.1 Add support for client certificate authentication WinSSL/SChannel currently makes use of the OS-level system and user certificate and private key stores. This does not allow the application or the user to supply a custom client certificate using curl or libcurl. Therefore support for the existing -E/--cert and --key options should be implemented by supplying a custom certificate to the SChannel APIs, see: - Getting a Certificate for Schannel https://msdn.microsoft.com/en-us/library/windows/desktop/aa375447.aspx 15.3 Add support for the --ciphers option The cipher suites used by WinSSL/SChannel are configured on an OS-level instead of an application-level. This does not allow the application or the user to customize the configured cipher suites using curl or libcurl. Therefore support for the existing --ciphers option should be implemented by mapping the OpenSSL/GnuTLS cipher suites to the SChannel APIs, see - Specifying Schannel Ciphers and Cipher Strengths https://msdn.microsoft.com/en-us/library/windows/desktop/aa380161.aspx 16. SASL 16.1 Other authentication mechanisms Add support for other authentication mechanisms such as OLP, GSS-SPNEGO and others. 16.2 Add QOP support to GSSAPI authentication Currently the GSSAPI authentication only supports the default QOP of auth (Authentication), whilst Kerberos V5 supports both auth-int (Authentication with integrity protection) and auth-conf (Authentication with integrity and privacy protection). 16.3 Support binary messages (i.e.: non-base64) Mandatory to support LDAP SASL authentication. 17. SSH protocols 17.1 Multiplexing SSH is a perfectly fine multiplexed protocols which would allow libcurl to do multiple parallel transfers from the same host using the same connection, much in the same spirit as HTTP/2 does. libcurl however does not take advantage of that ability but will instead always create a new connection for new transfers even if an existing connection already exists to the host. To fix this, libcurl would have to detect an existing connection and "attach" the new transfer to the existing one. 17.2 SFTP performance libcurl's SFTP transfer performance is sub par and can be improved, mostly by the approach mentioned in "1.6 Modified buffer size approach". 17.3 Support better than MD5 hostkey hash libcurl offers the CURLOPT_SSH_HOST_PUBLIC_KEY_MD5 option for verifying the server's key. MD5 is generally being deprecated so we should implement support for stronger hashing algorithms. libssh2 itself is what provides this underlying functionality and it supports at least SHA-1 as an alternative. SHA-1 is also being deprecated these days so we should consider workign with libssh2 to instead offer support for SHA-256 or similar. 17.4 Support CURLOPT_PREQUOTE The two other QUOTE options are supported for SFTP, but this was left out for unknown reasons! 18. Command line tool 18.1 sync "curl --sync http://example.com/feed[1-100].rss" or "curl --sync http://example.net/{index,calendar,history}.html" Downloads a range or set of URLs using the remote name, but only if the remote file is newer than the local file. A Last-Modified HTTP date header should also be used to set the mod date on the downloaded file. 18.2 glob posts Globbing support for -d and -F, as in 'curl -d "name=foo[0-9]" URL'. This is easily scripted though. 18.3 prevent file overwriting Add an option that prevents curl from overwriting existing local files. When used, and there already is an existing file with the target file name (either -O or -o), a number should be appended (and increased if already existing). So that index.html becomes first index.html.1 and then index.html.2 etc. 18.4 simultaneous parallel transfers The client could be told to use maximum N simultaneous parallel transfers and then just make sure that happens. It should of course not make more than one connection to the same remote host. This would require the client to use the multi interface. https://curl.haxx.se/bug/feature.cgi?id=1558595 Using the multi interface would also allow properly using parallel transfers with HTTP/2 and supporting HTTP/2 server push from the command line. 18.5 UTF-8 filenames in Content-Disposition RFC 6266 documents how UTF-8 names can be passed to a client in the Content-Disposition header, and curl does not support this. https://github.com/curl/curl/issues/1888 18.6 warning when setting an option Display a warning when libcurl returns an error when setting an option. This can be useful to tell when support for a particular feature hasn't been compiled into the library. 18.7 warning if curl version is not in sync with libcurl version This is usually a sign of a funny, weird or unexpected install situations that aren't always quickly nor easily detected by users. curl and libcurl are always released in sync and should use the same version numbers unless very special situations. 18.8 offer color-coded HTTP header output By offering different color output on the header name and the header contents, they could be made more readable and thus help users working on HTTP services. 18.9 Choose the name of file in braces for complex URLs When using braces to download a list of URLs and you use complicated names in the list of alternatives, it could be handy to allow curl to use other names when saving. Consider a way to offer that. Possibly like {partURL1:name1,partURL2:name2,partURL3:name3} where the name following the colon is the output name. See https://github.com/curl/curl/issues/221 18.10 improve how curl works in a windows console window If you pull the scrollbar when transferring with curl in a Windows console window, the transfer is interrupted and can get disconnected. This can probably be improved. See https://github.com/curl/curl/issues/322 18.11 -w output to stderr -w is quite useful, but not to those of us who use curl without -o or -O (such as for scripting through a higher level language). It would be nice to have an option that is exactly like -w but sends it to stderr instead. Proposed name: --write-stderr. See https://github.com/curl/curl/issues/613 18.12 keep running, read instructions from pipe/socket Provide an option that makes curl not exit after the last URL (or even work without a given URL), and then make it read instructions passed on a pipe or over a socket to make further instructions so that a second subsequent curl invoke can talk to the still running instance and ask for transfers to get done, and thus maintain its connection pool, DNS cache and more. 18.13 support metalink in http headers Curl has support for downloading a metalink xml file, processing it, and then downloading the target of the metalink. This is done via the --metalink option. It would be nice if metalink also supported downloading via metalink information that is stored in HTTP headers (RFC 6249). Theoretically this could also be supported with the --metalink option. See https://tools.ietf.org/html/rfc6249 See also https://lists.gnu.org/archive/html/bug-wget/2015-06/msg00034.html for an implematation of this in wget. 18.14 --fail without --location should treat 3xx as a failure To allow a command line like this to detect a redirect and consider it a failure: curl -v --fail -O https://example.com/curl-7.48.0.tar.gz ... --fail must treat 3xx responses as failures too. The least problematic way to implement this is probably to add that new logic in the command line tool only and not in the underlying CURLOPT_FAILONERROR logic. 18.15 --retry should resume When --retry is used and curl actually retries transfer, it should use the already transferred data and do a resumed transfer for the rest (when possible) so that it doesn't have to transfer the same data again that was already transferred before the retry. See https://github.com/curl/curl/issues/1084 18.16 send only part of --data When the user only wants to send a small piece of the data provided with --data or --data-binary, like when that data is a huge file, consider a way to specify that curl should only send a piece of that. One suggested syntax would be: "--data-binary @largefile.zip!1073741823-2147483647". See https://github.com/curl/curl/issues/1200 18.17 consider file name from the redirected URL with -O ? When a user gives a URL and uses -O, and curl follows a redirect to a new URL, the file name is not extracted and used from the newly redirected-to URL even if the new URL may have a much more sensible file name. This is clearly documented and helps for security since there's no surprise to users which file name that might get overwritten. But maybe a new option could allow for this or maybe -J should imply such a treatment as well as -J already allows for the server to decide what file name to use so it already provides the "may overwrite any file" risk. This is extra tricky if the original URL has no file name part at all since then the current code path will error out with an error message, and we can't *know* already at that point if curl will be redirected to a URL that has a file name... See https://github.com/curl/curl/issues/1241 18.18 retry on network is unreachable The --retry option retries transfers on "transient failures". We later added --retry-connrefused to also retry for "connection refused" errors. Suggestions have been brought to also allow retry on "network is unreachable" errors and while totally reasonable, maybe we should consider a way to make this more configurable than to add a new option for every new error people want to retry for? https://github.com/curl/curl/issues/1603 18.19 expand ~/ in config files For example .curlrc could benefit from being able to do this. See https://github.com/curl/curl/issues/2317 18.20 host name sections in config files config files would be more powerful if they could set different configurations depending on used URLs, host name or possibly origin. Then a default .curlrc could a specific user-agent only when doing requests against a certain site. 19. Build 19.1 roffit Consider extending 'roffit' to produce decent ASCII output, and use that instead of (g)nroff when building src/tool_hugehelp.c 19.2 Enable PIE and RELRO by default Especially when having programs that execute curl via the command line, PIE renders the exploitation of memory corruption vulnerabilities a lot more difficult. This can be attributed to the additional information leaks being required to conduct a successful attack. RELRO, on the other hand, masks different binary sections like the GOT as read-only and thus kills a handful of techniques that come in handy when attackers are able to arbitrarily overwrite memory. A few tests showed that enabling these features had close to no impact, neither on the performance nor on the general functionality of curl. 20. Test suite 20.1 SSL tunnel Make our own version of stunnel for simple port forwarding to enable HTTPS and FTP-SSL tests without the stunnel dependency, and it could allow us to provide test tools built with either OpenSSL or GnuTLS 20.2 nicer lacking perl message If perl wasn't found by the configure script, don't attempt to run the tests but explain something nice why it doesn't. 20.3 more protocols supported Extend the test suite to include more protocols. The telnet could just do FTP or http operations (for which we have test servers). 20.4 more platforms supported Make the test suite work on more platforms. OpenBSD and Mac OS. Remove fork()s and it should become even more portable. 20.5 Add support for concurrent connections Tests 836, 882 and 938 were designed to verify that separate connections aren't used when using different login credentials in protocols that shouldn't re-use a connection under such circumstances. Unfortunately, ftpserver.pl doesn't appear to support multiple concurrent connections. The read while() loop seems to loop until it receives a disconnect from the client, where it then enters the waiting for connections loop. When the client opens a second connection to the server, the first connection hasn't been dropped (unless it has been forced - which we shouldn't do in these tests) and thus the wait for connections loop is never entered to receive the second connection. 20.6 Use the RFC6265 test suite A test suite made for HTTP cookies (RFC 6265) by Adam Barth is available at https://github.com/abarth/http-state/tree/master/tests It'd be really awesome if someone would write a script/setup that would run curl with that test suite and detect deviances. Ideally, that would even be incorporated into our regular test suite. 21. Next SONAME bump 21.1 http-style HEAD output for FTP #undef CURL_FTP_HTTPSTYLE_HEAD in lib/ftp.c to remove the HTTP-style headers from being output in NOBODY requests over FTP 21.2 combine error codes Combine some of the error codes to remove duplicates. The original numbering should not be changed, and the old identifiers would be macroed to the new ones in an CURL_NO_OLDIES section to help with backward compatibility. Candidates for removal and their replacements: CURLE_FILE_COULDNT_READ_FILE => CURLE_REMOTE_FILE_NOT_FOUND CURLE_FTP_COULDNT_RETR_FILE => CURLE_REMOTE_FILE_NOT_FOUND CURLE_FTP_COULDNT_USE_REST => CURLE_RANGE_ERROR CURLE_FUNCTION_NOT_FOUND => CURLE_FAILED_INIT CURLE_LDAP_INVALID_URL => CURLE_URL_MALFORMAT CURLE_TFTP_NOSUCHUSER => CURLE_TFTP_ILLEGAL CURLE_TFTP_NOTFOUND => CURLE_REMOTE_FILE_NOT_FOUND CURLE_TFTP_PERM => CURLE_REMOTE_ACCESS_DENIED 21.3 extend CURLOPT_SOCKOPTFUNCTION prototype The current prototype only provides 'purpose' that tells what the connection/socket is for, but not any protocol or similar. It makes it hard for applications to differentiate on TCP vs UDP and even HTTP vs FTP and similar. 22. Next major release 22.1 cleanup return codes curl_easy_cleanup() returns void, but curl_multi_cleanup() returns a CURLMcode. These should be changed to be the same. 22.2 remove obsolete defines remove obsolete defines from curl/curl.h 22.3 size_t make several functions use size_t instead of int in their APIs 22.4 remove several functions remove the following functions from the public API: curl_getenv curl_mprintf (and variations) curl_strequal curl_strnequal They will instead become curlx_ - alternatives. That makes the curl app still capable of using them, by building with them from source. These functions have no purpose anymore: curl_multi_socket curl_multi_socket_all 22.5 remove CURLOPT_FAILONERROR Remove support for CURLOPT_FAILONERROR, it has gotten too kludgy and weird internally. Let the app judge success or not for itself. 22.6 remove CURLOPT_DNS_USE_GLOBAL_CACHE Remove support for a global DNS cache. Anything global is silly, and we already offer the share interface for the same functionality but done "right". 22.7 remove progress meter from libcurl The internally provided progress meter output doesn't belong in the library. Basically no application wants it (apart from curl) but instead applications can and should do their own progress meters using the progress callback. The progress callback should then be bumped as well to get proper 64bit variable types passed to it instead of doubles so that big files work correctly. 22.8 remove 'curl_httppost' from public curl_formadd() was made to fill in a public struct, but the fact that the struct is public is never really used by application for their own advantage but instead often restricts how the form functions can or can't be modified. Changing them to return a private handle will benefit the implementation and allow us much greater freedoms while still maintaining a solid API and ABI. PK0]Φ0X? ? RESOURCESnu[ _ _ ____ _ Project ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| This document lists documents and standards used by curl. RFC 959 - FTP Protocol RFC 1635 - How to Use Anonymous FTP RFC 1738 - Uniform Resource Locators RFC 1777 - Lightweight Directory Access Protocol (LDAP) RFC 1808 - Relative Uniform Resource Locators RFC 1867 - Form-based File Upload in HTML RFC 1950 - ZLIB Compressed Data Format Specification RFC 1951 - DEFLATE Compressed Data Format Specification RFC 1952 - GZIP File Format Specification RFC 1959 - LDAP URL Syntax RFC 2045-2049 - Everything you need to know about MIME! (needed for form based upload) RFC 2068 - HTTP 1.1 (obsoleted by RFC 2616) RFC 2104 - Keyed-Hashing for Message Authentication RFC 2109 - HTTP State Management Mechanism (cookie stuff) - Also, read Netscape's specification at https://curl.haxx.se/rfc/cookie_spec.html RFC 2183 - The Content-Disposition Header Field RFC 2195 - CRAM-MD5 Authentication RFC 2229 - A Dictionary Server Protocol RFC 2255 - Newer LDAP URL Format RFC 2231 - MIME Parameter Value and Encoded Word Extensions: Character Sets, Languages, and Continuations RFC 2388 - "Returning Values from Forms: multipart/form-data" Use this as an addition to the RFC1867 RFC 2396 - "Uniform Resource Identifiers: Generic Syntax and Semantics" This one obsoletes RFC 1738, but since RFC 1738 is often mentioned I've left it in this list. RFC 2428 - FTP Extensions for IPv6 and NATs RFC 2577 - FTP Security Considerations RFC 2616 - HTTP 1.1, the latest RFC 2617 - HTTP Authentication RFC 2718 - Guidelines for new URL Schemes RFC 2732 - Format for Literal IPv6 Addresses in URL's RFC 2818 - HTTP Over TLS (TLS is the successor to SSL) RFC 2821 - Simple Mail Transfer Protocol (SMTP) RFC 2964 - Use of HTTP State Management RFC 2965 - HTTP State Management Mechanism. Cookies. Obsoletes RFC2109 RFC 3207 - SMTP Over TLS RFC 4616 - PLAIN Authentication RFC 4954 - SMTP Authentication RFC 7932 - Brotli Compressed Data Format PK0]u ! !CHANGESnu[ _ _ ____ _ ___| | | | _ \| | / __| | | | |_) | | | (__| |_| | _ <| |___ \___|\___/|_| \_\_____| Changelog Version 7.61.1 (4 Sep 2018) Daniel Stenberg (4 Sep 2018) - THANKS: 7.61.1 status - RELEASE-NOTES: 7.61.1 - Curl_getoff_all_pipelines: ignore unused return values Since scan-build would warn on the dead "Dead store/Dead increment" Viktor Szakats (4 Sep 2018) - sftp: fix indentation Daniel Stenberg (4 Sep 2018) - [Przemysław Tomaszewski brought this change] sftp: don't send post-qoute sequence when retrying a connection Fixes #2939 Closes #2940 Kamil Dudka (3 Sep 2018) - url, vtls: make CURLOPT{,_PROXY}_TLS13_CIPHERS work This is a follow-up to PR #2607 and PR #2926. Closes #2936 Daniel Stenberg (3 Sep 2018) - [Jay Satiro brought this change] tool_operate: Add http code 408 to transient list for --retry - Treat 408 request timeout as transient so that curl will retry the request if --retry was used. Closes #2925 - [Jay Satiro brought this change] openssl: Fix setting TLS 1.3 cipher suites The flag indicating TLS 1.3 cipher support in the OpenSSL backend was missing. Bug: https://github.com/curl/curl/pull/2607#issuecomment-417283187 Reported-by: Kamil Dudka Closes #2926 - Curl_ntlm_core_mk_nt_hash: return error on too long password ... since it would cause an integer overflow if longer than (max size_t / 2). This is CVE-2018-14618 Bug: https://curl.haxx.se/docs/CVE-2018-14618.html Closes #2756 Reported-by: Zhaoyang Wu - [Rikard Falkeborn brought this change] http2: Use correct format identifier for stream_id Closes #2928 Marcel Raad (2 Sep 2018) - test1148: fix precheck output "precheck command error" is not very helpful. Daniel Stenberg (1 Sep 2018) - all: s/int/size_t cleanup Assisted-by: Rikard Falkeborn Closes #2922 - ssh-libssh: use FALLTHROUGH to silence gcc8 Jay Satiro (31 Aug 2018) - tool_operate: Fix setting proxy TLS 1.3 ciphers Daniel Stenberg (31 Aug 2018) - [Daniel Gustafsson brought this change] cookies: support creation-time attribute for cookies According to RFC6265 section 5.4, cookies with equal path lengths SHOULD be sorted by creation-time (earlier first). This adds a creation-time record to the cookie struct in order to make cookie sorting more deterministic. The creation-time is defined as the order of the cookies in the jar, the first cookie read fro the jar being the oldest. The creation-time is thus not serialized into the jar. Also remove the strcmp() matching in the sorting as there is no lexicographic ordering in RFC6265. Existing tests are updated to match. Closes #2524 Marcel Raad (31 Aug 2018) - Don't use Windows path %PWD for SSH tests All these tests failed on Windows because something like sftp://%HOSTIP:%SSHPORT%PWD/ expanded to sftp://127.0.0.1:1234c:/msys64/home/bla/curl and then curl complained about the port number ending with a letter. Use the original POSIX path instead of the Windows path created in checksystem to fix this. Closes https://github.com/curl/curl/pull/2920 Jay Satiro (29 Aug 2018) - CURLOPT_SSL_CTX_FUNCTION.3: clarify connection reuse warning Reported-by: Daniel Stenberg Closes https://github.com/curl/curl/issues/2916 Daniel Stenberg (28 Aug 2018) - THANKS-filter: dedup Daniel Jeliński - RELEASE-NOTES: synced - CURLOPT_ACCEPT_ENCODING.3: list them comma-separated [ci skip] - CURLOPT_SSL_CTX_FUNCTION.3: might cause unintended connection reuse [ci skip] Added a warning! Closes #2915 - curl: fix time-of-check, time-of-use race in dir creation Patch-by: Jay Satiro Detected by Coverity Fixes #2739 Closes #2912 - cmdline-opts/page-footer: fix edit mistake There was a missing newline. follow-up to a7ba60bb7250 - docs: clarify NO_PROXY env variable functionality Reported-by: Kirill Marchuk Fixes #2773 Closes #2911 Marcel Raad (24 Aug 2018) - lib1522: fix curl_easy_setopt argument type CURLOPT_POSTFIELDSIZE is a long option. - curl_threads: silence bad-function-cast warning As uintptr_t and HANDLE are always the same size, this warning is harmless. Just silence it using an intermediate uintptr_t variable. Closes https://github.com/curl/curl/pull/2908 Daniel Stenberg (24 Aug 2018) - README: add appveyor build badge [ci skip] Closes #2913 - [Ihor Karpenko brought this change] schannel: client certificate store opening fix 1) Using CERT_STORE_OPEN_EXISTING_FLAG ( or CERT_STORE_READONLY_FLAG ) while opening certificate store would be sufficient in this scenario and less-demanding in sense of required user credentials ( for example, IIS_IUSRS will get "Access Denied" 0x05 error for existing CertOpenStore call without any of flags mentioned above ), 2) as 'cert_store_name' is a DWORD, attempt to format its value like a string ( in "Failed to open cert store" error message ) will throw null pointer exception 3) adding GetLastError(), in my opinion, will make error message more useful. Bug: https://curl.haxx.se/mail/lib-2018-08/0198.html Closes #2909 - [Leonardo Taccari brought this change] gopher: Do not translate `?' to `%09' Since GOPHER support was added in curl `?' character was automatically translated to `%09' (`\t'). However, this behaviour does not seems documented in RFC 4266 and for search selectors it is documented to directly use `%09' in the URL. Apart that several gopher servers in the current gopherspace have CGI support where `?' is used as part of the selector and translating it to `%09' often leads to surprising results. Closes #2910 Marcel Raad (23 Aug 2018) - cookie tests: treat files as text Fixes test failures because of wrong line endings on Windows. Daniel Stenberg (23 Aug 2018) - libcurl-thread.3: expand somewhat on the NO_SIGNAL motivation Multi-threaded applictions basically MUST set CURLOPT_NO_SIGNAL to 1L to avoid the risk of getting a SIGPIPE. Either way, a multi-threaded application that uses libcurl/openssl needs to have a signhandler for or ignore SIGPIPE on its own. Based on discussions in #2800 Closes #2904 - RELEASE-NOTES: synced Marcel Raad (22 Aug 2018) - Tests: fixes for Windows - test 1268 requires unix sockets - test 2072 must be disabled also for MSYS/MinGW Daniel Stenberg (22 Aug 2018) - http2: abort the send_callback if not setup yet When Curl_http2_done() gets called before the http2 data is setup all the way, we cannot send anything and this should just return an error. Detected by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=10012 - http2: remove four unused nghttp2 callbacks Closes #2903 - x509asn1: use FALLTHROUGH ... as no other comments are accepted since 014ed7c22f51463 Marcel Raad (21 Aug 2018) - test1148: disable if decimal separator is not point Modifying the locale with environment variables doesn't work for native Windows applications. Just disable the test in this case if the decimal separator is something different than a point. Use a precheck with a small C program to achieve that. Closes https://github.com/curl/curl/pull/2786 - Enable more GCC warnings This enables the following additional warnings: -Wold-style-definition -Warray-bounds=2 instead of the default 1 -Wformat=2, but only for GCC 4.8+ as Wno-format-nonliteral is not respected for older versions -Wunused-const-variable, which enables level 2 instead of the default 1 -Warray-bounds also in debug mode through -ftree-vrp -Wnull-dereference also in debug mode through -fdelete-null-pointer-checks Closes https://github.com/curl/curl/pull/2747 - curl-compilers: enable -Wimplicit-fallthrough=4 for GCC This enables level 4 instead of the default level 3, which of the currently used comments only allows /* FALLTHROUGH */ to silence the warning. Closes https://github.com/curl/curl/pull/2747 - curl-compilers: enable -Wbad-function-cast on GCC This warning used to be enabled only for clang as it's a bit stricter on GCC. Silence the remaining occurrences and enable it on GCC too. Closes https://github.com/curl/curl/pull/2747 - configure: conditionally enable pedantic-errors Enable pedantic-errors for GCC >= 5 with --enable-werror. Before GCC 5, pedantic-errors was synonymous to -Werror=pedantic [0], which is still the case for clang [1]. With GCC 5, it became complementary [2]. Also fix a resulting error in acinclude.m4 as main's return type was missing, which is illegal in C99. [0] https://gcc.gnu.org/onlinedocs/gcc-4.9.0/gcc/Warning-Options.html [1] https://clang.llvm.org/docs/UsersManual.html#options-to-control-error-and-warning-messages [2] https://gcc.gnu.org/onlinedocs/gcc-5.1.0/gcc/Warning-Options.html Closes https://github.com/curl/curl/pull/2747 - Remove unused definitions Closes https://github.com/curl/curl/pull/2747 Daniel Stenberg (21 Aug 2018) - x509asn1: make several functions static and remove the private SIZE_T_MAX define and use the generic one. Closes #2902 - INTERNALS: require GnuTLS >= 2.11.3 Since the public pinning support was brought in e644866caf4. GnuTLS 2.11.3 was released in October 2010. Figured out in #2890 - http2: avoid set_stream_user_data() before stream is assigned ... before the stream is started, we have it set to -1. Fixes #2894 Closes #2898 - SSLCERTS: improve the openssl command line ... for extracting certs from a live HTTPS server to make a cacerts.pem from them. - docs/SECURITY-PROCESS: now we name the files after the CVE id - RELEASE-NOTES: synced - upload: change default UPLOAD_BUFSIZE to 64KB To make uploads significantly faster in some circumstances. Part 2 of #2888 Closes #2892 - upload: allocate upload buffer on-demand Saves 16KB on the easy handle for operations that don't need that buffer. Part 1 of #2888 - [Laurent Bonnans brought this change] vtls: reinstantiate engine on duplicated handles Handles created with curl_easy_duphandle do not use the SSL engine set up in the original handle. This fixes the issue by storing the engine name in the internal url state and setting the engine from its name inside curl_easy_duphandle. Reported-by: Anton Gerasimov Signed-of-by: Laurent Bonnans Fixes #2829 Closes #2833 - http2: make sure to send after RST_STREAM If this is the last stream on this connection, the RST_STREAM might not get pushed to the wire otherwise. Fixes #2882 Closes #2887 Researched-by: Michael Kaufmann - test1268: check the stderr output as "text" Follow-up to 099f37e9c57 Pointed-out-by: Marcel Raad - urldata: remove unused pipe_broke struct field This struct field is never set TRUE in any existing code path. This change removes the field completely. Closes #2871 - curl: warn the user if a given file name looks like an option ... simply because this is usually a sign of the user having omitted the file name and the next option is instead "eaten" by the parser as a file name. Add test1268 to verify Closes #2885 - http2: check nghttp2_session_set_stream_user_data return code Might help bug #2688 debugging Closes #2880 - travis: revert back to gcc-7 for coverage builds ... since the gcc-8 ones seem to fail frequently. Follow-up from b85207199544ca Closes #2886 - RELEASE-NOTES: synced ... and now listed in alphabetical order! - [Adrien brought this change] CMake: CMake config files are defining CURL_STATICLIB for static builds This change allows to use the CMake config files generated by Curl's CMake scripts for static builds of the library. The symbol CURL_STATIC lib must be defined to compile downstream, thus the config package is the perfect place to do so. Fixes #2817 Closes #2823 Reported-by: adnn on github Reviewed-by: Sergei Nikulov - TODO: host name sections in config files Kamil Dudka (14 Aug 2018) - ssh-libssh: fix infinite connect loop on invalid private key Added test 656 (based on test 604) to verify the fix. Bug: https://bugzilla.redhat.com/1595135 Closes #2879 - ssh-libssh: reduce excessive verbose output about pubkey auth The verbose message "Authentication using SSH public key file" was printed each time the ssh_userauth_publickey_auto() was called, which meant each time a packet was transferred over network because the API operates in non-blocking mode. This patch makes sure that the verbose message is printed just once (when the authentication state is entered by the SSH state machine). Daniel Stenberg (14 Aug 2018) - travis: disable h2 torture tests for "coverage" Since they started to fail almost 100% since a few days. Closes #2876 Marcel Raad (14 Aug 2018) - travis: update to GCC 8 Closes https://github.com/curl/curl/pull/2869 Daniel Stenberg (13 Aug 2018) - http: fix for tiny "HTTP/0.9" response Deal with tiny "HTTP/0.9" (header-less) responses by checking the status-line early, even before a full "HTTP/" is received to allow detecting 0.9 properly. Test 1266 and 1267 added to verify. Fixes #2420 Closes #2872 Kamil Dudka (13 Aug 2018) - docs: add disallow-username-in-url.d and haproxy-protocol.d on the list ... to make make the files appear in distribution tarballs Closes #2856 - .travis.yml: verify that man pages can be regenerated ... when curl is built from distribution tarball Closes #2856 Marcel Raad (11 Aug 2018) - Split non-portable part off test 1133 Split off testing file names with double quotes into new test 1158. Disable it for MSYS using a precheck as it doesn't support file names with double quotes (but Cygwin does, for example). Fixes https://github.com/curl/curl/issues/2796 Closes https://github.com/curl/curl/pull/2854 Jay Satiro (11 Aug 2018) - projects: Improve Windows perl detection in batch scripts - Determine if perl is in the user's PATH by running perl.exe. Prior to this change detection was done by checking the PATH for perl/ but that did not work in all cases (eg git install includes perl but not in perl/ path). Bug: https://github.com/curl/curl/pull/2865 Reported-by: Daniel Jeliński - [Michael Kaufmann brought this change] docs: Improve the manual pages of some callbacks - CURLOPT_HEADERFUNCTION: add newlines - CURLOPT_INTERLEAVEFUNCTION: fix the description of 'userdata' - CURLOPT_READDATA: mention crashes, same as in CURLOPT_WRITEDATA - CURLOPT_READFUNCTION: rename 'instream' to 'userdata' and explain how to set it Closes https://github.com/curl/curl/pull/2868 Marcel Raad (11 Aug 2018) - GCC: silence -Wcast-function-type uniformly Pointed-out-by: Rikard Falkeborn Closes https://github.com/curl/curl/pull/2860 - Silence GCC 8 cast-function-type warnings On Windows, casting between unrelated function types is fine and sometimes even necessary, so just use an intermediate cast to (void (*) (void)) to silence the warning as described in [0]. [0] https://gcc.gnu.org/onlinedocs/gcc-8.1.0/gcc/Warning-Options.html Closes https://github.com/curl/curl/pull/2860 Daniel Stenberg (11 Aug 2018) - CURLINFO_SIZE_UPLOAD: fix missing counter update Adds test 1522 for verification. Reported-by: cjmsoregan Fixes #2847 Closes #2864 - [Daniel Jelinski brought this change] Documentation: fix CURLOPT_SSH_COMPRESSION copy/paste bug Closes #2867 - RELEASE-NOTES: synced - openssl: fix potential NULL pointer deref in is_pkcs11_uri Follow-up to 298d2565e Coverity CID 1438387 Marcel Raad (10 Aug 2018) - travis: execute "set -eo pipefail" for coverage build Follow-up to 2de63ab179eb78630ee039ad94fb2a5423df522d and 0b87c963252d3504552ee0c8cf4402bd65a80af5. Closes https://github.com/curl/curl/pull/2862 Daniel Stenberg (10 Aug 2018) - lib1502: fix memory leak in torture test Reported-by: Marcel Raad Fixes #2861 Closes #2863 - docs: mention NULL is fine input to several functions Fixes #2837 Closes #2858 Reported-by: Markus Elfring - [Bas van Schaik brought this change] README.md: add LGTM.com code quality grade for C/C++ Closes #2857 - [Rikard Falkeborn brought this change] test1531: Add timeout Previously, the macro TEST_HANG_TIMEOUT was unused, but since there is looping going on, we might as well add timing instead of removing it. Closes #2853 - [Rikard Falkeborn brought this change] test1540: Remove unused macro TEST_HANG_TIMEOUT The macro has never been used, and it there is not really any place where it would make sense to add timing checks. Closes #2852 - [Rikard Falkeborn brought this change] asyn-thread: Remove unused macro The macro seems to never have been used. Closes #2852 - [Rikard Falkeborn brought this change] http_proxy: Remove unused macro SELECT_TIMEOUT Usage was removed in 5113ad0424044458ac497fa1458ebe0101356b22. Closes #2852 - [Rikard Falkeborn brought this change] formdata: Remove unused macro HTTPPOST_CONTENTTYPE_DEFAULT Its usage was removed in 84ad1fd3047815f9c6e78728bb351b828eac10b1. Closes #2852 - [Rikard Falkeborn brought this change] telnet: Remove unused macros TELOPTS and TELCMDS Their usage was removed in 3a145180cc754a5959ca971ef3cd243c5c83fc51. Closes #2852 - [Daniel Jelinski brought this change] openssl: fix debug messages Fixes #2806 Closes #2843 - configure: fix for -lpthread detection with OpenSSL and pkg-config ... by making sure it uses the -I provided by pkg-config! Reported-by: pszemus on github Fixes #2848 Closes #2850 - RELEASE-NOTES: synced - windows: follow up to the buffer-tuning 1ba1dba7 Somehow I didn't include the amended version of the previous fix. This is the missing piece. Pointed-out-by: Viktor Szakats - [Daniel Jelinski brought this change] windows: implement send buffer tuning Significantly enhances upload performance on modern Windows versions. Bug: https://curl.haxx.se/mail/lib-2018-07/0080.html Closes #2762 Fixes #2224 - [Anderson Toshiyuki Sasaki brought this change] ssl: set engine implicitly when a PKCS#11 URI is provided This allows the use of PKCS#11 URI for certificates and keys without setting the corresponding type as "ENG" and the engine as "pkcs11" explicitly. If a PKCS#11 URI is provided for certificate, key, proxy_certificate or proxy_key, the corresponding type is set as "ENG" if not provided and the engine is set to "pkcs11" if not provided. Acked-by: Nikos Mavrogiannopoulos Closes #2333 - [Ruslan Baratov brought this change] CMake: Respect BUILD_SHARED_LIBS Use standard CMake variable BUILD_SHARED_LIBS instead of introducing custom option CURL_STATICLIB. Use '-DBUILD_SHARED_LIBS=%SHARED%' in appveyor.yml. Reviewed-by: Sergei Nikulov Closes #2755 - [John Butterfield brought this change] cmake: bumped minimum version to 3.4 Closes #2753 - [John Butterfield brought this change] cmake: link curl to the OpenSSL targets instead of lib absolute paths Reviewed-by: Jakub Zakrzewski Reviewed-by: Sergei Nikulov Closes #2753 - travis: build darwinssl on macos 10.12 ... as building on 10.13.x before 10.13.4 leads to link errors. Assisted-by: Nick Zitzmann Fixes #2835 Closes #2845 - DEPRECATE: remove release date from 7.62.0 Since it will slip and the version is the important part there, not the date. - lib/Makefile: only do symbol hiding if told to This restores the ability to build a static lib with --disable-symbol-hiding to keep non-curl_ symbols. Researched-by: Dan Fandrich Reported-by: Ran Mozes Fixes #2830 Closes #2831 Marcel Raad (2 Aug 2018) - hostip: fix unused variable warning addresses is only used in an infof call, which is a macro expanding to nothing if CURL_DISABLE_VERBOSE_STRINGS is set. Daniel Stenberg (2 Aug 2018) - test1307: disabled Turns out that since we're using the native fnmatch function now when available, and they simply disagree on a huge number of test patterns that make it hard to test this function like this... Fixes #2825 - smb: don't mark it done in smb_do Follow-up to 09e401e01bf9. The SMB protocol handler needs to use its doing function too, which requires smb_do() to not mark itself as done... Closes #2822 - [Rikard Falkeborn brought this change] general: fix printf specifiers Closes #2818 - RELEASE-NOTES: synced - mailmap: Daniel Jelinski - [Harry Sintonen brought this change] HTTP: Don't attempt to needlessly decompress redirect body This change fixes a regression where redirect body would needlessly be decompressed even though it was to be ignored anyway. As it happens this causes secondary issues since there appears to be a bug in apache2 that it in certain conditions generates a corrupt zlib response. The regression was created by commit: dbcced8e32b50c068ac297106f0502ee200a1ebd Discovered-by: Harry Sintonen Closes #2798 - curl: use Content-Disposition before the "URL end" for -OJ Regression introduced in 7.61.0 Reported-by: Thomas Klausner Fixes #2783 Closes #2813 - [Daniel Jelinski brought this change] retry: return error if rewind was necessary but didn't happen Fixes #2801 Closes #2812 - http2: clear the drain counter in Curl_http2_done Reported-by: Andrei Virtosu Fixes #2800 Closes #2809 - smb: fix memory leak on early failure ... by making sure connection related data (->share) is stored in the connection and not in the easy handle. Detected by OSS-fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=9369 Fixes #2769 Closes #2810 - travis: run a 'make checksrc' too ... to make sure the examples are all checked. Closes #2811 Jay Satiro (29 Jul 2018) - examples/ephiperfifo: checksrc compliance - [Michael Kaufmann brought this change] sws: handle EINTR when calling select() Closes https://github.com/curl/curl/pull/2808 Daniel Stenberg (29 Jul 2018) - test1157: follow-up to 35ecffb9 Ignore the user-agent line. Pointed-out-by: Marcel Raad Michael Kaufmann (29 Jul 2018) - tests/http_pipe.py: Use /usr/bin/env to find python Daniel Stenberg (28 Jul 2018) - TODO: Support Authority Information Access certificate extension (AIA) Closes #2793 - conn_free: updated comment to clarify Let's call it disassociate instead of disconnect since the latter term is used so much for (TCP) connections already. - test1157: test -H from empty file Verifies bugfix #2797 - [Tobias Blomberg brought this change] curl: Fix segfault when -H @headerfile is empty The curl binary would crash if the -H command line option was given a filename to read using the @filename syntax but that file was empty. Closes #2797 - mime: check Curl_rand_hex's return code Bug: https://curl.haxx.se/mail/archive-2018-07/0015.html Reported-by: Jeffrey Walton Closes #2795 - [Josh Bialkowski brought this change] docs/examples: add hiperfifo example using linux epoll/timerfd Closes #2804 - [Darío Hereñú brought this change] docs/INSTALL.md: minor formatting fixes Closes #2794 - [Christopher Head brought this change] docs/CURLOPT_URL: fix indentation The statement, “The application does not have to keep the string around after setting this option,” appears to be indented under the RTMP paragraph. It actually applies to all protocols, not just RTMP. Eliminate the extra indentation. Closes #2788 - [Christopher Head brought this change] docs/CURLOPT_WRITEFUNCTION: size is always 1 For compatibility with `fwrite`, the `CURLOPT_WRITEFUNCTION` callback is passed two `size_t` parameters which, when multiplied, designate the number of bytes of data passed in. In practice, CURL always sets the first parameter (`size`) to 1. This practice is also enshrined in documentation and cannot be changed in future. The documentation states that the default callback is `fwrite`, which means `fwrite` must be a suitable function for this purpose. However, the documentation also states that the callback must return the number of *bytes* it successfully handled, whereas ISO C `fwrite` returns the number of items (each of size `size`) which it wrote. The only way these numbers can be equal is if `size` is 1. Since `size` is 1 and can never be changed in future anyway, document that fact explicitly and let users rely on it. Closes #2787 - [Carie Pointer brought this change] wolfSSL/CyaSSL: Fix memory leak in Curl_cyassl_random RNG structure must be freed by call to FreeRng after its use in Curl_cyassl_random. This call fixes Valgrind failures when running the test suite with wolfSSL. Closes #2784 - [Even Rouault brought this change] reuse_conn(): free old_conn->options This fixes a memory leak when CURLOPT_LOGIN_OPTIONS is used, together with connection reuse. I found this with oss-fuzz on GDAL and curl master: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=9582 I couldn't reproduce with the oss-fuzz original test case, but looking at curl source code pointed to this well reproducable leak. Closes #2790 Marcel Raad (25 Jul 2018) - [Daniel Jelinski brought this change] system_win32: fix version checking In the current version, VERSION_GREATER_THAN_EQUAL 6.3 will return false when run on windows 10.0. This patch addresses that error. Closes https://github.com/curl/curl/pull/2792 Daniel Stenberg (24 Jul 2018) - [Johannes Schindelin brought this change] auth: pick Bearer authentication whenever a token is available So far, the code tries to pick an authentication method only if user/password credentials are available, which is not the case for Bearer authentictation... Signed-off-by: Johannes Schindelin Closes #2754 - [Johannes Schindelin brought this change] auth: only ever pick CURLAUTH_BEARER if we *have* a Bearer token The Bearer authentication was added to cURL 7.61.0, but there is a problem: if CURLAUTH_ANY is selected, and the server supports multiple authentication methods including the Bearer method, we strongly prefer that latter method (only CURLAUTH_NEGOTIATE beats it), and if the Bearer authentication fails, we will never even try to attempt any other method. This is particularly unfortunate when we already know that we do not have any Bearer token to work with. Such a scenario happens e.g. when using Git to push to Visual Studio Team Services (which supports Basic and Bearer authentication among other methods) and specifying the Personal Access Token directly in the URL (this aproach is frequently taken by automated builds). Let's make sure that we have a Bearer token to work with before we select the Bearer authentication among the available authentication methods. Signed-off-by: Johannes Schindelin Closes #2754 Marcel Raad (22 Jul 2018) - test320: treat curl320.out file as binary Otherwise, LF line endings are converted to CRLF on Windows, but no conversion is done for the reply, so the test case fails. Closes https://github.com/curl/curl/pull/2776 Daniel Stenberg (22 Jul 2018) - vtls: set conn->data when closing TLS Follow-up to 1b76c38904f0. The VTLS backends that close down the TLS layer for a connection still needs a Curl_easy handle for the session_id cache etc. Fixes #2764 Closes #2771 Marcel Raad (21 Jul 2018) - tests: fixes for Windows line endlings Set mode="text" when line endings depend on the system representation. Closes https://github.com/curl/curl/pull/2772 - test214: disable MSYS2's POSIX path conversion for URL By default, the MSYS2 bash converts all backslashes to forward slashes in URLs. Disable this with MSYS2_ARG_CONV_EXCL for the test to pass. Ref https://github.com/msys2/msys2/wiki/Porting#filesystem-namespaces Daniel Stenberg (20 Jul 2018) - http2: several cleanups - separate easy handle from connections better - added asserts on a number of places - added sanity check of pipelines for debug builds Closes #2751 - smb_getsock: always wait for write socket too ... the protocol is doing read/write a lot, so it needs to write often even when downloading. A more proper fix could check for eactly when it wants to write and only ask for it then. Without this fix, an SMB download could easily get stuck when the event-driven API was used. Closes #2768 Marcel Raad (20 Jul 2018) - test1143: disable MSYS2's POSIX path conversion By default, the MSYS2 bash interprets http:/%HOSTIP:%HTTPPORT/want/1143 as a POSIX file list and converts it to a Windows file list. Disable this with MSYS2_ARG_CONV_EXCL for the test to pass. Ref https://github.com/msys2/msys2/wiki/Porting#filesystem-namespaces Closes https://github.com/curl/curl/pull/2765 Daniel Stenberg (18 Jul 2018) - RELEASE-NOTES: sync ... and work toward 7.61.1 - [Ruslan Baratov brought this change] CMake: Update scripts to use consistent style Closes #2727 Reviewed-by: Sergei Nikulov - header output: switch off all styles, not just unbold ... the "unbold" sequence doesn't work on the mac Terminal. Reported-by: Zero King Fixes #2736 Closes #2738 Nick Zitzmann (14 Jul 2018) - [Rodger Combs brought this change] darwinssl: add support for ALPN negotiation Marcel Raad (14 Jul 2018) - test1422: add required file feature curl configured with --enable-debug --disable-file currently complains on test1422: Info: Protocol "file" not supported or disabled in libcurl Make test1422 dependend on enabled FILE protocol to fix this. Fixes https://github.com/curl/curl/issues/2741 Closes https://github.com/curl/curl/pull/2742 Patrick Monnerat (12 Jul 2018) - content_encoding: accept up to 4 unknown trailer bytes after raw deflate data Some servers issue raw deflate data that may be followed by an undocumented trailer. This commit makes curl tolerate such a trailer of up to 4 bytes before considering the data is in error. Reported-by: clbr on github Fixes #2719 Daniel Stenberg (12 Jul 2018) - smb: fix memory-leak in URL parse error path Detected by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=9369 Closes #2740 Marcel Raad (12 Jul 2018) - schannel: enable CALG_TLS1PRF for w32api >= 5.1 The definition of CALG_TLS1PRF has been fixed in the 5.1 branch: https://osdn.net/projects/mingw/scm/git/mingw-org-wsl/commits/73aedcc0f2e6ba370de0d86ab878ad76a0dda7b5 Daniel Stenberg (12 Jul 2018) - docs/SECURITY-PROCESS: mention bounty, drop pre-notify + The hackerone bounty and its process - We don't and can't handle pre-notification - multi: always do the COMPLETED procedure/state It was previously erroneously skipped in some situations. libtest/libntlmconnect.c wrongly depended on wrong behavior (that it would get a zero timeout) when no handles are "running" in a multi handle. That behavior is no longer present with this fix. Now libcurl will always return a -1 timeout when all handles are completed. Closes #2733 - Curl_getoff_all_pipelines: improved for multiplexed On multiplexed connections, transfers can be removed from anywhere not just at the head as for pipelines. - ares: check for NULL in completed-callback - conn: remove the boolean 'inuse' field ... as the usage needs to be counted. - [Paul Howarth brought this change] openssl: assume engine support in 1.0.0 or later Commit 38203f1585da changed engine detection to be version-based, with a baseline of openssl 1.0.1. This does in fact break builds with openssl 1.0.0, which has engine support - the configure script detects that ENGINE_cleanup() is available - but doesn't get included to declare it. According to upstream documentation, engine support was added to mainstream openssl builds as of version 0.9.7: https://github.com/openssl/openssl/blob/master/README.ENGINE This commit drops the version test down to 1.0.0 as version 1.0.0d is the oldest version I have to test with. Closes #2732 Marcel Raad (11 Jul 2018) - schannel: fix MinGW compile break Original MinGW's w32api has a sytax error in its definition of CALG_TLS1PRF [0]. Don't use original MinGW w32api's CALG_TLS1PRF until this bug [1] is fixed. [0] https://osdn.net/projects/mingw/scm/git/mingw-org-wsl/blobs/d1d4a17e51a2b78e252ef0147d483267d56c90cc/w32api/include/wincrypt.h [1] https://osdn.net/projects/mingw/ticket/38391 Fixes https://github.com/curl/curl/pull/2721#issuecomment-403636043 Closes https://github.com/curl/curl/pull/2728 Daniel Stenberg (11 Jul 2018) - examples/crawler.c: move #ifdef to column 0 Apparently the C => HTML converter on the web site doesn't quite like it otherwise. Reported-by: Jeroen Ooms Version 7.61.0 (11 Jul 2018) Daniel Stenberg (11 Jul 2018) - release: 7.61.0 - TODO: Configurable loading of OpenSSL configuration file Closes #2724 - post303.d: clarify that this is an RFC violation ... and not the other way around, which this previously said. Reported-by: Vasiliy Faronov Fixes #2723 Closes #2726 - [Ruslan Baratov brought this change] CMake: remove redundant and old end-of-block syntax Reviewed-by: Jakub Zakrzewski Closes #2715 Jay Satiro (9 Jul 2018) - lib/curl_setup.h: remove unicode character Follow-up to 82ce416. Ref: https://github.com/curl/curl/commit/8272ec5#commitcomment-29646818 Daniel Stenberg (9 Jul 2018) - lib/curl_setup.h: remove unicode bom from 8272ec50f02 Marcel Raad (9 Jul 2018) - schannel: fix -Wsign-compare warning MinGW warns: /lib/vtls/schannel.c:219:64: warning: signed and unsigned type in conditional expression [-Wsign-compare] Fix this by casting the ptrdiff_t to size_t as we know it's positive. Closes https://github.com/curl/curl/pull/2721 - schannel: workaround for wrong function signature in w32api Original MinGW's w32api has CryptHashData's second parameter as BYTE * instead of const BYTE *. Closes https://github.com/curl/curl/pull/2721 - schannel: make more cipher options conditional They are not defined in the original MinGW's . Closes https://github.com/curl/curl/pull/2721 - curl_setup: include before Otherwise, only part of it gets pulled in through on original MinGW. Fixes https://github.com/curl/curl/issues/2361 Closes https://github.com/curl/curl/pull/2721 - examples: fix -Wformat warnings When size_t is not a typedef for unsigned long (as usually the case on Windows), GCC emits -Wformat warnings when using lu and lx format specifiers with size_t. Silence them with explicit casts to unsigned long. Closes https://github.com/curl/curl/pull/2721 Daniel Stenberg (9 Jul 2018) - smtp: use the upload buffer size for scratch buffer malloc ... not the read buffer size, as that can be set smaller and thus cause a buffer overflow! CVE-2018-0500 Reported-by: Peter Wu Bug: https://curl.haxx.se/docs/adv_2018-70a2.html - [Dave Reisner brought this change] scripts: include _curl as part of CLEANFILES Closes #2718 - [Nick Zitzmann brought this change] darwinssl: allow High Sierra users to build the code using GCC ...but GCC users lose out on TLS 1.3 support, since we can't weak-link enumeration constants. Fixes #2656 Closes #2703 - [Ruslan Baratov brought this change] CMake: Remove unused 'output_var' from 'collect_true' Variable 'output_var' is not used and can be removed. Function 'collect_true' renamed to 'count_true'. - [Ruslan Baratov brought this change] CMake: Remove unused functions Closes #2711 - KNOWN_BUGS: Stick to same family over SOCKS proxy - libssh: goto DISCONNECT state on error, not SSH_SESSION_FREE ... because otherwise not everything get closed down correctly. Fixes #2708 Closes #2712 - libssh: include line number in state change debug messages Closes #2713 - KNOWN_BUGS: Borland support is dropped, AIX problem is too old - [Jeroen Ooms brought this change] example/crawler.c: simple crawler based on libxml2 Closes #2706 - RELEASE-NOTES: synced - DEPRECATE: include year when specifying date - DEPRECATE: linkified - DEPRECATE: mention the PR that disabled axTLS - docs/DEPRECATE.md: spelling and minor formatting - DEPRECATE: new doc describing planned item removals Closes #2704 - [Gisle Vanem brought this change] telnet: fix clang warnings telnet.c(1401,28): warning: cast from function call of type 'int' to non-matching type 'HANDLE' (aka 'void *') [-Wbad-function-cast] Fixes #2696 Closes #2700 - docs: fix missed option name markups - [Gaurav Malhotra brought this change] openssl: Remove some dead code Closes #2698 - openssl: make the requested TLS version the *minimum* wanted The code treated the set version as the *exact* version to require in the TLS handshake, which is not what other TLS backends do and probably not what most people expect either. Reported-by: Andreas Olsson Assisted-by: Gaurav Malhotra Fixes #2691 Closes #2694 - RELEASE-NOTES: synced - openssl: allow TLS 1.3 by default Reported-by: Andreas Olsson Fixes #2692 Closes #2693 - [Adrian Peniak brought this change] CURLINFO_TLS_SSL_PTR.3: improve the example The previous example was a little bit confusing, because SSL* structure (or other "in use" SSL connection pointer) is not accessible after the transfer is completed, therefore working with the raw TLS library specific pointer needs to be done during transfer. Closes #2690 - travis: add a build using the synchronous name resolver ... since default uses the threaded one and we test the c-ares build already. Closes #2689 - configure: remove CURL_CHECK_NI_WITHSCOPEID too Since it isn't used either and requires the getnameinfo check Follow-up to 0aeca41702d2 - getnameinfo: not used Closes #2687 - easy_perform: use *multi_timeout() to get wait times ... and trim the threaded Curl_resolver_getsock() to return zero millisecond wait times during the first three milliseconds so that localhost or names in the OS resolver cache gets detected and used faster. Closes #2685 Max Dymond (27 Jun 2018) - configure: Add dependent libraries after crypto The linker is pretty dumb and processes things left to right, keeping a tally of symbols it hasn't resolved yet. So, we need -ldl to appear after -lcrypto otherwise the linker won't find the dl functions. Closes #2684 Daniel Stenberg (27 Jun 2018) - GOVERNANCE: linkify, changed some titles - GOVERNANCE: add maintainer details/duties - url: check Curl_conncache_add_conn return code ... it was previously unchecked in two places and thus errors could remain undetected and cause trouble. Closes #2681 - include/README: remove "hacking" advice, not the right place - RELEASE-NOTES: synced - CURLOPT_SSL_VERIFYPEER.3: fix syntax mistake Follow-up to b6a16afa0aa5 - netrc: use a larger buffer ... to work with longer passwords etc. Grow it from a 256 to a 4096 bytes buffer. Reported-by: Dario Nieuwenhuis Fixes #2676 Closes #2680 - [Patrick Schlangen brought this change] CURLOPT_SSL_VERIFYPEER.3: Add performance note Closes #2673 - [Javier Blazquez brought this change] multi: fix crash due to dangling entry in connect-pending list Fixes #2677 Closes #2679 - ConnectionExists: make sure conn->data is set when "taking" a connection Follow-up to 2c15693. Bug #2674 Closes #2675 - [Kevin R. Bulgrien brought this change] system.h: fix for gcc on 32 bit OpenServer Bug: https://curl.haxx.se/mail/lib-2018-06/0100.html - [Raphael Gozzo brought this change] cmake: allow multiple SSL backends This will make possible to select the SSL backend (using curl_global_sslset()) even when the libcurl is built using CMake Closes #2665 - url: fix dangling conn->data pointer By masking sure to use the *current* easy handle with extracted connections from the cache, and make sure to NULLify the ->data pointer when the connection is put into the cache to make this mistake easier to detect in the future. Reported-by: Will Dietz Fixes #2669 Closes #2672 - CURLOPT_INTERFACE.3: interface names not supported on Windows - travis: run more tests for coverage check ... run a few more tortured based and run all tests event-based. Closes #2664 - multi: fix memory leak when stopped during name resolve When the application just started the transfer and then stops it while the name resolve in the background thread hasn't completed, we need to wait for the resolve to complete and then cleanup data accordingly. Enabled test 1553 again and added test 1590 to also check when the host name resolves successfully. Detected by OSS-fuzz. Closes #1968 Viktor Szakats (15 Jun 2018) - maketgz: delete .bak files, fix indentation Ref: https://github.com/curl/curl/pull/2660 Closes https://github.com/curl/curl/pull/2662 Daniel Stenberg (15 Jun 2018) - runtests.pl: remove debug leftover from bb9a340c73f3 - curl-confopts.m4: fix typo from ed224f23d5beb Fixes my local configure to detect a custom installed c-ares without pkgconfig. - docs/RELEASE-PROCEDURE.md: renamed to use .md extension Closes #2663 - RELEASE-PROCEDURE: gpg sign the tags - RELEASE-NOTES: synced - CURLOPT_HTTPAUTH.3: CURLAUTH_BEARER was added in 7.61.0 - [Mamta Upadhyay brought this change] maketgz: fix sed issues on OSX maketgz creates release tarballs and removes the -DEV string in curl version (e.g. 7.58.0-DEV), else -DEV shows up on command line when curl is run. maketgz works fine on linux but fails on OSX. Problem is with the sed commands that use option -i without an extension. Maketgz expects GNU sed instead of BSD and this simply won't work on OSX. Adding a backup extension .bak after -i fixes this issue Running the script as if on OSX gives this error: sed: -e: No such file or directory Adding a .bak extension resolves it Closes #2660 - configure: enhance ability to detect/build with static openssl Fix the -ldl and -ldl + -lpthread checks for OpenSSL, necessary for building with static libs without pkg-config. Reported-by: Marcel Raad Fixes #2199 Closes #2659 - configure: use pkg-config for c-ares detection First check if there's c-ares information given as pkg-config info and use that as first preference. Reported-by: pszemus on github Fixes #2203 Closes #2658 - GOVERNANCE.md: explains how this project is run Closes #2657 - KNOWN_BUGS: NTLM doen't support password with § character Closes #2120 - KNOWN_BUGS: slow connect to localhost on Windows Closes #2281 - [Matteo Bignotti brought this change] mk-ca-bundle.pl: make -u delete certdata.txt if found not changed certdata.txt should be deleted also when the process is interrupted by "same certificate downloaded, exiting" The certdata.txt is currently kept on disk even if you give the -u option Closes #2655 - progress: remove a set of unused defines Reported-by: Peter Wu Closes #2654 - TODO: "Option to refuse usernames in URLs" done Implemented by Björn in 946ce5b61f - [Lyman Epp brought this change] Curl_init_do: handle NULL connection pointer passed in Closes #2653 - runtests: support variables in ... and make use of that to make 1455 work better without using a fixed local port number. Fixes #2649 Closes #2650 - Curl_debug: remove dead printhost code The struct field is never set (since 5e0d9aea3) so remove the use of it and remove the connectdata pointer from the prototype. Reported-by: Tejas Bug: https://curl.haxx.se/mail/lib-2018-06/0054.html Closes #2647 Viktor Szakats (12 Jun 2018) - schannel: avoid incompatible pointer warning with clang-6.0: ``` vtls/schannel_verify.c: In function 'add_certs_to_store': vtls/schannel_verify.c:212:30: warning: passing argument 11 of 'CryptQueryObject' from incompatible pointer type [-Wincompatible-pointer-types] &cert_context)) { ^ In file included from /usr/share/mingw-w64/include/schannel.h:10:0, from /usr/share/mingw-w64/include/schnlsp.h:9, from vtls/schannel.h:29, from vtls/schannel_verify.c:40: /usr/share/mingw-w64/include/wincrypt.h:4437:26: note: expected 'const void **' but argument is of type 'CERT_CONTEXT ** {aka struct _CERT_CONTEXT **}' WINIMPM WINBOOL WINAPI CryptQueryObject (DWORD dwObjectType, const void *pvObject, DWORD dwExpectedContentTypeFlags, DWORD dwExpectedFormatTypeFlags, DWORD dwFlags, ^~~~~~~~~~~~~~~~ ``` Ref: https://msdn.microsoft.com/library/windows/desktop/aa380264 Closes https://github.com/curl/curl/pull/2648 Daniel Stenberg (12 Jun 2018) - [Robert Prag brought this change] schannel: support selecting ciphers Given the contstraints of SChannel, I'm exposing these as the algorithms themselves instead; while replicating the ciphersuite as specified by OpenSSL would have been preferable, I found no way in the SChannel API to do so. To use this from the commandline, you need to pass the names of contants defining the desired algorithms. For example, curl --ciphers "CALG_SHA1:CALG_RSA_SIGN:CALG_RSA_KEYX:CALG_AES_128:CALG_DH_EPHEM" https://github.com The specific names come from wincrypt.h Closes #2630 - [Bernhard M. Wiedemann brought this change] test 46: make test pass after 2025 shifting the expiry date to 2037 for now to be before the possibly problematic year 2038 similar in spirit to commit e6293cf8764e9eecb Closes #2646 - [Marian Klymov brought this change] cppcheck: fix warnings - Get rid of variable that was generating false positive warning (unitialized) - Fix issues in tests - Reduce scope of several variables all over etc Closes #2631 - openssl: assume engine support in 1.0.1 or later Previously it was checked for in configure/cmake, but that would then leave other build systems built without engine support. While engine support probably existed prior to 1.0.1, I decided to play safe. If someone experience a problem with this, we can widen the version check. Fixes #2641 Closes #2644 - RELEASE-NOTES: synced - RELEASE-PROCEDURE: update the release calendar for 2019 - [Gisle Vanem brought this change] boringssl + schannel: undef X509_NAME in lib/schannel.h Fixes the build problem when both boringssl and schannel are enabled. Fixes #2634 Closes #2643 - [Vladimir Kotal brought this change] mk-ca-bundle.pl: leave certificate name untouched in decode() Closes #2640 - [Rikard Falkeborn brought this change] tests/libtests/Makefile.am: Add lib1521.c to CLEANFILES This removes the generated lib1521.c when running make clean. Closes #2633 - [Rikard Falkeborn brought this change] tests/libtest: Add lib1521 to nodist_SOURCES Since 467da3af0, lib1521.c is generated instead of checked in. According to the commit message, the intention was to remove it from the tarball as well. However, it is still present when running make dist. To remove it, add it to nodist_lib1521_SOURCES. This also means there is no need for the manually added dist-rule in the Makefile. Also update CMakelists.txt to handle the fact that we now may have nodist_SOURCES. - [Stephan Mühlstrasser brought this change] system.h: add support for IBM xlc C compiler Added a section to system.h guarded with __xlc__ for the IBM xml C compiler. Before this change the section titled 'generic "safe guess" on old 32 bit style' was used, which resulted in a wrong definition of CURL_TYPEOF_CURL_SOCKLEN_T, and for 64-bit also CURL_TYPEOF_CURL_OFF_T was wrong. Compilation warnings fixed with this change: CC libcurl_la-ftp.lo "ftp.c", line 290.55: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. "ftp.c", line 293.48: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. "ftp.c", line 1070.49: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. "ftp.c", line 1154.53: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. "ftp.c", line 1187.51: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. CC libcurl_la-connect.lo "connect.c", line 448.56: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. "connect.c", line 516.66: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. "connect.c", line 687.55: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. "connect.c", line 696.55: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. CC libcurl_la-tftp.lo "tftp.c", line 1115.33: 1506-280 (W) Function argument assignment between types "unsigned long* restrict" and "int*" is not allowed. Closes #2637 - cmdline-opts/cert-type.d: mention "p12" as a recognized type as well Viktor Szakats (3 Jun 2018) - spelling fixes Detected using the `codespell` tool (version 1.13.0). Also secure and fix an URL. Daniel Stenberg (2 Jun 2018) - axtls: follow-up spell fix of comment - axTLS: not considered fit for use URL: https://curl.haxx.se/mail/lib-2018-06/0000.html This is step one. It adds #error statements that require source edits to make curl build again if asked to use axTLS. At a later stage we might remove the axTLS specific code completely. Closes #2628 - build: remove the Borland specific makefiles According to the user survey 2018, not even one out of 670 users use them. Nobody on the mailing list spoke up for them either. Closes #2629 - curl_addrinfo: use same #ifdef conditions in source as header ... for curl_dofreeaddrinfo - multi: remove a DEBUGF() ... it might call infof() with a NULL first argument that isn't harmful but makes it not do anything. The infof() line is not very useful anymore, it has served it purpose. Good riddance! Fixes #2627 - [Alibek.Jorajev brought this change] CURLOPT_RESOLVE: always purge old entry first If there's an existing entry using the selected name. Closes #2622 - fnmatch: use the system one if available If configure detects fnmatch to be available, use that instead of our custom one for FTP wildcard pattern matching. For standard compliance, to reduce our footprint and to use already well tested and well exercised code. A POSIX fnmatch behaves slightly different than the internal function for a few test patterns currently and the macOS one yet slightly different. Test case 1307 is adjusted for these differences. Closes #2626 Patrick Monnerat (31 May 2018) - os400: add new option in ILE/RPG binding Follow-up to commit 946ce5b Daniel Stenberg (31 May 2018) - tests/libtest/.gitignore: follow-up fix to ignore lib5* too - KNOWN_BUGS: CURL_GLOBAL_SSL Closes #2276 - [Bernhard Walle brought this change] configure: check for declaration of getpwuid_r On our x86 Android toolchain, getpwuid_r is implemented but the header is missing: netrc.c:81:7: error: implicit declaration of function 'getpwuid_r' [-Werror=implicit-function-declaration] Unfortunately, the function is used in curl_ntlm_wb.c, too, so I moved the prototype to curl_setup.h. Signed-off-by: Bernhard Walle Closes #2609 - [Rikard Falkeborn brought this change] tests: update .gitignore for libtests Closes #2624 - [Rikard Falkeborn brought this change] strictness: correct {infof, failf} format specifiers Closes #2623 - [Björn Stenberg brought this change] option: disallow username in URL Adds CURLOPT_DISALLOW_USERNAME_IN_URL and --disallow-username-in-url. Makes libcurl reject URLs with a username in them. Closes #2340 - libcurl-security.3: improved layout for two rememdy lists - libcurl-security.3: refer to URL instead of in-source markdown file Viktor Szakats (30 May 2018) - curl.rc: embed manifest for correct Windows version detection * enable it in `src/Makefile.m32` * enable it in `winbuild/MakefileBuild.vc` if a custom manifest is _not_ enabled via the existing `EMBED_MANIFEST` option * enable it for all Windows CMake builds (also disable the built-in minimal manifest, added by CMake by default.) For other build systems, add the `-DCURL_EMBED_MANIFEST` option to the list of RC (Resource Compiler) flags to enable the manifest included in `src/curl.rc`. This may require to disable whatever automatic or other means in which way another manifest is added to `curl.exe`. Notice that Borland C doesn't support this method due to a long-pending resource compiler bug. Watcom C may also not handle it correctly when the `-zm` `wrc` option is used (this option may be unnecessary though) and regardless of options in certain earlier revisions of the 2.0 beta version. Closes https://github.com/curl/curl/pull/1221 Fixes https://github.com/curl/curl/issues/2591 Patrick Monnerat (30 May 2018) - os400: sync EBCDIC wrappers and ILE/RPG binding with latest options - os400: implement mime api EBCDIC wrappers Also sync ILE/RPG binding to define the new functions. Daniel Stenberg (29 May 2018) - setopt: add TLS 1.3 ciphersuites Adds CURLOPT_TLS13_CIPHERS and CURLOPT_PROXY_TLS13_CIPHERS. curl: added --tls13-ciphers and --proxy-tls13-ciphers Fixes #2435 Reported-by: zzq1015 on github Closes #2607 - configure: override AR_FLAGS to silence warning The automake default ar flags are 'cru', but the 'u' flag in there causes warnings on many modern Linux distros. Removing 'u' may have a minor performance impact on older distros but should not cause harm. Explained on the automake mailing list already back in April 2015: https://www.mail-archive.com/automake-patches@gnu.org/msg07705.html Reported-by: elephoenix on github Fixes #2617 Closes #2619 Sergei Nikulov (29 May 2018) - cmake: fixed comments in compile checks code Daniel Stenberg (29 May 2018) - INSTALL: LDFLAGS=-Wl,-R/usr/local/ssl/lib ... the older description doesn't work Reported-by: Peter Varga Fixes #2615 Closes #2616 - [Will Dietz brought this change] KNOWN_BUGS: restore text regarding #2101. This was added earlier but appears to have been removed accidentally. AFAICT this is very much still an issue. ----- I say "accidentally" because the text seems to have harmlessly snuck into [1] (which makes no mention of it). [1] was later reverted for unspecified reasons in [2], presumably because the mentioned issue was fixed or invalid. [1] de9fac00c40db321d44fa6fbab6eb62ec4c83998 [2] 16d1f369403cbb04bd7b085eabbeebf159473fc2 Closes #2618 - fnmatch: insist on escaped bracket to match A non-escaped bracket ([) is for a character group - as documented. It will *not* match an individual bracket anymore. Test case 1307 updated accordingly to match. Problem detected by OSS-Fuzz, although this fix is probably not a final fix for the notorious timeout issues. Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8525 Closes #2614 Patrick Monnerat (28 May 2018) - psl: use latest psl and refresh it periodically The latest psl is cached in the multi or share handle. It is refreshed before use after 72 hours. New share lock CURL_LOCK_DATA_PSL controls the psl cache sharing. If the latest psl is not available, the builtin psl is used. Reported-by: Yaakov Selkowitz Fixes #2553 Closes #2601 Daniel Stenberg (28 May 2018) - [Fabrice Fontaine brought this change] configure: fix ssh2 linking when built with a static mbedtls The ssh2 pkg-config file could contain the following lines when build with a static version of mbedtls: Libs: -L${libdir} -lssh2 /xxx/libmbedcrypto.a Libs.private: /xxx/libmbedcrypto.a This static mbedtls library must be used to correctly detect ssh2 support and this library must be copied in libcurl.pc otherwise compilation of any application (such as upmpdcli) with libcurl will fail when trying to found mbedtls functions included in libssh2. So, replace pkg-config --libs-only-l by pkg-config --libs. Fixes: - http://autobuild.buildroot.net/results/43e24b22a77f616d6198c10435dcc23cc3b9088a Signed-off-by: Fabrice Fontaine Closes #2613 - RELEASE-NOTES: synced - [Bernhard Walle brought this change] cmake: check for getpwuid_r The autotools-based build system does it, so we do it also in CMake. Bug: #2609 Signed-off-by: Bernhard Walle - cmdline-opts/gen.pl: warn if mutexes: or see-also: list non-existing options - [Frank Gevaerts brought this change] curl.1: Fix cmdline-opts reference errors. --data, --form, and --ntlm were declared to be mutually exclusive with non-existing options. --data and --form referred to --upload (which is short for --upload-file and therefore did work, so this one was merely a bit confusing), --ntlm referred to --negotiated instead of --negotiate. Closes #2612 - [Frank Gevaerts brought this change] docs: fix cmdline-opts metadata headers case consistency. Almost all headers start with an uppercase letter, but some didn't. - mailmap: Max Savenkov Sergei Nikulov (28 May 2018) - [Max Savenkov brought this change] Fix the test for fsetxattr and strerror_r tests in CMake to work without compiling Daniel Stenberg (27 May 2018) - mailmap: a Richard Alcock fixup - [Richard Alcock brought this change] schannel: add failf calls for client certificate failures Closes #2604 - [Richard Alcock brought this change] winbuild: In MakefileBuild.vc fix typo DISTDIR->DIRDIST Change requirement from $(DISTDIR) to $(DIRDIST) closes #2603 - [Richard Alcock brought this change] winbuild: only delete OUTFILE if it exists This removes the slightly annoying "Could not file LIBCURL_OBJS.inc" and "Could not find CURL_OBJS.inc.inc" message when building into a clean folder. closes #2602 - [Alejandro R. Sedeño brought this change] content_encoding: handle zlib versions too old for Z_BLOCK Fallback on Z_SYNC_FLUSH when Z_BLOCK is not available. Fixes #2606 Closes #2608 - multi: provide a socket to wait for in Curl_protocol_getsock ... even when there's no protocol specific handler setup. Bug: https://curl.haxx.se/mail/lib-2018-05/0062.html Reported-by: Sean Miller Closes #2600 - [Linus Lewandowski brought this change] httpauth: add support for Bearer tokens Closes #2102 - TODO: CURLINFO_PAUSE_STATE Closes #2588 Sergei Nikulov (24 May 2018) - cmake: set -d postfix for debug builds if not specified using -DCMAKE_DEBUG_POSTFIX explicitly fixes #2121, obsoletes #2384 Daniel Stenberg (23 May 2018) - configure: add basic test of --with-ssl prefix When given a prefix, the $PREFIX_OPENSSL/lib/openssl.pc or $PREFIX_OPENSSL/include/openssl/ssl.h files must be present or cause an error. Helps users detect when giving configure the wrong path. Reported-by: Oleg Pudeyev Assisted-by: Per Malmberg Fixes #2580 Patrick Monnerat (22 May 2018) - http resume: skip body if http code 416 (range error) is ignored. This avoids appending error data to already existing good data. Test 92 is updated to match this change. New test 1156 checks all combinations of --range/--resume, --fail, Content-Range header and http status code 200/416. Fixes #1163 Reported-By: Ithubg on github Closes #2578 Daniel Stenberg (22 May 2018) - tftp: make sure error is zero terminated before printfing it - configure: add missing m4/ax_compile_check_sizeof.m4 follow-up to mistake in 6876ccf90b4 Jay Satiro (22 May 2018) - [Johannes Schindelin brought this change] schannel: make CAinfo parsing resilient to CR/LF OpenSSL has supported --cacert for ages, always accepting LF-only line endings ("Unix line endings") as well as CR/LF line endings ("Windows line endings"). When we introduced support for --cacert also with Secure Channel (or in cURL speak: "WinSSL"), we did not take care to support CR/LF line endings, too, even if we are much more likely to receive input in that form when using Windows. Let's fix that. Happily, CryptQueryObject(), the function we use to parse the ca-bundle, accepts CR/LF input already, and the trailing LF before the END CERTIFICATE marker catches naturally any CR/LF line ending, too. So all we need to care about is the BEGIN CERTIFICATE marker. We do not actually need to verify here that the line ending is CR/LF. Just checking for a CR or an LF is really plenty enough. Signed-off-by: Johannes Schindelin Closes https://github.com/curl/curl/pull/2592 Daniel Stenberg (22 May 2018) - CURLOPT_ACCEPT_ENCODING.3: add brotli and clarify a bit - RELEASE-NOTES: synced - KNOWN_BUGS: mention the -O with %-encoded file names Closes #2573 - checksrc: make sure sizeof() is used *with* parentheses ... and unify the source code to adhere. Closes #2563 - curl: added --styled-output It is enabled by default, so --no-styled-output will switch off the detection/use of bold headers. Closes #2538 - curl: show headers in bold The feature is only enabled if the output is believed to be a tty. -J: There's some minor differences and improvements in -J handling, as now J should work with -i and it actually creates a file first using the initial name and then *renames* that to the one found in Content-Disposition (if any). -i: only shows headers for HTTP transfers now (as documented). Previously it would also show for pieces of the transfer that were HTTP (for example when doing FTP over a HTTP proxy). -i: now shows trailers as well. Previously they were not shown at all. --libcurl: the CURLOPT_HEADER is no longer set, as the header output is now done in the header callback. - configure: compile-time SIZEOF checks ... instead of exeucting code to get the size. Removes the use of LD_LIBRARY_PATH for this. Fixes #2586 Closes #2589 Reported-by: Bernhard Walle - configure: replace AC_TRY_RUN with CURL_RUN_IFELSE ... and export LD_LIBRARY_PATH properly. This is a follow-up from 2d4c215. Fixes #2586 Reported-by: Bernhard Walle - docs: clarify CURLOPT_HTTPGET somewhat Reported-by: bsammon on github Fixes #2590 - curl_fnmatch: only allow two asterisks for matching The previous limit of 5 can still end up in situation that takes a very long time and consumes a lot of CPU. If there is still a rare use case for this, a user can provide their own fnmatch callback for a version that allows a larger set of wildcards. This commit was triggered by yet another OSS-Fuzz timeout due to this. Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8369 Closes #2587 - checksrc: fix too long line follow-up to e05ad5d - [Aleks brought this change] docs: mention HAproxy protocol "version 1" ...as there's also a version 2. Closes #2579 - examples/progressfunc: make it build on older libcurls This example was changed in ce2140a8c1 to use the new microsecond based getinfo option. This change makes it conditionally keep using the older option so that the example still builds with older libcurl versions. Closes #2584 - stub_gssapi: fix numerous 'unused parameter' warnings follow-up to d9e92fd9fd1d - [Philip Prindeville brought this change] getinfo: add microsecond precise timers for various intervals Provide a set of new timers that return the time intervals using integer number of microseconds instead of floats. The new info names are as following: CURLINFO_APPCONNECT_TIME_T CURLINFO_CONNECT_TIME_T CURLINFO_NAMELOOKUP_TIME_T CURLINFO_PRETRANSFER_TIME_T CURLINFO_REDIRECT_TIME_T CURLINFO_STARTTRANSFER_TIME_T CURLINFO_TOTAL_TIME_T Closes #2495 - openssl: acknowledge --tls-max for default version too ... previously it only used the max setting if a TLS version was also explicitly asked for. Reported-by: byte_bucket Fixes #2571 Closes #2572 - bump: start working on the pending 7.61.0 - [Dagobert Michelsen brought this change] tests/libtest/Makefile: Do not unconditionally add gcc-specific flags The warning flag leads e.g. Sun Studio compiler to bail out. Closes #2576 - schannel_verify: fix build for non-schannel Jay Satiro (16 May 2018) - rand: fix typo - schannel: disable manual verify if APIs not available .. because original MinGW and old compilers do not have the Windows API definitions needed to support manual verification. - [Archangel_SDY brought this change] schannel: disable client cert option if APIs not available Original MinGW targets Windows 2000 by default, which lacks some APIs and definitions for this feature. Disable it if these APIs are not available. Closes https://github.com/curl/curl/pull/2522 Version 7.60.0 (15 May 2018) Daniel Stenberg (15 May 2018) - RELEASE-NOTES: 7.60.0 release - THANKS: added people from the curl 7.60.0 release - docs/libcurl/index.html: removed The HTML files are long gone from the dist, now remove the last HTML file pointing to those missing files. d - [steini2000 brought this change] http2: remove unused variable Closes #2570 - [steini2000 brought this change] http2: use easy handle of stream for logging - gcc: disable picky gcc-8 function pointer warnings in two places Reported-by: Rikard Falkeborn Bug: #2560 Closes #2569 - http2: use the correct function pointer typedef Fixes gcc-8 picky compiler warnings Reported-by: Rikard Falkeborn Bug: #2560 Closes #2568 - CODE_STYLE: mention return w/o parens, but sizeof with ... and remove the github markdown syntax so that it renders better on the web site. Also, don't use back-ticks inlined to allow the CSS to highlight source code better. - [Rikard Falkeborn brought this change] examples: Fix format specifiers Closes #2561 - [Rikard Falkeborn brought this change] tool: Fix format specifiers - [Rikard Falkeborn brought this change] ntlm: Fix format specifiers - [Rikard Falkeborn brought this change] tests: Fix format specifiers - [Rikard Falkeborn brought this change] lib: Fix format specifiers - contributors.sh: use "on github", not at - http2: getsock fix for uploads When there's an upload in progress, make sure to wait for the socket to become writable. Detected-by: steini2000 on github Bug: #2520 Closes #2567 - pingpong: fix response cache memcpy overflow Response data for a handle with a large buffer might be cached and then used with the "closure" handle when it has a smaller buffer and then the larger cache will be copied and overflow the new smaller heap based buffer. Reported-by: Dario Weisser CVE: CVE-2018-1000300 Bug: https://curl.haxx.se/docs/adv_2018-82c2.html - http: restore buffer pointer when bad response-line is parsed ... leaving the k->str could lead to buffer over-reads later on. CVE: CVE-2018-1000301 Assisted-by: Max Dymond Detected by OSS-Fuzz. Bug: https://curl.haxx.se/docs/adv_2018-b138.html Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=7105 Patrick Monnerat (13 May 2018) - cookies: do not take cookie name as a parameter RFC 6265 section 4.2.1 does not set restrictions on cookie names. This is a follow-up to commit 7f7fcd0. Also explicitly check proper syntax of cookie name/value pair. New test 1155 checks that cookie names are not reserved words. Reported-By: anshnd at github Fixes #2564 Closes #2566 Daniel Stenberg (12 May 2018) - smb: reject negative file sizes Assisted-by: Max Dymond Detected by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8245 - setup_transfer: deal with both sockets being -1 Detected by Coverity; CID 1435559. Follow-up to f8d608f38d00. It would index the array with -1 if neither index was a socket. - travis: add build using NSS Closes #2558 - [Sunny Purushe brought this change] openssl: change FILE ops to BIO ops To make builds with VS2015 work. Recent changes in VS2015 _IOB_ENTRIES handling is causing problems. This fix changes the OpenSSL backend code to use BIO functions instead of FILE I/O functions to circumvent those problems. Closes #2512 - travis: add a build using WolfSSL Assisted-by: Dan Fandrich Closes #2528 - RELEASE-NOTES: typo - RELEASE-NOTES: synced - [Daniel Gustafsson brought this change] URLs: fix one more http url This file wasn't included in commit 4af40b3646d3b09 which updated all haxx.se http urls to https. The file was committed prior to that update, but may have been merged after it and hence didn't get updated. Closes #2550 - github/lock: auto-lock closed issues after 90 days of inactivity - vtls: fix missing commas follow-up to e66cca046cef - vtls: use unified "supports" bitfield member in backends ... instead of previous separate struct fields, to make it easier to extend and change individual backends without having to modify them all. closes #2547 - transfer: don't unset writesockfd on setup of multiplexed conns Curl_setup_transfer() can be called to setup a new individual transfer over a multiplexed connection so it shouldn't unset writesockfd. Bug: #2520 Closes #2549 - [Frank Gevaerts brought this change] configure: put CURLDEBUG and DEBUGBUILD in lib/curl_config.h They are removed from the compiler flags. This ensures that make dependency tracking will force a rebuild whenever configure --enable-debug or --enable-curldebug changes. Closes #2548 - http: don't set the "rewind" flag when not uploading anything It triggers an assert. Detected by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8144 Closes #2546 - travis: add an mbedtls build Closes #2531 - configure: only check for CA bundle for file-using SSL backends When only building with SSL backends that don't use the CA bundle file (by default), skip the check. Fixes #2543 Fixes #2180 Closes #2545 - ssh-libssh.c: fix left shift compiler warning ssh-libssh.c:2429:21: warning: result of '1 << 31' requires 33 bits to represent, but 'int' only has 32 bits [-Wshift-overflow=] 'len' will never be that big anyway so I converted the run-time check to a regular assert. - [Stephan Mühlstrasser brought this change] URL: fix ASCII dependency in strcpy_url and strlen_url Commit 3c630f9b0af097663a64e5c875c580aa9808a92b partially reverted the changes from commit dd7521bcc1b7a6fcb53c31f9bd1192fcc884bd56 because of the problem that strcpy_url() was modified unilaterally without also modifying strlen_url(). As a consequence strcpy_url() was again depending on ASCII encoding. This change fixes strlen_url() and strcpy_url() in parallel to use a common host-encoding independent criterion for deciding whether an URL character must be %-escaped. Closes #2535 - [Denis Ollier brought this change] docs: remove extraneous commas in man pages Closes #2544 - RELEASE-NOTES: synced - Revert "TODO: remove configure --disable-pthreads" This reverts commit d5d683a97f9765bddfd964fe32e137aa6e703ed3. --disable-pthreads can be used to disable pthreads and get the threaded resolver to use the windows threading when building with mingw. - vtls: don't define MD5_DIGEST_LENGTH for wolfssl ... as it defines it (too) - TODO: remove configure --disable-pthreads Jay Satiro (2 May 2018) - [David Garske brought this change] wolfssl: Fix non-blocking connect Closes https://github.com/curl/curl/pull/2542 Daniel Stenberg (30 Apr 2018) - CURLOPT_URL.3: add ENCODING section [ci skip] Feedback-by: Michael Kilburn - KNOWN_BUGS: Client cert with Issuer DN differs between backends Closes #1411 - KNOWN_BUGS: Passive transfer tries only one IP address Closes #1508 - KNOWN_BUGS: --upload-file . hang if delay in STDIN Closes #2051 - KNOWN_BUGS: Connection information when using TCP Fast Open Closes #1332 - travis: enable libssh2 on both macos and Linux It seems to not be detected by default anymore (which is a bug I believe) Closes #2541 - TODO: Support the clienthello extension Closes #2299 - TODO: CLOEXEC Closes #2252 - tests: provide 'manual' as a feature to optionally require ... and make test 1026 rely on that feature so that --disable-manual builds don't cause test failures. Reported-by: Max Dymond and Anders Roxell Fixes #2533 Closes #2540 - CURLINFO_PROTOCOL.3: mention the existing defined names Jay Satiro (27 Apr 2018) - [Daniel Gustafsson brought this change] cookies: remove unused macro Commit 2bc230de63 made the macro MAX_COOKIE_LINE_TXT become unused, so remove as it's not part of the published API. Closes https://github.com/curl/curl/pull/2537 Daniel Stenberg (27 Apr 2018) - [Daniel Gustafsson brought this change] checksrc: force indentation of lines after an else This extends the INDENTATION case to also handle 'else' statements and require proper indentation on the following line. Also fixes the offending cases found in the codebase. Closes #2532 - http2: fix null pointer dereference in http2_connisdead This function can get called on a connection that isn't setup enough to have the 'recv_underlying' function pointer initialized so it would try to call the NULL pointer. Reported-by: Dario Weisser Follow-up to db1b2c7fe9b093f8 (never shipped in a release) Closes #2536 - http2: get rid of another strstr() Follow-up to 1514c44655e12e: replace another strstr() call done on a buffer that might not be zero terminated - with a memchr() call, even if we know the substring will be found. Assisted-by: Max Dymond Detected by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8021 Closes #2534 - cyassl: adapt to libraries without TLS 1.0 support built-in WolfSSL doesn't enable it by default anymore - configure: provide --with-wolfssl as an alias for --with-cyassl - RELEASE-NOTES: synced - [Daniel Gustafsson brought this change] os400.c: fix ASSIGNWITHINCONDITION checksrc warnings All occurrences of assignment within conditional expression in os400sys.c rewritten into two steps: first assignment and then the check on the success of the assignment. Also adjust related incorrect brace positions to match project indentation style. This was spurred by seeing "if((inp = input_token))", but while in there all warnings were fixed. There should be no functional change from these changes. Closes #2525 - [Daniel Gustafsson brought this change] cookies: ensure that we have cookies before writing jar The jar should be written iff there are cookies, so ensure that we still have cookies after expiration to avoid creating an empty file. Closes #2529 - strcpy_url: only %-encode values >= 0x80 OSS-Fuzz detected https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=8000 Broke in dd7521bcc1b7 - mime: avoid NULL pointer dereference risk Coverity detected, CID 1435120 Closes #2527 - [Stephan Mühlstrasser brought this change] ctype: restore character classification for non-ASCII platforms With commit 4272a0b0fc49a1ac0ceab5c4a365c9f6ab8bf8e2 curl-speficic character classification macros and functions were introduced in curl_ctype.[ch] to avoid dependencies on the locale. This broke curl on non-ASCII, e.g. EBCDIC platforms. This change restores the previous set of character classification macros when CURL_DOES_CONVERSIONS is defined. Closes #2494 - ftplistparser: keep state between invokes Fixes FTP wildcard parsing when done over a number of read buffers. Regression from f786d1f14 Reported-by: wncboy on github Fixes #2445 Closes #2526 - examples/http2-upload: expand buffer to avoid silly warning http2-upload.c:135:44: error: ‘%02d’ directive output may be truncated writing between 2 and 11 bytes into a region of size between 8 and 17 - examples/sftpuploadresume: typecast fseek argument to long /docs/examples/sftpuploadresume.c:102:12: warning: conversion to 'long int' from 'curl_off_t {aka long long int}' may alter its value - Revert "ftplistparser: keep state between invokes" This reverts commit abbc8457d85aca74b7cfda1d394b0844932b2934. Caused fuzzer problems on travis not seen when this was a PR! - Curl_memchr: zero length input can't match Avoids undefined behavior. Reported-by: Geeknik Labs - ftplistparser: keep state between invokes Fixes FTP wildcard parsing when doing over a number of read buffers. Regression from f786d1f14 Reported-by: wncboy on github Fixes #2445 Closes #2519 - ftplistparser: renamed some members and variables ... to make them better spell out what they're for. - RELEASE-NOTES: synced - [Christian Schmitz brought this change] curl_global_sslset: always provide available backends Closes #2499 - http2: convert an assert to run-time check Fuzzing has proven we can reach code in on_frame_recv with status_code not having been set, so let's detect that in run-time (instead of with assert) and error error accordingly. (This should no longer happen with the latest nghttp2) Detected by OSS-Fuzz Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=7903 Closes #2514 - curl.1: clarify that options and URLs can be mixed Fixes #2515 Closes #2517 Jay Satiro (23 Apr 2018) - [Archangel_SDY brought this change] CURLOPT_SSLCERT.3: improve WinSSL-specific usage info Ref: https://github.com/curl/curl/pull/2376#issuecomment-381858780 Closes https://github.com/curl/curl/pull/2504 - [Archangel_SDY brought this change] schannel: fix build error on targets <= XP - Use CRYPT_STRING_HEX instead of CRYPT_STRING_HEXRAW since XP doesn't support the latter. Ref: https://github.com/curl/curl/pull/2376#issuecomment-382153668 Closes https://github.com/curl/curl/pull/2504 Daniel Stenberg (23 Apr 2018) - Revert "ftplistparser: keep state between invokes" This reverts commit 8fb78f9ddc6d858d630600059b8ad84a80892fd9. Unfortunately this fix introduces memory leaks I've not been able to fix in several days. Reverting this for now to get the leaks fixed. Jay Satiro (21 Apr 2018) - tool_help: clarify --max-time unit of time is seconds Before: -m, --max-time