uclient-http: Implement error handling for header-sending

Add some error handling for errors during assembly of headers

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>

uclient: Handle memory allocation failure for url

Check ptr to url after allocation to prevent null pointer
dereference

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>

uclient-http: Handle memory allocation failure

Add null pointer check to allocation of uclient_http struct

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>

uclient-http: auth digest: Handle multiple possible memory allocation failures

Add null pointer checks to allocation of buffers for authentication parameters

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>

uclient-http: basic auth: Handle memory allocation failure

Allocation of the base64 buffer might fail, resulting in a null
ptr being passed to base64_encode as a target buffer

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>

uclient-fetch: Check for nullpointer returned by uclient_get_url_filename

uclient_get_url_filename may return a null ptr resulting in a null
ptr dereference by fprintf and open

Signed-off-by: Tobias Schramm <tobleminer@gmail.com>

uclient-fetch: use package name pattern in message for missing SSL library

When faced with this error message, I did not directly know what
package(s) to install. I first grepped the package list for packages
named libustream-ssl... found nothing, and finally realized, that here
only the library name was mentioned.

But when we mention the certificate packages, why not also directly
mention the SSL/TLS packages with a pattern. That will save one step
next time when stumbling here - at least for me ;-)

Signed-off-by: Michael Heimpold <mhei@heimpold.de>

uclient-fetch: fix overloading of output_file variable

When uclient-fetch is called with multiple URL's, it derives the
first filename from the URL. It then sets the global output_file
variable, causing a use-after-free bug on an attempt to use it as output
file for the next file.

Fix this by avoiding the overwrite entirely by only setting a local
variable

Reported-by: Philip Prindeville <philipp@redfish-solutions.com>
Signed-off-by: Felix Fietkau <nbd@nbd.name>

uclient-http: fix Host: header for literal IPv6 addresses

For literal IPv6 addresses, the host header must have the form

  Host: [...]

including the square brackets, as it may also contain a port. Some
webservers ignore the missing brackets, while others will return error 400.

IPv6 addresses are determined by searching for ':' characters in the host,
as neither IPv4 addresses nor DNS names can contain colons. An alternative
would be to add a flag to the uclient_url struct; but as this struct is
exposed in public headers, such a change could be considered an ABI change,
making a backport difficult.

Signed-off-by: Matthias Schiffer <mschiffer@universe-factory.net>
Acked-by: Jo-Philipp Wich <jo@mein.io>

uclient-fetch: read_data_cb: fix a potential buffer overflow

When a backend->read() is not implement or returns -1,
This -1 is interpreted as unsigned int resulting
in an access far behind the buffer.

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>

remove obsolete mac os x /opt/local include/library search path

Where necessary, this can easily be provided through CFLAGS/LDFLAGS

Signed-off-by: Felix Fietkau <nbd@nbd.name>

Fix unused results warnings

Fixes:

uclient-http.c:385:8: error: ignoring return value of 'fread', declared with attribute warn_unused_result [-Werror=unused-result]
   fread(&val, sizeof(val), 1, f);
        ^

uclient-fetch.c: In function 'main':
uclient-fetch.c:664:12: error: ignoring return value of 'asprintf', declared with attribute warn_unused_result [-Werror=unused-result]
    asprintf(&auth_str, "%s:%s", username, password);
            ^
uclient-fetch.c: In function 'read_data_cb':
uclient-fetch.c:269:9: error: ignoring return value of 'write', declared with attribute warn_unused_result [-Werror=unused-result]
    write(output_fd, buf, len);

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>

uclient-http: send correct "Host:" header if port is set

When connecting to a website with a special port, uclient-fetch connects
to the correct port, but is sending an incorrect Host: header without
the corresponding port.

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>

Fix printf format in previous commit.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>

Make missing SSL support message more explicit.

Signed-off-by: Jo-Philipp Wich <jo@mein.io>

http: add support for relative location on redirect

Signed-off-by: Felix Fietkau <nbd@nbd.name>

uclient-fetch: add support for --quiet

Signed-off-by: Felix Fietkau <nbd@nbd.name>

cmake: Find libubox/ustream-ssl.h header file

Add a CMake FIND_PATH and INCLUDE_DIRECTORIES searching for
libubox/ustream-ssl.h.  Some external toolchains which do not include
standard locations would fail to find the header otherwise.

Signed-off-by: Florian Fainelli <f.fainelli@gmail.com>

http: allow sending message body for DELETE request

Sending entity within DELETE is not forbidden by RFC 7231, see section
4.3.5. DELETE:
> A payload within a DELETE request message has no defined semantics;
> sending a payload body on a DELETE request might cause some existing
> implementations to reject the request.

Signed-off-by: Rafał Miłecki <zajec5@gmail.com>

uclient-fetch: fix inconsistencies in help text (thx, Hannu Nyman)

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: use usock_inet_timeout()

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: add support for specifying ipv4/ipv6 preference

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: truncate output files (unless resuming)

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: assume data EOF if the connection terminates

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: fix progress bar initialization with -O-

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: fix processing server http data separated by \n instead of \r\n

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: add proper error handling to uclient_http_redirect()

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: allow overwriting files if -O was used

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: fix short command line option for timeout

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: fix connection close handling on auth

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add missing newline after final progress bar update

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

core: fix unused but set variable

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

core: make __uclient_get_url static

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add progress bar support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add support for setting output directory

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-http: use blocking connect, switch to non-blocking later. fixes ipv6 -> ipv4 fallback

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: detect connect failure

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: ignore SIGPIPE

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add proxy support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: add proxy support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

core: add proxy support

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

core: fix memory leak if url change fails

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-http: make username/password variables local

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add support for resuming downloads

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: clean up output file selection

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: allow the header_done callback to reset the client state

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add support for --timeout

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add support for --spider

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add support for --post-data

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: only add default CA entries if no certificate was supplied and verify was requested

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add support for setting the user agent

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add support for fetching multiple urls

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: add user/password to command line help

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: load CA certificates

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: get remote address from usock instead of using getpeername

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: set server_name of the ssl context to support SNI

Signed-off-by: Alexander Couzens <lynxis@fe80.eu>

uclient-http: end custom header lines with CRLF

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-http: allow .header_done() callback to reset the connection

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

allow sending requests with DELETE method

Signed-off-by: Rafał Miłecki <zajec5@gmail.com>

use const for char buffer in uclient_write

We are not supposed to modify it and ustream accepts const already.

Signed-off-by: Rafał Miłecki <zajec5@gmail.com>

support for connection timeout

Signed-off-by: Rafał Miłecki <zajec5@gmail.com>

http: fix processing of digest auth parameters

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: terminate old connection in uclient_http_connect after incomplete request processing

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: do not call uclient_http_disconnect from uclient_notify_eof directly

Depending on the context, it can lead to crashes

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-http: only invoke data_sent callback if it is provided

Signed-off-by: Jo-Philipp Wich <jow@openwrt.org>

http: ignore empty lines before HTTP response

Fixes HTTP keepalive issues with some servers

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: fix segfault after destination was not reached

Signed-off-by: Luka Perkov <luka@openwrt.org>

http: implement data_sent write callback

Signed-off-by: Luka Perkov <luka@openwrt.org>

http: rework authentication handling - only retry internally for GET requests

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

uclient-fetch: define _GNU_SOURCE (used for asprintf)

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: print the URL while connecting

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: add support for --user and --password

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: always use O_EXCL when creating output files, use unlink if -O is specified

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: print download status

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: indicate an error if the connection was terminated prematurely

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

add uclient_get_url_filename helper function

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: print error messages on http error codes

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

fetch: remove http header output

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

rename uclient-example to uclient-fetch

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: rename callbacks

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

add support for PUT requests

Signed-off-by: Luka Perkov <luka@openwrt.org>

example: document -q

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: only close output fd after request completion if -O was not used

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: handle uclient connection errors

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: indicate http error presence with return code 8

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: close output file

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: only write output data on http 200/204

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: document -O

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: always exit on output file errors

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

add uclient_disconnect() to force a disconnect without further callbacks

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: write output data to a file

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: create only one request

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: update usage info

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: add quiet flag similar to wget

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

expose struct uclient_url

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: change -c to --ca-certificate to match wget

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: change -C to --no-check-certificate to match wget

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

example: make ustream-ssl optional (load via dlopen)

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: make ustream_ssl optional, only use provided ssl context

Signed-off-by: Felix Fietkau <nbd@openwrt.org>

http: get address at initial connect time

Signed-off-by: Felix Fietkau <nbd@openwrt.org>