Change the storage of easy handle option strings from an array sized for
all possible options to a hash set to reduce memory footprint.
Give the hash set initially room for 4 strings, with first allocation
happening when it goes beyond that. Measurements without test suite
and a forced fail on growing the set gives:
Size Result
2 1261 tests out of 1951 reported OK: 64%
4 1792 tests out of 1951 reported OK: 91%
8 1944 tests out of 1951 reported OK: 99%
16 1949 tests out of 1951 reported OK: 99%
32 single fail of 3211, unit test for u8_strset
Add u8_strset that keeps the tuples (uint8_t id, char *str)
and allows set/unset by `id`. Add that as data->set.strings.
Define MACROS
* CURL_EASY_STR(data, id) for access
* CURL_EASY_STR_SET(data, id, s) for setting, making a copy
* CURL_EASY_STR_SETN(data, id, s) for setting, no copy
* CURL_EASY_STR_CLEAR(data, id) for unsetting
* CURL_EASY_STR_CLEAR0(data, id) for unsetting and zero-ing value
Add `data->set.str_copypostfields` to handle former `STRING_COPYPOSTFIELDS`
string that was not always a string and could carry NUL bytes.
Add unit tests to test3211.
Closes#22628
struct SingleRequest tweaks
- 'upgr101' from enum upgrade101 => 'unsigned char', saves three bytes
- made some 'unsigned char' => uint8_t
- moved 'io_flags' to fill a hole on 64-bit arch
struct UserDefined tweaks
- Sort the fields by size. Larger to smaller. Helps avoding holes.
- httpsig_algorithm moved
- FTP uint8_t fields moved
- new_file_perms moved
- rtspreq moved
- sort fields on size
- urldata: drop 'struct Curl_data_priority'
It only had a single struct member 'weight'. Use that directly instead
to save indirections and struct alignments. Move field for size order.
- urldata: move the RTSP fields in the UrlState struct to be ordered by
size
struct PureInfo tweaks
- make 'pxcode' a uint8_t
- move 'conn_protocol' to fill a hole
- sort 'struct PureInfo' fields by size
Closes#22585
- convert the method switch() to a simple table
- avoid converting the methods from external to internal numbers, they were
the same anyway so keep the external ones, just use the old defines.
- fix range check. It wrongly used the method numbers as bitmask, which made
the check not work previously. Also error on OOM.
- Dropped the session-id check. It too wrongly did a bitmask check which was
wrong and never worked. When fixed, it broke test cases so I dropped the
entire check.
- split out rtsp_setup_request() from rtsp_do()
- replace the httpversion variable with a define
Closes#22505
That may hold credentials or other sensitive data, or where we want to
ensure the zeroing is not optimized out by the compiler.
Credits-to: Daniel Gustafsson
Ref: #13589 (original attempt)
Ref: #21588
Follow-up to #21645
Follow-up to 066478f634#21598Closes#21637
Add support for signing outgoing HTTP requests per RFC 9421 using
Ed25519 or HMAC-SHA256 algorithms.
New libcurl options:
- CURLOPT_HTTPSIG: signing algorithm ("ed25519" or "hmac-sha256")
- CURLOPT_HTTPSIG_KEY: path to hex-encoded key file
- CURLOPT_HTTPSIG_KEYID: key identifier for Signature-Input
- CURLOPT_HTTPSIG_HEADERS: space-separated components to sign
New CLI flags: --httpsig, --httpsig-key, --httpsig-keyid,
--httpsig-headers
The crypto layer follows the sha256.c multi-backend pattern with
implementations for OpenSSL (EVP_DigestSign) and wolfSSL
(wc_ed25519_sign_msg). HMAC-SHA256 uses the existing Curl_hmacit()
infrastructure which works on all backends.
Verified by test 5000 to 5021
Assisted-by: Daniel Stenberg
Signed-off-by: Sameeh Jubran <sameeh@wolfssl.com>
Closes#22386Closes#21239
Add call stacks to easy and multi instances. Record ongoing API calls
and callback invocations there to detect recursion and not allowed
invocations.
Define enums for easy, multi and callbacks in `api.h`. In `api.c` define
properties for these functions:
- can they recurse
- is the easy/multi handle destroyed during the call or should it be
good afterwards
- is the call allowed when a multi event callback is ongoing
- is the call allowed when a notification callback is ongoing
Entering a guard
- checks that passed CURL*/CURLM* are GOOD on entering
- checks that easy handle's `mid` is correct and it is known
for it in the multi.
- checks that call properties are obeyed (recursion, callback checks)
- checks that passed CURL*/CURLM* are GOOD on leaving, unless call is
known to kill it
Checks for ongoing callbacks inspect the whole call stack and catches
nested invocations (which our current flags can not).
Call stacks in easy/multi handle are fixed size and will deny recursion
when the limit is reached. The current limits are 7 for easy and 15 for
multi now.
Removes:
- multi->in_callback, check is done via call stack
- multi->in_ntfy_cb, check is done via call stack
The overhead in my tests seems minimal, if noticeable at all.
Closes#22237
This option is used for both TLS and SSH so it needs to be handled
even in TLS-disabled builds
Mention this in the man page as well.
Follow-up to 52fa8d9
Pointed out by Codex Security
Closes#22121
Some protocols (and servers) prefer to batch IO and will not send data
unless the window is of sufficient size. Set the burst rate for our
rate limits to a minimum of 32KB to prevent stalling.
Reported-by: Tatsuhiro Tsujikawa
Closes#22016
Adjust code to avoid `-Wformat-signedness` warnings, while making sure
that enums are always cast to a known type when passing them to `printf`
functions, to support compilers and compiler settings where enums are
not default-size signed ints.
- cast integers printed as hex to `unsigned`. (63 times, 20 of them in
`mbedtls.c`)
- cast misc enums to `int` for printing. (31 times)
- cast `CURL_LOCK_DATA_*` enums to `int`. (4 times)
- cast `CURL_FORMADD_*` enums to `int`. (13 times)
- cast `CURLSHE_*` enums to `int`. (3 times)
- cast `CURLUE_*` enums to `int`. (33 times)
- cast `CURLMSG_*` enums to `int`. (6 times)
- cast `CURLE_*` enums to `int`. (~380 times)
- unit1675: fix mask.
Follow-up to 7c34365cce#21879
Ref: #18343 (initial attempt)
Closes#20848
Add bit `native_ca_store_opt` to keep the setting of
CURLOPT_(PROXY_)SSL_OPTIONS and use that to calculate every easy
transfer if a native CA store shall be used or not.
This avoids `native_ca_store` getting stuck on TRUE after being set
once.
Closes#21902
- drop more uses of the word "just". (not enforced here)
- drop some uses of the "will" word.
- "then" -> "than".
- tests/http/testenv/curl.py: fix copy-paste typo in error message.
- pytest: replace `shutdownh` with `shutdown` in test names.
Spotted by GitHub Code Quality.
- comment typos.
- whitespace and newlines fixes.
Closes#21830
This patch adds two major proxy capabilities to curl (ngtcp2 QUIC):
- HTTP/3 Proxy CONNECT: Tunnel HTTP/1.1 or HTTP/2 traffic through an
HTTPS proxy that speaks HTTP/3 (QUIC) using the standard CONNECT
method over an HTTP/3 connection.
- MASQUE CONNECT-UDP: Tunnel HTTP/3 (QUIC) traffic through an HTTP
proxy (speaking HTTP/1.1, HTTP/2, or HTTP/3) using the extended
CONNECT method with the CONNECT-UDP protocol (RFC9297 & RFC9298).
Public API additions:
- `CURLPROXY_HTTPS3`: new proxy type constant for HTTP/3 proxy
- `--proxy-http3`: new CLI flag to negotiate HTTP/3 with HTTPS proxy
The implementation adds two new filters:
- `H3-PROXY` - enables negotiating HTTP/3 (QUIC) to the proxy and
running CONNECT/CONNECT-UDP through that proxy transport.
- `CAPSULE` - dedicated filter inserted between QUIC transport and
HTTP-PROXY to handle datagram capsule encapsulation/decapsulation.
Here is how the curl filter chaining looks in different scenarios:
- HTTP/3 Proxy CONNECT (tunneling TCP protocols over QUIC proxy):
conn -> HTTP/1.1 or HTTP/2 -> SSL -> HTTP-PROXY ->
H3-PROXY -> HAPPY-EYEBALLS -> UDP
- MASQUE CONNECT-UDP (tunneling QUIC over any proxy):
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H3-PROXY ->
HAPPY-EYEBALLS -> UDP
conn -> HTTP/3 -> CAPSULE -> HTTP-PROXY -> H1-PROXY or H2-PROXY ->
SSL -> HAPPY-EYEBALLS -> TCP
- Both features currently require the ngtcp2 QUIC backend.
- Both features are experimental (disabled by default). Enable with
`--enable-proxy-http3`(autotools) or `-DUSE_PROXY_HTTP3=ON`(CMake).
Tests:
- tests/unit/unit3400.c: Unit tests for capsule protocol encode/decode
- tests/http/test_60_h3_proxy.py: Comprehensive pytest integration suite
- tests/http/testenv/h2o.py: Managing h2o instances with HTTP/1.1, HTTP/2,
and HTTP/3 (QUIC) listeners, proxy.connect and proxy.connect-udp enabled.
References:
RFC 9297 - HTTP Datagrams and the Capsule Protocol
RFC 9298 - Proxying UDP in HTTP
RFC 9000 §16 — Variable-Length Integer Encoding
Signed-off-by: Aritra Basu <aritrbas+gh@cisco.com>
Closes#21153
The HTTP/2 feature is deprecated, few servers implement it and our
implementation is complicated by its state management. Make the two
CURLOPT_* involved a nop and deprecate them.
Closes#21723
In AppleSecTrust or NativeCA-enabled builds, make sure override it when
setting a custom `CURLOPT_PROXY_CAINFO_BLOB`.
Reported-by: Joshua Rogers (Aisle Research)
Follow-up to 1730407b74#18279
Follow-up to eefd03c572#18703Closes#21631
Make CURLOPT_GSSAPI_DELEGATION effective on Windows builds that use SSPI
(instead of a native GSS-API implementation), so Kerberos delegation can
be requested during SPNEGO/Negotiate authentication.
Closes#21528
This bit was used to do Digest authentication like Internet Explorer
before version 7 (released on October 18, 2006). Presumably no one uses
this anymore and since it is hard to use and does broken auth, starting
in 8.21.0 this bit does nothing (except setting the actual Digest bit).
Closes#21486
- passing an unknown string to CURLOPT_ECH now returns error
To properly allow applications to spot if they pass in a typo or
something to libcurl.
- CURLECH_DISABLE is now a plain zero internally, not a dedicated bit which
simplifies checks for when ECH is enabled
- Dropped the CURLECH_CLA_CFG bit, and just check STRING_ECH_CONFIG
- Turn grease/enable/hard into three different numerical values, no bitmask
needed
- Convert the struct field 'tls_ech' from an int to a byte.
Closes#21532
- enable three checks:
- bugprone-invalid-enum-default-initialization
- bugprone-sizeof-expression
- readability-inconsistent-declaration-parameter-name (strict)
- fix remaining discrepancies with arg names in prototypes
and implementation, in strict mode.
- document reason for some checks tested but not enabled.
Closes#20794
To sync names for the same macro logic between lib and src, and to move
it to the curlx namespace, to match `curlx_free()` that it's calling.
Closes#21151
- allow to specify when they are wanted on starting a resolve
- match dns cache entries accordingly. An entry which never
tried to get HTTPS-RRs is no answer for a resolve that wants
it.
- fix late arrivals of resolve answers to match the "async"
records that started them - if it still exists.
- provide for multiple "async" resolves in a transfer at the
same time. We may need to resolve an IP interface while the
main connection resolve has not finished yet.
- allow lookup of HTTPS-RR information as soon as it is
available, even if A/AAAA queries are still ongoing.
For this, the "async" infrastructure is changed:
- Defined bits for DNS queries `CURL_DNSQ_A`, `CURL_DNSQ_AAAA`
and `CURL_DNSQ_HTTPS`. These replace `ip_version` which says
nothing about HTTPS.
Use them in dns cache entries for matching.
- enhance the `async->id` to be a unique `uint32_t` for
resolves inside one multi. This is weak, as the id may
wrap around. However it is combined with the `mid` of
the easy handle, making collisions highly unlikely.
`data->state.async` is only accessed in few places where
the mid/async-id match is performed.
- vtls: for ECH supporting TLS backends (openssl, rustls, wolfssl),
retrieve the HTTPS-RR information from the dns connection filter.
Delay the connect if the HTTPS-RR is needed, but has not
been resolved yet.
The implementation of all this is complete for the threaded
resolver. c-ares resolver and DoH do not take advantage of
all new async features yet. To be done in separate PRs.
Details:
c-ares: cleanup settings and initialisation. Any ares channel
is only being created on starting a resolve and propagating
operations in setopt.c to the channel are not helpful.
Changed threaded+ares pollset handling so that they do not
overwrite each others `ASYNC_NAME` timeouts.
Add trace name 'threads' for tracing thread queue and
pool used by threaded resolver.
Closes#21175
Replace the `volatile int dirty` with a reference counter
protected by a mutex when available.
Solve the problem of when to call application's lock function
by adding a volatile flag that indicates a share has been added
to easy handles in its lifetime. That flag ever goes from
FALSE to TRUE, so volatile might work (in the absence of a mutex).
(The problem is that the lock/unlock functions need 2-3
`curl_share_setopt()` invocations to become usable and there
is no way of telling if the third will ever happen. Calling
the lock function before the 3rd setopt may crash the
application.)
When removing a share from an easy handle (or replacing it with
another share), detach the easy connection on a share with a
connection pool.
When cleaning up a share, allow this even if it is still used in
easy handles. It will be destroyed when the reference count
drops to 0.
Closes#20870
- Rename `Curl_resolv_unlink()` to `Curl_dns_entry_unlink()`.
- Change `Curl_dnscache_get()` to return CURLcode result. Returns
now `CURLE_COULDNT_RESOLVE_HOST` for "negative" cache entries.
- Add `Curl_dnscache_add_negative()` to put a "negative" entry
into the cache.
Closes#20864
- the value collided with the internal one used for WebSocket
- we add all new ones as internal bits
- bump to 64-bit internal type
- introduce new define for setting all protocols (CURLPROTO_64ALL)
Reported by Codex Security
Closes#20798
