Bumps quic-go to v0.59.1 (chungthuang fork rebased from upstream v0.45 onto
v0.59.1). Upstream removed the `logging` package and replaced its
callback-based ConnectionTracer with the structured `qlog`/`qlogwriter` event
API, which required migrating cloudflared's QUIC metrics collection.
Migrations:
- quic/tracing.go: connTracer no longer fills a logging.ConnectionTracer
callback struct. It implements qlogwriter.Trace + qlogwriter.Recorder and
dispatches qlog events (PacketSent, PacketReceived, MetricsUpdated, ...) to
the collector through RecordEvent. NewClientTracer now returns a function
compatible with quic.Config.Tracer.
- quic/metrics.go: collector methods take qlog types (qlog.Frame,
qlog.PacketType, qlog.MetricsUpdated, ...) and plain int64 in place of the
removed logging.ByteCount/Frame/RTTStats/TransportParameters.
- quic/conversion.go: PacketType, PacketDropReason and PacketLossReason are
strings upstream rather than numeric iotas, so the converters become
pass-through allowlists. CongestionState is also a string;
congestionStateToFloat maps it back to the numeric gauge values cloudflared
exports.
- quic.Connection/quic.Stream became *quic.Conn/*quic.Stream; updated
ConnWithCloser, SafeStreamCloser and the connection package accordingly.
Tests and generated mocks (mocks/mock_quic_connection.go) were adapted to
the new pointer-based API.
Closes TUN-10557
The bump of the QUIC library introduces a cyclic dependency between the connection and quic modules hence it is necessary to break this coupling.
Right now, the connection module depends on the quic module for the datagram v2/v3 and to which a QUIC connection (currently an interface) is passed.
As it is there is no issue however, under the hood, interface is a wrapper around an UDP connection and a QUIC connection meaning this type must be exposed to the quic module since the QUIC Connection will no longer be a interface but a struct.
Given the above, these changes introduce an interface, QUICConnection, with the surface used today in cloudflared and a struct, ConnWithCloser, that implements said interface within the quic module.
Closes TUN-10563
Introduce a new crypto/ package as the single source of truth for TLS
curve preferences used on every edge-facing connection, and adopt
X25519MLKEM768 as the primary post-quantum key exchange for both QUIC
and HTTP/2:
PQ Prefer (default): X25519MLKEM768, P256Kyber768Draft00, CurveP256
PQ Strict (--post-quantum): X25519MLKEM768, P256Kyber768Draft00
The curve list is identical under FIPS and non-FIPS builds, so
crypto.GetCurvePreferences takes only a features.PostQuantumMode and
returns a fresh slice on every call.
HTTP/2 now applies these curve preferences the same way QUIC does. The
previous PostQuantumStrict rejection in serveHTTP2 and the forced
QUIC-only selection in NewProtocolSelector are removed since both
transports support the same post-quantum curves; the needPQ parameter
is dropped from NewProtocolSelector accordingly.
Also fix a shared tls.Config race: both the QUIC and HTTP/2 paths now
Clone() the per-protocol entry from TunnelConfig.EdgeTLSConfigs before
mutating CurvePreferences instead of writing through the shared map
entry.
Legacy Kyber draft curve X25519Kyber768Draft00
and the unused removeDuplicates helper are removed along with the old
supervisor/pqtunnels.go / _test.go files.
AGENTS.md is updated with guidance on the new crypto/ package, the
cfdcrypto import alias, the tls.Config cloning rule, and the lint
workflow implications of .golangci.yaml's whole-files: true setting.
Corrects the pattern of using errgroup's and context cancellation to simplify the logic for canceling extra routines for the QUIC connection. This is because the extra context cancellation is redundant with the fact that the errgroup also cancels it's own provided context when a routine returns (error or not).
For the datagram handler specifically, since it can respond faster to a context cancellation from the QUIC connection, we wrap the error before surfacing it outside of the QUIC connection scope to the supervisor. Additionally, the supervisor will look for this error type to check if it should retry the QUIC connection. These two operations are required because the supervisor does not look for a context canceled error when deciding to retry a connection. If a context canceled from the datagram handler were to be returned up to the supervisor on the initial connection, the cloudflared application would exit. We want to ensure that cloudflared maintains connection attempts even if any of the services on-top of a QUIC connection fail (datagram handler in this case).
Additional logging is also introduced along these paths to help with understanding the error conditions from the specific handlers on-top of a QUIC connection.
Related CUSTESC-53681
Closes TUN-9610
Adds an OriginDialerService that takes over the roles of both DialUDP and DialTCP
towards the origin. This provides the possibility to leverage dialer "middleware"
to inject virtual origins, such as the DNS resolver service.
DNS Resolver service also gains access to the DialTCP operation to service TCP
DNS requests.
Minor refactoring includes changes to remove the needs previously provided by
the warp-routing configuration. This configuration cannot be disabled by cloudflared
so many of the references have been adjusted or removed.
Closes TUN-9470
Semgrep config / semgrep/ci (push) Has been cancelled
Check / check (1.22.x, macos-latest) (push) Has been cancelled
Check / check (1.22.x, ubuntu-latest) (push) Has been cancelled
Check / check (1.22.x, windows-latest) (push) Has been cancelled
Introduces a new `UDPOriginProxy` interface and `UDPOriginService`
to standardize how UDP connections are dialed to origins. Allows for
future overrides of the ingress service for specific dial destinations.
Simplifies dependency injection for UDP dialing throughout both datagram
v2 and v3 by using the same ingress service. Previous invocations called
into a DialUDP function in the ingress package that was a light
wrapper over `net.DialUDP`. Now a reference is passed into both datagram
controllers that allows more control over the DialUDP method.
Closes TUN-9469
Semgrep config / semgrep/ci (push) Has been cancelled
Check / check (1.22.x, ubuntu-latest) (push) Has been cancelled
Check / check (1.22.x, windows-latest) (push) Has been cancelled
Check / check (1.22.x, macos-latest) (push) Has been cancelled
## Summary
Update several moving parts of cloudflared build system:
* use goboring 1.24.2 in cfsetup
* update linter and fix lint issues
* update packages namely **quic-go and net**
* install script for macos
* update docker files to use go 1.24.1
* remove usage of cloudflare-go
* pin golang linter
Closes TUN-9016
Make sure to enforce snapshots of features and client information for each connection
so that the feature information can change in the background. This allows for new features
to only be applied to a connection if it completely disconnects and attempts a reconnect.
Updates the feature refresh time to 1 hour from previous cloudflared versions which
refreshed every 6 hours.
Closes TUN-9319
During a refresh of the supported features via the DNS TXT record,
cloudflared would update the internal feature list, but would not
propagate this information to the edge during a new connection.
This meant that a situation could occur in which cloudflared would
think that the client's connection could support datagram V3, and
would setup that muxer locally, but would not propagate that information
to the edge during a register connection in the `ClientInfo` of the
`ConnectionOptions`. This meant that the edge still thought that the
client was setup to support datagram V2 and since the protocols are
not backwards compatible, the local muxer for datagram V3 would reject
the incoming RPC calls.
To address this, the feature list will be fetched only once during
client bootstrapping and will persist as-is until the client is restarted.
This helps reduce the complexity involved with different connections
having possibly different sets of features when connecting to the edge.
The features will now be tied to the client and never diverge across
connections.
Also, retires the use of `support_datagram_v3` in-favor of
`support_datagram_v3_1` to reduce the risk of reusing the feature key.
The `dv3` TXT feature key is also deprecated.
Closes TUN-9291
Check / check (1.22.x, macos-latest) (push) Has been cancelled
Check / check (1.22.x, ubuntu-latest) (push) Has been cancelled
Check / check (1.22.x, windows-latest) (push) Has been cancelled
Semgrep config / semgrep/ci (push) Has been cancelled
## Summary
We have adapted our edge services to better know when they should flush on write. This is an important
feature to ensure response types like Server Side Events are not buffered, and instead are propagated to the eyeball
as soon as possible. This commit implements a similar logic for http2 tunnel protocol that we use in our edge
services. By adding the new events stream header for json `application/x-ndjson` and using the content-length
and transfer-encoding headers as well, following the RFC's:
- https://datatracker.ietf.org/doc/html/rfc7230#section-4.1
- https://datatracker.ietf.org/doc/html/rfc9112#section-6.1
Closes TUN-9255
## Summary
Within the work of FEDRamp it is necessary to change the HA SD lookup to use as srv `fed-v2-origintunneld`
This work assumes that the tunnel token has an optional endpoint field which will be used to modify the behaviour of the HA SD lookup.
Finally, the presence of the endpoint will override region to _fed_ and fail if any value is passed for the flag region.
Closes TUN-9007
Check / check (1.22.x, macos-latest) (push) Has been cancelled
Check / check (1.22.x, ubuntu-latest) (push) Has been cancelled
Check / check (1.22.x, windows-latest) (push) Has been cancelled
Semgrep config / semgrep/ci (push) Has been cancelled
## Summary
This commit renames the public variable that identifies the metadata key and value for the ConnectResponse structure when the flow was rate limited.
Closes TUN-8904
## Summary
Session is the concept used for UDP flows. Therefore, to make
the session limiter ambiguous for both TCP and UDP, this commit
renames it to flow limiter.
Closes TUN-8861
## Summary
In order to make cloudflared behavior more predictable and
prevent an exhaustion of resources, we have decided to add
session limits that can be configured by the user. This commit
adds the session limiter to the HTTP/TCP handling path.
For now the limiter is set to run only in unlimited mode.
## Summary
In order to make cloudflared behavior more predictable and
prevent an exhaustion of resources, we have decided to add
session limits that can be configured by the user. This first
commit introduces the session limiter and adds it to the UDP
handling path. For now the limiter is set to run only in
unlimited mode.
A new ICMPResponder interface is introduced to provide different
implementations of how the ICMP flows should return to the QUIC
connection muxer.
Improves usages of netip.AddrPort to leverage the embedded zone
field for IPv6 addresses.
Closes TUN-8640
## Summary
The new endpoint returns the current information to be used when calling the diagnostic procedure.
This also adds:
- add indexed connection info and method to extract active connections from connTracker
- add edge address to Event struct and conn tracker
- remove unnecessary event send
- add tunnel configuration handler
- adjust cmd and metrics to create diagnostic server
Closes TUN-8728
When a registration response from cloudflared gets lost on it's way back to the edge, the edge service will retry and send another registration request. Since cloudflared already has bound the local UDP socket for the provided request id, we want to re-send the registration response.
There are three types of retries that the edge will send:
1. A retry from the same QUIC connection index; cloudflared will just respond back with a registration response and reset the idle timer for the session.
2. A retry from a different QUIC connection index; cloudflared will need to migrate the current session connection to this new QUIC connection and reset the idle timer for the session.
3. A retry to a different cloudflared connector; cloudflared will eventually time the session out since no further packets will arrive to the session at the original connector.
Closes TUN-8709
The current supervisor serves the quic connection by performing all of the following in one method:
1. Dial QUIC edge connection
2. Initialize datagram muxer for UDP sessions and ICMP
3. Wrap all together in a single struct to serve the process loops
In an effort to better support modularity, each of these steps were broken out into their own separate methods that the supervisor will compose together to create the TunnelConnection and run its `Serve` method.
This also provides us with the capability to better interchange the functionality supported by the datagram session manager in the future with a new mechanism.
Closes TUN-8661
For macOS, we want to set the DF bit for the UDP packets used by the QUIC
connection; to achieve this, you need to explicitly set the network
to either "udp4" or "udp6". When determining which network type to pick
we need to use the edge IP address chosen to align with what the local
IP family interface we will use. This means we want cloudflared to bind
to local interfaces for a random port, so we provide a zero IP and 0 port
number (ex. 0.0.0.0:0). However, instead of providing the zero IP, we
can leave the value as nil and let the kernel decide which interface and
pick a random port as defined by the target edge IP family.
This was previously broken for IPv6-only edge connectivity on macOS and
all other operating systems should be unaffected because the network type
was left as default "udp" which will rely on the provided local or remote
IP for selection.
Closes TUN-8688
Some more legacy h2mux code to be cleaned up and moved out of the way.
The h2mux.Header used in the serialization for http2 proxied headers is moved to connection module. Additionally, the booleanfuse structure is also moved to supervisor as it is also needed. Both of these structures could be evaluated later for removal/updates, however, the intent of the proposed changes here is to remove the dependencies on the h2mux code and removal.
Approved-by: Chung-Ting Huang <chungting@cloudflare.com>
Approved-by: Luis Neto <lneto@cloudflare.com>
Approved-by: Gonçalo Garcia <ggarcia@cloudflare.com>
MR: https://gitlab.cfdata.org/cloudflare/tun/cloudflared/-/merge_requests/1576
Whenever cloudflared receives a SIGTERM or SIGINT it goes into graceful shutdown mode, which unregisters the connection and closes the control stream. Unregistering makes it so we no longer receive any new requests and makes the edge close the connection, allowing in-flight requests to finish (within a 3 minute period).
This was working fine for http2 connections, but the quic proxy was cancelling the context as soon as the controls stream ended, forcing the process to stop immediately.
This commit changes the behavior so that we wait the full grace period before cancelling the request
Revert "TUN-8621: Fix cloudflared version in change notes."
Revert "PPIP-2310: Update quick tunnel disclaimer"
Revert "TUN-8621: Prevent QUIC connection from closing before grace period after unregistering"
Revert "TUN-8484: Print response when QuickTunnel can't be unmarshalled"
Revert "TUN-8592: Use metadata from the edge to determine if request body is empty for QUIC transport"
Whenever cloudflared receives a SIGTERM or SIGINT it goes into graceful shutdown mode, which unregisters the connection and closes the control stream. Unregistering makes it so we no longer receive any new requests and makes the edge close the connection, allowing in-flight requests to finish (within a 3 minute period).
This was working fine for http2 connections, but the quic proxy was cancelling the context as soon as the controls stream ended, forcing the process to stop immediately.
This commit changes the behavior so that we wait the full grace period before cancelling the request
Adds new suite of metrics to capture the following for capnp rpcs operations:
- Method calls
- Method call failures
- Method call latencies
Each of the operations is labeled by the handler that serves the method and
the method of operation invoked. Additionally, each of these are split
between if the operation was called by a client or served.
Since legacy tunnels have been removed for a while now, we can remove
many of the capnp rpc interfaces that are no longer leveraged by the
legacy tunnel registration and authentication mechanisms.
Combines the tunnelrpc and quic/schema capnp files into the same module.
To help reduce future issues with capnp id generation, capnpids are
provided in the capnp files from the existing capnp struct ids generated
in the go files.
Reduces the overall interface of the Capnp methods to the rest of
the code by providing an interface that will handle the quic protocol
selection.
Introduces a new `rpc-timeout` config that will allow all of the
SessionManager and ConfigurationManager RPC requests to have a timeout.
The timeout for these values is set to 5 seconds as non of these operations
for the managers should take a long time to complete.
Removed the RPC-specific logger as it never provided good debugging value
as the RPC method names were not visible in the logs.
If cloudflared was unable to register the UDP session with the
edge, the socket would be left open to be eventually closed by the
OS, or garbage collected by the runtime. Considering that either of
these closes happened significantly after some delay, it was causing
cloudflared to hold open file descriptors longer than usual if continuously
unable to register sessions.
## Summary
To prevent bad eyeballs and severs to be able to exhaust the quic
control flows we are adding the possibility of having a timeout
for a write operation to be acknowledged. This will prevent hanging
connections from exhausting the quic control flows, creating a DDoS.
Miguel da Costa Martins Marcelino
lneto
João "Pisco" Fernandes
chungthuang
GitLab Maintenance Bot
Devin Carr
GoncaloGarcia