traefik/docs/content/migration/v2.md

652 lines
22 KiB
Markdown
Raw Normal View History

---
title: "Traefik Migration Documentation"
description: "Learn the steps needed to migrate to new Traefik Proxy v2 versions, i.e. v2.0 to v2.1 or v2.1 to v2.2. Read the technical documentation."
---
2019-12-12 16:06:05 +00:00
# Migration: Steps needed between the versions
## v2.0 to v2.1
### Kubernetes CRD
In v2.1, a new Kubernetes CRD called `TraefikService` was added.
While updating an installation to v2.1,
one should apply that CRD, and update the existing `ClusterRole` definition to allow Traefik to use that CRD.
2019-12-12 16:06:05 +00:00
2021-03-29 12:18:03 +00:00
To add that CRD and enhance the permissions, the following definitions need to be applied to the cluster.
2019-12-12 16:06:05 +00:00
```yaml tab="TraefikService"
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: traefikservices.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TraefikService
plural: traefikservices
singular: traefikservice
scope: Namespaced
```
```yaml tab="ClusterRole"
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
- apiGroups:
- ""
resources:
- services
- endpoints
- secrets
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
2019-12-12 16:06:05 +00:00
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- extensions
- networking.k8s.io
2019-12-12 16:06:05 +00:00
resources:
- ingresses/status
verbs:
- update
- apiGroups:
2023-03-20 14:38:08 +00:00
- traefik.io
2019-12-12 16:06:05 +00:00
- traefik.containo.us
resources:
- middlewares
2023-03-20 14:38:08 +00:00
- middlewaretcps
2020-03-05 10:48:04 +00:00
- ingressroutes
- traefikservices
- ingressroutetcps
2023-03-20 14:38:08 +00:00
- ingressrouteudps
2020-03-05 10:48:04 +00:00
- tlsoptions
2023-03-20 14:38:08 +00:00
- tlsstores
- serverstransports
2019-12-12 16:06:05 +00:00
verbs:
- get
- list
- watch
2020-03-05 10:48:04 +00:00
```
After having both resources applied, Traefik will work properly.
## v2.1 to v2.2
### Headers middleware: accessControlAllowOrigin
`accessControlAllowOrigin` is deprecated.
This field will be removed in future 2.x releases.
Please configure your allowed origins in `accessControlAllowOriginList` instead.
### Kubernetes CRD
In v2.2, new Kubernetes CRDs called `TLSStore` and `IngressRouteUDP` were added.
While updating an installation to v2.2,
one should apply that CRDs, and update the existing `ClusterRole` definition to allow Traefik to use that CRDs.
2021-03-29 12:18:03 +00:00
To add that CRDs and enhance the permissions, the following definitions need to be applied to the cluster.
2020-03-05 10:48:04 +00:00
```yaml tab="TLSStore"
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: tlsstores.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: TLSStore
plural: tlsstores
singular: tlsstore
scope: Namespaced
```
```yaml tab="IngressRouteUDP"
apiVersion: apiextensions.k8s.io/v1beta1
kind: CustomResourceDefinition
metadata:
name: ingressrouteudps.traefik.containo.us
spec:
group: traefik.containo.us
version: v1alpha1
names:
kind: IngressRouteUDP
plural: ingressrouteudps
singular: ingressrouteudp
scope: Namespaced
```
```yaml tab="ClusterRole"
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1beta1
metadata:
name: traefik-ingress-controller
rules:
2019-12-12 16:06:05 +00:00
- apiGroups:
2020-03-05 10:48:04 +00:00
- ""
2019-12-12 16:06:05 +00:00
resources:
2020-03-05 10:48:04 +00:00
- services
- endpoints
- secrets
2019-12-12 16:06:05 +00:00
verbs:
- get
- list
- watch
- apiGroups:
2020-03-05 10:48:04 +00:00
- extensions
- networking.k8s.io
2019-12-12 16:06:05 +00:00
resources:
2020-03-05 10:48:04 +00:00
- ingresses
2019-12-12 16:06:05 +00:00
verbs:
- get
- list
- watch
- apiGroups:
2020-03-05 10:48:04 +00:00
- extensions
- networking.k8s.io
2019-12-12 16:06:05 +00:00
resources:
2020-03-05 10:48:04 +00:00
- ingresses/status
2019-12-12 16:06:05 +00:00
verbs:
2020-03-05 10:48:04 +00:00
- update
2019-12-12 16:06:05 +00:00
- apiGroups:
- traefik.containo.us
resources:
2020-03-05 10:48:04 +00:00
- middlewares
2023-03-20 14:38:08 +00:00
- middlewaretcps
2020-03-05 10:48:04 +00:00
- ingressroutes
2019-12-12 16:06:05 +00:00
- traefikservices
2020-03-05 10:48:04 +00:00
- ingressroutetcps
- ingressrouteudps
- tlsoptions
- tlsstores
2023-03-20 14:38:08 +00:00
- serverstransports
2019-12-12 16:06:05 +00:00
verbs:
- get
- list
- watch
```
After having both resources applied, Traefik will work properly.
### Kubernetes Ingress
To enable HTTPS, it is not sufficient anymore to only rely on a TLS section in the Ingress.
#### Expose an Ingress on 80 and 443
Define the default TLS configuration on the HTTPS entry point.
```yaml tab="Ingress"
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
name: example
spec:
tls:
2021-06-24 09:28:05 +00:00
- secretName: my-tls-secret
rules:
- host: example.com
http:
paths:
- path: "/foo"
backend:
serviceName: example-com
servicePort: 80
```
Entry points definition and enable Ingress provider:
```yaml tab="File (YAML)"
# Static configuration
entryPoints:
web:
address: :80
websecure:
address: :443
http:
tls: {}
providers:
kubernetesIngress: {}
```
```toml tab="File (TOML)"
# Static configuration
[entryPoints.web]
address = ":80"
[entryPoints.websecure]
address = ":443"
[entryPoints.websecure.http]
[entryPoints.websecure.http.tls]
[providers.kubernetesIngress]
```
```bash tab="CLI"
# Static configuration
--entryPoints.web.address=:80
--entryPoints.websecure.address=:443
--entryPoints.websecure.http.tls=true
--providers.kubernetesIngress=true
```
#### Use TLS only on one Ingress
Define the TLS restriction with annotations.
```yaml tab="Ingress"
kind: Ingress
apiVersion: networking.k8s.io/v1beta1
metadata:
name: example-tls
annotations:
traefik.ingress.kubernetes.io/router.entrypoints: websecure
traefik.ingress.kubernetes.io/router.tls: "true"
spec:
tls:
2021-06-24 09:28:05 +00:00
- secretName: my-tls-secret
rules:
- host: example.com
http:
paths:
- path: ""
backend:
serviceName: example-com
servicePort: 80
```
Entry points definition and enable Ingress provider:
```yaml tab="File (YAML)"
# Static configuration
entryPoints:
web:
address: :80
websecure:
address: :443
providers:
kubernetesIngress: {}
```
```toml tab="File (TOML)"
# Static configuration
[entryPoints.web]
address = ":80"
[entryPoints.websecure]
address = ":443"
[providers.kubernetesIngress]
```
```bash tab="CLI"
# Static configuration
--entryPoints.web.address=:80
--entryPoints.websecure.address=:443
--providers.kubernetesIngress=true
```
2020-07-15 18:50:03 +00:00
## v2.2.2 to v2.2.5
### InsecureSNI removal
In `v2.2.2` we introduced a new flag (`insecureSNI`) which was available as a global option to disable domain fronting.
Since `v2.2.5` this global option has been removed, and you should not use it anymore.
### HostSNI rule matcher removal
In `v2.2.2` we introduced a new rule matcher (`HostSNI`) for HTTP routers which was allowing to match the Server Name Indication at the router level.
Since `v2.2.5` this rule has been removed for HTTP routers, and you should not use it anymore.
2020-07-15 18:50:03 +00:00
## v2.2 to v2.3
2020-08-17 10:02:03 +00:00
### X.509 CommonName Deprecation
The deprecated, legacy behavior of treating the CommonName field on X.509 certificates as a host name when no Subject Alternative Names are present, is now disabled by default.
It means that if one is using https with your backend servers, and a certificate with only a CommonName,
Traefik will not try to match the server name indication with the CommonName anymore.
It can be temporarily re-enabled by adding the value `x509ignoreCN=0` to the `GODEBUG` environment variable.
More information: https://golang.org/doc/go1.15#commonname
### File Provider
The file parser has been changed, since v2.3 the unknown options/fields in a dynamic configuration file are treated as errors.
### IngressClass
In `v2.3`, the support of `IngressClass`, which is available since Kubernetes version `1.18`, has been introduced.
In order to be able to use this new resource the [Kubernetes RBAC](../reference/dynamic-configuration/kubernetes-crd.md#rbac) must be updated.
2021-03-29 12:18:03 +00:00
## v2.3 to v2.4
### ServersTransport
In `v2.4.0`, the support of `ServersTransport` has been introduced.
It is therefore necessary to update [RBAC](../reference/dynamic-configuration/kubernetes-crd.md#rbac) and [CRD](../reference/dynamic-configuration/kubernetes-crd.md) definitions.
## v2.4.7 to v2.4.8
### Non-ASCII Domain Names
2021-04-14 10:20:03 +00:00
In `v2.4.8`, we introduced a new check on domain names used in HTTP router rule `Host` and `HostRegexp` expressions,
and in TCP router rule `HostSNI` expression.
This check ensures that provided domain names don't contain non-ASCII characters.
If not, an error is raised, and the associated router will be shown as invalid in the dashboard.
This new behavior is intended to show what was failing silently previously and to help troubleshooting configuration issues.
It doesn't change the support for non-ASCII domain names in routers rules, which is not part of the Traefik feature set so far.
In order to use non-ASCII domain names in a router's rule, one should use the Punycode form of the domain name.
For more information, please read the [HTTP routers rule](../routing/routers/index.md#rule) part or [TCP router rules](../routing/routers/index.md#rule_1) part of the documentation.
2021-03-31 07:43:04 +00:00
2021-04-14 10:20:03 +00:00
## v2.4.8 to v2.4.9
### Tracing Span
In `v2.4.9`, we changed span error to log only server errors (>= 500).
2021-04-21 09:32:14 +00:00
## v2.4.9 to v2.4.10
### K8S CrossNamespace
In `v2.4.10`, the default value for `allowCrossNamespace` has been changed to `false`.
### K8S ExternalName Service
In `v2.4.10`, by default, it is no longer authorized to reference Kubernetes ExternalName services.
To allow it, the `allowExternalNameServices` option should be set to `true`.
2021-07-13 16:12:29 +00:00
## v2.4 to v2.5
### Kubernetes CRD
In `v2.5`, the [Traefik CRDs](../reference/dynamic-configuration/kubernetes-crd.md#definitions) have been updated to support the new API version `apiextensions.k8s.io/v1`.
As required by `apiextensions.k8s.io/v1`, we have included the OpenAPI validation schema.
After deploying the new [Traefik CRDs](../reference/dynamic-configuration/kubernetes-crd.md#definitions), the resources will be validated only on creation or update.
Please note that the unknown fields will not be pruned when migrating from `apiextensions.k8s.io/v1beta1` to `apiextensions.k8s.io/v1` CRDs.
For more details check out the official [documentation](https://kubernetes.io/docs/tasks/extend-kubernetes/custom-resources/custom-resource-definitions/#specifying-a-structural-schema).
### Kubernetes Ingress
Traefik v2.5 moves forward for the Ingress provider to support Kubernetes v1.22.
Traefik now supports only v1.14+ Kubernetes clusters, which means the support of `extensions/v1beta1` API Version ingresses has been dropped.
The `extensions/v1beta1` API Version should now be replaced either by `networking.k8s.io/v1beta1` or by `networking.k8s.io/v1` (as of Kubernetes v1.19+).
The support of the `networking.k8s.io/v1beta1` API Version will stop in Kubernetes v1.22.
### Headers middleware: ssl redirect options
`sslRedirect`, `sslTemporaryRedirect`, `sslHost` and `sslForceHost` are deprecated in Traefik v2.5.
For simple HTTP to HTTPS redirection, you may use [EntryPoints redirections](../routing/entrypoints.md#redirection).
2021-06-11 13:30:05 +00:00
For more advanced use cases, you can use either the [RedirectScheme middleware](../middlewares/http/redirectscheme.md) or the [RedirectRegex middleware](../middlewares/http/redirectregex.md).
2021-05-28 07:24:14 +00:00
### Headers middleware: accessControlAllowOrigin
2021-07-15 12:40:13 +00:00
`accessControlAllowOrigin` is no longer supported in Traefik v2.5.
2021-08-13 15:42:09 +00:00
### X.509 CommonName Deprecation Bis
Following up on the deprecation started [previously](#x509-commonname-deprecation),
as the `x509ignoreCN=0` value for the `GODEBUG` is [deprecated in Go 1.17](https://tip.golang.org/doc/go1.17#crypto/x509),
2024-09-13 09:40:04 +00:00
the legacy behavior related to the CommonName field cannot be enabled at all anymore.
2021-09-10 12:58:13 +00:00
## v2.5.3 to v2.5.4
### Errors middleware
In `v2.5.4`, when the errors service is configured with the [`PassHostHeader`](../routing/services/index.md#pass-host-header) option to `true` (default),
the forwarded Host header value is now set to the client request Host value and not `0.0.0.0`.
Check out the [Errors middleware](../middlewares/http/errorpages.md#service) documentation for more details.
2021-10-06 09:55:12 +00:00
2021-09-10 12:58:13 +00:00
## v2.5 to v2.6
### HTTP/3
2021-09-10 12:58:13 +00:00
Traefik v2.6 introduces the `AdvertisedPort` option,
which allows advertising, in the `Alt-Svc` header, a UDP port different from the one on which Traefik is actually listening (the EntryPoint's port).
By doing so, it introduces a new configuration structure `http3`, which replaces the `enableHTTP3` option (which therefore doesn't exist anymore).
To enable HTTP/3 on an EntryPoint, please check out the [HTTP/3 configuration](../routing/entrypoints.md#http3) documentation.
### Kubernetes Gateway API Provider
2022-10-17 08:52:08 +00:00
In `v2.6`, the [Kubernetes Gateway API provider](../providers/kubernetes-gateway.md) now only supports the version [v1alpha2](https://gateway-api.sigs.k8s.io/v1alpha2/guides/) of the specification and
2022-09-19 09:26:08 +00:00
[route namespaces](https://gateway-api.sigs.k8s.io/v1alpha2/references/spec/#gateway.networking.k8s.io/v1beta1.RouteNamespaces) selectors, which requires Traefik to fetch and watch the cluster namespaces.
Therefore, the [RBAC](../reference/dynamic-configuration/kubernetes-gateway.md#rbac) and [CRD](../reference/dynamic-configuration/kubernetes-gateway.md#definitions) definitions must be updated.
## v2.6.0 to v2.6.1
2022-02-10 15:00:09 +00:00
### Metrics
In `v2.6.1`, the metrics system does not support any more custom HTTP method verbs to prevent potential metrics cardinality overhead.
In consequence, for metrics having the method label,
if the HTTP method verb of a request is not one defined in the set of common methods for [`HTTP/1.1`](https://developer.mozilla.org/en-US/docs/Web/HTTP/Methods)
or the [`PRI`](https://datatracker.ietf.org/doc/html/rfc7540#section-11.6) verb (for `HTTP/2`),
the value for the method label becomes `EXTENSION_METHOD`, instead of the request's one.
2022-02-10 15:00:09 +00:00
### Tracing
In `v2.6.1`, the Datadog tags added to a span changed from `service.name` to `traefik.service.name` and from `router.name` to `traefik.router.name`.
## v2.8
### TLS client authentication
2022-05-30 10:14:26 +00:00
In `v2.8`, the `caOptional` option is deprecated as TLS client authentication is a server side option.
This option available in the ForwardAuth middleware, as well as in the HTTP, Consul, Etcd, Redis, ZooKeeper, Marathon, Consul Catalog, and Docker providers has no effect and must not be used anymore.
### Consul Enterprise Namespaces
In `v2.8`, the `namespace` option of Consul and Consul Catalog providers is deprecated, please use the `namespaces` options instead.
### Traefik Pilot
In `v2.8`, the `pilot.token` and `pilot.dashboard` options are deprecated.
Please check our Blog for migration instructions later this year.
2022-08-09 15:36:08 +00:00
## v2.8.2
Since `v2.5.0`, the `PreferServerCipherSuites` is [deprecated and ignored](https://tip.golang.org/doc/go1.17#crypto/tls) by Go,
in `v2.8.2` the `preferServerCipherSuites` option is also deprecated and ignored in Traefik.
In `v2.8.2`, Traefik now reject certificates signed with the SHA-1 hash function. ([details](https://tip.golang.org/doc/go1.18#sha1))
2022-09-14 08:56:08 +00:00
## v2.9
### Traefik Pilot
In `v2.9`, Traefik Pilot support has been removed.
2023-03-20 14:38:08 +00:00
## v2.10
### Nomad Namespace
In `v2.10`, the `namespace` option of the Nomad provider is deprecated, please use the `namespaces` options instead.
2023-03-20 14:38:08 +00:00
### Kubernetes CRDs
In `v2.10`, the Kubernetes CRDs API Group `traefik.containo.us` is deprecated, and its support will end starting with Traefik v3. Please use the API Group `traefik.io` instead.
As the Kubernetes CRD provider still works with both API Versions (`traefik.io/v1alpha1` and `traefik.containo.us/v1alpha1`),
it means that for the same kind, namespace and name, the provider will only keep the `traefik.io/v1alpha1` resource.
2024-04-25 12:54:04 +00:00
In addition, the Kubernetes CRDs API Version `traefik.containo.us/v1alpha1` will not be supported in Traefik v3 itself.
2023-04-25 09:14:05 +00:00
Please note that it is a requirement to update the CRDs and the RBAC in the cluster before upgrading Traefik.
To do so, please apply the required [CRDs](https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml) and [RBAC](https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml) manifests for v2.10:
```bash
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/reference/dynamic-configuration/kubernetes-crd-rbac.yml
kubectl apply -f https://raw.githubusercontent.com/traefik/traefik/v2.10/docs/content/reference/dynamic-configuration/kubernetes-crd-definition-v1.yml
```
### Traefik Hub
2023-05-15 14:38:05 +00:00
In `v2.10`, Traefik Hub configuration has been removed because Traefik Hub v2 doesn't require this configuration.
## v2.11
### IPWhiteList (HTTP)
In `v2.11`, the `IPWhiteList` middleware is deprecated, please use the [IPAllowList](../middlewares/http/ipallowlist.md) middleware instead.
### IPWhiteList (TCP)
In `v2.11`, the `IPWhiteList` middleware is deprecated, please use the [IPAllowList](../middlewares/tcp/ipallowlist.md) middleware instead.
### TLS CipherSuites
> By default, cipher suites without ECDHE support are no longer offered by either clients or servers during pre-TLS 1.3 handshakes.
> This change can be reverted with the `tlsrsakex=1 GODEBUG` setting.
> (https://go.dev/doc/go1.22#crypto/tls)
The _RSA key exchange_ cipher suites are way less secure than the modern ECDHE cipher suites and exposes to potential vulnerabilities like [the Marvin Attack](https://people.redhat.com/~hkario/marvin).
Decision has been made to support ECDHE cipher suites only by default.
The following ciphers have been removed from the default list:
- `TLS_RSA_WITH_AES_128_CBC_SHA`
- `TLS_RSA_WITH_AES_256_CBC_SHA`
- `TLS_RSA_WITH_AES_128_GCM_SHA256`
- `TLS_RSA_WITH_AES_256_GCM_SHA384`
To enable these ciphers, please set the option `CipherSuites` in your [TLS configuration](https://doc.traefik.io/traefik/https/tls/#cipher-suites) or set the environment variable `GODEBUG=tlsrsakex=1`.
### Minimum TLS Version
> By default, the minimum version offered by `crypto/tls` servers is now TLS 1.2 if not specified with config.MinimumVersion,
> matching the behavior of crypto/tls clients.
> This change can be reverted with the `tls10server=1 GODEBUG` setting.
> (https://go.dev/doc/go1.22#crypto/tls)
To enable TLS 1.0, please set the option `MinVersion` to `VersionTLS10` in your [TLS configuration](https://doc.traefik.io/traefik/https/tls/#cipher-suites) or set the environment variable `GODEBUG=tls10server=1`.
## v2.11.1
### Maximum Router Priority Value
Before v2.11.1, the maximum user-defined router priority value is:
- `MaxInt32` for 32-bit platforms,
- `MaxInt64` for 64-bit platforms.
Please check out the [go documentation](https://pkg.go.dev/math#pkg-constants) for more information.
In v2.11.1, Traefik reserves a range of priorities for its internal routers and now,
the maximum user-defined router priority value is:
- `(MaxInt32 - 1000)` for 32-bit platforms,
- `(MaxInt64 - 1000)` for 64-bit platforms.
### EntryPoint.Transport.RespondingTimeouts.<Timeout>
Starting with `v2.11.1` the following timeout options are deprecated:
- `<entryPoint>.transport.respondingTimeouts.readTimeout`
- `<entryPoint>.transport.respondingTimeouts.writeTimeout`
- `<entryPoint>.transport.respondingTimeouts.idleTimeout`
They have been replaced by:
- `<entryPoint>.transport.respondingTimeouts.http.readTimeout`
- `<entryPoint>.transport.respondingTimeouts.http.writeTimeout`
- `<entryPoint>.transport.respondingTimeouts.http.idleTimeout`
### EntryPoint.Transport.RespondingTimeouts.TCP.LingeringTimeout
Starting with `v2.11.1` a new `lingeringTimeout` entryPoints option has been introduced, with a default value of 2s.
The lingering timeout defines the maximum duration between each TCP read operation on the connection.
As a layer 4 timeout, it applies during HTTP handling but respects the configured HTTP server `readTimeout`.
This change avoids Traefik instances with the default configuration hanging while waiting for bytes to be read on the connection.
We suggest to adapt this value accordingly to your situation.
The new default value is purposely narrowed and can close the connection too early.
Increasing the `lingeringTimeout` value could be the solution notably if you are dealing with the following errors:
- TCP: `Error while handling TCP connection: readfrom tcp X.X.X.X:X->X.X.X.X:X: read tcp X.X.X.X:X->X.X.X.X:X: i/o timeout`
- HTTP: `'499 Client Closed Request' caused by: context canceled`
- HTTP: `ReverseProxy read error during body copy: read tcp X.X.X.X:X->X.X.X.X:X: use of closed network connection`
## v2.11.2
### LingeringTimeout
Starting with `v2.11.2` the `<entrypoint>.transport.respondingTimeouts.tcp.lingeringTimeout` introduced in `v2.11.1` has been removed.
### RespondingTimeouts.TCP and RespondingTimeouts.HTTP
Starting with `v2.11.2` the `respondingTimeouts.tcp` and `respondingTimeouts.http` sections introduced in `v2.11.1` have been removed.
To configure the responding timeouts, please use the [`respondingTimeouts`](../routing/entrypoints.md#respondingtimeouts) section.
### EntryPoint.Transport.RespondingTimeouts.ReadTimeout
Starting with `v2.11.2` the entryPoints [`readTimeout`](../routing/entrypoints.md#respondingtimeouts) option default value changed to 60 seconds.
For HTTP, this option defines the maximum duration for reading the entire request, including the body.
For TCP, this option defines the maximum duration for the first bytes to be read on the connection.
The default value was previously set to zero, which means no timeout.
This change has been done to avoid Traefik instances with the default configuration to be hanging forever while waiting for bytes to be read on the connection.
Increasing the `readTimeout` value could be the solution notably if you are dealing with the following errors:
- TCP: `Error while handling TCP connection: readfrom tcp X.X.X.X:X->X.X.X.X:X: read tcp X.X.X.X:X->X.X.X.X:X: i/o timeout`
- HTTP: `'499 Client Closed Request' caused by: context canceled`
- HTTP: `ReverseProxy read error during body copy: read tcp X.X.X.X:X->X.X.X.X:X: use of closed network connection`
## v2.11.3
### Connection headers
In `v2.11.3`, the handling of the request Connection headers directives has changed to prevent any abuse.
Before, Traefik removed any header listed in the Connection header just before forwarding the request to the backends.
Now, Traefik removes the headers listed in the Connection header as soon as the request is handled.
As a consequence, middlewares do not have access to those Connection headers,
and a new option has been introduced to specify which ones could go through the middleware chain before being removed: `<entrypoint>.forwardedHeaders.connection`.
Please check out the [entrypoint forwarded headers connection option configuration](../routing/entrypoints.md#forwarded-headers) documentation.