Add Failover service
Co-authored-by: Kevin Pollet <pollet.kevin@gmail.com>
This commit is contained in:
parent
6622027c7c
commit
79aab5aab8
12 changed files with 583 additions and 3 deletions
|
@ -95,6 +95,12 @@
|
||||||
secure = true
|
secure = true
|
||||||
httpOnly = true
|
httpOnly = true
|
||||||
sameSite = "foobar"
|
sameSite = "foobar"
|
||||||
|
[http.services.Service04]
|
||||||
|
[http.services.Service04.failover]
|
||||||
|
service = "foobar"
|
||||||
|
fallback = "foobar"
|
||||||
|
|
||||||
|
[http.services.Service04.failover.healthCheck]
|
||||||
[http.middlewares]
|
[http.middlewares]
|
||||||
[http.middlewares.Middleware00]
|
[http.middlewares.Middleware00]
|
||||||
[http.middlewares.Middleware00.addPrefix]
|
[http.middlewares.Middleware00.addPrefix]
|
||||||
|
|
|
@ -95,6 +95,11 @@ http:
|
||||||
secure: true
|
secure: true
|
||||||
httpOnly: true
|
httpOnly: true
|
||||||
sameSite: foobar
|
sameSite: foobar
|
||||||
|
Service04:
|
||||||
|
failover:
|
||||||
|
service: foobar
|
||||||
|
fallback: foobar
|
||||||
|
healthCheck: {}
|
||||||
middlewares:
|
middlewares:
|
||||||
Middleware00:
|
Middleware00:
|
||||||
addPrefix:
|
addPrefix:
|
||||||
|
|
|
@ -228,6 +228,9 @@
|
||||||
| `traefik/http/services/Service03/weighted/sticky/cookie/name` | `foobar` |
|
| `traefik/http/services/Service03/weighted/sticky/cookie/name` | `foobar` |
|
||||||
| `traefik/http/services/Service03/weighted/sticky/cookie/sameSite` | `foobar` |
|
| `traefik/http/services/Service03/weighted/sticky/cookie/sameSite` | `foobar` |
|
||||||
| `traefik/http/services/Service03/weighted/sticky/cookie/secure` | `true` |
|
| `traefik/http/services/Service03/weighted/sticky/cookie/secure` | `true` |
|
||||||
|
| `traefik/http/services/Service04/failover/fallback` | `foobar` |
|
||||||
|
| `traefik/http/services/Service04/failover/healthCheck` | `` |
|
||||||
|
| `traefik/http/services/Service04/failover/service` | `foobar` |
|
||||||
| `traefik/tcp/middlewares/Middleware00/ipWhiteList/sourceRange/0` | `foobar` |
|
| `traefik/tcp/middlewares/Middleware00/ipWhiteList/sourceRange/0` | `foobar` |
|
||||||
| `traefik/tcp/middlewares/Middleware00/ipWhiteList/sourceRange/1` | `foobar` |
|
| `traefik/tcp/middlewares/Middleware00/ipWhiteList/sourceRange/1` | `foobar` |
|
||||||
| `traefik/tcp/routers/TCPRouter0/entryPoints/0` | `foobar` |
|
| `traefik/tcp/routers/TCPRouter0/entryPoints/0` | `foobar` |
|
||||||
|
|
|
@ -1212,6 +1212,139 @@ http:
|
||||||
url = "http://private-ip-server-2/"
|
url = "http://private-ip-server-2/"
|
||||||
```
|
```
|
||||||
|
|
||||||
|
### Failover (service)
|
||||||
|
|
||||||
|
A failover service job is to forward all requests to a fallback service when the main service becomes unreachable.
|
||||||
|
|
||||||
|
!!! info "Relation to HealthCheck"
|
||||||
|
|
||||||
|
The failover service relies on the HealthCheck system to get notified when its main service becomes unreachable,
|
||||||
|
which means HealthCheck needs to be enabled and functional on the main service.
|
||||||
|
However, HealthCheck does not need to be enabled on the failover service itself for it to be functional.
|
||||||
|
It is only required in order to propagate upwards the information when the failover itself becomes down
|
||||||
|
(i.e. both its main and its fallback are down too).
|
||||||
|
|
||||||
|
!!! info "Supported Providers"
|
||||||
|
|
||||||
|
This strategy can currently only be defined with the [File](../../providers/file.md) provider.
|
||||||
|
|
||||||
|
```yaml tab="YAML"
|
||||||
|
## Dynamic configuration
|
||||||
|
http:
|
||||||
|
services:
|
||||||
|
app:
|
||||||
|
failover:
|
||||||
|
service: main
|
||||||
|
fallback: backup
|
||||||
|
|
||||||
|
main:
|
||||||
|
loadBalancer:
|
||||||
|
healthCheck:
|
||||||
|
path: /status
|
||||||
|
interval: 10s
|
||||||
|
timeout: 3s
|
||||||
|
servers:
|
||||||
|
- url: "http://private-ip-server-1/"
|
||||||
|
|
||||||
|
backup:
|
||||||
|
loadBalancer:
|
||||||
|
servers:
|
||||||
|
- url: "http://private-ip-server-2/"
|
||||||
|
```
|
||||||
|
|
||||||
|
```toml tab="TOML"
|
||||||
|
## Dynamic configuration
|
||||||
|
[http.services]
|
||||||
|
[http.services.app]
|
||||||
|
[http.services.app.failover]
|
||||||
|
service = "main"
|
||||||
|
fallback = "backup"
|
||||||
|
|
||||||
|
[http.services.main]
|
||||||
|
[http.services.main.loadBalancer]
|
||||||
|
[http.services.main.loadBalancer.healthCheck]
|
||||||
|
path = "/health"
|
||||||
|
interval = "10s"
|
||||||
|
timeout = "3s"
|
||||||
|
[[http.services.main.loadBalancer.servers]]
|
||||||
|
url = "http://private-ip-server-1/"
|
||||||
|
|
||||||
|
[http.services.backup]
|
||||||
|
[http.services.backup.loadBalancer]
|
||||||
|
[[http.services.backup.loadBalancer.servers]]
|
||||||
|
url = "http://private-ip-server-2/"
|
||||||
|
```
|
||||||
|
|
||||||
|
#### Health Check
|
||||||
|
|
||||||
|
HealthCheck enables automatic self-healthcheck for this service,
|
||||||
|
i.e. if the main and the fallback services become unreachable,
|
||||||
|
the information is propagated upwards to its parent.
|
||||||
|
|
||||||
|
!!! info "All or nothing"
|
||||||
|
|
||||||
|
If HealthCheck is enabled for a given service, but any of its descendants does
|
||||||
|
not have it enabled, the creation of the service will fail.
|
||||||
|
|
||||||
|
HealthCheck on a Failover service can currently only be defined with the [File](../../providers/file.md) provider.
|
||||||
|
|
||||||
|
```yaml tab="YAML"
|
||||||
|
## Dynamic configuration
|
||||||
|
http:
|
||||||
|
services:
|
||||||
|
app:
|
||||||
|
failover:
|
||||||
|
healthCheck: {}
|
||||||
|
service: main
|
||||||
|
fallback: backup
|
||||||
|
|
||||||
|
main:
|
||||||
|
loadBalancer:
|
||||||
|
healthCheck:
|
||||||
|
path: /status
|
||||||
|
interval: 10s
|
||||||
|
timeout: 3s
|
||||||
|
servers:
|
||||||
|
- url: "http://private-ip-server-1/"
|
||||||
|
|
||||||
|
backup:
|
||||||
|
loadBalancer:
|
||||||
|
healthCheck:
|
||||||
|
path: /status
|
||||||
|
interval: 10s
|
||||||
|
timeout: 3s
|
||||||
|
servers:
|
||||||
|
- url: "http://private-ip-server-2/"
|
||||||
|
```
|
||||||
|
|
||||||
|
```toml tab="TOML"
|
||||||
|
## Dynamic configuration
|
||||||
|
[http.services]
|
||||||
|
[http.services.app]
|
||||||
|
[http.services.app.failover.healthCheck]
|
||||||
|
[http.services.app.failover]
|
||||||
|
service = "main"
|
||||||
|
fallback = "backup"
|
||||||
|
|
||||||
|
[http.services.main]
|
||||||
|
[http.services.main.loadBalancer]
|
||||||
|
[http.services.main.loadBalancer.healthCheck]
|
||||||
|
path = "/health"
|
||||||
|
interval = "10s"
|
||||||
|
timeout = "3s"
|
||||||
|
[[http.services.main.loadBalancer.servers]]
|
||||||
|
url = "http://private-ip-server-1/"
|
||||||
|
|
||||||
|
[http.services.backup]
|
||||||
|
[http.services.backup.loadBalancer]
|
||||||
|
[http.services.backup.loadBalancer.healthCheck]
|
||||||
|
path = "/health"
|
||||||
|
interval = "10s"
|
||||||
|
timeout = "3s"
|
||||||
|
[[http.services.backup.loadBalancer.servers]]
|
||||||
|
url = "http://private-ip-server-2/"
|
||||||
|
```
|
||||||
|
|
||||||
## Configuring TCP Services
|
## Configuring TCP Services
|
||||||
|
|
||||||
### General
|
### General
|
||||||
|
|
|
@ -35,6 +35,7 @@ type Service struct {
|
||||||
LoadBalancer *ServersLoadBalancer `json:"loadBalancer,omitempty" toml:"loadBalancer,omitempty" yaml:"loadBalancer,omitempty" export:"true"`
|
LoadBalancer *ServersLoadBalancer `json:"loadBalancer,omitempty" toml:"loadBalancer,omitempty" yaml:"loadBalancer,omitempty" export:"true"`
|
||||||
Weighted *WeightedRoundRobin `json:"weighted,omitempty" toml:"weighted,omitempty" yaml:"weighted,omitempty" label:"-" export:"true"`
|
Weighted *WeightedRoundRobin `json:"weighted,omitempty" toml:"weighted,omitempty" yaml:"weighted,omitempty" label:"-" export:"true"`
|
||||||
Mirroring *Mirroring `json:"mirroring,omitempty" toml:"mirroring,omitempty" yaml:"mirroring,omitempty" label:"-" export:"true"`
|
Mirroring *Mirroring `json:"mirroring,omitempty" toml:"mirroring,omitempty" yaml:"mirroring,omitempty" label:"-" export:"true"`
|
||||||
|
Failover *Failover `json:"failover,omitempty" toml:"failover,omitempty" yaml:"failover,omitempty" label:"-" export:"true"`
|
||||||
}
|
}
|
||||||
|
|
||||||
// +k8s:deepcopy-gen=true
|
// +k8s:deepcopy-gen=true
|
||||||
|
@ -76,6 +77,15 @@ func (m *Mirroring) SetDefaults() {
|
||||||
|
|
||||||
// +k8s:deepcopy-gen=true
|
// +k8s:deepcopy-gen=true
|
||||||
|
|
||||||
|
// Failover holds the Failover configuration.
|
||||||
|
type Failover struct {
|
||||||
|
Service string `json:"service,omitempty" toml:"service,omitempty" yaml:"service,omitempty" export:"true"`
|
||||||
|
Fallback string `json:"fallback,omitempty" toml:"fallback,omitempty" yaml:"fallback,omitempty" export:"true"`
|
||||||
|
HealthCheck *HealthCheck `json:"healthCheck,omitempty" toml:"healthCheck,omitempty" yaml:"healthCheck,omitempty" label:"allowEmpty" file:"allowEmpty" export:"true"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// +k8s:deepcopy-gen=true
|
||||||
|
|
||||||
// MirrorService holds the MirrorService configuration.
|
// MirrorService holds the MirrorService configuration.
|
||||||
type MirrorService struct {
|
type MirrorService struct {
|
||||||
Name string `json:"name,omitempty" toml:"name,omitempty" yaml:"name,omitempty" export:"true"`
|
Name string `json:"name,omitempty" toml:"name,omitempty" yaml:"name,omitempty" export:"true"`
|
||||||
|
@ -98,7 +108,7 @@ type WeightedRoundRobin struct {
|
||||||
|
|
||||||
// +k8s:deepcopy-gen=true
|
// +k8s:deepcopy-gen=true
|
||||||
|
|
||||||
// WRRService is a reference to a service load-balanced with weighted round robin.
|
// WRRService is a reference to a service load-balanced with weighted round-robin.
|
||||||
type WRRService struct {
|
type WRRService struct {
|
||||||
Name string `json:"name,omitempty" toml:"name,omitempty" yaml:"name,omitempty" export:"true"`
|
Name string `json:"name,omitempty" toml:"name,omitempty" yaml:"name,omitempty" export:"true"`
|
||||||
Weight *int `json:"weight,omitempty" toml:"weight,omitempty" yaml:"weight,omitempty" export:"true"`
|
Weight *int `json:"weight,omitempty" toml:"weight,omitempty" yaml:"weight,omitempty" export:"true"`
|
||||||
|
|
|
@ -285,6 +285,27 @@ func (in *ErrorPage) DeepCopy() *ErrorPage {
|
||||||
return out
|
return out
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||||
|
func (in *Failover) DeepCopyInto(out *Failover) {
|
||||||
|
*out = *in
|
||||||
|
if in.HealthCheck != nil {
|
||||||
|
in, out := &in.HealthCheck, &out.HealthCheck
|
||||||
|
*out = new(HealthCheck)
|
||||||
|
**out = **in
|
||||||
|
}
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
// DeepCopy is an autogenerated deepcopy function, copying the receiver, creating a new Failover.
|
||||||
|
func (in *Failover) DeepCopy() *Failover {
|
||||||
|
if in == nil {
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
out := new(Failover)
|
||||||
|
in.DeepCopyInto(out)
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
// DeepCopyInto is an autogenerated deepcopy function, copying the receiver, writing into out. in must be non-nil.
|
||||||
func (in *ForwardAuth) DeepCopyInto(out *ForwardAuth) {
|
func (in *ForwardAuth) DeepCopyInto(out *ForwardAuth) {
|
||||||
*out = *in
|
*out = *in
|
||||||
|
@ -1171,6 +1192,11 @@ func (in *Service) DeepCopyInto(out *Service) {
|
||||||
*out = new(Mirroring)
|
*out = new(Mirroring)
|
||||||
(*in).DeepCopyInto(*out)
|
(*in).DeepCopyInto(*out)
|
||||||
}
|
}
|
||||||
|
if in.Failover != nil {
|
||||||
|
in, out := &in.Failover, &out.Failover
|
||||||
|
*out = new(Failover)
|
||||||
|
(*in).DeepCopyInto(*out)
|
||||||
|
}
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
||||||
|
|
|
@ -69,6 +69,8 @@ func Test_buildConfiguration(t *testing.T) {
|
||||||
"traefik/http/services/Service03/weighted/services/0/weight": "42",
|
"traefik/http/services/Service03/weighted/services/0/weight": "42",
|
||||||
"traefik/http/services/Service03/weighted/services/1/name": "foobar",
|
"traefik/http/services/Service03/weighted/services/1/name": "foobar",
|
||||||
"traefik/http/services/Service03/weighted/services/1/weight": "42",
|
"traefik/http/services/Service03/weighted/services/1/weight": "42",
|
||||||
|
"traefik/http/services/Service04/failover/service": "foobar",
|
||||||
|
"traefik/http/services/Service04/failover/fallback": "foobar",
|
||||||
"traefik/http/middlewares/Middleware08/forwardAuth/authResponseHeaders/0": "foobar",
|
"traefik/http/middlewares/Middleware08/forwardAuth/authResponseHeaders/0": "foobar",
|
||||||
"traefik/http/middlewares/Middleware08/forwardAuth/authResponseHeaders/1": "foobar",
|
"traefik/http/middlewares/Middleware08/forwardAuth/authResponseHeaders/1": "foobar",
|
||||||
"traefik/http/middlewares/Middleware08/forwardAuth/authRequestHeaders/0": "foobar",
|
"traefik/http/middlewares/Middleware08/forwardAuth/authRequestHeaders/0": "foobar",
|
||||||
|
@ -688,6 +690,12 @@ func Test_buildConfiguration(t *testing.T) {
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
|
"Service04": {
|
||||||
|
Failover: &dynamic.Failover{
|
||||||
|
Service: "foobar",
|
||||||
|
Fallback: "foobar",
|
||||||
|
},
|
||||||
|
},
|
||||||
},
|
},
|
||||||
},
|
},
|
||||||
TCP: &dynamic.TCPConfiguration{
|
TCP: &dynamic.TCPConfiguration{
|
||||||
|
|
140
pkg/server/service/loadbalancer/failover/failover.go
Normal file
140
pkg/server/service/loadbalancer/failover/failover.go
Normal file
|
@ -0,0 +1,140 @@
|
||||||
|
package failover
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"errors"
|
||||||
|
"net/http"
|
||||||
|
"sync"
|
||||||
|
|
||||||
|
"github.com/traefik/traefik/v2/pkg/config/dynamic"
|
||||||
|
"github.com/traefik/traefik/v2/pkg/log"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Failover is an http.Handler that can forward requests to the fallback handler
|
||||||
|
// when the main handler status is down.
|
||||||
|
type Failover struct {
|
||||||
|
wantsHealthCheck bool
|
||||||
|
handler http.Handler
|
||||||
|
fallbackHandler http.Handler
|
||||||
|
// updaters is the list of hooks that are run (to update the Failover
|
||||||
|
// parent(s)), whenever the Failover status changes.
|
||||||
|
updaters []func(bool)
|
||||||
|
|
||||||
|
handlerStatusMu sync.RWMutex
|
||||||
|
handlerStatus bool
|
||||||
|
|
||||||
|
fallbackStatusMu sync.RWMutex
|
||||||
|
fallbackStatus bool
|
||||||
|
}
|
||||||
|
|
||||||
|
// New creates a new Failover handler.
|
||||||
|
func New(hc *dynamic.HealthCheck) *Failover {
|
||||||
|
return &Failover{
|
||||||
|
wantsHealthCheck: hc != nil,
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// RegisterStatusUpdater adds fn to the list of hooks that are run when the
|
||||||
|
// status of the Failover changes.
|
||||||
|
// Not thread safe.
|
||||||
|
func (f *Failover) RegisterStatusUpdater(fn func(up bool)) error {
|
||||||
|
if !f.wantsHealthCheck {
|
||||||
|
return errors.New("healthCheck not enabled in config for this failover service")
|
||||||
|
}
|
||||||
|
|
||||||
|
f.updaters = append(f.updaters, fn)
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
func (f *Failover) ServeHTTP(w http.ResponseWriter, req *http.Request) {
|
||||||
|
f.handlerStatusMu.RLock()
|
||||||
|
handlerStatus := f.handlerStatus
|
||||||
|
f.handlerStatusMu.RUnlock()
|
||||||
|
|
||||||
|
if handlerStatus {
|
||||||
|
f.handler.ServeHTTP(w, req)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
f.fallbackStatusMu.RLock()
|
||||||
|
fallbackStatus := f.fallbackStatus
|
||||||
|
f.fallbackStatusMu.RUnlock()
|
||||||
|
|
||||||
|
if fallbackStatus {
|
||||||
|
f.fallbackHandler.ServeHTTP(w, req)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
http.Error(w, http.StatusText(http.StatusServiceUnavailable), http.StatusServiceUnavailable)
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetHandler sets the main http.Handler.
|
||||||
|
func (f *Failover) SetHandler(handler http.Handler) {
|
||||||
|
f.handlerStatusMu.Lock()
|
||||||
|
defer f.handlerStatusMu.Unlock()
|
||||||
|
|
||||||
|
f.handler = handler
|
||||||
|
f.handlerStatus = true
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetHandlerStatus sets the main handler status.
|
||||||
|
func (f *Failover) SetHandlerStatus(ctx context.Context, up bool) {
|
||||||
|
f.handlerStatusMu.Lock()
|
||||||
|
defer f.handlerStatusMu.Unlock()
|
||||||
|
|
||||||
|
status := "DOWN"
|
||||||
|
if up {
|
||||||
|
status = "UP"
|
||||||
|
}
|
||||||
|
|
||||||
|
if up == f.handlerStatus {
|
||||||
|
// We're still with the same status, no need to propagate.
|
||||||
|
log.FromContext(ctx).Debugf("Still %s, no need to propagate", status)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
log.FromContext(ctx).Debugf("Propagating new %s status", status)
|
||||||
|
f.handlerStatus = up
|
||||||
|
|
||||||
|
for _, fn := range f.updaters {
|
||||||
|
// Failover service status is set to DOWN
|
||||||
|
// when main and fallback handlers have a DOWN status.
|
||||||
|
fn(f.handlerStatus || f.fallbackStatus)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetFallbackHandler sets the fallback http.Handler.
|
||||||
|
func (f *Failover) SetFallbackHandler(handler http.Handler) {
|
||||||
|
f.fallbackStatusMu.Lock()
|
||||||
|
defer f.fallbackStatusMu.Unlock()
|
||||||
|
|
||||||
|
f.fallbackHandler = handler
|
||||||
|
f.fallbackStatus = true
|
||||||
|
}
|
||||||
|
|
||||||
|
// SetFallbackHandlerStatus sets the fallback handler status.
|
||||||
|
func (f *Failover) SetFallbackHandlerStatus(ctx context.Context, up bool) {
|
||||||
|
f.fallbackStatusMu.Lock()
|
||||||
|
defer f.fallbackStatusMu.Unlock()
|
||||||
|
|
||||||
|
status := "DOWN"
|
||||||
|
if up {
|
||||||
|
status = "UP"
|
||||||
|
}
|
||||||
|
|
||||||
|
if up == f.fallbackStatus {
|
||||||
|
// We're still with the same status, no need to propagate.
|
||||||
|
log.FromContext(ctx).Debugf("Still %s, no need to propagate", status)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
log.FromContext(ctx).Debugf("Propagating new %s status", status)
|
||||||
|
f.fallbackStatus = up
|
||||||
|
|
||||||
|
for _, fn := range f.updaters {
|
||||||
|
// Failover service status is set to DOWN
|
||||||
|
// when main and fallback handlers have a DOWN status.
|
||||||
|
fn(f.handlerStatus || f.fallbackStatus)
|
||||||
|
}
|
||||||
|
}
|
163
pkg/server/service/loadbalancer/failover/failover_test.go
Normal file
163
pkg/server/service/loadbalancer/failover/failover_test.go
Normal file
|
@ -0,0 +1,163 @@
|
||||||
|
package failover
|
||||||
|
|
||||||
|
import (
|
||||||
|
"context"
|
||||||
|
"net/http"
|
||||||
|
"net/http/httptest"
|
||||||
|
"testing"
|
||||||
|
|
||||||
|
"github.com/stretchr/testify/assert"
|
||||||
|
"github.com/stretchr/testify/require"
|
||||||
|
"github.com/traefik/traefik/v2/pkg/config/dynamic"
|
||||||
|
)
|
||||||
|
|
||||||
|
type responseRecorder struct {
|
||||||
|
*httptest.ResponseRecorder
|
||||||
|
save map[string]int
|
||||||
|
sequence []string
|
||||||
|
status []int
|
||||||
|
}
|
||||||
|
|
||||||
|
func (r *responseRecorder) WriteHeader(statusCode int) {
|
||||||
|
r.save[r.Header().Get("server")]++
|
||||||
|
r.sequence = append(r.sequence, r.Header().Get("server"))
|
||||||
|
r.status = append(r.status, statusCode)
|
||||||
|
r.ResponseRecorder.WriteHeader(statusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFailover(t *testing.T) {
|
||||||
|
failover := New(&dynamic.HealthCheck{})
|
||||||
|
|
||||||
|
status := true
|
||||||
|
require.NoError(t, failover.RegisterStatusUpdater(func(up bool) {
|
||||||
|
status = up
|
||||||
|
}))
|
||||||
|
|
||||||
|
failover.SetHandler(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
|
||||||
|
rw.Header().Set("server", "handler")
|
||||||
|
rw.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
|
||||||
|
failover.SetFallbackHandler(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
|
||||||
|
rw.Header().Set("server", "fallback")
|
||||||
|
rw.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
|
||||||
|
recorder := &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
failover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 1, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 0, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
assert.True(t, status)
|
||||||
|
|
||||||
|
failover.SetHandlerStatus(context.Background(), false)
|
||||||
|
|
||||||
|
recorder = &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
failover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 0, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 1, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
assert.True(t, status)
|
||||||
|
|
||||||
|
failover.SetFallbackHandlerStatus(context.Background(), false)
|
||||||
|
|
||||||
|
recorder = &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
failover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 0, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 0, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, []int{503}, recorder.status)
|
||||||
|
assert.False(t, status)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFailoverDownThenUp(t *testing.T) {
|
||||||
|
failover := New(nil)
|
||||||
|
|
||||||
|
failover.SetHandler(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
|
||||||
|
rw.Header().Set("server", "handler")
|
||||||
|
rw.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
|
||||||
|
failover.SetFallbackHandler(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
|
||||||
|
rw.Header().Set("server", "fallback")
|
||||||
|
rw.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
|
||||||
|
recorder := &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
failover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 1, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 0, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
|
||||||
|
failover.SetHandlerStatus(context.Background(), false)
|
||||||
|
|
||||||
|
recorder = &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
failover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 0, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 1, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
|
||||||
|
failover.SetHandlerStatus(context.Background(), true)
|
||||||
|
|
||||||
|
recorder = &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
failover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 1, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 0, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
}
|
||||||
|
|
||||||
|
func TestFailoverPropagate(t *testing.T) {
|
||||||
|
failover := New(&dynamic.HealthCheck{})
|
||||||
|
failover.SetHandler(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
|
||||||
|
rw.Header().Set("server", "handler")
|
||||||
|
rw.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
failover.SetFallbackHandler(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
|
||||||
|
rw.Header().Set("server", "fallback")
|
||||||
|
rw.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
|
||||||
|
topFailover := New(nil)
|
||||||
|
topFailover.SetHandler(failover)
|
||||||
|
topFailover.SetFallbackHandler(http.HandlerFunc(func(rw http.ResponseWriter, req *http.Request) {
|
||||||
|
rw.Header().Set("server", "topFailover")
|
||||||
|
rw.WriteHeader(http.StatusOK)
|
||||||
|
}))
|
||||||
|
err := failover.RegisterStatusUpdater(func(up bool) {
|
||||||
|
topFailover.SetHandlerStatus(context.Background(), up)
|
||||||
|
})
|
||||||
|
require.NoError(t, err)
|
||||||
|
|
||||||
|
recorder := &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
topFailover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 1, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 0, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, 0, recorder.save["topFailover"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
|
||||||
|
failover.SetHandlerStatus(context.Background(), false)
|
||||||
|
|
||||||
|
recorder = &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
topFailover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 0, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 1, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, 0, recorder.save["topFailover"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
|
||||||
|
failover.SetFallbackHandlerStatus(context.Background(), false)
|
||||||
|
|
||||||
|
recorder = &responseRecorder{ResponseRecorder: httptest.NewRecorder(), save: map[string]int{}}
|
||||||
|
topFailover.ServeHTTP(recorder, httptest.NewRequest(http.MethodGet, "/", nil))
|
||||||
|
|
||||||
|
assert.Equal(t, 0, recorder.save["handler"])
|
||||||
|
assert.Equal(t, 0, recorder.save["fallback"])
|
||||||
|
assert.Equal(t, 1, recorder.save["topFailover"])
|
||||||
|
assert.Equal(t, []int{200}, recorder.status)
|
||||||
|
}
|
|
@ -29,7 +29,7 @@ type stickyCookie struct {
|
||||||
// (https://en.wikipedia.org/wiki/Earliest_deadline_first_scheduling)
|
// (https://en.wikipedia.org/wiki/Earliest_deadline_first_scheduling)
|
||||||
// Each pick from the schedule has the earliest deadline entry selected.
|
// Each pick from the schedule has the earliest deadline entry selected.
|
||||||
// Entries have deadlines set at currentDeadline + 1 / weight,
|
// Entries have deadlines set at currentDeadline + 1 / weight,
|
||||||
// providing weighted round robin behavior with floating point weights and an O(log n) pick time.
|
// providing weighted round-robin behavior with floating point weights and an O(log n) pick time.
|
||||||
type Balancer struct {
|
type Balancer struct {
|
||||||
stickyCookie *stickyCookie
|
stickyCookie *stickyCookie
|
||||||
wantsHealthCheck bool
|
wantsHealthCheck bool
|
||||||
|
@ -230,6 +230,7 @@ func (b *Balancer) AddService(name string, handler http.Handler, weight *int) {
|
||||||
if weight != nil {
|
if weight != nil {
|
||||||
w = *weight
|
w = *weight
|
||||||
}
|
}
|
||||||
|
|
||||||
if w <= 0 { // non-positive weight is meaningless
|
if w <= 0 { // non-positive weight is meaningless
|
||||||
return
|
return
|
||||||
}
|
}
|
||||||
|
|
|
@ -23,6 +23,7 @@ import (
|
||||||
"github.com/traefik/traefik/v2/pkg/safe"
|
"github.com/traefik/traefik/v2/pkg/safe"
|
||||||
"github.com/traefik/traefik/v2/pkg/server/cookie"
|
"github.com/traefik/traefik/v2/pkg/server/cookie"
|
||||||
"github.com/traefik/traefik/v2/pkg/server/provider"
|
"github.com/traefik/traefik/v2/pkg/server/provider"
|
||||||
|
"github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/failover"
|
||||||
"github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/mirror"
|
"github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/mirror"
|
||||||
"github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr"
|
"github.com/traefik/traefik/v2/pkg/server/service/loadbalancer/wrr"
|
||||||
"github.com/vulcand/oxy/roundrobin"
|
"github.com/vulcand/oxy/roundrobin"
|
||||||
|
@ -116,6 +117,13 @@ func (m *Manager) BuildHTTP(rootCtx context.Context, serviceName string) (http.H
|
||||||
conf.AddError(err, true)
|
conf.AddError(err, true)
|
||||||
return nil, err
|
return nil, err
|
||||||
}
|
}
|
||||||
|
case conf.Failover != nil:
|
||||||
|
var err error
|
||||||
|
lb, err = m.getFailoverServiceHandler(ctx, serviceName, conf.Failover)
|
||||||
|
if err != nil {
|
||||||
|
conf.AddError(err, true)
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
default:
|
default:
|
||||||
sErr := fmt.Errorf("the service %q does not have any type defined", serviceName)
|
sErr := fmt.Errorf("the service %q does not have any type defined", serviceName)
|
||||||
conf.AddError(sErr, true)
|
conf.AddError(sErr, true)
|
||||||
|
@ -125,6 +133,53 @@ func (m *Manager) BuildHTTP(rootCtx context.Context, serviceName string) (http.H
|
||||||
return lb, nil
|
return lb, nil
|
||||||
}
|
}
|
||||||
|
|
||||||
|
func (m *Manager) getFailoverServiceHandler(ctx context.Context, serviceName string, config *dynamic.Failover) (http.Handler, error) {
|
||||||
|
f := failover.New(config.HealthCheck)
|
||||||
|
|
||||||
|
serviceHandler, err := m.BuildHTTP(ctx, config.Service)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
f.SetHandler(serviceHandler)
|
||||||
|
|
||||||
|
updater, ok := serviceHandler.(healthcheck.StatusUpdater)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("child service %v of %v not a healthcheck.StatusUpdater (%T)", config.Service, serviceName, serviceHandler)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := updater.RegisterStatusUpdater(func(up bool) {
|
||||||
|
f.SetHandlerStatus(ctx, up)
|
||||||
|
}); err != nil {
|
||||||
|
return nil, fmt.Errorf("cannot register %v as updater for %v: %w", config.Service, serviceName, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
fallbackHandler, err := m.BuildHTTP(ctx, config.Fallback)
|
||||||
|
if err != nil {
|
||||||
|
return nil, err
|
||||||
|
}
|
||||||
|
|
||||||
|
f.SetFallbackHandler(fallbackHandler)
|
||||||
|
|
||||||
|
// Do not report the health of the fallback handler.
|
||||||
|
if config.HealthCheck == nil {
|
||||||
|
return f, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
fallbackUpdater, ok := fallbackHandler.(healthcheck.StatusUpdater)
|
||||||
|
if !ok {
|
||||||
|
return nil, fmt.Errorf("child service %v of %v not a healthcheck.StatusUpdater (%T)", config.Fallback, serviceName, fallbackHandler)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := fallbackUpdater.RegisterStatusUpdater(func(up bool) {
|
||||||
|
f.SetFallbackHandlerStatus(ctx, up)
|
||||||
|
}); err != nil {
|
||||||
|
return nil, fmt.Errorf("cannot register %v as updater for %v: %w", config.Fallback, serviceName, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return f, nil
|
||||||
|
}
|
||||||
|
|
||||||
func (m *Manager) getMirrorServiceHandler(ctx context.Context, config *dynamic.Mirroring) (http.Handler, error) {
|
func (m *Manager) getMirrorServiceHandler(ctx context.Context, config *dynamic.Mirroring) (http.Handler, error) {
|
||||||
serviceHandler, err := m.BuildHTTP(ctx, config.Service)
|
serviceHandler, err := m.BuildHTTP(ctx, config.Service)
|
||||||
if err != nil {
|
if err != nil {
|
||||||
|
@ -164,6 +219,7 @@ func (m *Manager) getWRRServiceHandler(ctx context.Context, serviceName string,
|
||||||
}
|
}
|
||||||
|
|
||||||
balancer.AddService(service.Name, serviceHandler, service.Weight)
|
balancer.AddService(service.Name, serviceHandler, service.Weight)
|
||||||
|
|
||||||
if config.HealthCheck == nil {
|
if config.HealthCheck == nil {
|
||||||
continue
|
continue
|
||||||
}
|
}
|
||||||
|
|
|
@ -39,8 +39,9 @@
|
||||||
<div class="text-subtitle2">Main Service</div>
|
<div class="text-subtitle2">Main Service</div>
|
||||||
<q-chip
|
<q-chip
|
||||||
dense
|
dense
|
||||||
class="app-chip app-chip-name">
|
class="app-chip app-chip-name app-chip-overflow">
|
||||||
{{ data.mirroring.service }}
|
{{ data.mirroring.service }}
|
||||||
|
<q-tooltip>{{ data.mirroring.service }}</q-tooltip>
|
||||||
</q-chip>
|
</q-chip>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
@ -80,6 +81,34 @@
|
||||||
</div>
|
</div>
|
||||||
</q-card-section>
|
</q-card-section>
|
||||||
|
|
||||||
|
<q-card-section v-if="data.failover && data.failover.service">
|
||||||
|
<div class="row items-start no-wrap">
|
||||||
|
<div class="col">
|
||||||
|
<div class="text-subtitle2">Main Service</div>
|
||||||
|
<q-chip
|
||||||
|
dense
|
||||||
|
class="app-chip app-chip-name app-chip-overflow">
|
||||||
|
{{ data.failover.service }}
|
||||||
|
<q-tooltip>{{ data.failover.service }}</q-tooltip>
|
||||||
|
</q-chip>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</q-card-section>
|
||||||
|
|
||||||
|
<q-card-section v-if="data.failover && data.failover.fallback">
|
||||||
|
<div class="row items-start no-wrap">
|
||||||
|
<div class="col">
|
||||||
|
<div class="text-subtitle2">Fallback Service</div>
|
||||||
|
<q-chip
|
||||||
|
dense
|
||||||
|
class="app-chip app-chip-name app-chip-overflow">
|
||||||
|
{{ data.failover.fallback }}
|
||||||
|
<q-tooltip>{{ data.failover.fallback }}</q-tooltip>
|
||||||
|
</q-chip>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
</q-card-section>
|
||||||
|
|
||||||
<q-separator v-if="sticky" />
|
<q-separator v-if="sticky" />
|
||||||
<StickyServiceDetails v-if="sticky" :sticky="sticky" :dense="dense"/>
|
<StickyServiceDetails v-if="sticky" :sticky="sticky" :dense="dense"/>
|
||||||
</q-scroll-area>
|
</q-scroll-area>
|
||||||
|
|
Loading…
Reference in a new issue