diff --git a/KUBERNETES.md b/KUBERNETES.md deleted file mode 100644 index 7642b22..0000000 --- a/KUBERNETES.md +++ /dev/null @@ -1,127 +0,0 @@ -# Kubernetes sablier Howto - -# Traefik parameters - -Its important to set allowEmptyServices to true, otherwhise the scale up will -not work because traefik cannot find the service if it was scaled down to zero. - - - "--pilot.token=xxxx" - - "--experimental.plugins.sablier.modulename=github.com/acouvreur/sablier/plugins/traefik" - - "--experimental.plugins.sablier.version=v0.1.1" - - "--providers.kubernetesingress.allowEmptyServices=true" - - If you are using the traefik helm chart its also important to set: - - experimental: - plugins: - enabled: true - -# Deployment - -In this example we will deploy the sablier into the namespace kube-system - - apiVersion: apps/v1 - kind: Deployment - metadata: - name: sablier - namespace: kube-system - labels: - app: sablier - spec: - replicas: 1 - selector: - matchLabels: - app: sablier - template: - metadata: - labels: - app: sablier - spec: - serviceAccountName: sablier - serviceAccount: sablier - containers: - - name: sablier - image: gchr.io/acouvreur/sablier - args: ["--swarmMode=false", "--kubernetesMode=true"] - ports: - - containerPort: 10000 - --- - apiVersion: v1 - kind: Service - metadata: - name: sablier - namespace: kube-system - spec: - selector: - app: sablier - ports: - - protocol: TCP - port: 10000 - targetPort: 10000 - -We have to create RBAC to allow the sablier to access the kubernetes API and get/update/patch the deployment resource - - apiVersion: v1 - kind: ServiceAccount - metadata: - name: sablier - namespace: kube-system - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRole - metadata: - name: sablier - namespace: kube-system - rules: - - apiGroups: - - apps - resources: - - statefulsets - - statefulsets/scale - - deployments - - deployments/scale - verbs: - - patch - - get - - update - --- - apiVersion: rbac.authorization.k8s.io/v1 - kind: ClusterRoleBinding - metadata: - name: sablier - namespace: kube-system - roleRef: - apiGroup: rbac.authorization.k8s.io - kind: ClusterRole - name: sablier - subjects: - - kind: ServiceAccount - name: sablier - namespace: kube-system - -## Creating a Middleware - -In this example we want to scale down the `code-server` deployment in the `codeserverns` namespace -First we need to create a traefik middleware for that: - - apiVersion: traefik.containo.us/v1alpha1 - kind: Middleware - metadata: - name: ondemand-codeserver - namespace: kube-system - spec: - plugin: - sablier: - name: deployment_codeserverns_code-server_1 - serviceUrl: 'http://sablier:10000' - timeout: 10m - -The format of the `name:` section is `___` where `_` is the delimiter. - -`KIND` can be either `deployment` or `statefulset` - -## Using the Middleware - -When using an Ingress (e.g. for code-server) you have to add the middleware in metadata.annotation: - - traefik.ingress.kubernetes.io/router.middlewares: kube-system-ondemand-codeserver@kubernetescrd diff --git a/README.md b/README.md index 573965d..073857c 100644 --- a/README.md +++ b/README.md @@ -1,135 +1,346 @@ -# Sablier +# ⏳ Sablier ![Github Actions](https://img.shields.io/github/workflow/status/acouvreur/sablier/Build?style=flat-square) ![Go Report](https://goreportcard.com/badge/github.com/acouvreur/sablier?style=flat-square) ![Go Version](https://img.shields.io/github/go-mod/go-version/acouvreur/sablier?style=flat-square) ![Latest Release](https://img.shields.io/github/release/acouvreur/sablier/all.svg?style=flat-square) -Sablier is an API that start containers on demand. +Sablier is an API that start containers for a given duration. + It provides an integrations with multiple reverse proxies and different loading strategies. -Sablier is a merge from https://github.com/acouvreur/traefik-ondemand-plugin/ and https://github.com/acouvreur/traefik-ondemand-service/. This repository was renamed to Sablier. - -Because Traefik doesn't support go module v2+ yet, this is re-released starting at v1.0.0 instead of my original plans as 2.0.0. +Which allows you to start your containers on demand and shut them down automatically as soon as there's no activity. ![Hourglass](./docs/img/hourglass.png) -- [Sablier](#sablier) - - [Getting started](#getting-started) - - [Features](#features) - - [CLI Usage](#cli-usage) - - [Configuration](#configuration) +- [⏳ Sablier](#-sablier) + - [⚡️ Quick start](#️-quick-start) + - [⚙️ Configuration](#️-configuration) + - [Dynamic loading](#dynamic-loading) + - [Dynamic Strategy Configuration](#dynamic-strategy-configuration) + - [Custom Themes](#custom-themes) + - [Blocking strategy](#blocking-strategy) - [Reverse proxies integration plugins](#reverse-proxies-integration-plugins) - [Traefik Integration](#traefik-integration) - - [Kubernetes](#kubernetes) - - [API](#api) + - [Traefik with Docker classic](#traefik-with-docker-classic) + - [Traefik with Docker Swarm](#traefik-with-docker-swarm) + - [Traefik with Kubernetes](#traefik-with-kubernetes) + - [Caddy Integration](#caddy-integration) + - [Credits](#credits) -## Getting started - -Binary +## ⚡️ Quick start ```bash +# Create and stop nginx container docker run -d --name nginx nginx docker stop nginx -./sablier start -curl 'http://localhost:10000/?name=nginx&timeout=1m' + +# Create and stop whoami container +docker run -d --name whoami containous/whoami:v1.5.0 +docker stop whoami + +# Start Sablier with the docker provider +docker run -v /var/run/docker.sock:/var/run/docker.sock -p 10000:10000 ghcr.io/acouvreur/sablier:latest --provider.name=docker + +# Start the containers, the request will hang until both containers are up and running +curl 'http://localhost:10000/api/strategies/blocking?names=nginx&names=whoami&session_duration=1m' +[ + { + "Instance": { + "Name": "whoami", + "CurrentReplicas": 1, + "Status": "ready", + "Message": "" + }, + "Error": null + }, + { + "Instance": { + "Name": "nginx", + "CurrentReplicas": 1, + "Status": "ready", + "Message": "" + }, + "Error": null + } +] ``` -Docker +## ⚙️ Configuration -```bash -docker run -d --name nginx nginx -docker stop nginx -docker run -v /var/run/docker.sock:/var/run/docker.sock -p 10000:10000 ghcr.io/acouvreur/sablier:latest --swarmode=false -curl 'http://localhost:10000/?name=nginx&timeout=1m' +| Cli | Yaml file | Environment variable | Default | Description | +| ---------------------------------------------- | -------------------------------------------- | -------------------------------------------- | ----------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- | +| `--provider.name` | `provider.name` | `PROVIDER_NAME` | `docker` | Provider to use to manage containers [docker swarm kubernetes] | +| `--server.base-path` | `server.base-path` | `SERVER_BASE_PATH` | `/` | The base path for the API | +| `--server.port` | `server.port` | `SERVER_PORT` | `10000` | The server port to use | +| `--sessions.default-duration` | `sessions.default-duration` | `SESSIONS_DEFAULT_DURATION` | `5m` | The default session duration | +| `--sessions.expiration-interval` | `sessions.expiration-interval` | `SESSIONS_EXPIRATION_INTERVAL` | `20s` | The expiration checking interval. Higher duration gives less stress on CPU. If you only use sessions of 1h, setting this to 5m is a good trade-off. | +| `--storage.file` | `storage.file` | `STORAGE_FILE` | | File path to save the state | +| `--strategy.blocking.default-timeout` | `strategy.blocking.default-timeout` | `STRATEGY_BLOCKING_DEFAULT_TIMEOUT` | `1m` | Default timeout used for blocking strategy | +| `--strategy.dynamic.custom-themes-path` | `strategy.dynamic.custom-themes-path` | `STRATEGY_DYNAMIC_CUSTOM_THEMES_PATH` | | Custom themes folder, will load all .html files recursively | +| `--strategy.dynamic.default-refresh-frequency` | `strategy.dynamic.default-refresh-frequency` | `STRATEGY_DYNAMIC_DEFAULT_REFRESH_FREQUENCY` | `5s` | Default refresh frequency in the HTML page for dynamic strategy | +| `--strategy.dynamic.default-theme` | `strategy.dynamic.default-theme` | `STRATEGY_DYNAMIC_DEFAULT_THEME` | `hacker-terminal` | Default theme used for dynamic strategy | + +## Dynamic loading + +**The Dynamic Strategy provides a waiting UI with multiple themes.** +This is best suited when this interaction is made through a browser. + +| Name | Preview | +| :---------------: | :-------------------------------------------------: | +| `ghost` | [![ghost](./docs/img/ghost.png) | +| `shuffle` | [![shuffle](./docs/img/shuffle.png) | +| `hacker-terminal` | [![hacker-terminal](./docs/img/hacker-terminal.png) | +| `matrix` | [![matrix](./docs/img/matrix.png) | + +### Dynamic Strategy Configuration + +| Cli | Yaml file | Environment variable | Default | Description | +| ---------------------------------------------- | -------------------------------------------- | -------------------------------------------- | ----------------- | --------------------------------------------------------------- | +| strategy | +| `--strategy.dynamic.custom-themes-path` | `strategy.dynamic.custom-themes-path` | `STRATEGY_DYNAMIC_CUSTOM_THEMES_PATH` | | Custom themes folder, will load all .html files recursively | +| `--strategy.dynamic.default-refresh-frequency` | `strategy.dynamic.default-refresh-frequency` | `STRATEGY_DYNAMIC_DEFAULT_REFRESH_FREQUENCY` | `5s` | Default refresh frequency in the HTML page for dynamic strategy | +| `--strategy.dynamic.default-theme` | `strategy.dynamic.default-theme` | `STRATEGY_DYNAMIC_DEFAULT_THEME` | `hacker-terminal` | Default theme used for dynamic strategy | + +### Custom Themes + +Use `--strategy.dynamic.custom-themes-path` to specify the folder containing your themes. + +Your theme will be rendered using a Go Template structure such as : + +```go +type TemplateValues struct { + DisplayName string + InstanceStates []RenderOptionsInstanceState + SessionDuration string + RefreshFrequency string + Version string +} ``` -## Features - -- Support for **Docker** containers -- Support for **Docker Swarm mode**, scale services -- Support for **Kubernetes** Deployments and Statefulsets -- Start your container/service on the first request -- Automatic **scale to zero** after configured timeout upon last request the service received -- Dynamic loading page (cloudflare or grafana cloud style) -- Customize dynamic and loading pages - -## CLI Usage - -``` -Usage: - sablier [command] - -Available Commands: - completion Generate the autocompletion script for the specified shell - help Help about any command - start Start the Sablier server - version Print the version Sablier - -Flags: - -h, --help help for sablier - -Use "sablier [command] --help" for more information about a command. +```go +type RenderOptionsInstanceState struct { + Name string + CurrentReplicas int + DesiredReplicas int + Status string + Error error +} ``` -Start options +- ⚠️ IMPORTANT ⚠️ You should always use `RefreshFrequency` like this: + ```html + + ... + + ... + + ``` + This will refresh the loaded page automatically every `RefreshFrequency`. +- You **cannot** load new themes added in the folder without restarting +- You **can** modify the existing themes files +- Why? Because we build a theme whitelist in order to prevent malicious payload crafting by using `theme=../../very_secret.txt` +- Custom themes **must end** with `.html` +- You can load themes by specifying their name and their relative path from the `--strategy.dynamic.custom-themes-path` value. + ```bash + /my/custom/themes/ + ├── custom1.html # custom1 + ├── custom2.html # custom2 + └── special + └── secret.html # special/secret + ``` +You can see the available themes from the API: ``` -Start the Sablier server - -Usage: - sablier start [flags] - -Flags: - -h, --help help for start - --provider.name string Provider to use to manage containers [docker swarm kubernetes] (default "docker") - --server.base-path string The base path for the API (default "/") - --server.port int The server port to use (default 10000) - --storage.file string File path to save the state +> curl 'http://localhost:10000/api/strategies/dynamic/themes' ``` - -## Configuration - -Sablier can be configured in that order: - -1. command line arguments -2. environment variable -3. config.yaml file - -```yaml -server: - port: 10000 - basePath: / -storage: - file: -provider: - name: docker # available providers are docker, swarm and kubernetes +```json +{ + "custom": [ + "custom" + ], + "embedded": [ + "ghost", + "hacker-terminal", + "matrix", + "shuffle" + ] +} ``` +## Blocking strategy + +**The Blocking Strategy waits for the instances to load before serving the request** +This is best suited when this interaction from an API. ## Reverse proxies integration plugins +- [Traefik](#traefik-integration) +- [Caddy]() + ### Traefik Integration -see [Traefik Integration](./plugins/traefik/README.md) +1. Add this snippet in the Traefik Static configuration - -## Kubernetes - -see [KUBERNETES.md](https://github.com/acouvreur/sablier/blob/main/KUBERNETES.md) - -## API - -``` -GET :10000/?name=&timeout= +```yaml +experimental: + plugins: + sablier: + moduleName: "github.com/acouvreur/sablier" + version: "v1.0.0" ``` -| Query param | Type | Description | -| ----------- | --------------- | ----------------------------------------------------------------------- | -| `name` | `string` | The docker container name, or the swarm service name | -| `timeout` | `time.Duration` | The duration after which the container/service will be scaled down to 0 | +2. Configure the plugin using the Dynamic Configuration. Example: -| Body | Status code | Description | -| ---------- | ------------ | ------------------------------------------------------------------------------ | -| `started` | 202 Created | The container/service is available | -| `starting` | 201 Accepted | The container/service has been scheduled for starting but is not yet available | +```yaml +http: + middlewares: + my-sablier: + plugin: + sablier: + sablierUrl: http://sablier:10000 + names: whoami,nginx # comma separated names + sessionDuration: 1m + # Dynamic strategy, provides the waiting webui + dynamic: + displayName: My Title + theme: hacker-terminal + # Blocking strategy, waits until services are up and running + # but will not wait more than `timeout` + blocking: + timeout: 1m +``` + +Or for Kubernetes CRD + +```yaml +apiVersion: traefik.containo.us/v1alpha1 +kind: Middleware +metadata: + name: my-sablier + namespace: my-namespace +spec: + plugin: + sablier: + sablierUrl: http://sablier:10000 + names: whoami,nginx # comma separated names + sessionDuration: 1m + # Dynamic strategy, provides the waiting webui + dynamic: + displayName: My Title + theme: hacker-terminal + # Blocking strategy, waits until services are up and running + # but will not wait more than `timeout` + blocking: + timeout: 1m +``` + +You can also checkout the End to End tests here: [plugins/traefik/e2e](./plugins/traefik/e2e/). + +#### Traefik with Docker classic + +⚠️ Limitations + +- Traefik will evict the container from its pool if it's `exited`. You must use the dynamic configuration. + +*docker-compose.yml* +```yaml +version: "3.9" + +services: + traefik: + image: traefik:2.9.1 + command: + - --entryPoints.http.address=:80 + - --providers.docker=true + - --providers.file.filename=/etc/traefik/dynamic-config.yml + - --experimental.plugins.sablier.moduleName=github.com/acouvreur/sablier/plugins/traefik + - --experimental.plugins.sablier.version=v1.0.0 + ports: + - "8080:80" + volumes: + - '/var/run/docker.sock:/var/run/docker.sock' + - './dynamic-config.yml:/etc/traefik/dynamic-config.yml' + + sablier: + image: ghcr.io/acouvreur/sablier:local + command: + - start + - --provider.name=docker + volumes: + - '/var/run/docker.sock:/var/run/docker.sock' + labels: + - traefik.enable=true + # Dynamic Middleware + - traefik.http.middlewares.dynamic.plugin.sablier.names=sablier-whoami-1 + - traefik.http.middlewares.dynamic.plugin.sablier.sablierUrl=http://sablier:10000 + - traefik.http.middlewares.dynamic.plugin.sablier.dynamic.sessionDuration=1m + - traefik.http.middlewares.dynamic.plugin.sablier.dynamic.displayName=Dynamic Whoami + + whoami: + image: containous/whoami:v1.5.0 + # Cannot use labels because as soon as the container is stopped, the labels are not treated by Traefik + # The route doesn't exist anymore. Use dynamic-config.yml file instead. + # labels: + # - traefik.enable + # - traefik.http.routers.whoami-dynamic.rule=PathPrefix(`/dynamic/whoami`) + # - traefik.http.routers.whoami-dynamic.middlewares=dynamic@docker +``` + +*dynamic-config.yaml* +```yaml +http: + services: + whoami: + loadBalancer: + servers: + - url: "http://whoami:80" + + routers: + whoami-dynamic: + rule: PathPrefix(`/dynamic/whoami`) + entryPoints: + - "http" + middlewares: + - dynamic@docker + service: "whoami" +``` + +#### Traefik with Docker Swarm + +- The value from the `names` section will do a strict match if possible, if it is not found it will match by suffix only if there's one match. + - `names=nginx` matches `nginx` from `MYSTACK_nginx` and `nginx` services + - `names=nginx` matches `MYSTACK_nginx` from `MYSTACK_nginx` and `nginx-2` services + +⚠️ Limitations + +- Traefik will evict the service from its pool as soon as the service is 0/0. You must add the [`traefik.docker.lbswarm`](https://doc.traefik.io/traefik/routing/providers/docker/#traefikdockerlbswarm) label. + ```yaml + services: + whoami: + image: containous/whoami:v1.5.0 + deploy: + replicas: 0 + labels: + - traefik.docker.lbswarm=true + ``` +- We cannot use [allowEmptyServices](https://doc.traefik.io/traefik/providers/docker/#allowemptyservices) because if you use the [blocking strategy](LINKHERE) you will receive a `503`. +- Replicas is set to 1 + +#### Traefik with Kubernetes + +- The format of the `names` section is `___` where `_` is the delimiter. + - Thus no `_` are allowed in `` +- `KIND` can be either `deployment` or `statefulset` + +⚠️ Limitations + +- Traefik will evict the service from its pool as soon as there is no endpoint available. You must use [`allowEmptyServices`](https://doc.traefik.io/traefik/providers/kubernetes-ingress/#allowemptyservices) +- Blocking Strategy is not yet supported because of how Traefik handles the pod ip. + +See [Kubernetes E2E Traefik Test script](./plugins/traefik/e2e/kubernetes.sh) to see how it is reproduced + +### Caddy Integration + +TODO ## Credits -[Hourglass icons created by Vectors Market - Flaticon](https://www.flaticon.com/free-icons/hourglass) \ No newline at end of file +- [Hourglass icons created by Vectors Market - Flaticon](https://www.flaticon.com/free-icons/hourglass) +- [tarampampam/error-pages](https://github.com/tarampampam/error-pages/) for the themes \ No newline at end of file diff --git a/docs/img/ghost.png b/docs/img/ghost.png new file mode 100644 index 0000000..67108e5 Binary files /dev/null and b/docs/img/ghost.png differ diff --git a/docs/img/hacker-terminal.png b/docs/img/hacker-terminal.png new file mode 100644 index 0000000..3e52216 Binary files /dev/null and b/docs/img/hacker-terminal.png differ diff --git a/docs/img/matrix.png b/docs/img/matrix.png new file mode 100644 index 0000000..e464b93 Binary files /dev/null and b/docs/img/matrix.png differ diff --git a/docs/img/shuffle.png b/docs/img/shuffle.png new file mode 100644 index 0000000..e4234bc Binary files /dev/null and b/docs/img/shuffle.png differ diff --git a/plugins/traefik/README.md b/plugins/traefik/README.md index 7780287..56b7225 100644 --- a/plugins/traefik/README.md +++ b/plugins/traefik/README.md @@ -1,99 +1,29 @@ # Traefik Sablier Plugin -Traefik middleware to start containers on demand. +## Plugin -![Demo](./img/ondemand.gif) - -## Usage - -### Plugin configuration - -#### Strategies - -**Dynamic Strategy (default)** - -_Serve an HTML page that self reload._ - -```yml -testData: - serviceUrl: http://sablier:10000 - name: whoami - timeout: 1m - waitui: true -``` - -**Blocking Strategy** - -Blocking strategy is enabled by setting `waitui` to `false`. - -Instead of displaying a self refreshing page, the request hangs until the service is ready to receive the request. - -The timeout is set by `blockdelay`. - -```yml -testData: - serviceUrl: http://sablier:10000 - name: whoami - timeout: 1m - waitui: false - blockdelay: 1m -``` - -*Typical use case: an API calling another API* - -#### Custom loading/error pages - -The `loadingpage` and `errorpage` keys in the plugin configuration can be used to override the default loading and error pages. - -The value should be a path where a template that can be parsed by Go's [html/template](https://pkg.go.dev/html/template) package can be found in the Traefik container. - -An example of both a loading page and an error page template can be found in the [pkg/pages/](pkg/pages/) directory in [loading.html](pkg/pages/loading.html) and [error.html](pkg/pages/error.html) respectively. - -The plugin will default to the built-in loading and error pages if these fields are omitted. - -You must include `` inside your html page to get auto refresh. - -**Example Configuration** - -```yml -testData: - serviceUrl: http://sablier:10000 - name: whoami - timeout: 1m - waitui: false - blockdelay: 1m - loadingpage: /etc/traefik/plugins/sablier/custompages/loading.html - errorpage: /etc/traefik/plugins/sablier/custompages/error.html -``` - -| Parameter | Type | Default | Required | Example | Description | -| ------------- | --------------- | ------- | -------- | ----------------------------------------------------------------------- | ------------------------------------------------------------------------------------- | -| `serviceUrl` | `string` | empty | yes | `http://sablier:10000` | The docker container name, or the swarm service name | -| `name` | `string` | empty | yes (except if `names` is set) | `whoami` | The container/service/kubernetes resource to be stopped (docker ps docker service ls) | -| `names` | `[]string` | [] | yes (except if `name` is set) | `[whoami-1, whoami-2]` | The containers/services to be stopped (docker ps docker service ls) | -| `timeout` | `time.Duration` | `1m` | no | `1m30s` | The duration after which the container/service will be scaled down to 0 | -| `waitui` | `bool` | `true` | no | `true` | Serves a self-refreshing html page when the service is scaled down to 0 | -| `displayname` | `string` | `the middleware name` | no | `My App` | Serves a self-refreshing html page when the service is scaled down to 0 | -| `blockdelay` | `time.Duration` | `1m` | no | `1m30s` | When `waitui` is `false`, wait for the service to be scaled up before `blockdelay` | -| `loadingpage` | `string` | empty | no | `/etc/traefik/plugins/sablier/custompages/loading.html` | The path in the traefik container for the **loading** page template | -| `errorpage` | `string` | empty | no | `/etc/traefik/plugins/sablier/custompages/error.html` | The path in the traefik container for the **error** page template | - -### sablier - -The [sablier](https://github.com/acouvreur/sablier) must be used to bypass [Yaegi](https://github.com/traefik/yaegi) limitations. - -Yaegi is the interpreter used by Traefik to load plugin and run them at runtime. - -The docker library that interacts with the docker deamon uses `unsafe` which must be specified when instanciating Yaegi. Traefik doesn't, and probably never will by default. - -## Examples - -- [Docker Classic](./examples/docker_classic/) -- [Docker Swarm](./examples/docker_swarm/) -- [Multiple Containers](./examples/multiple_containers/) -- [Kubernetes](./examples/kubernetes/) +The plugin is available in the Traefik [Plugin Catalog](https://plugins.traefik.io/plugins/633b4658a4caa9ddeffda119/sablier) ## Development -`export TRAEFIK_PILOT_TOKEN=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx` -`docker stack deploy -c docker-compose.yml DEV` +You can use this to load the plugin. + +```yaml +version: "3.7" + +services: + traefik: + image: traefik:2.9.1 + command: + - --experimental.localPlugins.sablier.moduleName=github.com/acouvreur/sablier + - --entryPoints.http.address=:80 + - --providers.docker=true + ports: + - "8080:80" + volumes: + - '/var/run/docker.sock:/var/run/docker.sock' + - '../../..:/plugins-local/src/github.com/acouvreur/sablier' + - './dynamic-config.yml:/etc/traefik/dynamic-config.yml' +``` + +But I recommend you to use the [`e2e`](./e2e/) folder. \ No newline at end of file