Update doc

README.md

@@ -16,6 +16,8 @@
 
 **Diun** :bell: is a CLI application written in [Go](https://golang.org/) to receive notifications :inbox_tray: when a Docker :whale: image is updated on a Docker registry. With Go, this app can be used across many platforms :game_die: and architectures. This support includes Linux, FreeBSD, macOS and Windows on architectures like amd64, i386, ARM and others.
 
+![](.res/screenshot.png)
+
 ## Features
 
 * Allow to watch a full Docker repository and report new tags
@@ -23,11 +25,12 @@
 * Internal cron implementation through go routines
 * Worker pool to parallelize analyses
 * Allow overriding image os and architecture
+* Multi providers available like [Docker](doc/providers/docker.md), [Swarm](doc/providers/swarm.md), [Static](doc/providers/static.md)...
 * Beautiful email report
 * Webhook notification
 * Enhanced logging
 * Timezone can be changed
-* :whale: Official [Docker image available](doc/install/docker.md)
+* Official [Docker image available](doc/install/docker.md)
 
 ## Documentation
 
@@ -37,8 +40,11 @@
   * [Linux service](doc/install/linux-service.md)
 * [Usage](doc/usage.md)
 * [Configuration](doc/configuration.md)
+* Providers
+  * [Docker](doc/providers/docker.md)
+  * [Swarm](doc/providers/swarm.md)
+  * [Static](doc/providers/static.md)
 * [Notifications](doc/notifications.md)
-* [TODO](doc/todo.md)
 
 ## How can I help ?
 

doc/configuration.md

@@ -40,7 +40,20 @@ regopts:
     password: bar2
     insecure_tls: true
 
-image:
+providers:
+  docker:
+    # Watch all containers on local Docker engine
+    local:
+      watch_stopped: true
+    # Watch only labeled containers on 10.0.0.1:2375
+    remote:
+      endpoint: tcp://10.0.0.1:2375
+      watch_by_default: true
+  swarm:
+    # Watch only labeled services on your local Docker Swarm
+    myswarm:
+      watch_by_default: true
+  static:
     # Watch latest tag of crazymax/nextcloud image on docker.io (DockerHub) with registry ID 'someregistryoptions'.
     - name: docker.io/crazymax/nextcloud:latest
       regopts_id: someregistryoptions
@@ -72,18 +85,15 @@ image:
 
 ## db
 
-* `db`
 * `path`: Path to Bolt database file where images manifests are stored (default: `diun.db`). Environment var `DIUN_DB` override this value.
 
 ## watch
 
-* `watch`
 * `workers`: Maximum number of workers that will execute tasks concurrently. _Optional_. (default: `10`).
 * `schedule`: [CRON expression](https://godoc.org/github.com/robfig/cron#hdr-CRON_Expression_Format) to schedule Diun watcher. _Optional_. (default: `0 * * * *`).
 
 ## notif
 
-* `notif`
 * `mail`
   * `enable`: Enable email reports (default: `false`).
   * `host`: SMTP server host (default: `localhost`). **required**
@@ -91,9 +101,12 @@ image:
   * `ssl`: SSL defines whether an SSL connection is used. Should be false in most cases since the auth mechanism should use STARTTLS (default: `false`).
   * `insecure_skip_verify`: Controls whether a client verifies the server's certificate chain and hostname (default: `false`).
   * `username`: SMTP username.
+  * `username_file`: Use content of secret file as SMTP username if `username` not defined.
   * `password`: SMTP password.
+  * `password_file`: Use content of secret file as SMTP password if `password` not defined.
   * `from`: Sender email address. **required**
   * `to`: Recipient email address. **required**
+
 * `webhook`
   * `enable`: Enable webhook notification (default: `false`).
   * `endpoint`: URL of the HTTP request. **required**
@@ -103,15 +116,33 @@ image:
 
 ## regopts
 
-* `regopts`: Map of registry options to use with images. Key is the ID and value is a struct with the following fields:
 * `username`: Registry username.
+* `username_file`: Use content of secret file as registry username if `username` not defined.
 * `password`: Registry password.
+* `password_file`: Use content of secret file as registry password if `password` not defined.
 * `timeout`: Timeout is the maximum amount of time for the TCP connection to establish. 0 means no timeout (default: `10`).
 * `insecure_tls`: Allow contacting docker registry over HTTP, or HTTPS with failed TLS verification (default: `false`).
 
-## image
+## providers
 
-* `image`: Slice of image to watch with the following fields:
+* `docker`: Map of Docker standalone engines to watch
+  * `<key>`: An unique identifier for this provider.
+    * `endpoint`: Server address to connect to. Local if empty. _Optional_
+    * `api_version`: Overrides the client version with the specified one. _Optional_
+    * `tls_certs_path`: Path to load the TLS certificates from. _Optional_
+    * `tls_verify`: Controls whether client verifies the server's certificate chain and hostname (default: `true`).
+    * `watch_by_default`: Enable watch by default. If false, containers that don't have `diun.enable=true` label will be ignored (default: `false`).
+    * `watch_stopped`: Include created and exited containers too (default: `false`).
+
+* `swarm`: Map of Docker Swarm to watch
+  * `<key>`: An unique identifier for this provider.
+    * `endpoint`: Server address to connect to. Local if empty. _Optional_
+    * `api_version`: Overrides the client version with the specified one. _Optional_
+    * `tls_certs_path`: Path to load the TLS certificates from. _Optional_
+    * `tls_verify`: Controls whether client verifies the server's certificate chain and hostname (default: `true`).
+    * `watch_by_default`: Enable watch by default. If false, services that don't have `diun.enable=true` label will be ignored (default: `false`).
+
+* `static`: Slice of static image to watch
   * `name`: Docker image name to watch using `registry/path:tag` format. If registry is omitted, `docker.io` will be used and if tag is omitted, `latest` will be used. **required**
   * `os`: OS to use. _Optional_. (default: `linux`).
   * `arch`: Architecture to use. _Optional_. (default: `amd64`).

doc/notifications.md

@@ -1,11 +1,23 @@
 # Notifications
 
+* [Mail](#mail)
+* [Webhook](#webhook)
+
+## Mail
+
+Here is an email sample if you add `mail` notification:
+
+![](../.res/notif-mail.png)
+
+## Webhook
+
 If you choose `webhook` notification, a HTTP request is sent with a JSON format response that looks like:
 
 ```json
 {
   "diun_version": "0.3.0",
   "status": "new",
+  "provider": "static-0",
   "image": "docker.io/crazymax/swarm-cronjob:0.2.1",
   "mime_type": "application/vnd.docker.distribution.manifest.v2+json",
   "digest": "sha256:5913d4b5e8dc15430c2f47f40e43ab2ca7f2b8df5eee5db4d5c42311e08dfb79",
@@ -14,7 +26,3 @@ If you choose `webhook` notification, a HTTP request is sent with a JSON format
   "os": "linux"
 }
 ```
-
-And here is an email sample if you add `mail` notification:
-
-![](../.res/notif-mail.png)

doc/providers/docker.md

@@ -0,0 +1,102 @@
+# Docker provider
+
+* [About](#about)
+* [Quick start](#quick-start)
+* [Configuration](#configuration)
+
+## About
+
+The Docker provider allows you to analyze the containers of your standalone Docker instance defined in the [Diun configuration](../configuration.md) to extract the images found and check for updates on the registry.
+
+## Quick start
+
+In this section we quickly go over a basic docker-compose file using your local docker provider.
+
+First of all, let's create a Diun configuration we named `diun.yml`:
+
+```yml
+watch:
+  workers: 20
+  schedule: "*/30 * * * *"
+
+providers:
+  docker:
+    mydocker:
+      watch_stopped: true
+```
+
+Here we use a single Docker provider with a minimum configuration to analyze labeled containers (watch by default disabled), even stopped ones, of your local Docker instance.
+
+Now let's create a simple docker-compose file with Diun and some simple services:
+
+```yml
+version: "3.5"
+
+services:
+  diun:
+    image: crazymax/diun:latest
+    volumes:
+      - "./data:/data"
+      - "./diun.yml:/diun.yml:ro"
+      - "/var/run/docker.sock:/var/run/docker.sock"
+    environment:
+      - "TZ=Europe/Paris"
+      - "LOG_LEVEL=info"
+      - "LOG_JSON=false"
+    restart: always
+
+  cloudflared:
+    image: crazymax/cloudflared:latest
+    ports:
+      - target: 5053
+        published: 5053
+        protocol: udp
+      - target: 49312
+        published: 49312
+        protocol: tcp
+    environment:
+      - "TZ=Europe/Paris"
+      - "TUNNEL_DNS_UPSTREAM=https://1.1.1.1/dns-query,https://1.0.0.1/dns-query"
+    labels:
+      - "diun.enable=true"
+      - "diun.watch_repo=true"
+    restart: always
+```
+
+As an example we use [crazymax/cloudflared:latest](https://github.com/crazy-max/docker-cloudflared) Docker image. A few [labels](#configuration) are added to configure the image analysis of this container for Diun. Now start this composition with `docker-composes up -d` and take a look at the logs:
+
+```
+$ docker-compose logs -f
+Attaching to bin_diun_1, cloudflared
+cloudflared    | time="2019-12-14T15:30:07+01:00" level=info msg="Adding DNS upstream" url="https://1.1.1.1/dns-query"
+cloudflared    | time="2019-12-14T15:30:07+01:00" level=info msg="Adding DNS upstream" url="https://1.0.0.1/dns-query"
+cloudflared    | time="2019-12-14T15:30:07+01:00" level=info msg="Starting metrics server" addr="[::]:49312"
+cloudflared    | time="2019-12-14T15:30:07+01:00" level=info msg="Starting DNS over HTTPS proxy server" addr="dns://0.0.0.0:5053"
+diun_1         | Sat, 14 Dec 2019 15:30:07 CET INF Starting Diun v2.0.0
+diun_1         | Sat, 14 Dec 2019 15:30:07 CET INF Found 1 docker provider(s) to analyze...
+diun_1         | Sat, 14 Dec 2019 15:30:10 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:latest provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.9.0 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.9.1 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.9.2 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.10.1 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.10.4 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.10.2 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.11.0 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:12 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.11.3 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:13 CET INF New image found id=mydocker image=docker.io/crazymax/cloudflared:2019.11.2 provider=docker
+diun_1         | Sat, 14 Dec 2019 15:30:13 CET INF Cron initialized with schedule */30 * * * *
+diun_1         | Sat, 14 Dec 2019 15:30:13 CET INF Next run in 29 minutes (2019-12-14 16:00:00 +0100 CET)
+```
+
+## Configuration
+
+In the same spirit as the [static provider](static.md), you can configure more finely the way to analyze the image of your container. But unlike the static provider, this is done via Docker labels:
+
+* `diun.enable`: Set to true to enable image analysis of this container. Required if `watch_by_default` is disabled for this provider.
+* `diun.os`: OS to use. _Optional_. (default: `linux`).
+* `diun.arch`: Architecture to use. _Optional_. (default: `amd64`).
+* `diun.regopts_id`: Registry options ID from [`regopts`](../configuration.md#regopts) to use.
+* `diun.watch_repo`: Watch all tags of this container image (default: `false`).
+* `diun.max_tags`: Maximum number of tags to watch if `diun.watch_repo` enabled. 0 means all of them (default: `0`).
+* `diun.include_tags`: Semi-colon separated list of regular expressions to include tags. Can be useful if you enable `diun.watch_repo`.
+* `diun.exclude_tags`: Semi-colon separated list of regular expressions to exclude tags. Can be useful if you enable `diun.watch_repo`.

doc/providers/static.md

@@ -0,0 +1,54 @@
+# Static provider
+
+* [About](#about)
+* [Quick start](#quick-start)
+
+## About
+
+The static provider is the most basic way to analyse Docker images. Nothing special to see here as everything is configured through the [providers field](../configuration.md#providers).
+
+## Quick start
+
+But let's take a look with a simple example:
+
+```yml
+db:
+  path: diun.db
+
+watch:
+  workers: 20
+  schedule: "* * * * *"
+
+regopts:
+  jfrog:
+    username: foo
+    password: bar
+
+providers:
+  static:
+    - name: crazymax/cloudflared
+      watch_repo: true
+    - name: docker.bintray.io/jfrog/xray-mongo:3.2.6
+      regopts_id: jfrog
+```
+
+Here we want to analyze all tags of `crazymax/cloudflared` and `docker.bintray.io/jfrog/xray-mongo:3.2.6` tag. Now let's start Diun:
+
+```
+$ diun --config diun.yml
+Sat, 14 Dec 2019 15:32:23 UTC INF Starting Diun 2.0.0
+Sat, 14 Dec 2019 15:32:23 UTC INF Found 2 static provider(s) to analyze...
+Sat, 14 Dec 2019 15:32:25 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:latest provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.11.3 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.11.0 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.10.1 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.9.0 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.9.2 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.10.2 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.11.2 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.9.1 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=crazymax/cloudflared image=docker.io/crazymax/cloudflared:2019.10.4 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF New image found id=docker.bintray.io/jfrog/xray-mongo:3.2.6 image=docker.bintray.io/jfrog/xray-mongo:3.2.6 provider=static
+Sat, 14 Dec 2019 15:32:28 UTC INF Cron initialized with schedule * * * * *
+Sat, 14 Dec 2019 15:32:28 UTC INF Next run in 31 seconds (2019-12-14 15:33:00 +0000 UTC)
+```

doc/providers/swarm.md

@@ -0,0 +1,108 @@
+# Swarm provider
+
+* [About](#about)
+* [Quick start](#quick-start)
+* [Configuration](#configuration)
+
+## About
+
+The Swarm provider is closely linked to the [Docker provider](docker.md) except that it allows you to analyze the services of your Swarm cluster defined in the [Diun configuration](../configuration.md) to extract the images found and check for updates on the registry.
+
+## Quick start
+
+In this section we quickly go over a basic stack using your local swarm cluster.
+
+First of all, let's create a Diun configuration we named `diun.yml`:
+
+```yml
+watch:
+  workers: 20
+  schedule: "*/30 * * * *"
+
+providers:
+  swarm:
+    myswarm:
+```
+
+Here we use a single Swarm provider with a minimum configuration to analyze labeled containers (watch by default disabled), of your local Swarm cluster.
+
+Now let's create a simple stack for Diun:
+
+```yml
+version: "3.5"
+
+services:
+  diun:
+    image: crazymax/diun:latest
+    volumes:
+      - "./data:/data"
+      - "./diun.yml:/diun.yml:ro"
+      - "/var/run/docker.sock:/var/run/docker.sock"
+    environment:
+      - "TZ=Europe/Paris"
+      - "LOG_LEVEL=info"
+      - "LOG_JSON=false"
+    deploy:
+      placement:
+        constraints:
+          - node.role == manager
+```
+
+And another one with a simple service:
+
+```yml
+version: "3.5"
+
+services:
+  nginx:
+    image: nginx
+    ports:
+      - target: 80
+        published: 80
+        protocol: udp
+    deploy:
+      mode: replicated
+      replicas: 2
+      labels:
+        - "diun.enable=true"
+        - "diun.watch_repo=true"
+```
+
+As an example we use [nginx](https://hub.docker.com/_/nginx/) Docker image. A few [labels](#configuration) are added to configure the image analysis of this service for Diun. We can now start these 2 stacks:
+
+```
+docker stack deploy -c diun.yml diun
+docker stack deploy -c nginx.yml nginx
+```
+
+And watch logs of Diun service:
+
+```
+$ docker service logs -f diun_diun
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:19:57 CET INF Starting Diun dev
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:19:57 CET INF Starting Diun...
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:19:57 CET INF Found 1 swarm provider(s) to analyze...
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:19:59 CET INF New image found id=myswarm image=docker.io/library/nginx:latest provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:01 CET INF New image found id=myswarm image=docker.io/library/nginx:1.9 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:01 CET INF New image found id=myswarm image=docker.io/library/nginx:1.9.4 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:01 CET INF New image found id=myswarm image=docker.io/library/nginx:1.9.8 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:01 CET INF New image found id=myswarm image=docker.io/library/nginx:1.9.7 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:01 CET INF New image found id=myswarm image=docker.io/library/nginx:1.9.9 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:01 CET INF New image found id=myswarm image=docker.io/library/nginx:1.9.6 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:01 CET INF New image found id=myswarm image=docker.io/library/nginx:1.9.5 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:mainline-alpine provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:alpine-perl provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:stable-perl provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:stable-alpine-perl provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:1 provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:perl provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:mainline-alpine-perl provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:stable provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:mainline-perl provider=swarm
+diun_diun.1.i1l4yuiafq6y@docker-desktop    | Sat, 14 Dec 2019 16:20:02 CET INF New image found id=myswarm image=docker.io/library/nginx:mainline provider=swarm
+...
+```
+
+## Configuration
+
+You can configure more finely the way to analyze the image of your service as for the [Docker provider](docker.md) with [Docker labels](docker.md#configuration).

doc/todo.md

@@ -1,5 +0,0 @@
-# TODO
-
-* [ ] Watch images inside Dockerfile and Compose files
-* [ ] Watch images from Docker daemon
-* [ ] Watch starred repo on DockerHub and Quay