diff --git a/README.md b/README.md index 2918618..526cace 100644 --- a/README.md +++ b/README.md @@ -13,21 +13,22 @@ This will build a container for backing up multiple types of DB Servers -Currently backs up CouchDB, InfluxDB, MySQL, Microsoft SQL, MongoDB, Postgres, Redis servers. +Backs up CouchDB, InfluxDB, MySQL, Microsoft SQL, MongoDB, Postgres, Redis servers. - dump to local filesystem or backup to S3 Compatible services, and Azure. -- select database user and password -- backup all databases, single, or multiple databases -- backup all to separate files or one singular file -- choose to have an MD5 or SHA1 sum after backup for verification -- delete old backups after specific amount of time -- choose compression type (none, gz, bz, xz, zstd) -- connect to any container running on the same system -- Script to perform restores -- Zabbix Monitoring capabilities -- select how often to run a dump -- select when to start the first dump, whether time of day or relative to container start time -- Execute script after backup for monitoring/alerting purposes +- multiple backup job support + - selectable when to start the first dump, whether time of day or relative to container start time + - selectable interval + - selectable omit scheduling during periods of time + - selectable database user and password + - selectable cleanup and archive capabilities + - selectable database name support - all databases, single, or multiple databases + - backup all to separate files or one singular file +- checksum support choose to have an MD5 or SHA1 hash generated after backup for verification +- compression support (none, gz, bz, xz, zstd) +- Zabbix Metrics support +- Hooks to execute pre and post backup job for customization purposes +- Companion script to aid in restores ## Maintainer @@ -49,20 +50,56 @@ Currently backs up CouchDB, InfluxDB, MySQL, Microsoft SQL, MongoDB, Postgres, R - [Environment Variables](#environment-variables) - [Base Images used](#base-images-used) - [Container Options](#container-options) - - [Database Specific Options](#database-specific-options) - - [For Influx DB2](#for-influx-db2) - - [Scheduling Options](#scheduling-options) - - [Backup Options](#backup-options) - - [Backing Up to S3 Compatible Services](#backing-up-to-s3-compatible-services) - - [Upload to a Azure storage account by `blobxfer`](#upload-to-a-azure-storage-account-by-blobxfer) + - [Job Defaults](#job-defaults) + - [Compression Options](#compression-options) + - [Scheduling Options](#scheduling-options) + - [Default Database Options](#default-database-options) + - [CouchDB](#couchdb) + - [InfluxDB](#influxdb) + - [MariaDB/MySQL](#mariadbmysql) + - [Microsoft SQL](#microsoft-sql) + - [MongoDB](#mongodb) + - [Postgresql](#postgresql) + - [Redis](#redis) + - [Default Storage Options](#default-storage-options) + - [Filesystem](#filesystem) + - [S3](#s3) + - [Azure](#azure) + - [Hooks](#hooks) + - [Path Options](#path-options) + - [Pre Backup](#pre-backup) + - [Post backup](#post-backup) + - [Job Backup Options](#job-backup-options) + - [Compression Options](#compression-options-1) + - [Scheduling Options](#scheduling-options-1) + - [Specific Database Options](#specific-database-options) + - [CouchDB](#couchdb-1) + - [InfluxDB](#influxdb-1) + - [MariaDB/MySQL](#mariadbmysql-1) + - [Microsoft SQL](#microsoft-sql-1) + - [MongoDB](#mongodb-1) + - [Postgresql](#postgresql-1) + - [Redis](#redis-1) + - [SQLite](#sqlite) + - [Specific Storage Options](#specific-storage-options) + - [Filesystem](#filesystem-1) + - [S3](#s3-1) + - [Azure](#azure-1) + - [Hooks](#hooks-1) + - [Path Options](#path-options-1) + - [Pre Backup](#pre-backup-1) + - [Post backup](#post-backup-1) + - [Notifications](#notifications) + - [Custom Notifications](#custom-notifications) + - [Email Notifications](#email-notifications) + - [Matrix Notifications](#matrix-notifications) + - [Mattermost Notifications](#mattermost-notifications) + - [Rocketchat Notifications](#rocketchat-notifications) - [Maintenance](#maintenance) - [Shell Access](#shell-access) - [Manual Backups](#manual-backups) + - [Manual Backups](#manual-backups-1) - [Restoring Databases](#restoring-databases) - - [Custom Scripts](#custom-scripts) - - [Path Options](#path-options) - - [Pre Backup](#pre-backup) - - [Post backup](#post-backup) - [Support](#support) - [Usage](#usage) - [Bugfixes](#bugfixes) @@ -70,8 +107,6 @@ Currently backs up CouchDB, InfluxDB, MySQL, Microsoft SQL, MongoDB, Postgres, R - [Updates](#updates) - [License](#license) -> **NOTE**: If you are using this with a docker-compose file along with a seperate SQL container, take care not to set the variables to backup immediately, more so have it delay execution for a minute, otherwise you will get a failed first backup. - ## Prerequisites and Assumptions - You must have a working connection to one of the supported DB Servers and appropriate credentials @@ -110,11 +145,10 @@ Images are built primarily for `amd64` architecture, and may also include builds ### Quick Start -- The quickest way to get started is using [docker-compose](https://docs.docker.com/compose/). See the examples folder for a working [docker-compose.yml](examples/docker-compose.yml) that can be modified for development or production use. +- The quickest way to get started is using [docker-compose](https://docs.docker.com/compose/). See the examples folder for a series of example compose.yml that can be modified for development or production use. - Set various [environment variables](#environment-variables) to understand the capabilities of this image. - Map [persistent storage](#data-volumes) for access to configuration and data files for backup. -- Make [networking ports](#networking) available for public access if necessary ### Persistent Storage @@ -139,119 +173,571 @@ Be sure to view the following repositories to understand all the customizable op #### Container Options -| Parameter | Description | Default | -| ----------------------- | -------------------------------------------------------------------------------------------------------------------------------- | --------------- | -| `BACKUP_LOCATION` | Backup to `FILESYSTEM`, `blobxfer` or `S3` compatible services like S3, Minio, Wasabi | `FILESYSTEM` | -| `MODE` | `AUTO` mode to use internal scheduling routines or `MANUAL` to simply use this as manual backups only executed by your own means | `AUTO` | -| `MANUAL_RUN_FOREVER` | `TRUE` or `FALSE` if you wish to try to make the container exit after the backup | `TRUE` | +| Parameter | Description | Default | +| ------------------------ | -------------------------------------------------------------------------------------------------------------------------------- | --------------- | +| `MODE` | `AUTO` mode to use internal scheduling routines or `MANUAL` to simply use this as manual backups only executed by your own means | `AUTO` | +| `USER_DBBACKUP` | The uid that the image should read and write files as (username is `dbbackup`) | `10000` | +| `GROUP_DBBACKUP` | The gid that the image should read and write files as (groupname is `dbbackup`) | `10000` | +| `LOG_PATH` | Path to log files | `/logs` | +| `TEMP_PATH` | Perform Backups and Compression in this temporary directory | `/tmp/backups/` | +| `MANUAL_RUN_FOREVER` | `TRUE` or `FALSE` if you wish to try to make the container exit after the backup | `TRUE` | +| `DEBUG_MODE` | If set to `true`, print copious shell script messages to the container log. Otherwise only basic messages are printed. | `FALSE` | +| `BACKUP_JOB_CONCURRENCY` | How many backup jobs to run concurrently | `1` | -| `DEBUG_MODE` | If set to `true`, print copious shell script messages to the container log. Otherwise only basic messages are printed. | `FALSE` | -| `LOG_PATH` | Path to log files | `/logs` | -| `LOG_FILE` | Logfile name | `dbbackup.log` | -| `CREATE_LATEST_SYMLINK` | Create a symbolic link pointing to last backup in this format: `latest-(DB_TYPE)-(DB_NAME)-(DB_HOST)` | `TRUE` | -| `PRE_SCRIPT` | Fill this variable in with a command to execute pre backing up | | -| `POST_SCRIPT` | Fill this variable in with a command to execute post backing up | | -| `USER_DBBACKUP` | The uid that the image should read and write files as (username is `dbbackup`) | `10000` | -| `GROUP_DBBACKUP` | The gid that the image should read and write files as (groupname is `dbbackup`) | `10000` | -| `SPLIT_DB` | For each backup, create a new archive. `TRUE` or `FALSE` (MySQL and Postgresql Only) | `TRUE` | -| `TEMP_LOCATION` | Perform Backups and Compression in this temporary directory | `/tmp/backups/` | +#### Job Defaults +If these are set and no other defaults or variables are set explicitly, they will be added to any of the backup jobs. -### Database Specific Options +| Variable | Description | Default | +| --------------------------------- | ------------------------------------------------------------------------------------- | ------------ | +| `DEFAULT_BACKUP_LOCATION` | Backup to `FILESYSTEM`, `blobxfer` or `S3` compatible services like S3, Minio, Wasabi | `FILESYSTEM` | +| `DEFAULT_CHECKSUM` | Either `MD5` or `SHA1` or `NONE` | `MD5` | +| `DEFAULT_LOG_LEVEL` | Log output on screen and in files `INFO` `NOTICE` `ERROR` `WARN` `DEBUG` | `notice` | +| `DEFAULT_SKIP_AVAILABILITY_CHECK` | Before backing up - skip connectivity check | `FALSE` | -| Parameter | Description | Default | `_FILE` | -| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- | ------- | -| `DB_AUTH` | (Mongo/PGSQL Only - Optional) Authentication Database | | | -| `DB_TYPE` | Type of DB Server to backup `couch` `influx` `mysql` `mssql` `pgsql` `mongo` `redis` `sqlite3` | | | -| `DB_HOST` | Server Hostname e.g. `mariadb`. For `sqlite3`, full path to DB file e.g. `/backup/db.sqlite3` | | x | -| `DB_NAME` | Schema Name e.g. `database` or `ALL` to backup all databases the user has access to. Backup multiple by separating with commas eg `db1,db2` | | x | -| `DB_NAME_EXCLUDE` | If using `ALL` - use this as to exclude databases separated via commas from being backed up | | x | -| `DB_USER` | username for the database(s) - Can use `root` for MySQL | | x | -| `DB_PASS` | (optional if DB doesn't require it) password for the database | | x | -| `DB_PORT` | (optional) Set port to connect to DB_HOST. Defaults are provided | varies | x | -| `INFLUX_VERSION` | What Version of Influx are you backing up from `1`.x or `2` series - AMD64 and ARM64 only for `2` | | | -| `MONGO_CUSTOM_URI` | If you wish to override the MongoDB Connection string enter it here e.g. `mongodb+srv://username:password@cluster.id.mongodb.net` | | x | -| | This environment variable will be parsed and populate the `DB_NAME` and `DB_HOST` variables to properly build your backup filenames. You can override them by making your own entries | | | +##### Compression Options -#### For Influx DB2 +| Variable | Description | Default | +| -------------------------------------- | ---------------------------------------------------------------------------------------------- | -------------- | +| `DEFAULT_COMPRESSION` | Use either Gzip `GZ`, Bzip2 `BZ`, XZip `XZ`, ZSTD `ZSTD` or none `NONE` | `ZSTD` | +| `DEFAULT_COMPRESSION_LEVEL` | Numerical value of what level of compression to use, most allow `1` to `9` | `3` | +| | except for `ZSTD` which allows for `1` to `19` | | +| `DEFAULT_GZ_RSYNCABLE` | Use `--rsyncable` (gzip only) for faster rsync transfers and incremental backup deduplication. | `FALSE` | +| `DEFAULT_ENABLE_PARALLEL_COMPRESSION` | Use multiple cores when compressing backups `TRUE` or `FALSE` | `TRUE` | +| `DEFAULT_PARALLEL_COMPRESSION_THREADS` | Maximum amount of threads to use when compressing - Integer value e.g. `8` | `autodetected` | -Your Organization will be mapped to `DB_USER` and your root token will need to be mapped to `DB_PASS`. You may use `DB_NAME=ALL` to backup the entire set of databases. For `DB_HOST` use syntax of `http(s)://db-name` +##### Scheduling Options -### Scheduling Options +| Variable | Description | Default | +| ----------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------- | +| `DEFAULT_BACKUP_INTERVAL` | How often to do a backup, in minutes after the first backup. Defaults to 1440 minutes, or once per day. | `1440` | +| `DEFAULT_BACKUP_BEGIN` | What time to do the initial backup. Defaults to immediate. (`+1`) | `+0` | +| | Must be in one of two formats: | | +| | Absolute HHMM, e.g. `2330` or `0415` | | +| | Relative +MM, i.e. how many minutes after starting the container, e.g. `+0` (immediate), `+10` (in 10 minutes), or `+90` in an hour and a half | | +| `DEFAULT_CLEANUP_TIME` | Value in minutes to delete old backups (only fired when backup interval executes) | `FALSE` | +| | 1440 would delete anything above 1 day old. You don't need to set this variable if you want to hold onto everything. | | +| `DEFAULT_ARCHIVE_TIME` | Value in minutes to move all files files older than (x) from | `DEFAULT_BACKUP_BLACKOUT_BEGIN` | Use `HHMM` notation to start a blackout period where no backups occur eg `0420` | | +| `DEFAULT_BACKUP_BLACKOUT_END` | Use `HHMM` notation to set the end period where no backups occur eg `0430` | | -| Parameter | Description | Default | -| --------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------- | -| `DB_DUMP_FREQ` | How often to do a dump, in minutes after the first backup. Defaults to 1440 minutes, or once per day. | `1440` | -| `DB_DUMP_BEGIN` | What time to do the first dump. Defaults to immediate. Must be in one of two formats | | -| | Absolute HHMM, e.g. `2330` or `0415` | | -| | Relative +MM, i.e. how many minutes after starting the container, e.g. `+0` (immediate), `+10` (in 10 minutes), or `+90` in an hour and a half | | -| `DB_DUMP_TARGET` | Directory where the database dumps are kept. | `${DB_DUMP_TARGET}/archive/` | -| `DB_DUMP_TARGET_PERMISSION` | Directory and File permissions to apply to files. | `700` | -| `DB_DUMP_TARGET_ARCHIVE` | Optional Directory where the database dumps archives are kept. | | -| `DB_CLEANUP_TIME` | Value in minutes to delete old backups (only fired when dump frequency fires). 1440 would delete anything above 1 day old. You don't need to set this variable if you want to hold onto everything. | `FALSE` | -| `DB_ARCHIVE_TIME` | Value in minutes to move all files files older than (x) from `DB_DUMP_TARGET` to `DB_DUMP_TARGET_ARCHIVE` - which is useful when pairing against an external backup system. | | +> You may need to wrap your `DEFAULT_BACKUP_BEGIN` value in quotes for it to properly parse. There have been reports of backups that start with a `0` get converted into a different format which will not allow the timer to start at the correct time. -- You may need to wrap your `DB_DUMP_BEGIN` value in quotes for it to properly parse. There have been reports of backups that start with a `0` get converted into a different format which will not allow the timer to start at the correct time. -### Backup Options +##### Default Database Options -| Parameter | Description | Default | `_FILE` | -| ------------------------------ | --------------------------------------------------------------------------------------------------------------------------- | ------------------------- | ------- | -| `COMPRESSION` | Use either Gzip `GZ`, Bzip2 `BZ`, XZip `XZ`, ZSTD `ZSTD` or none `NONE` | `ZSTD` | | -| `COMPRESSION_LEVEL` | Numerical value of what level of compression to use, most allow `1` to `9` except for `ZSTD` which allows for `1` to `19` - | `3` | | -| `ENABLE_PARALLEL_COMPRESSION` | Use multiple cores when compressing backups `TRUE` or `FALSE` | `TRUE` | | -| `PARALLEL_COMPRESSION_THREADS` | Maximum amount of threads to use when compressing - Integer value e.g. `8` | `autodetected` | | -| `GZ_RSYNCABLE` | Use `--rsyncable` (gzip only) for faster rsync transfers and incremental backup deduplication. e.g. `TRUE` | `FALSE` | | -| `ENABLE_CHECKSUM` | Generate either a MD5 or SHA1 in Directory, `TRUE` or `FALSE` | `TRUE` | | -| `CHECKSUM` | Either `MD5` or `SHA1` | `MD5` | | -| `EXTRA_OPTS` | If you need to pass extra arguments to the backup and database enumeration command, add them here e.g. `--extra-command` | | | -| `EXTRA_DUMP_OPTS` | If you need to pass extra arguments to the backup command only, add them here e.g. `--extra-command` | | | -| `EXTRA_ENUMERATION_OPTS` | If you need to pass extra arguments to the database enumeration command only, add them here e.g. `--extra-command` | | | -| `MYSQL_MAX_ALLOWED_PACKET` | Max allowed packet if backing up MySQL / MariaDB | `512M` | | -| `MYSQL_SINGLE_TRANSACTION` | Backup in a single transaction with MySQL / MariaDB | `TRUE` | | -| `MYSQL_STORED_PROCEDURES` | Backup stored procedures with MySQL / MariaDB | `TRUE` | | -| `MYSQL_ENABLE_TLS` | Enable TLS functionality for MySQL client | `FALSE` | | -| `MYSQL_TLS_VERIFY` | (optional) If using TLS (by means of MYSQL_TLS_* variables) verify remote host | `FALSE` | | -| `MYSQL_TLS_VERSION` | What TLS `v1.1` `v1.2` `v1.3` version to utilize | `TLSv1.1,TLSv1.2,TLSv1.3` | | -| `MYSQL_TLS_CA_FILE` | Filename to load custom CA certificate for connecting via TLS | `/etc/ssl/cert.pem` | x | -| `MYSQL_TLS_CERT_FILE` | Filename to load client certificate for connecting via TLS | | x | -| `MYSQL_TLS_KEY_FILE` | Filename to load client key for connecting via TLS | | x | +###### CouchDB -- When using compression with MongoDB, only `GZ` compression is possible. +| Variable | Description | Default | `_FILE` | +| -------------- | ------------ | ------- | ------- | +| `DEFAULT_PORT` | CouchDB Port | `5984` | x | -#### Backing Up to S3 Compatible Services +###### InfluxDB -If `BACKUP_LOCATION` = `S3` then the following options are used. +| Variable | Description | Default | `_FILE` | +| ------------------------ | ------------------------------------------------------------------------------------------------------- | ------- | ------- | +| `DEFAULT_PORT` | InfluxDB Port | | x | +| | Version 1.x | `8088` | | +| | Version 2.x | `8086` | | +| `DEFAULT_INFLUX_VERSION` | What Version of Influx are you backing up from `1`.x or `2` series - amd64 and aarch/armv8 only for `2` | `2` | | -| Parameter | Description | Default | `_FILE` | -| --------------------- | ----------------------------------------------------------------------------------------- | ------- | ------- | -| `S3_BUCKET` | S3 Bucket name e.g. `mybucket` | | x | -| `S3_KEY_ID` | S3 Key ID (Optional) | | x | -| `S3_KEY_SECRET` | S3 Key Secret (Optional) | | x | -| `S3_PATH` | S3 Pathname to save to (must NOT end in a trailing slash e.g. '`backup`') | | x | -| `S3_REGION` | Define region in which bucket is defined. Example: `ap-northeast-2` | | x | -| `S3_HOST` | Hostname (and port) of S3-compatible service, e.g. `minio:8080`. Defaults to AWS. | | x | -| `S3_PROTOCOL` | Protocol to connect to `S3_HOST`. Either `http` or `https`. Defaults to `https`. | `https` | x | -| `S3_EXTRA_OPTS` | Add any extra options to the end of the `aws-cli` process execution | | x | -| `S3_CERT_CA_FILE` | Map a volume and point to your custom CA Bundle for verification e.g. `/certs/bundle.pem` | | x | -| _*OR*_ | | | | -| `S3_CERT_SKIP_VERIFY` | Skip verifying self signed certificates when connecting | `TRUE` | | +###### MariaDB/MySQL -- When `S3_KEY_ID` and/or `S3_KEY_SECRET` is not set, will try to use IAM role assigned (if any) for uploading the backup files to S3 bucket. +| Variable | Description | Default | `_FILE` | +| ---------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------- | ------- | +| `DEFAULT_PORT` | MariaDB Port | `3306` | x | +| `DEFAULT_EXTRA_BACKUP_OPTS` | Pass extra arguments to the backup command only, add them here e.g. `--extra-command` | | | +| `DEFAULT_EXTRA_ENUMERATION_OPTS` | Pass extra arguments to the database enumeration command only, add them here e.g. `--extra-command` | | | +| `DEFAULT_EXTRA_OPTS` | Pass extra arguments to the backup and database enumeration command, add them here e.g. `--extra-command` | | | +| `DEFAULT_MYSQL_MAX_ALLOWED_PACKET` | Max allowed packet if backing up MySQL / MariaDB | `512M` | | +| `DEFAULT_MYSQL_SINGLE_TRANSACTION` | Backup in a single transaction with MySQL / MariaDB | `TRUE` | | +| `DEFAULT_MYSQL_STORED_PROCEDURES` | Backup stored procedures with MySQL / MariaDB | `TRUE` | | +| `DEFAULT_MYSQL_ENABLE_TLS` | Enable TLS functionality for MySQL/MariaDB client | `FALSE` | | +| `DEFAULT_MYSQL_TLS_VERIFY` | (optional) If using TLS (by means of MYSQL_TLS_* variables) verify remote host | `FALSE` | | +| `DEFAULT_MYSQL_TLS_VERSION` | What TLS `v1.1` `v1.2` `v1.3` version to utilize | `TLSv1.1,TLSv1.2,TLSv1.3` | | +| `DEFAULT_MYSQL_TLS_CA_FILE` | Filename to load custom CA certificate for connecting via TLS | `/etc/ssl/cert.pem` | x | +| `DEFAULT_MYSQL_TLS_CERT_FILE` | Filename to load client certificate for connecting via TLS | | x | +| `DEFAULT_MYSQL_TLS_KEY_FILE` | Filename to load client key for connecting via TLS | | x | -#### Upload to a Azure storage account by `blobxfer` -Support to upload backup files with [blobxfer](https://github.com/Azure/blobxfer) to the Azure file share storage. +###### Microsoft SQL -If `BACKUP_LOCATION` = `blobxfer` then the following options are used. +| Variable | Description | Default | `_FILE` | +| -------------- | ------------------ | ------- | ------- | +| `DEFAULT_PORT` | Microsoft SQL Port | `1433` | x | -| Parameter | Description | Default | `_FILE` | -| ------------------------------ | ------------------------------------------- | ------------------- | ------- | -| `BLOBXFER_STORAGE_ACCOUNT` | Microsoft Azure Cloud storage account name. | | x | -| `BLOBXFER_STORAGE_ACCOUNT_KEY` | Microsoft Azure Cloud storage account key. | | x | -| `BLOBXFER_REMOTE_PATH` | Remote Azure path | `/docker-db-backup` | x | +###### MongoDB -> This service uploads files from backup targed directory `DB_DUMP_TARGET`. -> If the a cleanup configuration in `DB_CLEANUP_TIME` is defined, the remote directory on Azure storage will also be cleaned automatically. +| Variable | Description | Default | `_FILE` | +| ------------------ | ------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------- | +| `DEFAULT_AUTH` | (Optional) Authentication Database | | x | +| `DEFAULT_PORT` | MongoDB Port | `27017` | x | +| `MONGO_CUSTOM_URI` | If you wish to override the MongoDB Connection string enter it here e.g. `mongodb+srv://username:password@cluster.id.mongodb.net` | | x | +| | This environment variable will be parsed and populate the `DB_NAME` and `DB_HOST` variables to properly build your backup filenames. | | | +| | You can override them by making your own entries | | | + +###### Postgresql + +| Variable | Description | Default | `_FILE` | +| -------------------------------- | --------------------------------------------------------------------------------------------------------- | ------- | ------- | +| `DEFAULT_AUTH` | (Optional) Authentication Database | | x | +| `DEFAULT_EXTRA_BACKUP_OPTS` | Pass extra arguments to the backup command only, add them here e.g. `--extra-command` | | | +| `DEFAULT_EXTRA_ENUMERATION_OPTS` | Pass extra arguments to the database enumeration command only, add them here e.g. `--extra-command` | | | +| `DEFAULT_EXTRA_OPTS` | Pass extra arguments to the backup and database enumeration command, add them here e.g. `--extra-command` | | | +| `DEFAULT_PORT` | PostgreSQL Port | `5432` | x | + +###### Redis + +| Variable | Description | Default | `_FILE` | +| -------------------------------- | --------------------------------------------------------------------------------------------------- | ------- | ------- | +| `DEFAULT_PORT` | Default Redis Port | `6379` | x | +| `DEFAULT_EXTRA_ENUMERATION_OPTS` | Pass extra arguments to the database enumeration command only, add them here e.g. `--extra-command` | | | + + +##### Default Storage Options + +Options that are related to the value of `DEFAULT_BACKUP_LOCATION` + +###### Filesystem + +If `DEFAULT_BACKUP_LOCTION` = `FILESYSTEM` then the following options are used: + +| Variable | Description | Default | +| --------------------------------- | ----------------------------------------------------------------------------------------------------- | ------------------------------------- | +| `DEFAULT_CREATE_LATEST_SYMLINK` | Create a symbolic link pointing to last backup in this format: `latest-(DB_TYPE)-(DB_NAME)-(DB_HOST)` | `TRUE` | +| `DEFAULT_FILESYSTEM_PATH` | Directory where the database dumps are kept. | `/backup` | +| `DEFAULT_FILESYSTEM_ARCHIVE_PATH` | Optional Directory where the database dumps archives are kept | `${DEFAULT_FILESYSTEM_PATH}/archive/` | +| `DEFAULT_FILESYSTEM_PERMISSION` | Directory and File permissions to apply to files. | `700` | + +###### S3 + +If `DEFAULT_BACKUP_LOCATION` = `S3` then the following options are used: + +| Parameter | Description | Default | `_FILE` | +| ----------------------------- | ----------------------------------------------------------------------------------------- | ------- | ------- | +| `DEFAULT_S3_BUCKET` | S3 Bucket name e.g. `mybucket` | | x | +| `DEFAULT_S3_KEY_ID` | S3 Key ID (Optional) | | x | +| `DEFAULT_S3_KEY_SECRET` | S3 Key Secret (Optional) | | x | +| `DEFAULT_S3_PATH` | S3 Pathname to save to (must NOT end in a trailing slash e.g. '`backup`') | | x | +| `DEFAULT_S3_REGION` | Define region in which bucket is defined. Example: `ap-northeast-2` | | x | +| `DEFAULT_S3_HOST` | Hostname (and port) of S3-compatible service, e.g. `minio:8080`. Defaults to AWS. | | x | +| `DEFAULT_S3_PROTOCOL` | Protocol to connect to `DEFAULT_S3_HOST`. Either `http` or `https`. Defaults to `https`. | `https` | x | +| `DEFAULT_S3_EXTRA_OPTS` | Add any extra options to the end of the `aws-cli` process execution | | x | +| `DEFAULT_S3_CERT_CA_FILE` | Map a volume and point to your custom CA Bundle for verification e.g. `/certs/bundle.pem` | | x | +| _*OR*_ | | | | +| `DEFAULT_S3_CERT_SKIP_VERIFY` | Skip verifying self signed certificates when connecting | `TRUE` | | + +- When `DEFAULT_S3_KEY_ID` and/or `DEFAULT_S3_KEY_SECRET` is not set, will try to use IAM role assigned (if any) for uploading the backup files to S3 bucket. + +###### Azure + +If `DEFAULT_BACKUP_LOCATION` = `blobxfer` then the following options are used:. + +| Parameter | Description | Default | `_FILE` | +| -------------------------------------- | ------------------------------------------- | ------------------- | ------- | +| `DEFAULT_BLOBXFER_STORAGE_ACCOUNT` | Microsoft Azure Cloud storage account name. | | x | +| `DEFAULT_BLOBXFER_STORAGE_ACCOUNT_KEY` | Microsoft Azure Cloud storage account key. | | x | +| `DEFAULT_BLOBXFER_REMOTE_PATH` | Remote Azure path | `/docker-db-backup` | x | + +> This service uploads files from backup targed directory `DEFAULT_FILESYSTEM_PATH`. +> If the a cleanup configuration in `DEFAULT_CLEANUP_TIME` is defined, the remote directory on Azure storage will also be cleaned automatically. + +##### Hooks + +###### Path Options + +| Parameter | Description | Default | +| ------------------------------ | --------------------------------------------------------------------------- | ----------------------- | +| `DEFAULT_SCRIPT_LOCATION_PRE` | Location on filesystem inside container to execute bash scripts pre backup | `/assets/scripts/pre/` | +| `DEFAULT_SCRIPT_LOCATION_POST` | Location on filesystem inside container to execute bash scripts post backup | `/assets/scripts/post/` | +| `DEFAULT_PRE_SCRIPT` | Fill this variable in with a command to execute pre backing up | | +| `DEFAULT_POST_SCRIPT` | Fill this variable in with a command to execute post backing up | | + +###### Pre Backup + +If you want to execute a custom script before a backup starts, you can drop bash scripts with the extension of `.sh` in the location defined in `DB01_SCRIPT_LOCATION_PRE`. See the following example to utilize: + +```bash +$ cat pre-script.sh +##!/bin/bash + +# #### Example Pre Script +# #### $1=DBXX_TYPE (Type of Backup) +# #### $2=DBXX_HOST (Backup Host) +# #### $3=DBXX_NAME (Name of Database backed up +# #### $4=BACKUP START TIME (Seconds since Epoch) +# #### $5=BACKUP FILENAME (Filename) + +echo "${1} Backup Starting on ${2} for ${3} at ${4}. Filename: ${5}" +``` + + ## script DBXX_TYPE DBXX_HOST DBXX_NAME STARTEPOCH BACKUP_FILENAME + ${f} "${backup_job_db_type}" "${backup_job_db_host}" "${backup_job_db_name}" "${backup_routines_start_time}" "${target}" + +Outputs the following on the console: + +`mysql Backup Starting on example-db for example at 1647370800. Filename: mysql_example_example-db_202200315-000000.sql.bz2 + +###### Post backup + +If you want to execute a custom script at the end of a backup, you can drop bash scripts with the extension of `.sh` in the location defined in `DB01_SCRIPT_LOCATION_POST`. Also to support legacy users `/assets/custom-scripts` is also scanned and executed.See the following example to utilize: + +```bash +$ cat post-script.sh +##!/bin/bash + +# #### Example Post Script +# #### $1=EXIT_CODE (After running backup routine) +# #### $2=DBXX_TYPE (Type of Backup) +# #### $3=DBXX_HOST (Backup Host) +# #### #4=DBXX_NAME (Name of Database backed up +# #### $5=BACKUP START TIME (Seconds since Epoch) +# #### $6=BACKUP FINISH TIME (Seconds since Epoch) +# #### $7=BACKUP TOTAL TIME (Seconds between Start and Finish) +# #### $8=BACKUP FILENAME (Filename) +# #### $9=BACKUP FILESIZE +# #### $10=HASH (If CHECKSUM enabled) +# #### $11=MOVE_EXIT_CODE + +echo "${1} ${2} Backup Completed on ${3} for ${4} on ${5} ending ${6} for a duration of ${7} seconds. Filename: ${8} Size: ${9} bytes MD5: ${10}" +``` + + ## script EXIT_CODE DB_TYPE DB_HOST DB_NAME STARTEPOCH FINISHEPOCH DURATIONEPOCH BACKUP_FILENAME FILESIZE CHECKSUMVALUE + ${f} "${exit_code}" "${dbtype}" "${backup_job_db_host}" "${backup_job_db_name}" "${backup_routines_start_time}" "${backup_routines_finish_time}" "${backup_routines_total_time}" "${target}" "${FILESIZE}" "${checksum_value}" "${move_exit_code} + +Outputs the following on the console: + +`0 mysql Backup Completed on example-db for example on 1647370800 ending 1647370920 for a duration of 120 seconds. Filename: mysql_example_example-db_202200315-000000.sql.bz2 Size: 7795 bytes Hash: 952fbaafa30437494fdf3989a662cd40 0` + +If you wish to change the size value from bytes to megabytes set environment variable `DB01_SIZE_VALUE=megabytes` + +You must make your scripts executable otherwise there is an internal check that will skip trying to run it otherwise. +If for some reason your filesystem or host is not detecting it right, use the environment variable `DB01_POST_SCRIPT_SKIP_X_VERIFY=TRUE` to bypass. + + +#### Job Backup Options + +If `DEFAULT_` variables are set and you do not wish for the settings to carry over into your jobs, you can set the appropriate environment variable with the value of `unset`. +Otherwise, override them per backup job. Additional backup jobs can be scheduled by using `DB02_`,`DB03_`,`DB04_` ... prefixes. See [Specific Database Options](#specific-database-options) which may overrule this list. + +| Parameter | Description | Default | `_FILE` | +| ----------- | ---------------------------------------------------------------------------------------------- | ------- | ------- | +| `DB01_TYPE` | Type of DB Server to backup `couch` `influx` `mysql` `mssql` `pgsql` `mongo` `redis` `sqlite3` | | | +| `DB01_HOST` | Server Hostname e.g. `mariadb`. For `sqlite3`, full path to DB file e.g. `/backup/db.sqlite3` | | x | +| `DB01_NAME` | Schema Name e.g. `database` | | x | +| `DB01_USER` | username for the database(s) - Can use `root` for MySQL | | x | +| `DB01_PASS` | (optional if DB doesn't require it) password for the database | | x | +| | | + +| Variable | Description | Default | +| ------------------------------ | --------------------------------------------------------------------------------------------------------- | ------------ | +| `DB01_BACKUP_LOCATION` | Backup to `FILESYSTEM`, `blobxfer` or `S3` compatible services like S3, Minio, Wasabi | `FILESYSTEM` | +| `DB01_CHECKSUM` | Either `MD5` or `SHA1` or `NONE` | `MD5` | +| `DB01_EXTRA_BACKUP_OPTS` | Pass extra arguments to the backup command only, add them here e.g. `--extra-command` | | +| `DB01_EXTRA_ENUMERATION_OPTS` | Pass extra arguments to the database enumeration command only, add them here e.g. `--extra-command` | | +| `DB01_EXTRA_OPTS` | Pass extra arguments to the backup and database enumeration command, add them here e.g. `--extra-command` | | +| `DB01_LOG_LEVEL` | Log output on screen and in files `INFO` `NOTICE` `ERROR` `WARN` `DEBUG` | `debug` | +| `DB01_SKIP_AVAILABILITY_CHECK` | Before backing up - skip connectivity check | `FALSE` | + +##### Compression Options + +| Variable | Description | Default | +| ----------------------------------- | ---------------------------------------------------------------------------------------------- | -------------- | +| `DB01_COMPRESSION` | Use either Gzip `GZ`, Bzip2 `BZ`, XZip `XZ`, ZSTD `ZSTD` or none `NONE` | `ZSTD` | +| `DB01_COMPRESSION_LEVEL` | Numerical value of what level of compression to use, most allow `1` to `9` | `3` | +| | except for `ZSTD` which allows for `1` to `19` | | +| `DB01_GZ_RSYNCABLE` | Use `--rsyncable` (gzip only) for faster rsync transfers and incremental backup deduplication. | `FALSE` | +| `DB01_ENABLE_PARALLEL_COMPRESSION` | Use multiple cores when compressing backups `TRUE` or `FALSE` | `TRUE` | +| `DB01_PARALLEL_COMPRESSION_THREADS` | Maximum amount of threads to use when compressing - Integer value e.g. `8` | `autodetected` | + +##### Scheduling Options + +| Variable | Description | Default | +| ---------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- | ------- | +| `DB01_BACKUP_INTERVAL` | How often to do a backup, in minutes after the first backup. Defaults to 1440 minutes, or once per day. | `1440` | +| `DB01_BACKUP_BEGIN` | What time to do the initial backup. Defaults to immediate. (`+1`) | `+0` | +| | Must be in one of two formats: | | +| | Absolute HHMM, e.g. `2330` or `0415` | | +| | Relative +MM, i.e. how many minutes after starting the container, e.g. `+0` (immediate), `+10` (in 10 minutes), or `+90` in an hour and a half | | +| `DB01_CLEANUP_TIME` | Value in minutes to delete old backups (only fired when backup interval executes) | `FALSE` | +| | 1440 would delete anything above 1 day old. You don't need to set this variable if you want to hold onto everything. | | +| `DB01_ARCHIVE_TIME` | Value in minutes to move all files files older than (x) from `DB01_BACKUP_FILESYSTEM_PATH` | | +| | to `DB01_BACKUP_FILESYSTEM_ARCHIVE_PATH` - which is useful when pairing against an external backup system. | | +| `DB01_BACKUP_BLACKOUT_BEGIN` | Use `HHMM` notation to start a blackout period where no backups occur eg `0420` | | +| `DB01_BACKUP_BLACKOUT_END` | Use `HHMM` notation to set the end period where no backups occur eg `0430` | | + +##### Specific Database Options + +###### CouchDB + +| Variable | Description | Default | `_FILE` | +| ----------- | ------------ | ------- | ------- | +| `DB01_PORT` | CouchDB Port | `5984` | x | + +###### InfluxDB + +| Variable | Description | Default | `_FILE` | +| --------------------- | ------------------------------------------------------------------------------------------------------- | ------- | ------- | +| `DB01_PORT` | InfluxDB Port | | x | +| | Version 1.x | `8088` | | +| | Version 2.x | `8086` | | +| `DB01_INFLUX_VERSION` | What Version of Influx are you backing up from `1`.x or `2` series - amd64 and aarch/armv8 only for `2` | `2` | | + +> Your Organization will be mapped to `DB_USER` and your root token will need to be mapped to `DB_PASS`. +> You may use `DB_NAME=ALL` to backup the entire set of databases. +> For `DB_HOST` use syntax of `http(s)://db-name` + + +###### MariaDB/MySQL + +| Variable | Description | Default | `_FILE` | +| ------------------------------- | --------------------------------------------------------------------------------------------------------- | ------------------------- | ------- | +| `DB01_EXTRA_OPTS` | Pass extra arguments to the backup and database enumeration command, add them here e.g. `--extra-command` | | +| `DB01_EXTRA_BACKUP_OPTS` | Pass extra arguments to the backup command only, add them here e.g. `--extra-command` | | +| `DB01_EXTRA_ENUMERATION_OPTS` | Pass extra arguments to the database enumeration command only, add them here e.g. `--extra-command` | | +| `DB01_NAME` | Schema Name e.g. `database` or `ALL` to backup all databases the user has access to. | | | +| | Backup multiple by separating with commas eg `db1,db2` | | x | +| `DB01_NAME_EXCLUDE` | If using `ALL` - use this as to exclude databases separated via commas from being backed up | | x | +| `DB01_PORT` | MariaDB Port | `3306` | x | +| `DB01_MYSQL_MAX_ALLOWED_PACKET` | Max allowed packet if backing up MySQL / MariaDB | `512M` | | +| `DB01_MYSQL_SINGLE_TRANSACTION` | Backup in a single transaction with MySQL / MariaDB | `TRUE` | | +| `DB01_MYSQL_STORED_PROCEDURES` | Backup stored procedures with MySQL / MariaDB | `TRUE` | | +| `DB01_MYSQL_ENABLE_TLS` | Enable TLS functionality for MySQL/MariaDB client | `FALSE` | | +| `DB01_MYSQL_TLS_VERIFY` | (optional) If using TLS (by means of MYSQL_TLS_* variables) verify remote host | `FALSE` | | +| `DB01_MYSQL_TLS_VERSION` | What TLS `v1.1` `v1.2` `v1.3` version to utilize | `TLSv1.1,TLSv1.2,TLSv1.3` | | +| `DB01_MYSQL_TLS_CA_FILE` | Filename to load custom CA certificate for connecting via TLS | `/etc/ssl/cert.pem` | x | +| `DB01_MYSQL_TLS_CERT_FILE` | Filename to load client certificate for connecting via TLS | | x | +| `DB01_MYSQL_TLS_KEY_FILE` | Filename to load client key for connecting via TLS | | x | + + +###### Microsoft SQL + +| Variable | Description | Default | `_FILE` | +| ----------- | ------------------ | ------- | ------- | +| `DB01_PORT` | Microsoft SQL Port | `1433` | x | + +###### MongoDB + +| Variable | Description | Default | `_FILE` | +| ----------------------- | ------------------------------------------------------------------------------------------------------------------------------------ | ------- | ------- | +| `DB01_AUTH` | (Optional) Authentication Database | | | +| `DB01_PORT` | MongoDB Port | `27017` | x | +| `DB01_MONGO_CUSTOM_URI` | If you wish to override the MongoDB Connection string enter it here e.g. `mongodb+srv://username:password@cluster.id.mongodb.net` | | x | +| | This environment variable will be parsed and populate the `DB_NAME` and `DB_HOST` variables to properly build your backup filenames. | | | +| | You can override them by making your own entries | | | + +###### Postgresql + +| Variable | Description | Default | `_FILE` | +| ----------------------------- | --------------------------------------------------------------------------------------------------------- | ------- | ------- | +| `DB01_AUTH` | (Optional) Authentication Database | | | +| `DB01_EXTRA_OPTS` | Pass extra arguments to the backup and database enumeration command, add them here e.g. `--extra-command` | | +| `DB01_EXTRA_BACKUP_OPTS` | Pass extra arguments to the backup command only, add them here e.g. `--extra-command` | | +| `DB01_EXTRA_ENUMERATION_OPTS` | Pass extra arguments to the database enumeration command only, add them here e.g. `--extra-command` | | +| `DB01_NAME` | Schema Name e.g. `database` or `ALL` to backup all databases the user has access to. | | | +| | Backup multiple by separating with commas eg `db1,db2` | | x | +| `DB01_PORT` | PostgreSQL Port | `5432` | x | + +###### Redis + +| Variable | Description | Default | `_FILE` | +| ------------------------ | --------------------------------------------------------------------------------------------------------- | ------- | ------- | +| `DB01_EXTRA_OPTS` | Pass extra arguments to the backup and database enumeration command, add them here e.g. `--extra-command` | | +| `DB01_EXTRA_BACKUP_OPTS` | Pass extra arguments to the backup command only, add them here e.g. `--extra-command` | | +| `DB01_PORT` | Redis Port | `6379` | x | + +###### SQLite + +| Variable | Description | Default | `_FILE` | +| ----------- | -------------------------------------------------------- | ------- | ------- | +| `DB01_HOST` | Enter the full path to DB file e.g. `/backup/db.sqlite3` | | x | + +##### Specific Storage Options + +Options that are related to the value of `DB01_BACKUP_LOCATION` + +###### Filesystem + +If `DB01_BACKUP_LOCTION` = `FILESYSTEM` then the following options are used: + +| Variable | Description | Default | +| ------------------------------ | ----------------------------------------------------------------------------------------------------- | --------------------------------- | +| `DB01_CREATE_LATEST_SYMLINK` | Create a symbolic link pointing to last backup in this format: `latest-(DB_TYPE)-(DB_NAME)-(DB_HOST)` | `TRUE` | +| `DB01_FILESYSTEM_PATH` | Directory where the database dumps are kept. | `/backup` | +| `DB01_FILESYSTEM_ARCHIVE_PATH` | Optional Directory where the database dumps archives are kept | `${DB01_FILESYSTEM_PATH/archive/` | +| `DB01_FILESYSTEM_PERMISSION` | Directory and File permissions to apply to files. | `700` | + +###### S3 + +If `DB01_BACKUP_LOCATION` = `S3` then the following options are used: + +| Parameter | Description | Default | `_FILE` | +| -------------------------- | ----------------------------------------------------------------------------------------- | ------- | ------- | +| `DB01_S3_BUCKET` | S3 Bucket name e.g. `mybucket` | | x | +| `DB01_S3_KEY_ID` | S3 Key ID (Optional) | | x | +| `DB01_S3_KEY_SECRET` | S3 Key Secret (Optional) | | x | +| `DB01_S3_PATH` | S3 Pathname to save to (must NOT end in a trailing slash e.g. '`backup`') | | x | +| `DB01_S3_REGION` | Define region in which bucket is defined. Example: `ap-northeast-2` | | x | +| `DB01_S3_HOST` | Hostname (and port) of S3-compatible service, e.g. `minio:8080`. Defaults to AWS. | | x | +| `DB01_S3_PROTOCOL` | Protocol to connect to `DB01_S3_HOST`. Either `http` or `https`. Defaults to `https`. | `https` | x | +| `DB01_S3_EXTRA_OPTS` | Add any extra options to the end of the `aws-cli` process execution | | x | +| `DB01_S3_CERT_CA_FILE` | Map a volume and point to your custom CA Bundle for verification e.g. `/certs/bundle.pem` | | x | +| _*OR*_ | | | | +| `DB01_S3_CERT_SKIP_VERIFY` | Skip verifying self signed certificates when connecting | `TRUE` | | + +> When `DB01_S3_KEY_ID` and/or `DB01_S3_KEY_SECRET` is not set, will try to use IAM role assigned (if any) for uploading the backup files to S3 bucket. + +###### Azure + +If `DB01_BACKUP_LOCATION` = `blobxfer` then the following options are used:. + +| Parameter | Description | Default | `_FILE` | +| ----------------------------------- | ------------------------------------------- | ------------------- | ------- | +| `DB01_BLOBXFER_STORAGE_ACCOUNT` | Microsoft Azure Cloud storage account name. | | x | +| `DB01_BLOBXFER_STORAGE_ACCOUNT_KEY` | Microsoft Azure Cloud storage account key. | | x | +| `DB01_BLOBXFER_REMOTE_PATH` | Remote Azure path | `/docker-db-backup` | x | + +> This service uploads files from backup target directory `DB01_BACKUP_FILESYSTEM_PATH`. +> If the a cleanup configuration in `DB01_CLEANUP_TIME` is defined, the remote directory on Azure storage will also be cleaned automatically. + +##### Hooks + +###### Path Options + +| Parameter | Description | Default | +| --------------------------- | --------------------------------------------------------------------------- | ----------------------- | +| `DB01_SCRIPT_LOCATION_PRE` | Location on filesystem inside container to execute bash scripts pre backup | `/assets/scripts/pre/` | +| `DB01_SCRIPT_LOCATION_POST` | Location on filesystem inside container to execute bash scripts post backup | `/assets/scripts/post/` | +| `DB01_PRE_SCRIPT` | Fill this variable in with a command to execute pre backing up | | +| `DB01_POST_SCRIPT` | Fill this variable in with a command to execute post backing up | | + +###### Pre Backup + +If you want to execute a custom script before a backup starts, you can drop bash scripts with the extension of `.sh` in the location defined in `DB01_SCRIPT_LOCATION_PRE`. See the following example to utilize: + +```bash +$ cat pre-script.sh +##!/bin/bash + +# #### Example Pre Script +# #### $1=DB01_TYPE (Type of Backup) +# #### $2=DB01_HOST (Backup Host) +# #### $3=DB01_NAME (Name of Database backed up +# #### $4=BACKUP START TIME (Seconds since Epoch) +# #### $5=BACKUP FILENAME (Filename) + +echo "${1} Backup Starting on ${2} for ${3} at ${4}. Filename: ${5}" +``` + + ## script DB01_TYPE DB01_HOST DB01_NAME STARTEPOCH BACKUP_FILENAME + ${f} "${backup_job_db_type}" "${backup_job_db_host}" "${backup_job_db_name}" "${backup_routines_start_time}" "${target}" + +Outputs the following on the console: + +`mysql Backup Starting on example-db for example at 1647370800. Filename: mysql_example_example-db_202200315-000000.sql.bz2 + +###### Post backup + +If you want to execute a custom script at the end of a backup, you can drop bash scripts with the extension of `.sh` in the location defined in `DB01_SCRIPT_LOCATION_POST`. Also to support legacy users `/assets/custom-scripts` is also scanned and executed.See the following example to utilize: + +```bash +$ cat post-script.sh +##!/bin/bash + +# #### Example Post Script +# #### $1=EXIT_CODE (After running backup routine) +# #### $2=DB_TYPE (Type of Backup) +# #### $3=DB_HOST (Backup Host) +# #### #4=DB_NAME (Name of Database backed up +# #### $5=BACKUP START TIME (Seconds since Epoch) +# #### $6=BACKUP FINISH TIME (Seconds since Epoch) +# #### $7=BACKUP TOTAL TIME (Seconds between Start and Finish) +# #### $8=BACKUP FILENAME (Filename) +# #### $9=BACKUP FILESIZE +# #### $10=HASH (If CHECKSUM enabled) +# #### $11=MOVE_EXIT_CODE + +echo "${1} ${2} Backup Completed on ${3} for ${4} on ${5} ending ${6} for a duration of ${7} seconds. Filename: ${8} Size: ${9} bytes MD5: ${10}" +``` + + ## script EXIT_CODE DB_TYPE DB_HOST DB_NAME STARTEPOCH FINISHEPOCH DURATIONEPOCH BACKUP_FILENAME FILESIZE CHECKSUMVALUE + ${f} "${exit_code}" "${dbtype}" "${dbhost}" "${dbname}" "${backup_routines_start_time}" "${backup_routines_finish_time}" "${backup_routines_total_time}" "${target}" "${FILESIZE}" "${checksum_value}" "${move_exit_code} + +Outputs the following on the console: + +`0 mysql Backup Completed on example-db for example on 1647370800 ending 1647370920 for a duration of 120 seconds. Filename: mysql_example_example-db_202200315-000000.sql.bz2 Size: 7795 bytes Hash: 952fbaafa30437494fdf3989a662cd40 0` + +If you wish to change the size value from bytes to megabytes set environment variable `DB01_SIZE_VALUE=megabytes` + +You must make your scripts executable otherwise there is an internal check that will skip trying to run it otherwise. +If for some reason your filesystem or host is not detecting it right, use the environment variable `DB01_POST_SCRIPT_SKIP_X_VERIFY=TRUE` to bypass. + + +#### Notifications + +This image has capabilities on sending notifications via a handful of services when a backup job fails. This is a global option that cannot be individually set per backup job. + +| Parameter | Description | Default | +| ---------------------- | --------------------------------------------------------------------------------- | ------- | +| `ENABLE_NOTIFICATIONS` | Enable Notifications | `FALSE` | +| `NOTIFICATION_TYPE` | `CUSTOM` `EMAIL` `MATRIX` `MATTERMOST` `ROCKETCHAT` - Seperate Multiple by commas | | + +##### Custom Notifications + +The following is sent to the custom script. Use how you wish: + +```` +$1 unix timestamp +$2 logfile +$3 errorcode +$4 subject +$5 body/error message +```` + +| Parameter | Description | Default | +| ---------------------------- | ------------------------------------------------------- | ------- | +| `NOTIFICATION_CUSTOM_SCRIPT` | Path and name of custom script to execute notification. | | + + +##### Email Notifications +| Parameter | Description | Default | `_FILE` | +| ----------- | ----------------------------------------------------------------------------------------- | ------- | ------- | +| `MAIL_FROM` | What email address to send mail from for errors | | | +| `MAIL_TO` | What email address to send mail to for errors. Send to multiple by seperating with comma. | | | +| `SMTP_HOST` | What SMTP server to use for sending mail | | x | +| `SMTP_PORT` | What SMTP port to use for sending mail | | x | + +##### Matrix Notifications + +Fetch a `MATRIX_ACCESS_TOKEN`: + +```` +curl -XPOST -d '{"type":"m.login.password", "user":"myuserid", "password":"mypass"}' "https://matrix.org/_matrix/client/r0/login" +```` + +Copy the JSON response `access_token` that will look something like this: + +```` +{"access_token":"MDAxO...blahblah","refresh_token":"MDAxO...blahblah","home_server":"matrix.org","user_id":"@myuserid:matrix.org"} +```` + +| Parameter | Description | Default | `_FILE` | +| --------------------- | ---------------------------------------------------------------------------------------- | ------- | ------- | +| `MATRIX_HOST` | URL (https://matrix.example.com) of Matrix Homeserver | | x | +| `MATRIX_ROOM` | Room ID eg `\!abcdef:example.com` to send to. Send to multiple by seperating with comma. | | x | +| `MATRIX_ACCESS_TOKEN` | Access token of user authorized to send to room | | x | + +##### Mattermost Notifications +| Parameter | Description | Default | `_FILE` | +| ------------------------ | -------------------------------------------------------------------------------------------- | ------- | ------- | +| `MATTERMOST_WEBHOOK_URL` | Full URL to send webhook notifications to | | x | +| `MATTERMOST_RECIPIENT` | Channel or User to send Webhook notifications to. Send to multiple by seperating with comma. | | x | +| `MATTERMOST_USERNAME` | Username to send as eg `tiredofit` | | x | + +##### Rocketchat Notifications +| Parameter | Description | Default | `_FILE` | +| ------------------------ | -------------------------------------------------------------------------------------------- | ------- | ------- | +| `ROCKETCHAT_WEBHOOK_URL` | Full URL to send webhook notifications to | | x | +| `ROCKETCHAT_RECIPIENT` | Channel or User to send Webhook notifications to. Send to multiple by seperating with comma. | | x | +| `ROCKETCHAT_USERNAME` | Username to send as eg `tiredofit` | | x | ## Maintenance @@ -269,6 +755,9 @@ Manual Backups can be performed by entering the container and typing `backup-now - Recently there was a request to have the container work with Kubernetes cron scheduling. This can theoretically be accomplished by setting the container `MODE=MANUAL` and then setting `MANUAL_RUN_FOREVER=FALSE` - You would also want to disable a few features from the upstream base images specifically `CONTAINER_ENABLE_SCHEDULING` and `CONTAINER_ENABLE_MONITORING`. This should allow the container to start, execute a backup by executing and then exit cleanly. An alternative way to running the script is to execute `/etc/services.available/10-db-backup/run`. +### Manual Backups +Manual Backups can be performed by entering the container and typing `backup-now`. This will execute all the backup tasks that are scheduled by means of the `BACKUPXX_` variables. Alternatively if you wanted to execute a job on its own you could simply type `backup01-now` (or whatever your number would be). There is no concurrency, and jobs will be executed sequentially. + ### Restoring Databases Entering in the container and executing `restore` will execute a menu based script to restore your backups - MariaDB, Postgres, and Mongo supported. @@ -292,75 +781,7 @@ The script can also be executed skipping the interactive mode by using the follo If you only enter some of the arguments you will be prompted to fill them in. -### Custom Scripts -#### Path Options - -| Parameter | Description | Default | -| ---------------------- | --------------------------------------------------------------------------- | ----------------------- | -| `SCRIPT_LOCATION_PRE` | Location on filesystem inside container to execute bash scripts pre backup | `/assets/scripts/pre/` | -| `SCRIPT_LOCATION_POST` | Location on filesystem inside container to execute bash scripts post backup | `/assets/scripts/post/` | - -#### Pre Backup - -If you want to execute a custom script before a backup starts, you can drop bash scripts with the extension of `.sh` in the location defined in `SCRIPT_LOCATION_PRE`. See the following example to utilize: - -```bash -$ cat pre-script.sh -##!/bin/bash - -# #### Example Pre Script -# #### $1=DB_TYPE (Type of Backup) -# #### $2=DB_HOST (Backup Host) -# #### $3=DB_NAME (Name of Database backed up -# #### $4=BACKUP START TIME (Seconds since Epoch)ff -# #### $5=BACKUP FILENAME (Filename) - -echo "${1} Backup Starting on ${2} for ${3} at ${4}. Filename: ${5}" -``` - - ## script DB_TYPE DB_HOST DB_NAME STARTEPOCH BACKUP_FILENAME - ${f} "${dbtype}" "${dbhost}" "${dbname}" "${backup_start_time}" "${target}" - -Outputs the following on the console: - -`mysql Backup Starting on example-db for example at 1647370800. Filename: mysql_example_example-db_202200315-000000.sql.bz2 - -#### Post backup - -If you want to execute a custom script at the end of a backup, you can drop bash scripts with the extension of `.sh` in the location defined in `SCRIPT_LOCATION_POST`. Also to support legacy users `/assets/custom-scripts` is also scanned and executed.See the following example to utilize: - -```bash -$ cat post-script.sh -##!/bin/bash - -# #### Example Post Script -# #### $1=EXIT_CODE (After running backup routine) -# #### $2=DB_TYPE (Type of Backup) -# #### $3=DB_HOST (Backup Host) -# #### #4=DB_NAME (Name of Database backed up -# #### $5=BACKUP START TIME (Seconds since Epoch) -# #### $6=BACKUP FINISH TIME (Seconds since Epoch) -# #### $7=BACKUP TOTAL TIME (Seconds between Start and Finish) -# #### $8=BACKUP FILENAME (Filename) -# #### $9=BACKUP FILESIZE -# #### $10=HASH (If CHECKSUM enabled) -# #### $11=MOVE_EXIT_CODE - -echo "${1} ${2} Backup Completed on ${3} for ${4} on ${5} ending ${6} for a duration of ${7} seconds. Filename: ${8} Size: ${9} bytes MD5: ${10}" -``` - - ## script EXIT_CODE DB_TYPE DB_HOST DB_NAME STARTEPOCH FINISHEPOCH DURATIONEPOCH BACKUP_FILENAME FILESIZE CHECKSUMVALUE - ${f} "${exit_code}" "${dbtype}" "${dbhost}" "${dbname}" "${backup_start_timme}" "${backup_finish_time}" "${backup_total_time}" "${target}" "${FILESIZE}" "${checksum_value}" "${move_exit_code} - -Outputs the following on the console: - -`0 mysql Backup Completed on example-db for example on 1647370800 ending 1647370920 for a duration of 120 seconds. Filename: mysql_example_example-db_202200315-000000.sql.bz2 Size: 7795 bytes Hash: 952fbaafa30437494fdf3989a662cd40 0` - -If you wish to change the size value from bytes to megabytes set environment variable `SIZE_VALUE=megabytes` - -You must make your scripts executable otherwise there is an internal check that will skip trying to run it otherwise. -If for some reason your filesystem or host is not detecting it right, use the environment variable `POST_SCRIPT_SKIP_X_VERIFY=TRUE` to bypass. ## Support diff --git a/install/assets/defaults/10-db-backup b/install/assets/defaults/10-db-backup index d87ba1b..f1b2554 100644 --- a/install/assets/defaults/10-db-backup +++ b/install/assets/defaults/10-db-backup @@ -13,7 +13,6 @@ DB_DUMP_TARGET_PERMISSION=${DB_DUMP_TARGET_PERMISSION:-"700"} DB_DUMP_TARGET_ARCHIVE=${DB_DUMP_TARGET_ARCHIVE:-"${DB_DUMP_TARGET}/archive/"} ENABLE_CHECKSUM=${ENABLE_CHECKSUM:-"TRUE"} ENABLE_PARALLEL_COMPRESSION=${ENABLE_PARALLEL_COMPRESSION:-"TRUE"} -LOG_FILE=${LOG_FILE:-"dbbackup.log"} LOG_LEVEL=${LOG_LEVEL:-"debug"} LOG_PATH=${LOG_PATH:-"/logs"} MANUAL_RUN_FOREVER=${MANUAL_RUN_FOREVER:-"TRUE"} diff --git a/install/assets/functions/10-db-backup b/install/assets/functions/10-db-backup index 6b1fc23..00c7ac2 100644 --- a/install/assets/functions/10-db-backup +++ b/install/assets/functions/10-db-backup @@ -19,7 +19,7 @@ bootstrap_filesystem() { if [ "$(stat -c %U "${LOG_PATH}")" != "dbbackup" ] ; then chown dbbackup:dbbackup "${LOG_PATH}" ; fi if [ "$(stat -c %a "${LOG_PATH}")" != "755" ] ; then chmod -R 755 "${LOG_PATH}" ; fi - create_logrotate dbbackup "${LOG_PATH}"/"${LOG_FILE}" none dbbackup dbbackup + echo "0 0 * * * /usr/sbin/logrotate_dbbackup >/dev/null 2>&1" > /assets/cron/dbbackup_logrotate if [ ! -d "${TEMP_LOCATION}" ]; then mkdir -p "${TEMP_LOCATION}" @@ -963,7 +963,7 @@ write_log() { case "${LOG_LEVEL,,}" in "debug" ) print_debug "$@" - echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [debug] $@" | run_as_user tee -a "${LOG_PATH}"/"${LOG_FILE}" > /dev/null + echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [debug] $@" | run_as_user tee -a "${LOG_PATH}"/"${DB_TYPE}"_"${DB_NAME}"_"${DB_HOST}"_"${now}".log > /dev/null ;; esac ;; @@ -972,21 +972,21 @@ write_log() { case "${LOG_LEVEL,,}" in "debug" | "notice" | "warn" | "error") print_error "$@" - echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [error] $@" | run_as_user tee -a "${LOG_PATH}"/"${LOG_FILE}" > /dev/null + echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [error] $@" | run_as_user tee -a "${LOG_PATH}"/"${DB_TYPE}"_"${DB_NAME}"_"${DB_HOST}"_"${now}".log > /dev/null ;; esac ;; info ) shift 1 print_info "$@" - echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [info] $@" | run_as_user tee -a "${LOG_PATH}"/"${LOG_FILE}" > /dev/null + echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [info] $@" | run_as_user tee -a "${LOG_PATH}"/"${DB_TYPE}"_"${DB_NAME}"_"${DB_HOST}"_"${now}".log > /dev/null ;; notice ) shift 1 case "${LOG_LEVEL,,}" in "debug" | "notice" ) print_notice "$@" - echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [notice] $@" | run_as_user tee -a "${LOG_PATH}"/"${LOG_FILE}" > /dev/null + echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [notice] $@" | run_as_user tee -a "${LOG_PATH}"/"${DB_TYPE}"_"${DB_NAME}"_"${DB_HOST}"_"${now}".log > /dev/null ;; esac ;; @@ -995,7 +995,7 @@ write_log() { case "${LOG_LEVEL,,}" in "debug" | "notice" | "warn" ) print_warn "$@" - echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [warn] $@" | run_as_user tee -a "${LOG_PATH}"/"${LOG_FILE}" > /dev/null + echo "$(date +'%Y-%m-%d %H:%M:%S %Z') [warn] $@" | run_as_user tee -a "${LOG_PATH}"/"${DB_TYPE}"_"${DB_NAME}"_"${DB_HOST}"_"${now}".log > /dev/null ;; esac ;;