Release 3.4.2 - See CHANGELOG.md

Improve release notes for 3.4.1

CHANGELOG.md

@@ -1,3 +1,26 @@
+## 3.4.2 2022-09-19 <dave at tiredofit dot ca>
+
+   ### Changed
+      - Skip availability check for Mongo Atlas connections
+
+
+## 3.4.1 2022-09-13 <dave at tiredofit dot ca>
+
+### Added
+- Introduce environment variables for SCRIPT_LOCATION_POST and SCRIPT_LOCATION_PRE for better seperation
+   
+### Deprecated
+
+- Introduce deprecation warning for the custom script paths `/assets/custom-scripts` and `/assets/custom-scripts/pre`. These
+paths will continue to work for now but support may be removed in the next major version release. To support the new
+default paths your scripts should be moved as follows:
+
+  |Script Type|Old Path (Deprecated)|New Environment Variable|Environment Value Default|
+  |-----------|--------|-------------------------|----------------|
+  |Pre|`/assets/custom-scripts/pre`|SCRIPT_LOCATION_PRE|`/assets/scripts/pre`|
+  |Post|`/assets/custom-scripts`|SCRIPT_LOCATION_POST|`/assets/scripts/post`|
+
+
 ## 3.4.0 2022-09-12 <dave at tiredofit dot ca>
 
    ### Added

README.md

@@ -59,6 +59,7 @@ Currently backs up CouchDB, InfluxDB, MySQL, MongoDB, Postgres, Redis servers.
   - [Manual Backups](#manual-backups)
   - [Restoring Databases](#restoring-databases)
   - [Custom Scripts](#custom-scripts)
+    - [Path Options](#path-options)
     - [Pre Backup](#pre-backup)
     - [Post backup](#post-backup)
 - [Support](#support)
@@ -106,11 +107,11 @@ Images are built primarily for `amd64` architecture, and may also include builds
 ### Persistent Storage
 
 The following directories are used for configuration and can be mapped for persistent storage.
-| Directory                    | Description                                                                         |
+| Directory              | Description                                                                         |
-| ---------------------------- | ----------------------------------------------------------------------------------- |
+| ---------------------- | ----------------------------------------------------------------------------------- |
-| `/backup`                    | Backups                                                                             |
+| `/backup`              | Backups                                                                             |
-| `/assets/custom-scripts/pre` | *Optional* Put custom scripts in this directory to execute before backup operations |
+| `/assets/scripts/pre`  | *Optional* Put custom scripts in this directory to execute before backup operations |
-| `/assets/custom-scripts`     | *Optional* Put custom scripts in this directory to execute after backup operations  |
+| `/assets/scripts/post` | *Optional* Put custom scripts in this directory to execute after backup operations  |
 
 ### Environment Variables
 
@@ -133,8 +134,8 @@ Be sure to view the following repositories to understand all the customizable op
 | `MANUAL_RUN_FOREVER` | `TRUE` or `FALSE` if you wish to try to make the container exit after the backup                                                 | `TRUE`          |
 | `TEMP_LOCATION`      | Perform Backups and Compression in this temporary directory                                                                      | `/tmp/backups/` |
 | `DEBUG_MODE`         | If set to `true`, print copious shell script messages to the container log. Otherwise only basic messages are printed.           | `FALSE`         |
-| `PRE_SCRIPT`         | Fill this variable in with a command to execute post the script backing up                                                       |                 |
+| `PRE_SCRIPT`         | Fill this variable in with a command to execute pre backing up                                                                   |                 |
-| `POST_SCRIPT`        | Fill this variable in with a command to execute post the script backing up                                                       |                 |
+| `POST_SCRIPT`        | Fill this variable in with a command to execute post backing up                                                                  |                 |
 | `SPLIT_DB`           | For each backup, create a new archive. `TRUE` or `FALSE` (MySQL and Postgresql Only)                                             | `TRUE`          |
 
 ### Database Specific Options
@@ -156,15 +157,18 @@ Be sure to view the following repositories to understand all the customizable op
 Your Organization will be mapped to `DB_USER` and your root token will need to be mapped to `DB_PASS`. You may use `DB_NAME=ALL` to backup the entire set of databases. For `DB_HOST` use syntax of `http(s)://db-name`
 
 ### Scheduling Options
-| Parameter         | Description                                                                                                                                                                                        | Default |
+| Parameter          | Description                                                                                                                                                                                        | Default   |
-| ----------------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------- |
+|--------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-----------|
-| `DB_DUMP_FREQ`    | How often to do a dump, in minutes after the first backup. Defaults to 1440 minutes, or once per day.                                                                                              | `1440`  |
+| `DB_DUMP_FREQ`     | How often to do a dump, in minutes after the first backup. Defaults to 1440 minutes, or once per day.                                                                                              | `1440`    |
-| `DB_DUMP_BEGIN`   | What time to do the first dump. Defaults to immediate. Must be in one of two formats                                                                                                               |         |
+| `DB_DUMP_BEGIN`    | What time to do the first dump. Defaults to immediate. Must be in one of two formats                                                                                                               |           |
-|                   | Absolute HHMM, e.g. `2330` or `0415`                                                                                                                                                               |         |
+|                    | Absolute HHMM, e.g. `2330` or `0415`                                                                                                                                                               |           |
-|                   | Relative +MM, i.e. how many minutes after starting the container, e.g. `+0` (immediate), `+10` (in 10 minutes), or `+90` in an hour and a half                                                     |         |
+|                    | Relative +MM, i.e. how many minutes after starting the container, e.g. `+0` (immediate), `+10` (in 10 minutes), or `+90` in an hour and a half                                                     |           |
-| `DB_CLEANUP_TIME` | Value in minutes to delete old backups (only fired when dump freqency fires). 1440 would delete anything above 1 day old. You don't need to set this variable if you want to hold onto everything. | `FALSE` |
+| `DB_DUMP_TARGET`   | Directory where the database dumps are kept.                                                                                                                                                       | `/backup` |
+| `DB_CLEANUP_TIME`  | Value in minutes to delete old backups (only fired when dump freqency fires). 1440 would delete anything above 1 day old. You don't need to set this variable if you want to hold onto everything. | `FALSE`   |
+
 
 - You may need to wrap your `DB_DUMP_BEGIN` value in quotes for it to properly parse. There have been reports of backups that start with a `0` get converted into a different format which will not allow the timer to start at the correct time.
+
 ### Backup Options
 | Parameter                      | Description                                                                                                                  | Default        |
 | ------------------------------ | ---------------------------------------------------------------------------------------------------------------------------- | -------------- |
@@ -237,33 +241,41 @@ If you only enter some of the arguments you will be prompted to fill them in.
 
 ### Custom Scripts
 
+#### Path Options
+
+| Parameter              | Description                                                                 | Default                 |
+| ---------------------- | --------------------------------------------------------------------------- | ----------------------- |
+| `SCRIPT_LOCATION_PRE`  | Location on filesystem inside container to execute bash scripts pre backup  | `/assets/scripts/pre/`  |
+| `SCRIPT_LOCATION_POST` | Location on filesystem inside container to execute bash scripts post backup | `/assets/scripts/post/` |
+
 #### Pre Backup
-If you want to execute a custom script before a backup starts, you can drop bash scripts with the extension of `.sh` in `/assets/custom/pre`. See the following example to utilize:
+If you want to execute a custom script before a backup starts, you can drop bash scripts with the extension of `.sh` in the location defined in `SCRIPT_LOCATION_PRE`. See the following example to utilize:
 
 ````bash
 $ cat pre-script.sh
 ##!/bin/bash
 
 # #### Example Pre Script
-# #### $2=DB_TYPE (Type of Backup)
+# #### $1=DB_TYPE (Type of Backup)
-# #### $3=DB_HOST (Backup Host)
+# #### $2=DB_HOST (Backup Host)
-# #### #4=DB_NAME (Name of Database backed up
+# #### $3=DB_NAME (Name of Database backed up
-# #### $5=BACKUP START TIME (Seconds since Epoch)
+# #### $4=BACKUP START TIME (Seconds since Epoch)ff
-# #### $8=BACKUP FILENAME (Filename)
+# #### $5=BACKUP FILENAME (Filename)
 
-echo "${2} Backup Starting on ${3} for ${4} on ${5} ending ${6} for a duration of ${7} seconds. Filename: ${8} Size: ${9} bytes MD5: ${10}"
+echo "${1} Backup Starting on ${2} for ${3} at ${4}. Filename: ${5}"
 ````
 
-      ## script EXIT_CODE DB_TYPE DB_HOST DB_NAME STARTEPOCH BACKUP_FILENAME
+    ## script DB_TYPE DB_HOST DB_NAME STARTEPOCH BACKUP_FILENAME
-      ${f} "${exit_code}" "${dbtype}" "${dbhost}" "${dbname}" "${backup_start_time}" "${target}"
+    ${f} "${dbtype}" "${dbhost}" "${dbname}" "${backup_start_time}" "${target}"
 
 
 Outputs the following on the console:
 
-`mysql Backup Starting on example-db for example on 1647370800. Filename: mysql_example_example-db_202200315-000000.sql.bz2
+`mysql Backup Starting on example-db for example at 1647370800. Filename: mysql_example_example-db_202200315-000000.sql.bz2
+
 
 #### Post backup
-If you want to execute a custom script at the end of backup, you can drop bash scripts with the extension of `.sh` in `/assets/custom`. See the following example to utilize:
+If you want to execute a custom script at the end of a backup, you can drop bash scripts with the extension of `.sh` in the location defined in `SCRIPT_LOCATION_POST`. Also to support legacy users `/assets/custom-scripts` is also scanned and executed.See the following example to utilize:
 
 ````bash
 $ cat post-script.sh

install/assets/defaults/10-db-backup

@@ -17,6 +17,8 @@ MYSQL_STORED_PROCEDURES=${MYSQL_STORED_PROCEDURES:-"TRUE"}
 PARALLEL_COMPRESSION_THREADS=${PARALLEL_COMPRESSION_THREADS:-"$(nproc)"}
 S3_CERT_SKIP_VERIFY=${S3_CERT_SKIP_VERIFY:-"TRUE"}
 S3_PROTOCOL=${S3_PROTOCOL:-"https"}
+SCRIPT_LOCATION_PRE=${SCRIPT_LOCATION_PRE:-"/assets/scripts/pre/"}
+SCRIPT_LOCATION_POST=${SCRIPT_LOCATION_POST:-"/assets/scripts/post/"}
 SIZE_VALUE=${SIZE_VALUE:-"bytes"}
 SPLIT_DB=${SPLIT_DB:-"TRUE"}
 TEMP_LOCATION=${TEMP_LOCATION:-"/tmp/backups"}

install/assets/functions/10-db-backup

@@ -375,12 +375,14 @@ check_availability() {
             esac
         ;;
         "mongo" )
-            counter=0
+            if [ "${MONGO_HOST_TYPE,,}" != "atlas" ] ; then
-            while ! (nc -z ${DB_HOST} ${DB_PORT}) ; do
+                counter=0
-                sleep 5
+                while ! (nc -z ${DB_HOST} ${DB_PORT}) ; do
-                (( counter+=5 ))
+                    sleep 5
-                print_warn "Mongo Host '${DB_HOST}' is not accessible, retrying.. ($counter seconds so far)"
+                    (( counter+=5 ))
-            done
+                    print_warn "Mongo Host '${DB_HOST}' is not accessible, retrying.. ($counter seconds so far)"
+                done
+            fi
         ;;
         "mysql" )
             counter=0
@@ -666,8 +668,14 @@ pre_dbbackup() {
     fi
 
     ### Pre Backup Custom Script Support
-    if [ -d "/assets/custom-scripts/pre/" ] ; then
+    if [ -d "/assets/custom-scripts/pre" ] && dir_notempty "/assets/custom-scripts/pre" ; then
-        for f in $(find /assets/custom-scripts/pre/ -name \*.sh -type f); do
+        print_warning "Found Custom Post Scripts in /assets/custom-scripts/pre - Automatically moving them to '${SCRIPT_LOCATION_PRE}'"
+        mkdir -p "${SCRIPT_LOCATION_PRE}"
+        silent cp /assets/custom-scripts/pre/* "${SCRIPT_LOCATION_PRE}"
+    fi
+
+    if [ -d "${SCRIPT_LOCATION_PRE}" ] && dir_notempty "${SCRIPT_LOCATION_PRE}" ; then
+        for f in $(find ${SCRIPT_LOCATION_PRE} -name \*.sh -type f); do
             if var_true "${PRE_SCRIPT_SKIP_X_VERIFY}" ; then
                 ${f} "${dbtype}" "${DB_HOST}" "${1}" "${dbbackup_start_time}" "${target}"
             else
@@ -711,8 +719,14 @@ post_dbbackup() {
     fi
 
     ### Post Backup Custom Script Support
-    if [ -d "/assets/custom-scripts/" ] ; then
+    if [ -d "/assets/custom-scripts/" ] && dir_notempty "/assets/custom-scripts" ; then
-        for f in $(find /assets/custom-scripts/ -name \*.sh -type f); do
+        print_warning "Found Custom Post Scripts in /assets/custom-scripts/ - Automatically moving them to '${SCRIPT_LOCATION_POST}'"
+        mkdir -p "${SCRIPT_LOCATION_POST}"
+        silent cp /assets/custom-scripts/* "${SCRIPT_LOCATION_POST}"
+    fi
+
+    if [ -d "${SCRIPT_LOCATION_POST}" ] && dir_notempty "${SCRIPT_LOCATION_POST}" ; then
+        for f in $(find ${SCRIPT_LOCATION_POST} -name \*.sh -type f); do
             if var_true "${POST_SCRIPT_SKIP_X_VERIFY}" ; then
                 ${f} "${exit_code}" "${dbtype}" "${DB_HOST}" "${1}" "${dbbackup_start_time}" "${dbbackup_finish_time}" "${dbbackup_total_time}" "${target}" "${filesize}" "${checksum_value}"
             else