Skip to content

Commit

Permalink
Merge pull request #52 from BCDevOps/mssql
Browse files Browse the repository at this point in the history
Add support for MS SQL
  • Loading branch information
WadeBarnes authored Jun 8, 2020
2 parents 1f36e01 + 80de210 commit 33fdad0
Show file tree
Hide file tree
Showing 10 changed files with 329 additions and 13 deletions.
19 changes: 14 additions & 5 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Backup Container
description: A simple containerized backup solution for backing up one or more postgres or mongo databases to a secondary location.
description: A simple containerized backup solution for backing up one or more supported databases to a secondary location.
author: WadeBarnes
resourceType: Components
personas:
Expand All @@ -12,19 +12,25 @@ labels:
- backups
- postgres
- mongo
- mssql
- database
---
[![License](https://img.shields.io/badge/License-Apache%202.0-blue.svg)](LICENSE)

# Backup Container
[Backup Container](https://github.com/BCDevOps/backup-container) is a simple containerized backup solution for backing up one or more postgres or mongo databases to a secondary location. _Code and documentation was originally pulled from the [HETS Project](https://github.com/bcgov/hets)_
[Backup Container](https://github.com/BCDevOps/backup-container) is a simple containerized backup solution for backing up one or more supported databases to a secondary location. _Code and documentation was originally pulled from the [HETS Project](https://github.com/bcgov/hets)_

# Supported Databases
MongoDB
MSSQL - Currently MSSQL requires that the nfs db volume be shared with the database for backups to function correctly.
PostgresSQL

# Backup Container Options
You can run the Backup Container for postgres and mongo databases separately or in a mixed environment.
You can run the Backup Container for supported databases separately or in a mixed environment.
For a mixed environment:
1) You MUST use the recommended `backup.conf` configuration.
2) Within the `backup.conf`, you MUST specify the `DatabaseType` for each listed database.
3) You will need to create two builds and two deployment configs. One for a postgres backup container and the other for a mongo backup container.
3) You will need to create a build and deployment config for each type of supported backup container in use.
4) Mount the same `backup.conf` file (ConfigMap) to each deployed container.

## Backups in OpenShift
Expand Down Expand Up @@ -76,7 +82,7 @@ Together, the scripts and templates provided in the [openshift](./openshift) dir

The following environment variables are defaults used by the `backup` app.

**NOTE**: These environment variables MUST MATCH those used by the postgresql container(s) you are planning to backup.
**NOTE**: These environment variables MUST MATCH those used by the database container(s) you are planning to backup.

| Name | Default (if not set) | Purpose |
| ---- | ------- | ------- |
Expand Down Expand Up @@ -283,6 +289,9 @@ Plugin Examples:
- [backup.mongo.plugin](./docker/backup.mongo.plugin)
- Mongo backup implementation.

- [backup.mssql.plugin](./docker/backup.mssql.plugin)
- MSSQL backup implementation.

- [backup.null.plugin](./docker/backup.null.plugin)
- Sample/Template backup implementation that simply outputs log messages for the various operations.

Expand Down
6 changes: 4 additions & 2 deletions config/backup.conf
Original file line number Diff line number Diff line change
Expand Up @@ -9,9 +9,9 @@
# - <Hostname/>:<Port/>/<DatabaseName/>
# - <DatabaseType>=<Hostname/>/<DatabaseName/>
# - <DatabaseType>=<Hostname/>:<Port/>/<DatabaseName/>
# <DatabaseType> can be postgres or mongo
# <DatabaseType> can be postgres, mongo or mssql
# <DatabaseType> MUST be specified when you are sharing a
# single backup.conf file between postgres and mongo
# single backup.conf file between postgres, mongo and mssql
# backup containers. If you do not specify <DatabaseType>
# the listed databases are assumed to be valid for the
# backup container in which the configuration is mounted.
Expand All @@ -21,6 +21,7 @@
# - postgres=postgresql:5432/my_database
# - mongo=mongodb/my_database
# - mongo=mongodb:27017/my_database
# - mssql=mssql_server:1433/my_database
# -----------------------------------------------------------
# Cron Scheduling:
# -----------------------------------------------------------
Expand All @@ -42,6 +43,7 @@
# postgres=postgresql:5432/TheOrgBook_Database
# mongo=mender-mongodb:27017/useradm
# postgres=wallet-db/tob_issuer
# mssql=pims-db-dev:1433/pims
#
# 0 1 * * * default ./backup.sh -s
# 0 4 * * * default ./backup.sh -s -v all
Expand Down
50 changes: 50 additions & 0 deletions docker/Dockerfile_MSSQL
Original file line number Diff line number Diff line change
@@ -0,0 +1,50 @@
FROM mcr.microsoft.com/mssql/rhel/server:2019-CU1-rhel-8

# Change timezone to PST for convenience
ENV TZ=PST8PDT

# Set the workdir to be root
WORKDIR /

# Load the backup scripts into the container (must be executable).
COPY backup.* /

COPY webhook-template.json /

# ========================================================================================================
# Install go-crond (from https://github.com/BCDevOps/go-crond)
# - Adds some additional logging enhancements on top of the upstream project;
# https://github.com/webdevops/go-crond
#
# CRON Jobs in OpenShift:
# - https://blog.danman.eu/cron-jobs-in-openshift/
# --------------------------------------------------------------------------------------------------------
ARG SOURCE_REPO=BCDevOps
ARG GOCROND_VERSION=0.6.3
ADD https://github.com/$SOURCE_REPO/go-crond/releases/download/$GOCROND_VERSION/go-crond-64-linux /usr/bin/go-crond

USER root

RUN chmod ug+x /usr/bin/go-crond
# ========================================================================================================

# ========================================================================================================
# Perform operations that require root privilages here ...
# --------------------------------------------------------------------------------------------------------
RUN echo $TZ > /etc/timezone
# ========================================================================================================
COPY uid_entrypoint /opt/mssql-tools/bin/
RUN chmod -R a+rwx /opt/mssql-tools/bin/uid_entrypoint

ENV PATH=${PATH}:/opt/mssql/bin:/opt/mssql-tools/bin
RUN mkdir -p /var/opt/mssql/data && \
chmod -R g=u /var/opt/mssql /etc/passwd

# Important - Reset to the base image's user account.
USER 10001

# Set the default CMD.
CMD sh /backup.sh

### user name recognition at runtime w/ an arbitrary uid - for OpenShift deployments
ENTRYPOINT [ "/opt/mssql-tools/bin/uid_entrypoint" ]
12 changes: 12 additions & 0 deletions docker/backup.container.utils
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,16 @@ function isMongo(){
)
}

function isMsSql(){
(
if isInstalled "sqlcmd"; then
return 0
else
return 1
fi
)
}

function getContainerType(){
(
local _containerType=${UNKNOWN_DB}
Expand All @@ -31,6 +41,8 @@ function getContainerType(){
_containerType=${POSTGRE_DB}
elif isMongo; then
_containerType=${MONGO_DB}
elif isMsSql; then
_containerType=${MSSQL_DB}
else
_containerType=${UNKNOWN_DB}
_rtnCd=1
Expand Down
229 changes: 229 additions & 0 deletions docker/backup.mssql.plugin
Original file line number Diff line number Diff line change
@@ -0,0 +1,229 @@
#!/bin/bash
# =================================================================================================================
# MSSQL Backup and Restore Functions:
# - Dynamically loaded as a plug-in
# -----------------------------------------------------------------------------------------------------------------
export serverDataDirectory="/var/opt/mssql/data"

function onBackupDatabase(){
(
local OPTIND
local unset flags
while getopts : FLAG; do
case $FLAG in
? ) flags+="-${OPTARG} ";;
esac
done
shift $((OPTIND-1))

_databaseSpec=${1}
_backupFile=${2}

_hostname=$(getHostname ${_databaseSpec})
_database=$(getDatabaseName ${_databaseSpec})
_port=$(getPort ${_databaseSpec})
_portArg=${_port:+",${_port}"}
_username=$(getLocalOrDBUsername ${_databaseSpec} ${_hostname})
_password=$(getPassword ${_databaseSpec})
echoGreen "Backing up '${_hostname}${_port:+:${_port}}${_database:+/${_database}}' to '${_backupFile}' ..."

#TODO: add support for backing up transaction log as well.
sqlcmd -S ${_hostname}${_portArg} -U ${_username} -P ${_password} -Q "BACKUP DATABASE ${_database} TO DISK = N'${_fileName}' WITH NOFORMAT, NOINIT, SKIP, NOREWIND, NOUNLOAD, STATS = 10"
return ${?}
)
}

function onRestoreDatabase(){
(
local OPTIND
local unset flags
while getopts : FLAG; do
case $FLAG in
? ) flags+="-${OPTARG} ";;
esac
done
shift $((OPTIND-1))

_databaseSpec=${1}
_fileName=${2}
_adminPassword=${3}

_hostname=$(getHostname ${flags} ${_databaseSpec})
_database=$(getDatabaseName ${_databaseSpec})
_port=$(getPort ${flags} ${_databaseSpec})
_portArg=${_port:+",${_port}"}
_username=$(getLocalOrDBUsername ${_databaseSpec} ${_hostname})
_password=$(getPassword ${_databaseSpec})
echo -e "Restoring '${_fileName}' to '${_hostname}${_port:+:${_port}}${_database:+/${_database}}' ...\n" >&2

#force single user mode on database to ensure restore works properly
sqlcmd -S ${_hostname}${_portArg} -U ${_username} -P ${_password} -Q "ALTER DATABASE ${_database} SET SINGLE_USER WITH ROLLBACK AFTER 30;RESTORE DATABASE ${_database} FROM DISK = N'${_fileName}' WITH FILE = 1, NOUNLOAD, REPLACE, STATS=5;ALTER DATABASE ${_database} SET MULTI_USER"
return ${?}
)
}

function onStartServer(){
(
local OPTIND
local unset flags
while getopts : FLAG; do
case $FLAG in
? ) flags+="-${OPTARG} ";;
esac
done
shift $((OPTIND-1))

_databaseSpec=${1}

/opt/mssql/bin/sqlservr --accept-eula &
)
}

function onStopServer(){
(
local OPTIND
local unset flags
while getopts : FLAG; do
case $FLAG in
? ) flags+="-${OPTARG} ";;
esac
done
shift $((OPTIND-1))

_hostname=$(getHostname ${flags} ${_databaseSpec})
_port=$(getPort ${flags} ${_databaseSpec})
_portArg=${_port:+",${_port}"}
_username=$(getLocalOrDBUsername ${_databaseSpec} ${_hostname})
_password=$(getPassword ${_databaseSpec})

sqlcmd -S ${_hostname}${_portArg} -U ${_username} -P ${_password} -Q "SHUTDOWN"
)
}

function onCleanup(){
(
if ! dirIsEmpty ${serverDataDirectory}; then
# Delete the database files and configuration
echo -e "Cleaning up ...\n" >&2
rm -rf ${serverDataDirectory}/*
else
echo -e "Already clean ...\n" >&2
fi
)
}

function onPingDbServer(){
(
local OPTIND
local unset flags
while getopts : FLAG; do
case $FLAG in
? ) flags+="-${OPTARG} ";;
esac
done
shift $((OPTIND-1))

_databaseSpec=${1}

_hostname=$(getHostname ${flags} ${_databaseSpec})
_database=$(getDatabaseName ${_databaseSpec})
_port=$(getPort ${flags} ${_databaseSpec})
_portArg=${_port:+",${_port}"}
_username=$(getLocalOrDBUsername ${_databaseSpec} ${_hostname})
_password=$(getPassword ${_databaseSpec})

if sqlcmd -S ${_hostname}${_portArg} -U ${_username} -P ${_password} -Q "SELECT 1" >/dev/null 2>&1; then
return 0
else
return 1
fi
)
}

function onVerifyBackup(){
(
local OPTIND
local unset flags
while getopts : FLAG; do
case $FLAG in
? ) flags+="-${OPTARG} ";;
esac
done
shift $((OPTIND-1))

_databaseSpec=${1}
_hostname=$(getHostname -l ${_databaseSpec})
_database=$(getDatabaseName ${_databaseSpec})
_port=$(getPort -l ${_databaseSpec})
_portArg=${_port:+",${_port}"}
_username=$(getLocalOrDBUsername ${_databaseSpec} ${_hostname})
_password=$(getPassword ${_databaseSpec})

tables=$(sqlcmd -S ${_hostname}${_portArg} -U ${_username} -P ${_password} -d ${_database} -Q "SELECT table_name FROM information_schema.tables WHERE TABLE_CATALOG = '${_database}' AND table_type='BASE TABLE';")
rtnCd=${?}

# Get the size of the restored database
if (( ${rtnCd} == 0 )); then
size=$(getDbSize -l "${_databaseSpec}")
rtnCd=${?}
fi

if (( ${rtnCd} == 0 )); then
numResults=$(echo "${tables}"| wc -l)
if [[ ! -z "${tables}" ]] && (( numResults >= 1 )); then
# All good
verificationLog="\nThe restored database contained ${numResults} tables, and is ${size} in size."
else
# Not so good
verificationLog="\nNo tables were found in the restored database ${_database}."
rtnCd="3"
fi
fi

echo ${verificationLog}
return ${rtnCd}
)
}

function onGetDbSize(){
(
local OPTIND
local unset flags
while getopts : FLAG; do
case $FLAG in
? ) flags+="-${OPTARG} ";;
esac
done
shift $((OPTIND-1))

_databaseSpec=${1}

_hostname=$(getHostname ${flags} ${_databaseSpec})
_database=$(getDatabaseName ${_databaseSpec})
_port=$(getPort ${flags} ${_databaseSpec})
_portArg=${_port:+",${_port}"}
_username=$(getLocalOrDBUsername ${_databaseSpec} ${_hostname})
_password=$(getPassword ${_databaseSpec})

size=$(sqlcmd -S ${_hostname}${_portArg} -U ${_username} -P ${_password} -d ${_database} -Q "SELECT CONVERT(VARCHAR,SUM(size)*8/1024)+' MB' AS 'size' FROM sys.master_files m INNER JOIN sys.databases d ON d.database_id = m.database_id WHERE d.name = '${_database}' AND m.type_desc = 'ROWS' GROUP BY d.name")
rtnCd=${?}

echo ${size}
return ${rtnCd}
)
}

function getLocalOrDBUsername(){
(
_databaseSpec=${1}
_localhost="127.0.0.1"
_hostname=${2}
if [ "$_hostname" == "$_localhost" ]; then
_username=sa
else
_username=$(getUsername ${_databaseSpec})
fi
echo ${_username}
)
}
# =================================================================================================================
Loading

0 comments on commit 33fdad0

Please sign in to comment.