task/FP-1120 & task/FP-1136: Allow standalone services (e.g. CMS/Docs…

…/Portal) (#19) * ignore .idea IDE project directoy * Breaking up services into independent composable files to facilitate fully customizable suite of services. Including a default configuration for commonly deployed services * updating Makefile with full path to overrides compose overrides * Including env file in override * use relative paths * Use home environment variable to help find configs * use home var to find configs * moving base compose files * gitignoring custom overrides--not tracked here * Updating readme and changing the default docs command to build * Update Makefile with default CAMINO_HOME Co-authored-by: Sal Tijerina <[email protected]> * task/fp 1120 1135: Update samples, gitignore, and fix CAMINO_HOME (#20) * update samples; organize .gitgnore * formatting * move CAMINO_HOME declaration to top * fix CAMINO_HOME var * ignore all but example in conf/portal * add sample "CAMINO_HOME" setting Co-authored-by: Sal Tijerina <[email protected]>
TACC · Nov 1, 2021 · 26a10b4 · 26a10b4
1 parent 2398e75
commit 26a10b4
Show file tree

Hide file tree

Showing 19 changed files with 386 additions and 331 deletions.
diff --git a/.gitignore b/.gitignore
@@ -3,6 +3,7 @@ __pycache__/
 *.py[cod]
 *$py.class
 .idea/
+
 # C extensions
 *.so
 
@@ -31,17 +32,17 @@ MANIFEST
 local_settings.py
 db.sqlite3
 db.sqlite3-journal
-rabbitmq.env
-elasticsearch.yml
-*secrets.py
-*secrets.env
-*.secret
-*settings_secret.py
-*settings_secret.env
+
+# Camino configuraitons from Core-Portal-Deployments
 .env
 conf/camino/*
 !conf/camino/*sample.*
-camino_env
-compose_file
-make_env
 
+# Secrets and Configurations
+*settings_secret.py
+*settings_custom.py
+*settings_local.py
+*secrets.py
+rabbitmq.env
+conf/portal/*
+!conf/portal/*example.*
diff --git a/Makefile b/Makefile
@@ -1,7 +1,25 @@
 #!make
 ENV_FILE ?= ./.env
 include $(ENV_FILE)
-DOCKER_COMPOSE :=  docker-compose -f docker-compose.yml -f ./conf/camino/$(COMPOSE_FILE) --env-file=$(ENV_FILE)
+
+# CAMINO_HOME defaults to the path to this Makefile
+ifndef CAMINO_HOME
+override CAMINO_HOME := $(dir $(lastword $(abspath $(MAKEFILE_LIST))))
+endif
+
+ifndef BASE_COMPOSE_FILE
+override BASE_COMPOSE = -f ${CAMINO_HOME}/conf/compose/docker-compose.core.cms.yml
+else
+override BASE_COMPOSE = -f ${CAMINO_HOME}/conf/compose/${BASE_COMPOSE_FILE}
+endif
+
+ifndef COMPOSE_FILE
+override COMPOSE_OVERRIDE =
+else
+override COMPOSE_OVERRIDE = -f ${CAMINO_HOME}/conf/camino/${COMPOSE_FILE}
+endif
+
+DOCKER_COMPOSE :=  docker-compose ${BASE_COMPOSE} ${COMPOSE_OVERRIDE} --env-file=$(ENV_FILE)
 
 .PHONY: deploy-core
 deploy-core:

diff --git a/README.md b/README.md
@@ -13,40 +13,44 @@ A compose file exists for each environment to orchestrate deployments. Each appl
 Setup:
 Clone the Camino repo to the path `/opt/portal`
 
-At the root of the cloned project create the env file `.env` and add the name of the environment file to be used to manage deployments. The env files are named according the deployment environment they manage, like `dev.env, pprd.env, prod.env` and live in the directory `conf/camino`. With the env file set, make will load the environment variables and pass them to the compose file.
+At the root of the cloned project create the env file `.env` and add the environment variables
+that indicate which docker-compose files should be used. With the env file set, Make will 
+load the environment variables and pass them to the compose file.
 
-Ex. To manage and deploy dev, use the filename `dev.env`, this file will be passed to docker-compose with `--env-file=./conf/camino/dev.env`
+The .env file can be used to [populate values in the compose file](https://docs.docker.com/compose/environment-variables/)
+and in conjunction with the Makefile helps create custom service sets.
 
-This file can be used to [populate values in the compose file](https://docs.docker.com/compose/environment-variables/)
+### Quick-Start Example
 
-To deploy changes to a service, the image name and tag (or digest) for the service should be set in the appropriate compose file (or, if using a var, the matching env file in `conf/camino`). This means that the image must be pre-built and its tag or digest known before deployment.
-
-For example, to deploy a new pre-built cms image to production, the image in docker-compose.yml would be updated like below:
-
-```
-services:
-  cms:
-    image: taccwma/frontera-cms:{tag}
-    volumes:
+1. Clone this repo
+2. Create a `.env` file and populate with:
 ```
-to
-```
-services:
-  cms:
-    image: taccwma/frontera-cms:{tag}
-    volumes:
+BASE_COMPOSE_FILE=docker-compose.nginx.yml
+COMPOSE_FILE=
+CAMINO_HOME=${PWD}
 ```
+3. Run `make up`
+4. Navigate to `localhost`
 
-To update a dev environment, a similar change would happen in the docker-compose-dev.yml file.
+Services can be customized without modifying base
+compose files by creating overrides files in `conf/camino/`
 
-Additionally, to facilitate continuous integration without constant updates to the Camino repo, a service in a compose file can be tagged with `:latest`. New images built and published with the tag `:latest` can then be deployed without requiring updates to Camino.
+##### To override the version of nginx image used in the previous example
 
+1. Create file `docker-compose.nginx.overrides.yml` In `conf/camino` and paste in:
 ```
+version: "3.8"
 services:
-  cms:
-    image: taccwma/frontera-cms:latest
-    volumes:
+  nginx:
+    image: nginx:stable
 ```
+2. In the .env file add entry, `COMPOSE_FILE=docker-compose.nginx.overrides.yml`
+3. run make up
+4. Note that the nginx image with tag `stable` is being pulled down
+5. Navigating to `localhost` now displays the default page for the new image
+
+To facilitate continuous integration without updating the Camino repo, 
+a service in a compose file can be tagged with `:latest`. New images built and published with the tag `:latest` can then be deployed without requiring updates to Camino.
 
 
 Deploying with Jenkins:

diff --git a/conf/camino/sample.env b/conf/camino/sample.env
@@ -5,3 +5,5 @@ COMPOSE_FILE=
 CMS_TAG=
 PORTAL_TAG=
 SSH_HOST=
+CUSTOM_SETTINGS_PORTAL=
+CAMINO_HOME=/opt/portal/Camino
diff --git a/conf/cms/secrets.sample.py b/conf/cms/secrets.sample.py
@@ -1,28 +1,43 @@
-_SECRET_KEY = 'change_me'
-_DEBUG = True
-_ALLOWED_HOSTS = ['*']
-_SITE_ID = 1
-_CMS_TEMPLATES =  (
-    ## Customize this
-    ('base.html', 'Base'),
-    ('cms_menu.html', 'CMS Menu'),
-    ('fullwidth.html', 'Fullwidth'),
-    ('sidebar_left.html', 'Sidebar Left'),
-    ('sidebar_right.html', 'Sidebar Right')
-)
+# SECRET SETTINGS VALUES.
+# SAMPLE.TACC.UTEXAS.EDU
 
-# As needed:
-#_CACHES = {
-#    'default': {
-#        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
-#        'LOCATION': 'frontera_portal_memcached:11211',
-#    }
-#}
+########################
+# DJANGO SETTINGS
+########################
 
+SECRET_KEY = 'change_me'
+ALLOWED_HOSTS = ['sample.tacc.utexas.edu']
+CEP_AUTH_VERIFICATION_ENDPOINT = 'https://sample.tacc.utexas.edu'
 
-_DATABASE_ENGINE = 'django.db.backends.postgresql'
-_DATABASE_NAME = 'postgres'
-_DATABASE_USERNAME = 'postgres'
-_DATABASE_PASSWORD = 'secretpw'
-_DATABASE_HOST = 'postgres_host'
-_DATABASE_PORT = 5432
+########################
+# DATABASE SETTINGS
+########################
+
+DATABASES = {
+    'default': {
+        'ENGINE': 'django.db.backends.postgresql_psycopg2',
+        'PORT': '5432',
+        'NAME': 'sample_cms',
+        'USER': 'sample_cms',
+        'PASSWORD': 'change_me',
+        'HOST': 'portal_postgres'
+    }
+}
+
+########################
+# ELASTICSEARCH
+########################
+
+ES_HOSTS = 'https://esclient:9200'
+ES_AUTH = 'sample:change_me'
+ES_INDEX_PREFIX = 'sample-{}'
+ES_DOMAIN = 'https://sample.tacc.utexas.edu'
+
+HAYSTACK_CONNECTIONS = {
+    'default': {
+        'ENGINE': 'haystack.backends.elasticsearch_backend.ElasticsearchSearchEngine',
+        'URL': ES_HOSTS,
+        'INDEX_NAME': ES_INDEX_PREFIX.format('cms'),
+        'KWARGS': {'http_auth': ES_AUTH }
+    }
+}
diff --git a/conf/compose/docker-compose.cms.yml b/conf/compose/docker-compose.cms.yml
@@ -0,0 +1,16 @@
+version: "3.8"
+services:
+  cms:
+    image: ${CMS_IMAGE}:${CMS_TAG}
+    volumes:
+      - ${CAMINO_HOME}/conf/cms/secrets.py:/code/taccsite_cms/secrets.py
+      - ${CAMINO_HOME}/conf/uwsgi/uwsgi_cms.ini:/code/uwsgi.ini
+      - /var/www/portal/cms/static:/code/static
+      - /var/www/portal/cms/media:/code/media
+    command: ["uwsgi", "--ini", "uwsgi.ini"]
+    container_name: portal_cms
+    logging:
+      driver: syslog
+      options:
+        tag: portal_cms
+    restart: always
diff --git a/conf/compose/docker-compose.core.cms.yml b/conf/compose/docker-compose.core.cms.yml
@@ -0,0 +1,63 @@
+version: "3.8"
+services:
+  nginx:
+    extends:
+      file: ${CAMINO_HOME}/conf/compose/docker-compose.nginx.yml
+      service: nginx
+    volumes:
+      - ${CAMINO_HOME}/conf/nginx/nginx.conf:/etc/nginx/nginx.conf
+      - ${CAMINO_HOME}/conf/uwsgi/uwsgi_params:/etc/nginx/uwsgi_params
+      - /etc/ssl/dhparam.pem:/etc/ssl/dhparam.pem
+      - /var/www/portal/cms/static:/var/www/portal/cms/static
+      - /var/www/portal/cms/media:/var/www/portal/cms/media
+      - /var/www/portal/portal/media:/var/www/portal/portal/media
+      - /var/www/portal/portal/static:/var/www/portal/portal/static
+    ports:
+      - 80:80
+      - 443:443
+    container_name: portal_nginx
+    logging:
+      driver: syslog
+      options:
+        tag: portal_nginx
+    restart: always
+
+  memcached:
+    extends:
+      file: docker-compose.memcached.yml
+      service: memcached
+
+  redis:
+    extends:
+      file: docker-compose.redis.yml
+      service: redis
+
+  rabbitmq:
+    extends:
+      file: docker-compose.rabbitmq.yml
+      service: rabbitmq
+
+  websockets:
+    extends:
+      file: docker-compose.websockets.yml
+      service: websockets
+
+  core:
+    extends:
+      file: docker-compose.core.yml
+      service: core
+
+  workers:
+    extends:
+      file: docker-compose.workers.yml
+      service: workers
+
+  cms:
+    extends:
+      file: docker-compose.cms.yml
+      service: cms
+
+volumes:
+  portal_es_data:
+  portal_redis_data:
+  portal_rabbitmq_data:
diff --git a/conf/compose/docker-compose.core.docs.cms.yml b/conf/compose/docker-compose.core.docs.cms.yml
@@ -0,0 +1,78 @@
+version: "3.8"
+services:
+  nginx:
+    extends:
+      file: ${CAMINO_HOME}/conf/compose/docker-compose.nginx.yml
+      service: nginx
+    volumes:
+      - ${CAMINO_HOME}/conf/nginx/nginx.conf:/etc/nginx/nginx.conf
+      - ${CAMINO_HOME}/conf/uwsgi/uwsgi_params:/etc/nginx/uwsgi_params
+      - /etc/ssl/dhparam.pem:/etc/ssl/dhparam.pem
+      - /var/www/portal/cms/static:/var/www/portal/cms/static
+      - /var/www/portal/cms/media:/var/www/portal/cms/media
+      - /var/www/portal/portal/media:/var/www/portal/portal/media
+      - /var/www/portal/portal/static:/var/www/portal/portal/static
+    ports:
+      - 80:80
+      - 443:443
+    container_name: portal_nginx
+    logging:
+      driver: syslog
+      options:
+        tag: portal_nginx
+    restart: always
+
+  docs:
+    extends:
+      file: docker-compose.docs.yml
+      service: docs
+    image: taccwma/frontera-docs:latest
+    volumes:
+      - /var/www/portal/docs:/docs/site
+    command: ["mkdocs", "build"]
+    container_name: portal_user_guide
+    logging:
+      driver: syslog
+      options:
+        tag: portal_user_guide
+
+  memcached:
+    extends:
+      file: docker-compose.memcached.yml
+      service: memcached
+
+  redis:
+    extends:
+      file: docker-compose.redis.yml
+      service: redis
+
+  rabbitmq:
+    extends:
+      file: docker-compose.rabbitmq.yml
+      service: rabbitmq
+
+  websockets:
+    extends:
+      file: docker-compose.websockets.yml
+      service: websockets
+
+  core:
+    extends:
+      file: docker-compose.core.yml
+      service: core
+
+  workers:
+    extends:
+      file: docker-compose.workers.yml
+      service: workers
+
+  cms:
+    extends:
+      file: docker-compose.cms.yml
+      service: cms
+
+volumes:
+  portal_es_data:
+  portal_redis_data:
+  portal_rabbitmq_data:
+
diff --git a/conf/compose/docker-compose.core.yml b/conf/compose/docker-compose.core.yml
@@ -0,0 +1,19 @@
+version: "3.8"
+services:
+  core:
+    image: ${PORTAL_IMAGE}:${PORTAL_TAG}
+    volumes:
+      - ${CAMINO_HOME}/conf/portal/settings_secret.py:/srv/www/portal/server/portal/settings/settings_secret.py
+      - ${CAMINO_HOME}/conf/portal/settings_custom.py:/srv/www/portal/server/portal/settings/settings_custom.py
+      - /var/www/portal/portal/media:/srv/www/portal/server/media
+      - /var/www/portal/portal/static:/srv/www/portal/server/static
+      - ${CAMINO_HOME}/conf/uwsgi/uwsgi_core.ini:/srv/www/portal/server/conf/uwsgi/uwsgi_core.ini
+    stdin_open: true
+    tty: true
+    command: uwsgi --ini /srv/www/portal/server/conf/uwsgi/uwsgi_core.ini
+    container_name: portal_django
+    logging:
+      driver: syslog
+      options:
+        tag: portal_django
+    restart: always
diff --git a/conf/compose/docker-compose.docs.yml b/conf/compose/docker-compose.docs.yml
@@ -0,0 +1,6 @@
+version: "3.8"
+services:
+  docs:
+    image: taccwma/frontera-docs:latest
+    command: ["mkdocs", "build"]
+    container_name: portal_user_guide