I came into a new project recently. One of the challenges was that the CICD pieces almost worked for developers, but fell short and all of the QA was being done because of that on the developer’s local laptop.
It works on my local… Actually and in fact.
Each developer would announce to the group new environmental variables required to run the app, and the developers present would add that env var to their personal .bashrc. Any developer not present, or missing the announcement of the change in slack or by email, would discover it on their own when their local stopped working after pulling down the latest code.
At some point his had to stop. The env var sets were used in docker-compose, to bring in environmental vars for the container image, in gitlab cicd pipelines, providing the env vars to test and to build images and deploy to s3, and in deployments to Kubernetes.
The resolution was top have a single source of truth for env vars, one for each environment. And to then create ansible templates to create docker-compose files when building images, env var files to source in gitlab CICD pipelines and then in kubernetes deployments.
I created a common directory to hold the ansible plays and templates, repo/common. Inside that directory are the ansible plays and templates for gitlab environmental vars, docker-compose creation, and creating the deployment yaml for kubernetes.
environmental vars for gitlab-ci.yml jobs
[python]
dsm_macbook:common mm26994 $ cat annuity_create_gatsby_vars.j2
#!/bin/bash
# local env var file to source for gatsby build process
# brings in the same env var file as the kubernetes deployment
{% for environ in app_environs -%}
export {{ environ[‘envname’] }}='{{ environ[‘envvalue’] }}’
{% endfor -%}
[/python]
I love jinja templating. This looping construct takes each entry in the environmental vars single source of truth file and puts it into a .profile or .bashrc type file which can then be sourced inline in the gitlab cicd job to instantiate the env vars in the cicd shell.
The play is:
[python]
—
# ansible playbook to populate env vars file to source for gatby
– hosts: localhost
become: yes
become_user: root
tasks:
# vars are scoped as variable_name
– name: annuity create gatsby vars | pull in vars
include_vars:
file: ‘{{ ENV_VAR_FILE }}’
# create the deployment file from template
# template references variables from imported file
# edits are made in that file
– name: annuity create gatsby vars | template the gatsby envvars source file
template:
src: ../common/annuity_create_gatsby_vars.j2
dest: ‘{{ ENVVAR_FILE_GATSBY }}’
mode: 0755
# local file <environment>_env_var is available
dsm_macbook:common mm26994 $
[/python]
This imports the env var file (which is passed from the gitlab-ci.yml file job script code as ansible-playbook –extra-vars “vars are defined in here”
The env file looks like
[python]
dsm_macbook:uatjr mm26994 $ cat uatjr_env_vars.yml
—
app_environs:
– envname: ‘ENV_NAME’
envvalue: ‘uatjr’
– envname: ‘QUILT_PORT’
envvalue: ‘4000’
– envname: ‘QUILT_ENDPOINT’
envvalue: ‘/graphme’
etc…
app_certs:
– envname: ‘PUBLIC_CERT’
envvalue: |-
|-
—–BEGIN CERTIFICATE—–
multiline key hash…
—–END CERTIFICATE—–
– envname: ‘PRIVATE_KEY’
envvalue: |-
|-
—–BEGIN RSA PRIVATE KEY—–
multiline key hash…
—–END RSA PRIVATE KEY—–
REPLICAS: ‘3’
DEPLOYMENT_NAME: ‘annuity-uat-deploy’
CONTAINER_NAME: ‘annuity-uat-cont’
[/python]
docker-compose
The same pattern works for docker-compose files.
The template – actually there are two different templates, one for production and one for everything else.
[python]
dsm_macbook:common mm26994 $ cat annuity_create_docker-compose.dev.j2
version: ‘3.0’
services:
postgres:
image: postgres
restart: always
ports:
– ‘5432:5432’
environment:
POSTGRES_USER: admin
POSTGRES_PASSWORD: password
POSTGRES_DB: annuity
logging:
driver: awslogs
options:
awslogs-group: /ecs/annuity-dev
awslogs-region: aws_region
awslogs-stream-prefix: annuity
annuity:
image: ecr repo address:${CI_COMMIT_SHORT_SHA}
build:
context: ../../
dockerfile: docker/dev/Dockerfile
restart: ‘no’
environment:
{% for environ in app_environs -%}
{{ environ[‘envname’] }}: ‘{{ environ[‘envvalue’] }}’
{% endfor -%}
# throwaway line to buffer indent artifacts…
depends_on:
– postgres
ports:
– ‘3400:3400’
– ‘8080:8080’
logging:
driver: awslogs
options:
awslogs-group: /ecs/annuity-dev
awslogs-region: us-east-2
awslogs-stream-prefix: annuity
entrypoint: run
[/python]
and the play…
[python]
dsm_macbook:common mm26994 $ cat annuity_create_docker-compose.yml
—
# ansible playbook to populate docker-compose.yml template
– hosts: localhost
become: yes
become_user: root
tasks:
# vars are scoped as variable_name
– name: annuity create docker-compose.yml | pull in vars
include_vars:
file: ‘{{ ENV_VAR_FILE }}’
# create the docker-compose.yml file from template
# template references the variable_name from imported file
# edits are made in that file
– name: annuity create docker-compopse.yml | template the deployment
template:
# annuity_create_docker-compose.master.j2 (master)
# annuity_create_docker-compose.dev.j2 (dev)
src: ‘{{ DOCKER_COMPOSE_TEMPLATE }}’
# DOCKERE_COMPOSE_FILE will be the relative path to drop the docker-compose.yml file,
# e.g., ../../docker/dev/docker-compose.yml
dest: ‘{{ DOCKER_COMPOSE_FILE }}’
mode: 0644
# this will be called in .gitlab-ci.yml, after cd /docker/[master|dev]
# so creates a local docker-compose.yml
[/python]
there is a different template for production as that has slightly different pathing and requirements.
kubernetes deployment
Same pattern
[python]
dsm_macbook:common mm26994 $ cat annuity_create_deployment.
annuity_create_deployment.j2 annuity_create_deployment.yml
dsm_macbook:common mm26994 $ cat annuity_create_deployment.j2
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
name: {{ DEPLOYMENT_NAME }}
spec:
replicas: {{ REPLICAS }}
selector:
matchLabels:
app: annual
template:
metadata:
labels:
app: annual
spec:
containers:
– name: {{ CONTAINER_NAME }}
image: ECR repo:{{ DEPLOYMENT_IMAGE }}
imagePullPolicy: Always
env:
{% for environ in app_environs -%}
– name: {{ environ[‘envname’] }}
value: ‘{{ environ[‘envvalue’] }}’
{% endfor -%}
{% for cert in app_certs -%}
– name: {{ cert[‘envname’] }}
value: {{ cert[‘envvalue’] }}
{% endfor -%}
# this line buffers indent artifact…
command: [‘/bin/sh’]
args: [‘-c’, ‘start:server’]
ports:
– name: serverport
containerPort: 3400
– name: restport
containerPort: 7070
[/python]
single source of truth
From this devs can edit a single file and it will be distributed as needed out into gitlab, docker and kubernetes.
— Doug