…, or Templating GitHub Branch Source Org config.xml
I created a yaml file ghe_config.yml to provide the values for a jinja template which created the GitHub Branch Source Organization config.xml for a Github organization on Jenkins. These were brought together by an ansible play that explicitly pulled in the ghe_config.yml values and then pushes those values into the template.
The play is fairly simple, and directly addresses the ghe_config.yml file.
---
- hosts: localhost
become: yes
become_user: root
tasks:
# import the nebula-ghe yaml file
# located by the call to ansible-playbook, e.g., ansible-playbook --extra-vars "yamldir=${YAMLDIR}"
- name: ghe template | import yaml
include_vars:
file: "./ghe-config.yml"
name: ghe
- name: ghe template | display all variables/facts
debug:
var: hostvars[inventory_hostname]
tags: debug_info
- name: ghe template | check for ghe-displayname
debug:
msg: "ghe_displayname set as {{ ghe.ghe_displayname }}"
# template this bad boy
- name: ghe template | create ghe branch source config.xml for this org
template:
src: /var/lib/jenkins/cicdTemplates/nebula-ghe-config.xml.j2
dest: "./nebula-ghe-config.xml"
That’s the simple part. This play, and the template file nebula-ghe-config.xml.j2 and the ghe_config.yml, plus a script and a couple of more ansible plays allow the CICD Discover plugin to accommodate GitHub Branch Source Organization jobs, creating them automagically on the fly…
Crap. Let me back up a bit. I had a developer team that really really wanted to stay with GitHub Branch Source Organization jobs. These are multibranch pipeline jobs organized by directly communicating and scanning a GitHub organization and creating a separate branch job and Pull Request job on the fly. It streamlines the interaction between GitHub and Jenkins UI tremendously, but also imposes a state on Jenkins, in that one assumption is that there is a single Jenkins master. Period. Only one that would ever contain this organization.
I replace a Jenkins buildfarm by changing the cname and pointing it to the new AWS cloud formation. Thus, we often have two buildfarm clouds up at the same time, one newer and addressed by the cname, the other staying at the previous version as backup, and leapfrogging between the two. The GitHub Branch Source configuration allows nightly scanning of the organization to catch any changes needing to be built not captured by a direct notifyCommit message from GitHub. If you do not stop the nightly polling of the GitHub servers by the local configuration for the GitHub Branch Source Organization, here’s the sequence of what unexpectedly goes wrong.
First your active Jenkins, the one currently holding the name, receives the notifyCommit generated by a pull request and sets up a job and builds. Great.
Then that evening the second server, not actively taking traffic but with the nightly polling still running, discovers that commit by asking GitHub for what’s changed in this organization? It then builds and potentially deploys code. And you get a call from a dev or operations guy going what happened???
But it’s pretty, and I made it work inside a buildfarm with a single Jenkins. So far there’s no way I’ve discovered to have this work on a stateless buildfarm with multiple Jenkins controllers and have it do any controller-initiated scan of the GitHub organization. The notifyCommit hits the ELB, passes back to a controller, builds. If any of the other controllers later do a scan they build, too.
This is not a plugin we really want except under very specific circumstances, like an intractable developer. In fact I think I’ll separate the GitHub Branch Source code stuff back out of CICD Discover and make a second plugin to capture the code, then revert CICD Discover to stateless kick ass pipeline creation on the fly.
The ansible play above imports the ghe_config.xml variable file directly. That play addresses a template nebula-ghe-config.xml.j2:
{{ ghe.ghe_description }} {{ ghe.ghe_displayname }} {% if ghe.ghe_org_libraries is defined -%} {% if ghe.ghe_credentials is defined -%} {% for credentials in ghe.ghe_credentials -%} svc-nebulacicd svc-nebulacicd {password} {% endfor -%} {% endif -%} {{ credentials.id }} {{ credentials.description }} {{ credentials.username }} {{ credentials.password }} {% endif -%} {% for libs in ghe.ghe_org_libraries -%} {% endfor -%} {{ libs.libname }} {{ libs.generated_retrieve_id }} {{ libs.project_repo }} {{ libs.lib_credentials_id }} {% if libs.behaviors_discover_branches == 'true' or libs.behaviors_discover_tags == 'true' -%}{% if libs.behaviors_discover_branches == 'true' -%} {% else -%}{% endif -%} {% if libs.behaviors_discover_tags == 'true' -%} {% endif -%} {% endif -%} {{ libs.default_version }} {{ libs.load_implicitly }} {{ libs.allow_override_default_version }} {{ libs.include_at_lib_changes_in_recent_job_changes }} {{ ghe.ghe_docker_label }} {{ ghe.ghe_branch_build_regex }} false {% if ghe.ghe_trigger_spec is defined and ghe.ghe_trigger_interval is defined -%} {{ ghe.ghe_prune_dead_branches }} {{ ghe.ghe_days_to_keep_old_items }} {{ ghe.ghe_max_num_old_items_to_keep }} {% else -%} {{ ghe.ghe_trigger_spec }} {{ ghe.ghe_trigger_interval }} {% endif -%} false {{ ghe.ghe_repo_owner }} {{ ghe.ghe_api_endpoint }} {{ ghe.ghe_repo_assigned_credential }} {{ ghe.ghe_branch_discover }} {{ ghe.ghe_origin_pr_discovery }} {% if ghe.ghe_adv_clone_options == 'true' -%} {{ ghe.ghe_fork_pr_discovery }} {% endif -%} {% if ghe.ghe_localbranch is defined -%} {{ ghe.ghe_adv_clone_opt_shallow }} {{ ghe.ghe_adv_clone_opt_notags }} {{ ghe.ghe_adv_clone_opt_ref }} {{ ghe.ghe_adv_clone_opt_depth }} {{ ghe.ghe_adv_clone_opt_honorrefspec }} {% endif -%} {% if ghe.ghe_ssh_checkout_trait_credentials is defined -%} ** {% else %} {{ ghe.ghe_ssh_checkout_trait_credentials }} {% endif %} {% if ghe.ghe_wipe_workspace is defined -%} {% if ghe.ghe_wipe_workspace == 'true' -%} {% endif -%} {% endif -%} {% if ghe.ghe_project_recognizers is defined -%} {% for recognizers in ghe.ghe_project_recognizers -%} cicd/pipeline/Jenkinsfile {% endfor -%} {% endif -%} {{ recognizers.scriptpath }}
and the ghe_config.yml variables file imported directly is
--- # this yaml file aligns with the Jenkins UI for Github Branch Source Configuration # ------------------------------------ # # General # ------------------------------------ # # Name in the UI is derived from the directory at /var/lib/jenkins/jobs/organization_name and is not templatized # template code: #{{ ghe.ghe_description }} #{{ ghe.ghe_displayname }} # --- displayname if left blank defaults to folder name "ghe_displayname": "Nebula-GHE-Org-Example" # description "ghe_description": "" # ------------------------------------ # # Projects # ------------------------------------ # ### credentials # template code: ## #"ghe_credentials": [ # { # id: "dmunsinger", # description: "", # username: "dmunsinger20", # password: "{defined}" # } # ] ### Api Endpoint # template code: ## # {% if ghe.ghe_credentials is defined -%} # {% for credentials in ghe.ghe_credentials -%} #svc-nebulacicd ## svc-nebulacicd #{password} ## # {% endfor -%} # {% endif -%} #{{ credentials.id }} #{{ credentials.description }} #{{ credentials.username }} #{{ credentials.password }} #{{ ghe.ghe_api_endpoint }} ghe_api_endpoint: "https://git.ouroath.com/api/v3" ### Credentials (assign credential to use to scan repo) # template code: #{{ ghe.ghe_repo_assigned_credential }} ghe_repo_assigned_credential: "svc-nebulacicd" ### repo owner # template code: #{{ ghe.ghe_repo_owner }} ghe_repo_owner: "Nebula-GHE-Org-Example" #### Behaviors ### Discover Branches # three possible variations for each behavior, 1, 2 or 3. # strategy id 1: Exclude branches that are also filed as PRs # strategy id 2: Only branches that are also files as PRs # strategy id 3: All branches # template code: ## ghe_branch_discover: "1" ### Discover Pull Requests from Origin # strategy id 1: Merging the pull request with the current target branch revision # strategy id 2: The current pull request revision # strategy id 3: Both the current pull request revision and the pull request merged with the current target branch revision # template code: #{{ ghe.ghe_branch_discover }} ## ghe_origin_pr_discovery: "1" ### Discover Pull Requests from Forks # strategy id 1: Merging the pull request with the current target branch revision # strategy id 2: The current pull request revision # strategy id 3: Both the current pull request revision and the pull request merged with the current target branch revision # template code: #{{ ghe.ghe_origin_pr_discovery }} ## ghe_fork_pr_discovery: "1" # trust permission #{{ ghe.ghe_fork_pr_discovery }} ## # this is not templatized - nothing configured in the Jenkins UI in this section affects the config,xml ### --- Additional --- ## Advanced Clone Behaviors # template code: # {% if ghe.ghe_adv_clone_options == 'true' -%} # # # {% endif -%} # ghe_adv_clone_options set to anything but "true" causes this block to disappear from the config file # clone options # ghe_adv_clone_options would be set to true if valid and false, say, to make invalid # this value must be set ghe_adv_clone_options: "false" # these value would be required IF ghe_adv_clone_options were set to true... #ghe_adv_clone_opt_shallow: "false" #ghe_adv_clone_opt_notags: "false" #ghe_adv_clone_opt_ref: "" #ghe_adv_clone_opt_depth: "0" #ghe_adv_clone_opt_honorrefspec: "false" ## Checkout to matching local branch # template code: # {% if ghe.ghe_localbranch is defined -%} ## #{{ ghe.ghe_adv_clone_opt_shallow }} #{{ ghe.ghe_adv_clone_opt_notags }} #{{ ghe.ghe_adv_clone_opt_ref }} #{{ ghe.ghe_adv_clone_opt_depth }} #{{ ghe.ghe_adv_clone_opt_honorrefspec }} ## # {% endif -%} # comment out this value to remove this block from the config file # local branch #ghe_localbranch: '**' ## Checkout over SSH # "use build agent's key" causes the config to be ## #{{ ghe.ghe_localbranch }} ## configuring a key looks like # # # template code: # {% if ghe.ghe_ssh_checkout_trait_credentials is defined -%} #c0e21fda-697c-433c-8f1f-74e9f018b0aa ## # {%- else -%} #{{ ghe.ghe_ssh_checkout_trait_credentials }} ## {% endif -%} # comment out this value to revert to local build agent's key block # assign a valid reference to credentials to bring in the configured credentials block # this would assign the svc-nebulacicd user for ssh checkouts #ghe_ssh_checkout_trait_credentials: "c0e21fda-697c-433c-8f1f-74e9f018b0aa" ghe_ssh_checkout_trait_credentials: "c0e21fda-697c-433c-8f1f-74e9f018b0aa" ## Wipe Workspace and Force Clone # define as true or anything but true # template code: # {% if ghe.ghe_wipe_workspace == 'true' -%} # # # {% endif -%} ghe_wipe_workspace: 'false' ### Project Recognizers # template code: ## # # # {% if ghe.ghe_project_recognizers is defined -%} # {% for recognizers in ghe.ghe_project_recognizers -%} #cicd/pipeline/Jenkinsfile ## # {% endfor -%} # {% endif -%} # one or more project recognizers can be configured here. # The default "cicd/pipeline/Jenkinsfile" will always be present #"ghe_project_recognizers": [ # { # scriptpath: "Jenkinsfile" # } # ] # ----------------------------------- # Scan Org Triggers # ------------------------------------ # # template code: # {% if ghe.ghe_trigger_spec is defined and ghe.ghe_trigger_interval is defined -%} #{{ recognizers.scriptpath }} ## # {% else -%} ## #{{ ghe.ghe_trigger_spec }} #{{ ghe.ghe_trigger_interval }} ## {% endif -%} # # both values must be configured or the block reverts to " " ghe_trigger_spec: "H H * * *" ghe_trigger_interval: "86400000" # ------------------------------------ # # Orphaned Items Strategy # ------------------------------------ # # # ## Discard Old Items ghe_prune_dead_branches: "true" ## Days to keep old items ghe_days_to_keep_old_items: "5" ## Max # of old items to keep ghe_max_num_old_items_to_keep: "5" # ------------------------------------ # # ***Health Metrics # ------------------------------------ # # at present nothing done in the UI afects this block # worst child health check is the default and is neithr added nor removed by UI # ------------------------------------ # # ***Properties # ------------------------------------ # # not currently configurable (20180808) # ------------------------------------ # # Pipeline Libraries # ------------------------------------ # # template code: # {% if ghe.ghe_org_libraries is defined -%} #{{ ghe.ghe_prune_dead_branches }} #{{ ghe.ghe_days_to_keep_old_items }} #{{ ghe.ghe_max_num_old_items_to_keep }} ## # {% endif -%} # if there are not configurations for ghe_org_libraries, the block goes away # at present (since that's all that have been used so far) only "Discover branches" and "Discover tags" are available as behaviors # if more are needed, open a ticket to have them added, with exact specifics # org library config #"ghe_org_libraries": [ # { # libname: "library_to_bring_in", # default_version: "master", # load_implicitly: "false", # allow_override_default_version: "true", # include_at_lib_changes_in_recent_job_changes: "true", # project_repo: "git@gitserver:organization/repo.git", # lib_credentials_id: "your_creds_id", # behaviors_discover_branches: "true", # behaviors_discover_tags: "true", # generated_retrieve_id: "committed" # } # ] # notes: # generated_retrieval_id - this is a number generated by jenkins when it creates a library block # to create this id configure your library in a Jenkins UI # and capture the# {% for libs in ghe.ghe_org_libraries -%} # ## # {% endfor -%} #{{ libs.libname }} ## ## #{{ libs.generated_retrieve_id }} #{{ libs.project_repo }} #{{ libs.lib_credentials_id }} # {% if libs.behaviors_discover_branches == 'true' or libs.behaviors_discover_tags == 'true' -%} ## {% if libs.behaviors_discover_branches == 'true' -%} # # {% else -%} ## {% endif -%} # {% if libs.behaviors_discover_tags == 'true' -%} # # {% endif -%} # # {% endif -%} # {{ libs.default_version }} #{{ libs.load_implicitly }} #{{ libs.allow_override_default_version }} #{{ libs.include_at_lib_changes_in_recent_job_changes }} #string right above the some git url block # ------------------------------------ # # Pipeline Model Definition (Docker) # ------------------------------------ # # template code: ## # docker label ghe_docker_label: "" # ------------------------------------ # # Automatic Branch Project Triggering # ------------------------------------ # # template code: #{{ ghe.docker_label }} ## # # branch build regex ghe_branch_build_regex: '(^PR-\d*)' # ------------------------------------ # # *** - indicates a section that exists in the UI, but doesn't change the underlying config.xml # future plugin endpoints{{ ghe.branch_build_regex }} #
There’s one other under the hood manipulation – the config.xml contains a list of regex expressions to identify builds to trigger – that’s:
ghe_branch_build_regex: '(^PR-\d*)'
When the job initially comes up and scans, if those regex’s are in place, the organization builds whatever matches. Since this is brand new when the CICD Discover plugin (soon to be a separate plugin – say, CICDGitHubBranchSourceDiscover Plugin) creates the config, and I pass a flag to the shell script that executes the creation of the config (running the playbook).
The third arg ($3) to this script is yes or no regex in the config.xml. I run the script twice inside the CICD Discover plugin. The first with a “no” argument, removes the values in that key, and creates the organization structure and then scans the GutHub organization with no regex build triggers. The second pass calls the script with a “yes” and brings back the regex values in the key, creating a complete config.xml and updating the job with it. Adding the regex triggers back doesn’t seem to cause the scan-and-just-build-everything behavior.
#! /bin/bash
# script to run the ansible-playbook and cobble together the config.xml for github branch source configs
# called by cicd discover
# verify args
if [[ (-z "$1") || (-z $2) ]]
then
echo "USAGE: $0 [noregex]"
exit 1
fi
# ProcessBuilder templateme = new ProcessBuilder(jen_home + "/scripts/" + "javaTemplateMeAConfigXmlFromYaml.sh", gitCommitId, gitProject);
# gitCommitId
gitCommitId=${1}
# gitProject
gitProject=${2}
# if no regex is present, it is not assigned, just used to remove the regex from the config before creating it
# File g = new File("/tmp/jenkins/gheorg/" + gitProject + "/" + gitCommitId + "/nebula-ghe/ghe-config.yml");
DIR="/tmp/jenkins/gheorg/${gitProject}/${gitCommitId}/nebula-ghe"
# this should already exist and have the ghe-config.yml file in place,
# then this script is called by cicd discover...
# log
LOG="${DIR}/javaTemplateMeConfigXml.sh.log"
date +%Y%m%d%H%M%S > $LOG
echo "to the directory!!!" >> $LOG
cd ${DIR} >> $LOG 2>&1
# if $3 = noregex, run this to remove regex first
if [[ -n $3 ]]; then
echo "found 3rd arg: $3" >> $LOG
if [[ "${3}" == 'noregex' ]]; then # we remove the regex this run
echo "removing regex: $3" >> $LOG
cp /etc/ansible/config_playbooks/nebula-ghe-regex-* ${DIR} >> $LOG 2>&1
ansible-playbook nebula-ghe-regex-out.yml >> $LOG 2>&1 #removes rhe regex
else
echo "sent third arg: $3: but this is not noregex string" >> $LOG
fi
fi
# copy the ansible play into this directory
echo "copy ansible play into temp dir..." >> $LOG
cp -f /etc/ansible/config_playbooks/nebula-ghe-template-engine.yml ${DIR} >> $LOG 2>&1
echo "run the play..." >> $LOG
ansible-playbook nebula-ghe-template-engine.yml >> $LOG 2>&1
# is the file there?
if [[ -f ${DIR}/nebula-ghe-config.xml ]]; then
echo "file created nebula-ghe-config SUCCESS" >> $LOG
exit 0
else
echo "FAIL file not found nebula-ghe-config.xml" >> $LOG
exit 1
fi
Then there are two really simple ansible plays manipulating the regex.
nebula-ghe-regex-out.yml, removes the regex from the config.xml by manipulating the value in the ghe_config.yml values file
---
- hosts: localhost
become: yes
become_user: root
tasks:
- name: regex out | backup original ghe-config.yml
shell: 'cp ghe-config.yml ghe-config.yml.holdmesafe'
- name: regex out | remove regex from branches tag
lineinfile:
path: ghe-config.yml
regexp: '^ghe_branch_build_regex:.*$'
line: 'ghe_branch_build_regex: ""'
and nebula-ghe-regex-put-back.yml, which puts the value back by copying back the backup file created by the first play
---
- hosts: localhost
become: yes
become_user: root
tasks:
- name: regex put back | put back the yml file
shell: 'cp -f ghe-config.yml.holdmesafe ghe-config.yml'
We probably won’t push this into the product going forward – it doesn’t match with a stateless Jenkins that just runs and it requires making the Jenkins UI available and accessible for devs and defeats the black box the product should be to scale and self-heal.
— doug