leveraging Automation / CI/CD pipeline for Day2 activities? If yes what are those activities?
We leveraged Jenkins for CI/CD
The Benefits
Efficient Utilization of resource – OpenStack allows us to set the desired level of over-subscription for CPU, Memory and Storage resources, and then to continuously monitor current allocation and utilization levels to tweak this accordingly, ultimately to get optimal utilization of resources.
End-To-End Automation – The entire test lifecycle is 100% automated, with very little manual intervention. This means very high developer productivity.
Better Visibility – Our DevOps folks have much better visibility into artifacts for current running builds and workloads. They are available for debugging and testing both via UI and API when needed.
The Ingredients (Requirements)
An OpenStack deployment configured with sufficient compute, memory, storage infrastructure to run our Jenkins CI/CD setup.
Vanilla VM images with base Operating System installed for our specific use case
Jenkins to drive our CI/CD workflow pipeline
Our Version Control System (VCS)GIT to manage our source code.
Specs for one of our internal environment
Show entriesSearch:
Total
Server config
Memory
Storage
Price
100Ghz CPU 500GB Memory and 2.5TB Storage
5 AMD Opteron G5 Model Servers - 2 Sockets 3GHz x 12 cores each (HP Proliant DL385)
100 GB Memory each DDR3 SDRAM
500 GB Storage
~4K price
Over-Allocation:
Show entriesSearch:
Compute
Memory
Storage
We heavily over-allocate compute in this environment - 5x or 6x but can go up to 16x
Memory overallocation is allowed up to 1.5x but in practice, we try not to reach that level as it results in slow-downs of non-test workloads
Storage is COW and heavily utilized. In our setup, Storage tends to be the bottleneck
Workload
Show entriesSearch:
VMs run each day
VMs run at any given point
Common VM configs
~1000 VM instances get started and torn down each day
~200-300 VMs running at any given point
1 vCPU - 1GB Mem - 10GB Storage, 1 vCPU - 4GB Mem - 8GB Storage
Procecedure:-
1) A running OCATA Openstack
2) Configure our Continuous Integration (CI) Server Jenkins
The Workflow
Step 1 – Developer pushes a changeset to Git
The moment a developer is ready to push a change to Git, a private branch gets created with a specific name pattern.
The Jenkins builder is watching for activity on branches with this pattern. Once noticed, it pulls the changes from the branch to start the build chain process.
Step 2 – Run Unit-Tests and create build artifacts.
The first step of the chain is to do basic validation of the change by running unit tests. Jenkins slave is spawned to run unit tests on the component that the software changeset touches, as well as each dependent components. Any failures in unit tests halt the job immediately and the developer is notified.
If all the unit tests pass, build scripts specific to Platform9 are run to create build artifacts corresponding to our deployment.
NOTE This part will be specific to our own environment.
Step 3 – Deploy a Virtual Machine ‘Pod’
At this point, we are ready to trigger our integration tests. The Jenkins builder talks to our internal deployment of Platform9 and creates a Virtual Machine ‘Pod’. A Pod is simply a collection of golden Virtual Machine images needed to test out a given change. We standardize on a single Pod configuration for Platform9, you might have multiple Pods created in our environment – each specific to a given use case.
Platform9’s Pod consists of:
One or more Platform9 ‘controller’ VMs, deployed with Platform9 specific code
One or more database VMs
One or more hypervisor VMs to act as ‘hosts’ for our testing
The Jenkins builder communicates with Platform9 via REST APIs and deploys the Pod.
Step 4 – Configure the Virtual Machine ‘Pod’
Once the Pod is deployed, Jenkins builder communicates with Platform9 via REST APIs to configure the following:
– The Virtual Machines deployed within the Pod are Tagged with information about the specific build version being created, name of the user deploying the pod, etc. This data is useful for later tracking of the running Pods.
– Each VM in the Pod is customized with a default hostname, injected with ssh keys and a starter Ansible script to enable further customization via Ansible. Platform9 makes all this customization easy and possible.
– Once the Pod is up and running, Ansible scripts running in the Jenkins builder communicate with the Pod and customizes the VMs to have a specific identity. This is the step that deploys Platform9 specific packages in one or more of the VMs in the Pod.
Step 5 – Create multiple copies of the ‘Pod’- run integration tests
Once the Pod is configured and ready, Jenkins builder snapshots the Pod to create new set of golden images. It then creates multiple copies of the Pod. Each individual copy is utilized for running a specific set of integration tests.
At this point, the Jenkins builder powers-on multiple Pod copies in parallel. Each Pod is then utilized to simulate a different configuration and run corresponding integration tests in parallel.
Step 6 – Pass or fail
If all integration tests pass, the changeset is automatically pushed into main git branch. For any failures, the appropriate developer is notified. All the build configurations are saved so that the developer can log into the Platform9 UI and work on appropriate Pods to debug the issues.
Step 7 – Cleanup
Finally a scheduled job runs periodic cleanup of usused VMs based on the tags.
