Introduction to GitOps and Continuous Delivery
GitOps is a deployment methodology that leverages Git as the single source of truth for infrastructure and applications. Born from the need to improve container orchestration in Kubernetes environments, it allows developers to handle operations tasks through declarative specifications.
Unlike traditional DevOps, GitOps specifically focuses on continuous deployment by explicitly defining desired states in repositories, enabling automated, reliable deployment workflows for cloud-native applications. This approach streamlines the development cycle while providing greater consistency, auditability, and deployment accuracy.
What is GitOps?
GitOps is a method for managing Kubernetes clusters and delivering applications introduced in 2017 by Weaveworks. GitOps makes declarative infrastructure and applications operate by utilizing Git as a single source of truth. Software agents can notify on any divergence between Git and what's running in a cluster with GitOps. If there's a difference, Kubernetes reconcilers will update or roll back the cluster based on the situation. Developers utilize familiar tools to create pull requests to accelerate and simplify application deployments and operations duties to Kubernetes when Git is at the heart of your delivery pipelines.
GitOps is a method used for Continuous Delivery. It uses Git as a single source of truth for infrastructures like declarative infrastructure and applications. On Git, set delivery pipelines, make pull requests, accelerate and simplify both the application deployments and the operations task to Kubernetes. By using Git, create new features in the Applications rather than focusing on operations tasks.
You only need to update the repository to deploy a new application or update an old one; the automated process will handle the rest. It's like having cruise control when it comes to managing your production applications.
Here's how different components relate to GitOps:
-
Code: We can write our application code and push it to Git.
-
Build (Pipeline as Code): Build pipeline code (YAML) in Git to build code using CI tools.
-
Provision (Infrastructure as Code): Write a terraform file or Ansible playbook in Git to provision a Kubernetes cluster.
-
Deploy: Write a deployment manifest for Kubernetes or based on the application.
GitOps Principles
GitOps is based on several core principles:
-
The Entire System is Declarative - Declarative means that configuration is guaranteed by a set of facts instead of by a set of instructions. Kubernetes is a modern cloud-native tool that is "declarative," and that can be treated as code. A set of facts rather than a set of instructions guarantees configuration. You have a single source of truth with your application's declarations versioned in Git.
-
The Required Version System is Git - Git is a single source where everything is stored, including various features such as rollbacks, commit history, revert, rebase, etc. Users can use SSH keys to sign commits to enforcing robust security mechanisms and code protection.
-
Approved Changes Automatically Applied to the System - Once PR or MR is reviewed and merged in the configuration, it gets directly applied to the system. This allows changes to be automatically applied, creating a separation between what users do and how they do it.
-
Software Agents Ensure Efficiency and Correctness - Software agents inform users whenever reality doesn't match expectations. The use of agents also ensures that the entire system is self-healing. And by self-healing, this doesn't just mean when nodes or pods fail—Kubernetes handles those. They act as the feedback and control loop for such operations.
GitOps Deployment Models: Push vs. Pull-based Approaches
The deployment approach for GitOps can be implemented in two ways: push-based and pull-based deployments. The distinction between the two deployment types is how the deployment environment matches the desired infrastructure. Pull-based approaches should be chosen whenever possible because they are deemed more secure and better practices for implementing GitOps.
Push-based Deployments
Popular CI/CD technologies like Jenkins, CircleCI, and Travis CI implement the Push-based deployment technique. The application's source code and the Kubernetes YAML required to deploy the app are stored in the application repository. The build pipeline is initiated whenever the application code is updated, and it generates the container images before updating the environment configuration repository with new deployment descriptors.
Changes to the environment configuration repository trigger the deployment pipeline. This pipeline is in charge of applying the environment configuration repository's manifests to the infrastructure. It is critical to provide credentials to the deployment environment while using this method. When conducting automated provisioning of cloud infrastructure, a Push-based deployment is unavoidable in some instances.
Another thing to bear in mind is that the deployment pipeline is only activated when the environment repository changes when utilizing this strategy. It is unable to detect any deviations in the surroundings from the ideal state. This means there must be some form of monitoring in place so that if the environment does not match what is stated in the environment repository, action can be taken.
Pull-based Deployments
The pull-based deployment method follows the same principles as the push-based strategy, but the deployment pipeline is different. When new code is pushed to an application repository, for example, traditional CI/CD pipelines are triggered by an external event. The operator is introduced with the pull-based deployment method. It replaces the pipeline by regularly comparing the planned state in the environment repository to the actual state in the deployed infrastructure. When the operator notices a discrepancy, the infrastructure is updated to reflect the environment repository. In addition, the image registry can be checked to see if there are any new versions of images to deploy.
This option, like the push-based deployment, updates the environment anytime the repository changes. Changes in the operator, on the other hand, might be seen in both directions. Any changes to the deployed infrastructure that aren't documented in the environment repository are reversed. Making all direct changes to the cluster impossible ensures that all changes are traceable in the Git log.
The difficulty of push-based deployments, in which the environment is only updated when the environment repository is updated, is solved with this shift in direction. This does not, however, imply that you may go without any surveillance. Most operators support sending emails or Slack notifications if the environment cannot be brought to the intended state for whatever reason, such as when a container image cannot be pulled. Furthermore, you should set up monitoring for the operator, as there is no way to automate the deployment process without it.
The operator should always be located in the same environment or cluster as the application that is being deployed. This avoids the god-mode found with the push-based method, where the CI/CD pipeline knows the credentials for making deployments. External services do not need to know any credentials if the deploying instance is located within the same environment. The deployment platform's authorization mechanism can be used to limit the permissions for completing deployments. This has a significant impact on security. RBAC configurations and service accounts can be used with Kubernetes.
GitOps and Kubernetes Integration
GitOps treats each software or infrastructure component as one or more files in a version control system (VCS), with an automated process to synchronize the state between version control and the runtime environment. An orchestration system like Kubernetes is essential for achieving this. Without Kubernetes, the overall infrastructure, with multiple, incompatible technologies, can be too complex to manage. It helps address these issues and has influenced the evolution of IaC practices. The growth of Kubernetes enabled the creation of tools for implementing GitOps.
Configuration of GitOps with Kubernetes
Workflow changes with Kubernetes:
-
Developers push their code to the application repository.
-
Code changes trigger the build pipeline, resulting in the application's build.
-
Build pipeline publishes a container image with a new application version.
-
The environment repository stores the Kubernetes configuration manifest file, updated to use the new container image version.
Kubernetes GitOps Checklist
Here's a checklist for implementing GitOps with Kubernetes:
Kubernetes
Adopting an essential core technology for containers and infrastructure management is a crucial demand. It helps organizations manage workflows efficiently, accelerate application development, and meet market demand faster. Besides coordinating containers, Kubernetes can also manage hardware and offers essential middleware elements for managing data.
Team Knowledge and Culture
To implement the methodology of DevOps/GitOps in the organization, the development team must be aware and willing to become accustomed to new processes to save actual time. The team should handle experimentation, trials, mistakes, and training to understand new workflows, tools, and techniques while moving forward to new methods.
GitOps Pipeline
First, it's not the CI system conducting the deployment. Second, a GitOps agent (such as Harness Continuous Deployment or ArgoCD) should be deployed in the Kubernetes clusters (such as AKS, EKS, Bare Metal, OpenShift, etc.). Last but not least, introducing a configuration repository that will declare the state in which the user wants that environment to be.
Delivery
The process of implementing and upholding security compliance best practices or regulations in the software delivery pipeline is known as trusted delivery. These barriers are implemented utilizing policy-as-code and take the form of codified policies in the software development lifecycle.
Flux for Implementing GitOps
Flux is one of the GitOps operators for Kubernetes. It is an open-source tool developed by Weaveworks and later donated to the Cloud Native Computing Foundation. It runs in the cluster used to manage, and its primary function is to watch a remote repository describing the cluster state. It reconciles changes to converge the current cluster state to the desired state.
Flux provides:
-
GitOps for both apps and infrastructure
-
Push to Git, and Flux does the rest
-
Works with your existing tools
-
Designed with security in mind
-
Works with any Kubernetes and all standard Kubernetes tooling
-
Multi-Tenancy (and "Multi-everything")
-
Alerts and notifications
Flux Configuration
When you're getting ready to install Flux, there are a few components:
-
The command-line tool
-
Connecting to GitHub
-
GitHub credentials
-
Deploying Flux
GitOps Kubernetes Deployment Workflow
GitOps uses a version control system (such as Git) to house all information, documentation, and code for a Kubernetes deployment and then uses automated directors to deploy changes to the cluster. CI means changes are made on the master branch of Git. CD means Kubernetes updates itself based on changes in the master.
The workflow is as follows:
-
The developer changes the code and pushes it to Git
-
This triggers the build to form the latest docker image
-
That image is pushed to the Docker registry
-
Once the image is uploaded to the repository, this docker image will be replaced in the YAML file by the config updater service, including all the YAML files pushed
-
After updating all the YAML files, code and these files are pushed to the Git repository and forwarded to the staging server for review by managers
-
If the manager is satisfied with all, they merge code to the main branch and thus deploy it in the production Kubernetes cluster
Continuous Delivery and GitOps Workflow
The GitOps core machinery is in its Continuous Integrations/Continuous Delivery tooling, with continuous deployment (CD) supporting Git-cluster synchronization being the most important aspect. Workflow for creating or updating a new feature is as follows:
-
For review, a pull request for a new feature is submitted to GitHub.
-
A colleague looks at the code and approves it. The code is merged to Git when it has been reviewed and re-approved.
-
The Git merge starts the CI and builds a process that tests before building a new image and storing it in a registry.
-
The Cloud Deployment Automator (or other GitOps tools like Flux or ArgoCD) monitors the image registry for new images, pulls them from the registry, and updates their YAML in the config repo.
-
The Cloud Deployment Synchronizer (installed on the cluster) detects the cluster's out-of-date status. It deploys the new feature to production after pulling the revised manifests from the config repo.
GitOps for Machine Learning (ML) Pipelines
With technological advancements, Machine Learning is being incorporated into GitOps to automate or modernize operations even more. It lets developers know the changes in the source code and enables them to deploy updates many times a day.
Machine learning practitioners are using intelligent applications to push GitOps and cloud-native technology forward and try to gain a competitive edge over their competitors.
Bringing GitOps to ML introduces progressive delivery and deployment frameworks, which helps implement workflows for making the transition from an older version to the latest version and can be avoided if a significant error occurs.
ML Pipeline Components
The Basic Machine Learning pipelines consist of Data sets, Models, and Algorithms. To understand the ML pipeline for GitOps, let's discuss some in-depth functional aspects of the components:
-
Data Engineer - The Data sets are prepared and configured by a data engineer for other data modeling.
-
Data Scientist - The Data Scientist is responsible for writing the algorithms for aligning the data sets and models for accurate deployment and user experience.
Reasons to use GitOps for ML
Some benefits of GitOps with Machine Learning Operability can simplify tasks and enhance operational efficiency with:
-
Deployment Validation - Capturing the Web Page and converting it into an image for testing works to identify defects along the development journey. The image gets divided into multiple UX components and used to check other components and train the model for better and more efficient testing. The testing is based on scoring. A score closer to 0 signifies the test case is close to failure, and a score near 1 displays the test case meets the required criteria.
-
Monitoring and Alerts - Machine Learning can identify disk usage and generate alerts from the system to display production errors. It works on previous successful test cases, and understanding the KPIs enables a practical monitoring approach.
-
Tracking and Auditing - When Everything is on Git, then all the testing of the components can be tracked and audited. Also, with Kubernetes, the developer can allow for changes to the runs with minimal changes to the code.
Progressive Delivery Frameworks for ML
If everything is in Git, it can be tracked, audited, reviewed, and reproducible. Here are some Progressive Delivery Strategies for ML:
-
Blue/Green Deployment Strategy - In this deployment strategy, the newer version of the application and the older version of the application get deployed at the same time. The users have access to the older version, and the QA team gets access to the newer version on separate servers to check and test automation. When the testing is done, the older version is replaced with the newer version to give access to the new features to the users.
-
A/B Deployment Strategy - This deployment strategy is quite different from the Blue/Green Deployment Strategy. Rather than automating the tests, a small number of users are given access to the new version without being noticed. If the user testing goes well without any queries, the newer version is rolled out for the rest of the users. Also, it is sometimes referred to as 'dark deployment.'
-
Canary Testing - To test the new functionality of the application in the backend, the canary deployment strategy is traditionally used right now. In this approach, there are two identical servers. One is open to all users; generally, the older version is being used on one, whereas on the other, the new features are rolled out to a specific number of users. If no errors are reported, the new version of the application is gradually rolled out and ultimately replaces the old version for the users.
Benefits of GitOps
GitOps offers numerous benefits for organizations implementing it:
-
Productivity Improvements
Mean Time to Deployment is sped up using continuous deployment automation and an integrated feedback control loop. The team can ship 30-100 times more changes per day, resulting in a 2-3-fold increase in overall development production.
-
Developer Experience Enhancement
Without knowing the internals of Kubernetes, developers may utilize familiar technologies like Git to manage upgrades and features to Kubernetes more quickly. Newly hired developers can come up to speed fast and start working within days rather than months.
-
Increased Reliability
You get stable and repeatable rollbacks with Git's revert/rollback and fork capabilities. Because your entire system is defined in Git, you can recover from a meltdown using a single source of truth, cutting your mean time to recovery (MTTR) from hours to minutes.
-
Standardization and Consistency
You have consistent end-to-end processes throughout your business because GitOps provides a single model for making infrastructure, app, and Kubernetes add-on modifications. Pull requests entirely drive your continuous integration and continuous deployment pipelines, and your operations chores are reproducible thanks to Git.
-
Easier Compliance and Auditing
Compliance and auditing are simplified because modifications are tracked and logged securely. You may also compare a trustworthy definition of the state of the cluster with the actual running cluster using comparison tools.
-
Improved Security and Compliance
GitOps allows using a single platform for infrastructure change management, reducing downtime and minimizing outages, allowing developers to continue working without compromise.
-
Faster Deployments
GitOps allows organizations to use open-source technology to perform continuous integrations for fast and reliable deployments. Delivery releases make teams more flexible when responding to customers with minimal delay.
-
Cost-Effectiveness
It reduces operational overhead, lowers downtime & vastly improves productivity.
Challenges and Solutions in GitOps Implementation
Challenges
-
CI and CD Separation - The CI pipeline handles the building and testing of code and a separate CD pipeline for deployment. Still, the need for GitOps tools to accomplish the separation makes it difficult to implement. The main challenge is conducting tests after deployment, as Operations need visibility into the source code.
-
Multi-Environments Management - GitOps makes it challenging to manage multiple environments. All the environment details are in the Git repository; each time the environment changes, a new Git push has to be executed. When many production environments are involved, creating different branches in Git becomes complicated, and tracking all the branches can be difficult.
-
Difficult to Rollback - In Git, using a past commit helps for immediate rollbacks in GitOps as the complete history is stored in Git, but multiple past commits can create problems in setting up the rollback process. Different software development teams will have different preferences for performing rollbacks. GitOps does not have a standard practice for rollbacks. Automating rollbacks needs to be better defined.
-
Hard to Audit - In a large software team, it is good to have a specialized tool on top of the deployment platform; it is tough to answer them by only having access to a Git repository and its history. It refers to the lack of visibility.
-
Managing Implicit Infrastructure - Difficult to manage implicit infrastructure, its state, and configurations. It makes it difficult to investigate how and where your application is running. Sometimes make rollback challenging to roll back faulty changes done.
-
Eventual Consistency - Even with the centralized configuration, the actual configuration and a declared one can drift with manual operations.
-
Umbrella Chart Issues - Umbrella chart helps to deploy a full dependent stack over a cluster. But sometimes, it becomes punishing to reconfigure the charts, interrupt the isolation of charts, and increase load over the cluster.
Solutions
ArgoCD for GitOps
ArgoCD follows the declarative approach and acts as an agent to provide continuous delivery automation tools in the GitOps lifecycle. The primary purpose of ArgoCD is to provide support to GitOps as a continuous delivery tool in the Kubernetes environment. The heart of ArgoCD is the Application Controller, that continuously monitors the running applications state and compares that with the live application state against the desired target state defined in the Git repository.
ArgoCD Components:
-
API Server: When users interact with ArgoCD, either GUI or CLI, The interaction is possible with the API server component of ArgoCD.
-
Repository Server: Internal service that maintains a local cache of Git repository. Mainly responsible for generating and returning Kubernetes manifest.
-
Application Server: The Kubernetes controller continuously monitors the live state of the application with the desired state of application inside the git repository. It detects the OutofSync application and takes the correct action.
When deploying multiple applications over the cluster, the app-of-apps pattern will be the best solution. This directly fetches applications from the Git repository. This method overcomes all the challenges encountered in umbrella charts like it maintains the isolation of applications and supports accessible reconfiguration applications.
Your Git repository layout should look like this:
├── Chart.yaml
├── templates
│ ├── guestbook.yaml
│ ├── helm-dependency.yaml
│ ├── helm-guestbook.yaml
│ └── customize-guestbook.yaml
└── values.yaml
Now you have to manage only the above-given files instead of managing each application's helm chart.
GitOps Tools and Best Practices
Best Tools for GitOps
- Terraform
- CloudFormation
- Spinnaker
- Puppet
- Chef
- Ansible
- GitLab
- Kubernetes
- Hasura
- kubediff
- Ansible diff
- terradiff
Best Practices of GitOps
The core best practices of GitOps are:
-
Declarative Infrastructure as Code
-
Immutable deployment
-
Code & config version controlled
-
Rollback functionality
-
Operational changes made by Pull Request; don't let CI push updates into the Kubernetes cluster or use kubectl by hand
-
Build each container image just once and 'promote' it through each test sequence/environment, do not rebuild each time. Must update declarative configuration changes in Git
-
Monitorable & observable services & infrastructure
-
Different tools detect/alert divergence
-
Sync tools enable convergence
-
Completely automated delivery pipeline
Use Cases of GitOps
Static Websites
For deployment of complex static websites that comprise a set of markdown files which makes site pages easier to edit than raw HTML. It needs a build to make it publishable. Or even make modifications in the images to be available in different sizes to make it look good on various devices. For all this, building up a site or optimizing an image can be achieved by GitOps or even copying the deployment to the webserver.
Writing Books
As books are text-centric, they can be aligned with a VCS system & the GitOps pipeline can be set up as soon as you get done with writing your book. The pipeline can automatically check grammatical & spelling checks & further build them in various formats like doc, pdf, ePUB & many more. Further, it can publish it to your cloud drive or a static website.
GitOps for Documentation
It's a great fit for product documentation & is commonly used for this purpose as the document is also text-based and can be created as markdown or Ascii docs can easily be stored in a VCS system like bitbucket or GitHub. Then a CI tool fetches the changes and deploys the new version of the documentation. The deploy stage can upload the latest version on various platforms as new changes are committed to the documentation.
ArgoCD Use Cases
-
Developer pushes updated application source code to the git repo.
-
Git repo contains the Gitlab CI, which will build docker files, push docker image to the registry and update kubernetes manifest files(Present in different repo or branch) with the updated docker image.
-
The kubernetes manifest files git repo is registered with the ArgoCD application with auto-sync on.
-
Now the ArgoCDs' component Application Controller compares the current application state with the desired state of the application from Git.
-
The updated application is deployed over the kubernetes version with the help of ArgoCD.
-
Now when each time the application code changes over git repo, steps 1→ 5 apply in sequence.
Security and Compliance in GitOps
Adding Compliance and Security to GitOps is crucial for enterprise organizations that must meet stringent requirements and government laws.
A standard list of software delivery compliance and security requirements is as follows:
-
Access Control - Control who can deploy what in which environment.
-
Audit Trail - Keep track of all changes to the environments, including who made what changes and why.
-
Approval Process - Changes to specific settings require approval from authorized persons before they can be implemented.
How GitOps Solves Compliance Complexities
-
Access Control - Only those with write access to an environment's configuration Git repo are allowed to deploy to that environment.
-
Approval Process - For sensitive situations, we can allow Jenkins to create a Pull Request (PR) for the development team, but only authorized individuals can merge it. This results in a very smooth approval procedure with minimal setup time.
-
Audit - Git is naturally good at keeping track of changes because it is a version control system. Each Git commit contains a message that describes the change and identifies the person who made it.
Final Thoughts on GitOps Processes
In this comprehensive guide, we've learned about GitOps, why it is called the Single Source of Truth, and its numerous benefits. We explored different types of deployments (push and pull), the workflow of CI/CD with GitOps, and how GitOps integrates with Kubernetes.
We also examined the application of GitOps to Machine Learning pipelines, discussing how ML and GitOps can work together to improve development and deployment processes. The various challenges in implementing GitOps were addressed, along with solutions like ArgoCD that can help overcome these challenges.
GitOps, with the help of tools like ArgoCD, provides great support for modern development workflows. Git plays the role of continuous integration, and tools like ArgoCD handle continuous deployment. Together, they solve many real-life problems of seamless application deployment, forming a comprehensive solution for Continuous Integration and Continuous Deployment.
More Ways to Explore Us
Kubernetes Consulting Services and Solutions Company
GitOps for Kubernetes | The Ultimate Guide
Machine Learning Development Services
