Continuous Integration and DevOps Tools Setup and Tips
Hands on DevOps Coaching provided on AWS and Azure Cloud platforms. please contact at info.devopstraining@gmail.com for more info. You can also reach out Ria +1 (732) 799-8493
Helm is a package manager for Kubernetes. Helm is the K8s equivalent of yum or apt. It accomplishes the same goals as Linux system package managers like APT or YUM: managing the installation of applications and dependencies behind the scenes and hiding the complexity from the user.
Why use Helm?
As the Kubernetes platform and ecosystem continued to expand, deploying one and only one Kubernetes configuration file (ie: a single YAML) was not the norm anymore. As number of K8S deployment files increased, how to manage those files? Helm solves that problem.
Watch the steps in YouTube channel
Helm Charts
Helm uses a packaging format called Charts.A Helm Chart is a collection of files that describe a set of Kubernetes resources.Helm Charts helps you define, install, and upgrade even the most complex Kubernetes application. Charts are easy to create, version, share, and publish.
Helm Kubernetes Integration
In helm 3 there is no tiller component. Helm client directly interacts with the Kubernetes API for the helm chart deployment.
Helm 3 can be installed many ways. We will install Helm 3 using scripts option.
We will learn how to automate Docker builds using Jenkins and Deploy into Kubernetes Cluster in AWS Cloud. We will use kubectl command to deploy Docker images into EKS cluster. We will use Python based application. I have already created a repo with source code + Dockerfile. The repo also have Jenkinsfile for automating the following:
- Automating builds using Jenkins - Automating Docker image creation - Automating Docker image upload into Docker registry - Automating Deployments to Kubernetes Cluster using kubectl CLI plug-in
Pre-requistes: 1. EKS Cluster is setup and running. Click here to learn how to create EKS cluster.
2. Jenkins Master is up and running. 3. Docker, Docker pipeline and Kubectl CLI plug-ins are installed in Jenkins
How to setup SonaType Nexus 3 using Docker compose?
Nexus is open source, binary repository manager and Java based tool. It can be installed quickly using Docker with less manual steps.
Watch Steps in YouTube channel:
Pre-requistes:
Ubuntu EC2 up and running with at least t2.medium(4GB RAM), 2GB will not work
Port 8081 is opened in security firewall rule
Perform System update
sudo apt-get update
Install Docker
sudo apt-get install docker -y
Install Docker-Compose
sudo apt-get install docker-compose -y
Add current user to Docker group
sudo usermod -aG docker $USER
What is Docker Compose?
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.
The purpose of docker-compose is to function as docker cli but to issue multiple commands much more quickly. To make use of docker-compose, you need to encode the commands you were running before into a docker-compose.yml file
Run docker-compose up and Compose starts and runs your entire app.
Create docker-compose.yml
this yml has all configuration for installing Nexus on Ubuntu EC2.
sudo vi docker-compose.yml
(Copy the below code high-lighted in yellow color)
version: "3"
services:
nexus:
image: sonatype/nexus3
restart: always
volumes:
- "nexus-data:/sonatype-work"
ports:
- "8081:8081"
volumes:
nexus-data: {}
Save the file by entering :wq!
Now execute the compose file using Docker compose command to start Nexus Container
sudo docker-compose up -d
-d means detached mode
Make sure Nexus 3 is up and running
sudo docker-compose logs --follow
How to get Nexus admin password? sure Nexus 3 is up and running
Once you see the message, that's it. Nexus 3 is been setup successfully. press control C and enter.
Now access Nexus UI by going to browser and enter public dns name with port 8081
Now to go to browser --> http://change to_nexus_publicdns_name:8081
We need to login to docker container to get admin password.
Here are the top 10 DevOps Tools to focus on to put your DevOps learning in fast-track and kick start your career quickly as a Cloud or DevOps engineer in about 8 weeks from now.
How to setup SonarQube using Docker and Docker compose?
SonarQube is static code analyis tool. It can be installed quickly using Docker with less manual steps.
Pre-requistes:
Ubuntu EC2 up and running with at least t2.small
Port 9000 is opened in security firewall rule
Make sure maximum number of
sudo vi /etc/sysctl.conf
Add the following lines to the bottom of that file:
vm.max_map_count=262144 fs.file-max=65536
To make sure changes are getting into effect:
sudo sysctl -p
Install Docker
sudo apt-get install docker -y
Install Docker-Compose
sudo apt-get install docker-compose -y
What is Docker Compose?
Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.
The purpose of docker-compose is to function as docker cli but to issue multiple commands much more quickly. To make use of docker-compose, you need to encode the commands you were running before into a docker-compose.yml file
Run docker-compose up and Compose starts and runs your entire app.
Create docker-compose.yml
this yml has all configuration for installing both SonarQube and Postgresql:
sudo vi docker-compose.yml
(Copy the below code high-lighted in yellow color)
We will be learning how to provision resources in AWS cloud using Terraform and Jenkins. We will also learn how to store terraform state info remotely in AWS S3 bucket.
We will create S3 bucket for storing terraform state info and Dynamo DB table for locking capability.
We will try to create an EC2 instance and S3 Bucket using Terraform and Jenkins in AWS cloud. Look at the diagram that describes the whole flow.
Watch these steps in action in YouTube channel:
Pre-requistes:
Create S3 bucket for storing TF state
Create dynamo DB table for providing lock capability
Make sure you have necessary IAM role created with right policy and attached to Jenkins EC2 instance. see below for the steps to create IAM role.
I have provided my public repo as an example which you can use.
Step # 1 - Create S3 Bucket:
Login to AWS, S3. Click on create S3 bucket.
Give unique name to the bucket, name needs to be unique.
Block all public access, enable bucket versioning as well.
Enable encryption.
Step # 2 - Create DynamoDB Table
Create a new table with LockID as partition Key
Step - 3 Create IAM role to provision EC2 instance in AWS
Select AWS service, EC2, Click on Next Permissions
Type EC2 and choose AmazonEC2FullAccess as policy and type S3 and add AmazonS3FullAccess, type Dynamo
Attach three policies
Click on Next tags, Next Review
give some role name and click on Create role.
Step 4 - Assign IAM role to EC2 instance
Go back to Jenkins EC2 instance, click on EC2 instance, Security, Modify IAM role
Type your IAM role name my-ec2-terraform-role and Save to attach that role to EC2 instance.
Step 5 - Create a new Jenkins Pipeline
Give a name to the pipeline you are creating.
Step 6 - Add parameters to the pipeline
Click checkbox - This project is parameterized, choose Choice Parameter
Enter name as action
type apply and enter and type destroy as choices as it is shown below(it should be in two lines)
Go to Pipeline section
Add below pipeline code and modify per your GitHub repo configuration.
pipeline {
agent any
stages {
stage('Checkout') {
steps {
checkout scm
}
}
stage ("terraform init") {
steps {
sh ('terraform init -reconfigure')
}
}
stage ("terraform plan") {
steps {
sh ('terraform plan')
}
}
stage ("terraform Action") {
steps {
echo "Terraform action is --> ${action}"
sh ('terraform ${action} --auto-approve')
}
}
}
}
Click on Build with Parameters and choose apply to build the infrastructure or choose destroy if you like to destroy the infrastructure you have built.
Click on Build With Parameters,
choose apply from the dropdown
Now you should see the console output if you choose apply.
Pipeline will look like below:
Login to AWS console
Login to S3 Bucket, you should see terraform state info is also added
How to Destroy all the resources created using Terraform?
run the Jenkins Pipeline with destroy option. This should destroy all the resources you have created using Terraform.
We will be learning an interesting use case to provision resources in AWS cloud using Terraform and Azure DevOps. We will also learn how to store terraform state info remotely in AWS S3 bucket.
We will create S3 bucket for storing terraform state info and Dynamo DB table for providing state lock capability.
We will try to create an EC2 instance and S3 Bucket using Terraform and Azure DevOps in AWS cloud. Look at the diagram that describes the whole flow.
Create AWS service connection in Azure DevOps for Terraform to use
Create service connection for connecting to GitHub
Create S3 bucket for storing TF state
Create dynamo DB table for providing lock capability
I have provided my public repo as an example which you can use.
Step # 1 - Create S3 Bucket:
Login to AWS, S3. Click on create S3 bucket.
Give unique name to the bucket, name needs to be unique.
Block all public access, enable bucket versioning as well.
Enable encryption.
Step # 2 - Create DynamoDB Table
Create a new table with LockID as partition Key
Step # 3 - Create Service connection to connect to AWS from Azure DevOps
Go to Azure Devops, select your project. Project Settings
Click Service Connections
Select AWS for Terraform
Enter Access Key, Secret Key and region code, enter name for service connection and choose Grant Access to all pipelines.
Click Save.
Create Service connection for connecting to GitHub
Save
Step 4 - Create a new Release Pipeline
Click on Releases, New, choose New Release pipeline
Select empty job.
Click on Add Artifacts
Choose GitHub, select Github service connection, select the repo
Click on Add tasks, type terraform
choose the task
Add task, search for install terraform and select installer task
It should show something like this:
Add another terraform task for init
Search for terraform and add task
select AWS from drop down, choose init as command and add -reconfigure as additional command arguments. Select AWS service connection and enter bucket name
Add another task for plan, select right values.
enter -out dev-plan
Add another task by cloning for apply.
enter "dev-plan" as additional arguments
Click on Save.
Create Release, now job should be running.
Login to AWS--> S3 Bucket, you should see terraform state info
How to destroy all the resources created using Terraform?
Clone the current infra setup release pipeline.
Modify the pipeline name.
Add
Modify the apply task to as shown in the diagram
enter -destroy as additional argument.
Click on Create Release to make sure all the resources are destroyed.
