Tuesday, August 29, 2023

Automate Azure cloud infrastructure setup using Ansible and Azure DevOps pipeline | How to integrate Ansible with Azure DevOps | Integrating Ansible with Azure DevOps Pipelines |

Ansible is an open-source, configuration management tool that automates cloud provisioning, configuration management, and application deployments. Using Ansible you can provision virtual machines, containers, network, and complete cloud infrastructures. 

Automate Azure cloud infrastructure setup using Ansible and Azure pipeline



Integrate Ansible with Azure Cloud
Integrating Ansible with Microsoft Azure allows you to automate and manage your Azure infrastructure using Ansible playbooks and modules. Ansible provides a collection of Azure-specific modules that enable you to provision and configure resources in Azure.


To configure Azure credentials, you need the following information:

  • Your Azure subscription ID and tenant ID
  • The service principal application ID and secret

Pre-requisites:

  • Azure account subscription, click here if you don't have one.
  • Service principal to create any resources in Azure cloud using Azure cloud shell or Azure CLI

Create Azure Service Principal

Run the following commands to create an Azure Service Principal:

az ad sp create-for-rbac --name <service-principal-name> \ 
--role Contributor \ 
--scopes /subscriptions/<subscription_id>
Save the above output in a file as you will not be able retrieve later.
Create an Ansible playbook

Create a simple playbook to create resource group in Azure. Make sure you modify the name of the resource group and location below.

---

- hosts: localhost

  connection: local

  tasks:

    - name: Creating resource group

      azure_rm_resourcegroup:

        name: "my-rg12"

        location: "eastus"


Create Azure YAML build pipeline:

Login to Azure Devops --> https://dev.azure.com

Select project dashboard.

Go to Pipelines -> New pipeline --> Click on Azure Repos Git or any SCM where you have playbooks stored. Select repo, click on Starter pipeline.

Add below four pipeline variables with value received from service principal creation.

AZURE_SUBSCRIPTION_ID
AZURE_CLIENT_ID
AZURE_SECRET
AZURE_TENANT
Add below tasks:
  • Install Ansible on build agent
  • Install Ansible rm module on build agent
  • Execute Ansible playbook for creating resource group in Azure cloud.
trigger:
- main
pr: none # Disable PR triggers, can be adjusted as needed
pool:
vmImage: 'ubuntu-latest'
steps:
- script: |
# Install Ansible
pip3 install "ansible==2.9.17"
displayName: 'Install Ansible'
- script: |
# Install Ansible rm module
pip3 install ansible[azure]
displayName: 'Install Ansible rm module'
- script: |
# Run Ansible playbook to create resource group
ansible-playbook create-rg.yml
displayName: 'Run Ansible Playbook'
env:
AZURE_SUBSCRIPTION_ID: $(AZURE_SUBSCRIPTION_ID)
AZURE_CLIENT_ID: $(AZURE_CLIENT_ID)
AZURE_SECRET: $(AZURE_SECRET)
AZURE_TENANT: $(AZURE_TENANT)

Save the pipeline and run it.


Now Login to Azure cloud to see if the resource group have been created.


Watch Steps in YouTube channel:

Thursday, August 24, 2023

Install Ansible on Red Hat Linux | How to setup Ansible on Red Hat Linux VM | Ansible install on Azure Linux Virtual Machine | Ansible Azure Integration

How to setup Ansible on Red Hat Linux VM and Integrate with Azure Cloud?

Ansible is #1 configuration management tool. It can also be used for infrastructure provisioning as well. or You can use Ansible in combination of Terraform which can take care of infra automation and Ansible can do configuration management. We will be setting up Ansible on Red Hat VM in Azure cloud And create some resources in Azure Cloud by using Ansible playbooks.


 
Ansible Architecture:
 

The best way to install Ansible in Linux is to use PIP, a package manager for Python.

Pre-requisites:
How to setup Ansible on Red Hat Linux VM

Watch Steps in YouTube channel:

Change host name to AnsibleMgmtNode
sudo hostnamectl set-hostname 
AnsibleMgmtNode

Update Repository
sudo yum update -y

Install Python-pip3
sudo yum install python3-pip -y

Upgrade pip3 sudo pip3 install --upgrade pip


# Install Ansible pip3 install "ansible==2.9.17"



check Ansible version
ansible --version


# Install Ansible azure_rm module for interacting with Azure.
pip3 install ansible[azure]

Authenticate with Azure


To configure Azure credentials, you need the following information:

  • Your Azure subscription ID and tenant ID
  • The service principal application ID and secret

Create an Azure Service Principal

Login to Azure first
az login
Enter Microsoft credentials

Run the following commands to create an Azure Service Principal:

az ad sp create-for-rbac --name <service-principal-name> \ 
--role Contributor \ 
--scopes /subscriptions/<subscription_id>
Save the above output in a file as you will not be able retrieve later.
Configure the Ansible credentials using one of the following techniques:

Option 1: Create Ansible credentials file

In this section, you create a local credentials file to provide credentials to Ansible. For security reasons, credential files should only be used in development environments.

mkdir ~/.azure 

vi ~/.azure/credentials


Insert the following lines into the file. Replace the placeholders with the service principal values.
[default] subscription_id=<subscription_id> client_id=<service_principal_app_id> secret=<service_principal_password> tenant=<service_principal_tenant_id>

Option 2: Define Ansible environment variables

On the host virtual machine, export the service principal values to configure your Ansible credentials.

export AZURE_SUBSCRIPTION_ID=<subscription_id> export AZURE_CLIENT_ID=<service_principal_app_id> export AZURE_SECRET=<service_principal_password> export AZURE_TENANT=<service_principal_tenant_id>

Test Ansible installation

You now have a virtual machine with Ansible installed and configured!

This section shows how to create a test resource group within your new Ansible configuration. If you don't need to do that, you can skip this section.

Option 1: Use an ad-hoc ansible command

Run the following ad-hoc Ansible command to create a resource group:

ansible localhost -m azure_rm_resourcegroup -a "name=my-rg123 location=eastus"

Option 2: Write and run an Ansible playbook

Create a simple playbook to create resource group in Azure.

sudo vi create-rg.yml

---

- hosts: localhost

  connection: local

  tasks:

    - name: Creating resource group

      azure_rm_resourcegroup:

        name: "myResourceGroup"

        location: "eastus"

Execute the playbook using ansible-playbook command.

ansible-playbook create-rg.yml

Now Login to Azure cloud to see if the resource group have been created.



Clean up Resources

Save the following code as delete-rg.yml

sudo vi delete-rg.yml

--- - hosts: localhost tasks: - name: Deleting resource group - "{{ name }}" azure_rm_resourcegroup: name: "{{ name }}" state: absent register: rg - debug: var: rg

ansible-playbook delete-rg.yml --extra-vars "name=myResourceGroup"

check in Azure cloud to see if the resource group have been deleted.

Saturday, August 12, 2023

How to secure Azure DevOps Pipelines | Azure DevOps Build Pipelines Best Practices

Securing your Azure DevOps pipelines is crucial to protect your code, data, and infrastructure. Here are several best practices to consider for securing your Azure DevOps pipelines:

1. Pipeline Security:

  • Use Version Control: Store your pipeline definitions and code in version control repositories. This ensures version history, change tracking, and controlled access.
  • Secure Branch Policies: Implement branch policies to control who can make changes to specific branches. This prevents unauthorized or unreviewed changes from being deployed.
  • Use Parameterized Builds: Avoid hardcoding sensitive information like credentials directly into pipeline definitions. Use pipeline variables or parameterized builds instead.
  • Limit Access: Follow the principle of least privilege. Only grant necessary permissions to users and groups. Use role-based access control (RBAC) to manage access to pipelines and resources.
  • Monitor and Audit: Regularly review access logs, audit logs, and pipeline run history to identify suspicious activities.

2. Secrets Management:

  • Use Variable Groups: Azure DevOps provides variable groups that allow you to store sensitive information securely. These can be linked to pipelines without exposing the values directly in pipeline definitions.
  • Use Secure File Variables: For sensitive files, use secure file variables to securely store and manage encrypted files that your pipelines can use during execution.
  • Azure Key Vault Integration: Integrate Azure Key Vault to store and manage secrets, keys, and certificates. You can then reference these secrets in your pipelines securely.

3. Secure Pipeline Code:

  • Use Private Repositories: Store sensitive code in private repositories. Avoid exposing secrets in public repositories.
  • Encryption: Encrypt sensitive data at rest and during transmission. Use HTTPS for repository connections and secure protocols for data storage.
  • Secure Code Review: Implement code review processes to catch and address security issues in pipeline definitions.

4. Secure Pipeline Execution:

  • Agent Security: Secure the build and deployment agents by regularly updating them, applying security patches, and using agents in private networks when possible.
  • Use Service Principals: If your pipeline needs access to Azure resources, use service principals with the least privilege required. Avoid using personal credentials.
  • Secure Pipeline Variables: When using pipeline variables, ensure that they are set to "Secret" to hide their values in logs and run history.

5. Compliance:

  • Regulatory Compliance: If your organization needs to adhere to specific regulatory standards, ensure that your pipelines and processes align with those standards.
  • Compliance Scanning: Use security and compliance scanning tools to identify vulnerabilities in your pipeline code and configurations.

6. Continuous Improvement:

  • Stay Updated: Regularly review Azure DevOps documentation, security best practices, and new features to stay informed about security improvements.
  • Incident Response Plan: Develop an incident response plan to handle security breaches or unauthorized access.

By following these best practices, you can significantly enhance the security of your Azure DevOps pipelines and safeguard your development and deployment processes. Remember that security is an ongoing effort, and it's important to regularly assess and update your security measures as needed.

Thursday, June 15, 2023

How to Deploy Springboot Microservices App into AKS cluster using Jenkins Pipeline and Helm | Deploy Microservices into AKS cluster using Helm Charts and Jenkins Pipeline

We are going to learn how to Automate build and deploy of Springboot Microservices App into Azure Kubernetes Cluster(AKS) using Helm and Jenkins pipeline.


Sample springboot App Code:

I have created a sample Springboot App setup in GitHub. Click here to access code base in GitHub. 

Jenkins pipeline will:

- Automate maven build(jar) using Jenkins
- Automate Docker image creation
- Automate Docker image upload into Azure container registry
- Automate Springboot docker container Deployments to Azure Kubernetes Cluster using Helm charts


Pre-requisites:
2.  Install Docker in Jenkins and Jenkins have proper permission to perform Docker builds
3. Install Azure CLI on your Jenkins machine. (We will be creating AKS cluster from Jenkins machine running in Azure Cloud)
4. Helm installed on Jenkins instance
5. Install Kubectl on Jenkins instance
6. AKS cluster needs to be up running. You can create AKS cluster, ACR Repo using shell script provided in my website.
7. Make sure to Install Docker, Docker pipeline 


8. ACR is also setup in Azure cloud. 
10. Dockerfile created along with the application source code for springboot App.

The Code for this video is here:

Implementation steps:

  1. Create a resource group, AKS cluster and Azure container registry 
  2. Provide pull access for AKS to pull image from ACR 
  3. Create a namespace for helm deployment
  4. Create a helm chart for spring boot app
  5. Create a Jenkins pipeline with below stages:
    • checkout
    • Build Jar
    • Build docker image
    • Upload image to ACR
    • Deploy to AKS using helm
  6. Run the pipeline to deploy springboot app into AKS
  7. Verify deployments in the namespace in AKS
  8. Access the app in the browser
Create AKS cluster from Jenkins Virtual machine
Login to Jenkins virtual machine. switch as jenkins user
sudo su - jenkins

Authenticate to Azure Cloud by typing:
az login


Now to the browser, type https://microsoft.com/devicelogin
enter the code as received from previous step:

Enter your microsoft credentials, click continue per below screen:

You will be shown below screen:

 
Create a shell script based on the script provided here

vi create-aks.sh
copy and paste entire script provided in this link.

Execute shell script to create the following in Azure clid:
  • Resource group
  • AKS cluster
  • ACR repo
  • provide pull access to AKS for pulling docker image from ACR
  • namespace in AKS cluster for deploying our springboot app
sh create-aks.sh
This will take 2 to 5 mins to create the resources in Azure cloud.

Make sure cluster is up and running with worker nodes
kubectl get nodes


Create Helm chart using helm command
Go to your root of repo where you have source code for your springboot application. Create helm chart by executing below command:

helm create mychart
tree mychart
Execute the above command to see the files created.



Add Docker image details to download from ECR before deploying to EKS cluster
open mychart/values.yaml. change per below values:

image:
repository: myacrrepo531.azurecr.io/myacrrepo531
tag: ""


Enter service type as LoadBalancer
And also
open mychart/templates/deployment.yaml and change containerPort to 8080


Save the files, commit and push into repo.
Step # 1 - Create Maven3 variable under Global tool configuration in Jenkins
Make sure you create Maven3 variable under Global tool configuration. 



Step #2 - Create Credentials to connect to ACR from Jenkins

Go to Azure Portal console, go to container registry
Settings--> Access keys
Get the username and password 
Go to Jenkins-> Manage Jenkins. Create credentials.


Enter ID as ACR and enter some text for description and Save.


Step # 3 - Create a pipeline in Jenkins
Create a new pipeline job.

Step # 4 - Copy the pipeline code from below
Make sure you change values as per your settings highlighted in yellow below:

pipeline {
  tools {
        maven 'Maven3'
    }
    agent any
        environment {
        //once you create ACR in Azure cloud, use that here
        registryName = "myacrrepo531"
        //- update your credentials ID after creating credentials for connecting to ACR
        registryCredential = 'ACR'
        dockerImage = ''
        registryUrl = 'myacrrepo531.azurecr.io'
    }
    
    stages {
        stage('checkout') {
            steps {
                checkout([$class: 'GitSCM', branches: [[name: '*/master']], extensions: [], userRemoteConfigs: [[url: 'check_out_from_your_repo_after_forking_my_repo']]])
            }
        }
        
        stage ('Build Jar') {
        steps {
            sh 'mvn clean install'           
        }
     }
     
    stage ('Build Docker image') {
        steps {
                script {
                    dockerImage = docker.build registryName
                }
            }
        }
        
    // Uploading Docker images into ACR
        stage('Upload Image to ACR') {
         steps{   
             script {
                docker.withRegistry( "http://${registryUrl}", registryCredential ) {
                dockerImage.push("$BUILD_NUMBER")
                }
            }
          }
        }
        
        stage ('Helm Deploy') {
          steps {
            script {
                sh "helm upgrade first --install mychart --namespace helm-deployment --set image.tag=$BUILD_NUMBER"
                }
            }
        }
    }
}

Step # 5 - Build the pipeline

Step # 6 - Verify deployments in AKS
Execute the below command to list the helm deployments:
helm ls -n helm-deployment


kubectl get pods -n helm-deployment

kubectl get services -n helm-deployment

Steps # 7 - Access Springboot App Deployed in AKS cluster
Once deployment is successful, go to browser and enter above load balancer URL mentioned above

You should see page like below:

Clean up the Cluster:

To avoid charges from Azure, you should clean up unneeded resources. When the cluster is no longer needed, use the az group delete command to remove the resource group, container service, and all related resources. 

az group delete --name resource-group-name --yes --no-wait


Watch steps in YouTube Channel: