How to install Puppet on Ubuntu 16.0.4 | How to configure Puppet Master and manage nodes on Ubuntu 16.0.4 on Amazon EC2?

Puppet uses Client/Server model. The server does all the automation of tasks on nodes/servers that have a client(agent) installed. The work of the Puppet agent is to send facts to the puppet master and request a catalog based on certain interval level(default time 30 mins). Once it receives a catalog, Puppet agent applies it to the node by checking each resource the catalog describes. It makes relevant changes to attain the desired state. The work of the Puppet master is to control configuration information.  Each managed agent node requests its own configuration catalog from the master.

Please find the steps needed for Integrating Puppet master and agent on Ubuntu 16.0.4:

Pre-requisites:

a) One Ubuntu instance for Puppet Master - this instance should have 4 GB RAM. so instance type should be at least medium. Master ubuntu EC2 should have a security group to ensure that below ports are open in security firewall
        * TCP 8140 - Agents will talk to the master on this port(puppet enterprise)
        * TCP 22 - To login to the server/instance using SSH

b) one Ubuntu instance as node which will have agent installed - This can be micro instance.

1. Steps for Puppet Master
:

curl -O https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb
sudo dpkg -i puppetlabs-release-pc1-xenial.deb
sudo apt-get update
sudo apt-get install puppetserver

sudo ufw allow 8140
sudo systemctl enable puppetserver
          (the above command is to start the service during starting the Ubuntu instance)

sudo systemctl start puppetserver
          (The above command is for starting the server and this may take some time)
sudo systemctl status puppetserver
       you should see a message like
       puppet systemd[1]: Started puppetserver Service.
    That's it puppet master is up and running.

Now press q to come out of window.

2. Steps for Puppet Agent

Step 2.1 First edit the hosts file on the puppet agent by modifying /etc/hosts
sudo nano /etc/hosts
# Please add Puppet Master server IP address and space and enter puppet
puppet_master_ip_address   puppet

(please do not use public DNS name, use only private IP address)  

Press Ctrl O for saving and then enter
Press Ctrl X for exit after saving.

Step 2.2 — Installing Puppet Agent on server node that Puppet master will manage
wget https://apt.puppetlabs.com/puppetlabs-release-pc1-xenial.deb 

sudo dpkg -i puppetlabs-release-pc1-xenial.deb

sudo apt-get update

sudo apt-get install puppet-agent -y

sudo systemctl enable puppet

sudo systemctl restart puppet

sudo systemctl status puppet

 




Now press q to come out of window.

Step 3 - Signing certificates on Puppet Master
The first time you run the Puppet agent, it generates an SSL certificate and sends a signing request to the Puppet master. After the Puppet master signs the agent's certificate, it will be able to communicate with and control the agent node.

First list the unsigned certificates on puppet master EC2 instance

sudo /opt/puppetlabs/bin/puppetserver ca list

The above command will list agent ip address.
  "your_puppet_Agent_Ec2_private_dns_name"  (SHA256) 46:19:79:3F:70:19:0A:FB:DA:3D:C8:74:47:EF:C8:B0:05:8A:06:50:2B:40:B3:B9:26:35:F6:96:17:85:5E:7C

Now sign the Puppet agent IP address.
sudo /opt/puppetlabs/bin/puppetserver ca sign --certname "your_puppet_Agent_Ec2_private_dns_name"
 
Note: (this is NOT required)
To sign the certificates all, execute the below command.
sudo /opt/puppetlabs/bin/puppetserver ca sign —all

Revoke Certificates (NOT required)
sudo /opt/puppetlabs/bin/puppetserver ca clean hostname

Step 4 - Verifying installation by creating Manifests in Puppet Master

The puppet manifest file is the actual file which contains the configuration details for the agents. This file is centrally stored at Puppet Master.

sudo nano /etc/puppetlabs/code/environments/production/manifests/site.pp
#copy the below yellow lines in the above file
    file {'/tmp/puppet_test.txt':                        # resource type file and filename
    ensure  => present,                             # make sure it exists
    mode    => '0644',                              # file permissions
  content => "Hello from Puppet master to agent on ${ipaddress_eth0}!\n",  # Print the eth0 IP fact
    }

Press Ctrl O for saving and then enter
Press Ctrl X for exit after saving.

Step 5 - Apply Manifests in Puppet Agent
apply the changes in puppet agent by executing below command:
sudo /opt/puppetlabs/bin/puppet agent --test

 


You should see a file being modified in /tmp/puppet_works.txt in agent(node).
You can confirm by typing this command on puppet node 

sudo cat /tmp/puppet_test.txt
Hello from Puppet master to agent on IP_address!!

That's it! you have set up Puppet Master and configured agent on the target node successfully!

Puppet syntax check validator

In order to check the syntax of puppet manifests, use the below command to run the syntax check:

puppet parser validate  manifest_name

e.g.
puppet parser validate  /etc/puppetlabs/code/environments/production/manifests/site.pp

How to configure Jenkins master/slave nodes in Ubuntu | Jenkins master and slave setup on Ubuntu

Jenkins has powerful feature of master slave architecture which enables distributed builds. This article we will learn how to establish Jenkins Master and slave nodes on Ubuntu machines.

This lab demo is available on YouTube:

Few info on Jenkins master and slave setup:

Jenkins Master

Your main Jenkins server is the Master. The Master’s job is to handle:

• Scheduling build jobs.
• Dispatching builds to the slaves for the actual execution.
• Monitor the slaves (possibly taking them online and offline as required).
• Recording and presenting the build results.
• A Master instance of Jenkins can also execute build jobs directly.

Jenkins Slave

A Slave is a Java executable that runs on a remote machine. Following are the characteristics of Jenkins Slaves:

• It hears requests from the Jenkins Master instance.
• Slaves can run on a variety of operating systems.
• The job of a Slave is to do as they are told to, which involves executing build jobs dispatched by the Master.
• You can configure a project to always run on a particular Slave machine, or a particular type of Slave machine, or simply let Jenkins pick the next available Slave.

Lets see how to configure both Jenkins master and slave nodes on Ubuntu EC2.

Step 1 - Jenkins master node configuration

(If you have already Jenkins up and running, this step # 1 is not required, Go to next step # 2, slave node configuration)

Install first Java by following below steps. Make sure port 8080 is opened in security group.

sudo apt-get update
sudo apt-get install default-jdk -y

Jenkins (make sure you open port number)

wget -q -O - https://pkg.jenkins.io/debian/jenkins.io.key | sudo apt-key add -
echo deb http://pkg.jenkins.io/debian-stable binary/ | sudo tee /etc/apt/sources.list.d/jenkins.list
sudo apt-get update
sudo apt-get install jenkins -y
now to go to browser --> http://server_ip_address:8080/
Copy the password from this location
By entering
sudo cat /var/lib/jenkins/secrets/initialAdminPassword
Paste the password and click on install suggested plug-ins.

Create SSH keys in master node by executing below command:

ssh-keygen (no need to enter any password, just enter three times)

Step # 2 Slave node configuration(you need new micro Ubuntu 18.0.4 instance for this slave)
only port 22 needs to be open

Install Java

sudo apt-get update
sudo apt-get install default-jdk -y

Install Maven
sudo apt-get install maven -y

Create User as Jenkins

sudo useradd -m jenkins

sudo -u jenkins mkdir /home/jenkins/.ssh

Add SSH Keys from Master to Slave 
Execute the below command in Jenkins master Ec2 to print the ssh keys.
sudo cat ~/.ssh/id_rsa.pub
If you do not have keys, create using ssh-keygen command.
Copy the output of the above command:

Now go to Slave node and execute the below command
sudo -u jenkins vi /home/jenkins/.ssh/authorized_keys

This will be empty file, now copy the public keys from master into above file.
Once you pasted the public keys in the above file in Slave, come out of the file by entering wq!

Now go into master node
ssh jenkins@slave_node_ip

this is to make sure master is able to connect slave node. once you are successfully logged into slave, type exit to come out of slave.

Now copy the SSH keys into /var/lib/jenkins/.ssh by executing below command in master(make sure you exited from slave by typing exit command:

sudo cp ~/.ssh/known_hosts  /var/lib/jenkins/.ssh

Because jenkins master will look keys from the above folder.

Step # 3 Register slave node in Jenkins:
Now to go Jenkins Master, manage jenkins, manage nodes.

Click on new node. give name and check permanent agent. Click Ok to create the slave node.
in the next screen
give name and no of executors as 1. enter /home/jenkins as remote directory.
select launch method as Launch slaves nodes via SSH.
enter Slave node ip address as Host.

click on credentials. Enter user name as jenkins. Make jenkins lowercase as it is shown.
 Kind as SSH username with private key. enter private key of master node directly by executing below command:

sudo cat ~/.ssh/id_rsa
(Make sure you copy the whole key including the below without missing anything)
-----BEGIN RSA PRIVATE KEY-----

-----END RSA PRIVATE KEY-----

click Save.
select Host key verification strategy as "manually trusted key verification strategy".

Click Save. Now click on Log in drop down like shown below.

It should connect and show like this...no red marked error..

Now you can kick start building the jobs, you will see Jenkins master runs jobs in slave nodes.

How to configure webhooks in Bitbucket to trigger a build in Jenkins? How to trigger automated builds in Jenkins through Bitbucket?

Jenkins jobs can be triggered many ways. Here are those ways:

1. pull - using poll scm
2. Webhooks (push mechanism) - by triggering a build from Bitbucket or GitHub for every repository changes.
3. through slack channel. Click here to learn about trigger Jenkins job using Slack.

Webhooks are triggers that enables developers to trigger Jenkins jobs automatically every time there is a code change.

we will see in this article how to trigger a(push) build for every change in bitbucket repository:

Changes needed in Jenkins

1. Click on Manage Jenkins.   
2. Click on Configure Global Security. 
3. Uncheck the option for Prevent Cross Site Request Forgery exploits
4. In matrix-based security change: 

for Anonymous user, do the below:

Overall - Read   
Job - Build  
Job - Read   

Job - Workspace

Select the job you would like to configure webhook is for. Choose configure Also you need to have token created for the job you would like to trigger. Click on the build job. Go to triggers section and click on Trigger builds remotely (e.g., from scripts)

make sure you enter token name as MyToken in

 Authentication Token field. 

Also uncheck poll SCM option(if it was selected earlier) 

Changes in Bitbucket

1. go to bitbucket, choose the repository, go to settings, click on web hooks.

2. enter title, url which is your jenkins job url - append build/?token=myToken
 example the url should be like this - http://jenkins_public_dns_server_url:8080/job/myFirstAutomateJob/build?token=myToken

Sample url is given below:

http://jenkins_url:8080/job/myFirstAutomateJob/build?token=myToken

For e.g.,
jenkins public dns name - jenkins_url:8080
Freestyle job name -  myFirstAutomateJob
token= myToken
3. status should be active
4. click on skip certificate verification
5. triggers --> repository push

Now make a code change in bitbucket to see if that triggers a build in Jenkins automatically.

Tomcat 8 hung on Ubuntu 16.0.4

Apply the below fix when you have tomcat not starting issues:

Actually, by setting the following in /etc/default/tomcat8, I was fine:
JAVA_OPTS="-Djava.security.egd=file:/dev/./urandom -Djava.awt.headless=true -Xmx1024m -XX:MaxPermSize=512m -XX:+UseConcMarkSweepGC"

sonar-maven-plugin:2.6:sonar: java.lang.UnsupportedClassVersionError: org/sonar/api/utils/SonarException : Unsupported major.minor version 52.0

This issue can be resolved by adding JDK version to 1.8 in POm.xml

Please apply the below fix in POM.xml

<build>
    <finalName>appname</finalName>
  <plugins>
  <plugin>
    <groupId>org.apache.maven.plugins</groupId>
     <artifactId>maven-compiler-plugin</artifactId>
     <version>3.6.1</version>
     <configuration>
     <source>1.8</source>
     <target>1.8</target>
     </configuration>
  </plugin>
  <plugin>
    <groupId>org.codehaus.mojo</groupId>
    <artifactId>sonar-maven-plugin</artifactId>
    <version>2.6</version>
  </plugin>
</plugins>
  </build>

Basic provisioning example of EC2 instance using Ansible on AWS

---
 - name: Basic provisioning example
   hosts: local
   connection: local
   gather_facts: False
   tags: provisioning

   vars:
     keypair: MyinfraCodeKey
     instance_type: t2.micro
     image: ami-916f59f4
     wait: yes
     group: webserver
     count: 1
     region: us-east-2
     security_group: ansible-webserver-1
   
   tasks:

     - name: Create a security group
       local_action: 
         module: ec2_group
         name: "{{ security_group }}"
         description: Security Group for webserver Servers
         region: "{{ region }}"
         rules:
            - proto: tcp
              from_port: 22
              to_port: 22
              cidr_ip: 0.0.0.0/0
            - proto: tcp
              from_port: 80
              to_port: 80
              cidr_ip: 0.0.0.0/0
            - proto: tcp
              from_port: 443
              to_port: 443
              cidr_ip: 0.0.0.0/0
         rules_egress:
            - proto: all
              cidr_ip: 0.0.0.0/0
       register: basic_firewall
     
     - name: Launch the new EC2 Instance
       local_action:  ec2 
                      group={{ security_group }} 
                      instance_type={{ instance_type}} 
                      image={{ image }} 
                      wait=true 
                      region={{ region }} 
                      keypair={{ keypair }}
                      count={{count}}
       register: ec2

     - name: Add the newly created EC2 instance(s) to the local host group (located inside the directory)
       local_action: lineinfile
                     dest="/etc/ansible/hosts"
                     regexp={{ item.public_ip }}
                     insertafter="[webserver]" line={{ item.public_ip }}
       with_items: "{{ ec2.instances }}"

How to install Apache Maven on Redhat Enterprise Linux Instance?

Here are the steps for installing Apache Maven on RHEL:
Make sure you install Java first by following below steps:

sudo yum -y install wget
sudo wget --no-check-certificate --no-cookies --header 'Cookie: oraclelicense=accept-securebackup-cookie' 'http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.rpm'
sudo rpm -i jdk-8u141-linux-x64.rpm

Now steps for installing Apache Maven:
sudo wget http://repos.fedorapeople.org/repos/dchen/apache-maven/epel-apache-maven.repo -O /etc/yum.repos.d/epel-apache-maven.repo
sudo sed -i s/\$releasever/6/g /etc/yum.repos.d/epel-apache-maven.repo
sudo yum install -y apache-maven
mvn --version

the above command should print the below message:

Apache Maven 3.5.2 (138edd61fd100ec658bfa2d307c43b76940a5d7d; 2017-10-18T07:58:13Z)
Maven home: /usr/share/apache-maven
Java version: 1.7.0_181, vendor: Oracle Corporation
Java home: /usr/lib/jvm/java-1.7.0-openjdk-1.7.0.181-2.6.14.8.el7_5.x86_64/jre
Default locale: en_US, platform encoding: UTF-8
OS name: "linux", version: "3.10.0-862.el7.x86_64", arch: "amd64", family: "unix"

How to change default port number in Sonatype Nexus 2? - Change default port number in SonaType Nexus 2

The default port number for Sonatype Nexus is 8081. It can be changed though.
If you would like to change it, modify in below location:
$install-dir/etc/org.sonatype.nexus.cfg file
application-port=8082

Click here to see steps for how to install Nexus on Redhat.
Click here to see steps for how to install Nexus on Ubuntu.

How to install Sonarqube on Redhat enterprise Linux - Install Sonarqube on Redhat enterprise Linux (RHEL)

SonarQube is popular static code analysis tool. It is Java based tool. Here are the steps for installing Sonarqube.

first you need to install a database as SonarQube uses database to store code quality report. We can use MySQL, Postgres, Oracle or MSSQL. Here are the steps for setting up SonarQube using MySQL database :

sudo yum update -y

sudo yum install wget -y

MySQL installation steps:

sudo wget http://repo.mysql.com/mysql-community-release-el7-5.noarch.rpm  

sudo rpm -ivh mysql-community-release-el7-5.noarch.rpm

sudo yum install mysql-server -y

Start MySQL service

sudo systemctl start mysqld

sudo mysql_secure_installation

enter n as we will not be setting any admin password for user root.

Would you like to setup VALIDATE PASSWORD component?
type n

now set up password for root user

Remove anonymous users?
type y
for Disallow root login remotely?
type y
for Remove test database and access to it?
type y
for Reload privileges tables
type y

Setup Sonarqube user and password

mysql -u root -p

SHOW GLOBAL VARIABLES LIKE 'storage_engine';

CREATE USER 'sonar'@'localhost' IDENTIFIED BY 'password';

CREATE DATABASE sonar;

GRANT ALL PRIVILEGES ON sonar.* TO 'sonar'@'localhost';

now type exit to come out of MySQL.

Sonarqube web server installation steps:

Start with java installation.

sudo wget --no-check-certificate --no-cookies --header 'Cookie: oraclelicense=accept-securebackup-cookie' 'http://download.oracle.com/otn-pub/java/jdk/8u141-b15/336fa29ff2bb4ef291e347e091f7f4a7/jdk-8u141-linux-x64.rpm'

sudo rpm -i jdk-8u141-linux-x64.rpm

now type java -version to make sure Java 8 is installed.

Download SonarQube

cd /opt 

sudo wget https://binaries.sonarsource.com/Distribution/sonarqube/sonarqube-6.7.6.zip

sudo yum install unzip -y

sudo unzip sonarqube-6.7.6.zip

sudo mv sonarqube-6.7.6 sonarqube

Modify Sonar Conf file

sudo vi /opt/sonarqube/conf/sonar.properties

uncomment the below lines by removing #:

sonar.jdbc.username=sonar

sonar.jdbc.password=password

sonar.jdbc.url=jdbc:mysql://localhost:3306/sonar?useUnicode=true&characterEncoding=utf8&rewriteBatchedStatements=true&useConfigs=maxPerformance

sonar.web.host=0.0.0.0

sonar.web.javaOpts=-Xmx512m -Xms256m -XX:+HeapDumpOnOutOfMemoryError

sonar.ce.javaOpts=-Xmx512m -Xms256m -XX:+HeapDumpOnOutOfMemoryError

Run Sonarqube as sonar user
create a user called sonar by executing below command:
sudo useradd sonar

sudo chown -R sonar:sonar /opt/sonarqube

Edit the sonar.sh start script and change the #RUN_AS_USER to be RUN_AS_USER=sonar

sudo vi /opt/sonarqube/bin/linux-x86-64/sonar.sh

Change 
RUN_AS_USER=sonar

now start the server 

sudo sh /opt/sonarqube/bin/linux-x86-64/sonar.sh start

for any issues, look at logs at 

tail -f /opt/sonarqube/logs/sonar.log

Verify by going to browser http://hostname:9000

 

Docker installation on Ubuntu 16.0.4 server

Docker can be installed on Ubuntu 16.0.4 by following below steps:
 
sudo curl -fsSLhttps://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

sudo add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable"

sudo apt-get update

sudo apt-cache policy docker-ce

sudo apt-get install -y docker-ce

sudo systemctl status docker

ps -ef | grep docker

docker –version

docker run hello-world