github repo: https://github.com/Pardeep32/reddit-clone-k8s.git
Launch an AWS T2 Large Instance. Use the image as Ubuntu. You can create a new key pair or use an existing one. Enable HTTP and HTTPS settings in the Security Group and open all ports and storage of 30 gb.
go to IAM and create a role for this:
Granting administrative access to this role is not recommended, but for the purposes of this scenario, I'll provide access.
Now associate this role with the instance.
Modify the security group that is associated with instance:
In security group: 8080 is for jenkins, 9000 is for sonarqube and 3000 is for application.
Now on ec2 instance terminal install following:
Java
jenkins
Docker
Trivy
Terraform
Kubectl
AWScli
Vim jenkins.sh
#!/bin/bash
# Update package repositories
sudo apt update
# Install necessary dependencies
sudo apt install -y fontconfig openjdk-17-jre
# Verify Java installation
java -version
# Download Jenkins key and add it to the keyring
sudo wget -O /usr/share/keyrings/jenkins-keyring.asc https://pkg.jenkins.io/debian-stable/jenkins.io-2023.key
# Add Jenkins repository to sources.list.d
echo "deb [signed-by=/usr/share/keyrings/jenkins-keyring.asc] https://pkg.jenkins.io/debian-stable binary/" | sudo tee /etc/apt/sources.list.d/jenkins.list > /dev/null
# Update package repositories again to include Jenkins repository
sudo apt-get update
# Install Jenkins
sudo apt-get install -y jenkins
give executable permission to jenkins.sh and run it.
chmod +x jenkins.sh
sh jenkins.sh
Retrieve the public IP of the EC2 instance and access port 8080 in a new browser window/tab
Copy the password and paste it into the Jenkins login screen to unlock the system. Then proceed to install the necessary plugins.
Now install docker
sudo apt-get update
sudo apt-get install docker.io -y
sudo usermod -aG docker $USER #my case is ubuntu
newgrp docker
sudo chmod 777 /var/run/docker.sock
Now run the sonar container on ec2 terminal:
docker run -d --name sonar -p 9000:9000 sonarqube:lts-community
Install Trivy, Kubectl,Terraform on Ec2 terminal:
vim script.sh
sudo apt-get install wget apt-transport-https gnupg lsb-release -y
wget -qO - https://aquasecurity.github.io/trivy-repo/deb/public.key | gpg --dearmor | sudo tee /usr/share/keyrings/trivy.gpg > /dev/null
echo "deb [signed-by=/usr/share/keyrings/trivy.gpg] https://aquasecurity.github.io/trivy-repo/deb $(lsb_release -sc) main" | sudo tee -a /etc/apt/sources.list.d/trivy.list
sudo apt-get update
sudo apt-get install trivy -y
# Install Terraform
sudo apt install wget -y
wget -O- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list
sudo apt update && sudo apt install terraform
# Install kubectl
sudo apt update
sudo apt install curl -y
curl -LO https://dl.k8s.io/release/$(curl -L -s https://dl.k8s.io/release/stable.txt)/bin/linux/amd64/kubectl
sudo install -o root -g root -m 0755 kubectl /usr/local/bin/kubectl
kubectl version --client
# Install AWS CLI
curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
sudo apt-get install unzip -y
unzip awscliv2.zip
sudo ./aws/install
Next, we will log in to Jenkins and start to configure our Pipeline in Jenkins
Install Plugins like JDK, Sonarqube Scanner, NodeJs, OWASP Dependency Check
Install Plugin
Goto Manage Jenkins →Plugins → Available Plugins →
Install below plugins
1 → Eclipse Temurin Installer (Install without restart)
2 → SonarQube Scanner (Install without restart)
3 → NodeJs Plugin (Install Without restart)
4 → Docker
5 → Docker commons
6 → Docker pipeline
7 → Docker API
8 → Docker Build step
9 → Owasp Dependency Check
10 → Terraform
11 → Kubernetes
12 → Kubernetes CLI
13 → Kubernetes Client API
14 → Kubernetes Pipeline DevOps steps
After installing plugins it is a best practice to restart jenkins:
Configure Java, Sonarqube scanner, Nodejs, Dependency check, Docker and terrform in Global Tool Configuration
Goto Manage Jenkins → Tools → Install JDK(17) and NodeJs(16)→ Click on Apply and Save
Go to snaorqube:
Click on Administration → Security → Users → Click on Tokens and Update Token → Give it a name → and click on Generate Token
Copy this token .
go to jenkins -> manage jenkins-> credentials-> global credentials:
Sonar token in Jenkins' credentials allows for secure authentication and interaction between Jenkins and SonarQube, enabling seamless integration of static code analysis into the CI/CD pipeline.
Running SonarQube analyses: Jenkins often triggers builds and runs tests as part of its continuous integration process. By providing a Sonar token, Jenkins can authenticate with SonarQube to execute static code analysis during the build process.
Publishing analysis results: After running static code analysis using SonarQube, Jenkins needs to publish the analysis results back to the SonarQube server. Authentication via the Sonar token ensures that Jenkins has the necessary permissions to publish the results.
Accessing SonarQube APIs: Jenkins may need to access SonarQube APIs for various tasks, such as retrieving analysis reports, project metadata, or triggering analyses programmatically. Providing a Sonar token allows Jenkins to authenticate when making API requests to SonarQube.
Add docker's credentials too.
Integrate the Sonar token into Jenkins' system, granting SonarQube access to a specific URL. This token will authorize SonarQube to conduct comprehensive scans of the entire codebase.
In the Sonarqube Dashboard add a quality gate also
Administration–> Configuration–>Webhooks
Create EKS Cluster from Jenkins. Select new item
CHANGE YOUR S3 BUCKET NAME IN THE BACKEND.TF (in your repo and make sure that s3 bucket is already exist in your aws account in your region)
Now create a new job for the Eks provision
pipeline{
agent any
stages {
stage('Checkout from Git'){
steps{
git branch: 'main', url: 'https://github.com/Pardeep32/reddit-clone-k8s.git'
}
}
stage('Terraform version'){
steps{
sh 'terraform --version'
}
}
stage('Terraform init'){
steps{
dir('Eks-terraform') {
sh 'terraform init'
}
}
}
stage('Terraform validate'){
steps{
dir('Eks-terraform') {
sh 'terraform validate'
}
}
}
stage('Terraform plan'){
steps{
dir('Eks-terraform') {
sh 'terraform plan'
}
}
}
stage('Terraform apply/destroy'){
steps{
dir('Eks-terraform') {
sh 'terraform ${action} --auto-approve'
}
}
}
}
}
Create 2nd item:
Dockerfile:
pipeline{
agent any
tools{
jdk 'jdk17'
nodejs 'node16'
}
environment {
SCANNER_HOME=tool 'sonar-scanner'
}
stages {
stage('clean workspace'){
steps{
cleanWs()
}
}
stage('Checkout from Git'){
steps{
git branch: 'main', url: 'https://github.com/Pardeep32/reddit-clone-k8s.git'
}
}
stage('Install Dependencies') {
steps {
sh "npm install"
}
}
stage("Sonarqube Analysis "){
steps{
withSonarQubeEnv('sonar-server') {
sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Reddit \
-Dsonar.projectKey=Reddit '''
}
}
}
stage("quality gate"){
steps {
script {
waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token'
}
}
}
stage('OWASP FS SCAN') {
steps {
dependencyCheck additionalArguments: '--scan ./ --disableYarnAudit --disableNodeAudit', odcInstallation: 'DP-Check'
dependencyCheckPublisher pattern: '**/dependency-check-report.xml'
}
}
stage('TRIVY FS SCAN') {
steps {
sh "trivy fs . > trivyfs.txt"
}
}
stage("Docker Build & Push"){
steps{
script{
withDockerRegistry(credentialsId: 'docker', toolName: 'docker'){
sh "docker build -t reddit ."
sh "docker tag reddit pardeepkaur/reddit:latest "
sh "docker push pardeepkaur/reddit:latest "
}
}
}
}
stage("TRIVY"){
steps{
sh "trivy image pardeepkaur/reddit:latest > trivy.txt"
}
}
stage('Deploy to container'){
steps{
sh 'docker run -d --name reddit -p 3000:3000 pardeepkaur/reddit:latest'
}
}
}
}
Before run this pipeline just make sure to change the image name in deployment.yaml file on github.
Make sure that instead of pardeepkaur write your dockerhub username, only then you are allowed to push your image to your dockerhub.
You can see the report has been generated and the status shows as passed. You can see that there are 6.8k lines it scanned. You will see that in status, a graph will also be generated and Vulnerabilities.
To see a detailed report, you can go to issues.
Before run the pipeline again make sure to delete the previous running conatiner.
Now deploy the container in cluster: go to the Jenkins Instance
Give this command
aws eks update-kubeconfig --name EKS_CLOUD --region ca-central-1
It will Generate an Kubernetes configuration file.
Here is the path for config file.
cd .kube
cat config
copy the file that generates
Save it in your local file explorer, at your desired location with any name as REDDIT-SECRET file.
apiVersion: v1
clusters:
- cluster:
certificate-authority-data: LS0tLS1CRUdJTiBDRVJUSUZJQ0FURS0tLS0tCk1JSURCVENDQWUyZ0F3SUJBZ0lJZFYzdm1kOEZkbEF3RFFZSktvWklodmNOQVFFTEJRQXdGVEVUTUJFR0ExVUUKQXhNS2EzVmlaWEp1WlhSbGN6QWVGdzB5TkRBek1qVXlNekkwTkRsYUZ3MHpOREF6TWpNeU16STVORGxhTUJVeApFekFSQmdOVkJBTVRDbXQxWW1WeWJtVjBaWE13Z2dFaU1BMEdDU3FHU0liM0RRRUJBUVVBQTRJQkR3QXdnZ0VLCkFvSUJBUUNrbzFySlFvSlRTZEwza2tnblNTUG15RFdsbFBramFqc1N4MUgzOHAwY29KOGlKNWdXbXNtR1pHUEkKdXl5aURnaVRnZmIxQTlwN25ybnJQNFFIaWRwKzJpaVRIMDFUYkphVWxEYXpDd3RCbU93eEUzZzl4aDNMQVhiaQpSTVpRL1FyVGY3MzI4ZnkyLzZYeVhUb0E2WGM2SnUxVkJPNzdDREI0R2RBK0didWwzdi9OYVRURXBYb0RtSGpECkdOOEtCamNIb1IyeE0rOGdQdlozakZqQzkvdlR3ajV5SHdxZFhkZjB4S1dOYkVHUlZDeTFtU1ZFemRBWVVET3MKLzh4QWdwNUhuK1RzeUd1dXhkeG04RUpGQVpiL2V5ekhVbkNJYzE3dXF0S09jQmppaU1XbXdvblh4clVvUHU0NgpFcmVoZXdzOWx3TG50cHl6dCtPYW05ak1mMWVYQWdNQkFBR2pXVEJYTUE0R0ExVWREd0VCL3dRRUF3SUNwREFQCkJnTlZIUk1CQWY4RUJUQURBUUgvTUIwR0ExVWREZ1FXQkJRVUQ5YkU4TXVTU1dMUk5qcmJFMlhFa1RSKzV6QVYKQmdOVkhSRUVEakFNZ2dwcmRXSmxjbTVsZEdWek1BMEdDU3FHU0liM0RRRUJDd1VBQTRJQkFRQmpJTktwUC9KNApXdVIzbGN4NCtJLzBQakd1T25OVXNNOWZqZXVaUGxBWXlLY1dSang1WUo2KzQrVGpuaTZCQXhnMktmOXU1SUtFCmQ5M2lZeGpWUUJYcEhnaWJUbnBEdy81dG0yMXRnRjAyKzZBVlRCZ0VzaWdMN2tSSWpuMzVxYi9aL1ZGd2lUZG0KaEdza0k1anovbFl0ZDJYNzUrL2ZGSlNFc3FnZndaZVJOdWhvaUJwL1djY1VldklzVDUyU21BVzI2VjhXei9UUQpEdFp6Z0dVdWFyQUxoZTYxZFQxUG9ST2tUc1A4UDV4RGF6WkxWK0pScThlRHVJSjh6MTVrd3pSdkxnaXFTZXNMCm91Ym5vUkt0akZrenJ1Nk50bVVUUDkzeG12dndDTnkwbjFYcmpnOS9LUGh5MVoralltazR0NzFCTGFBVGdqTFUKU2wxQVgvTWNGZnRvCi0tLS0tRU5EIENFUlRJRklDQVRFLS0tLS0K
server: https://700B1E8EFEA7F39C799D3C007FE6B4F9.yl4.ca-central-1.eks.amazonaws.com
name: arn:aws:eks:ca-central-1:654654392783:cluster/EKS_CLOUD
contexts:
- context:
cluster: arn:aws:eks:ca-central-1:654654392783:cluster/EKS_CLOUD
user: arn:aws:eks:ca-central-1:654654392783:cluster/EKS_CLOUD
name: arn:aws:eks:ca-central-1:654654392783:cluster/EKS_CLOUD
current-context: arn:aws:eks:ca-central-1:654654392783:cluster/EKS_CLOUD
kind: Config
preferences: {}
users:
- name: arn:aws:eks:ca-central-1:654654392783:cluster/EKS_CLOUD
user:
exec:
apiVersion: client.authentication.k8s.io/v1beta1
args:
- --region
- ca-central-1
- eks
- get-token
- --cluster-name
- EKS_CLOUD
- --output
- json
command: aws
jenkins->manage jenkins->credentials->system->global credentails: use secret file and upload REDDIT-SECRET file here.
After configuring the Kubernetes cluster, you can add a stage to your Jenkins pipeline to deploy your application onto the Kubernetes cluster. Here's how you can do it:
stage('Deploy to kubernets'){
steps{
script{
withKubeConfig(caCertificate: '', clusterName: '', contextName: '', credentialsId: 'k8s', namespace: '', restrictKubeConfigAccess: false, serverUrl: '') {
sh 'kubectl apply -f deployment.yml'
sh 'kubectl apply -f service.yml'
sh 'kubectl apply -f ingress.yml'
}
}
}
}
Go to gmail account-> manage your google account->security->app passwords (generate a password that will add to jenkins credentials)
Add these credentials in Mange jenkins -> credentials -> system -> Global credentials: provide gmail id and password that is recently generated, id must be mail.
Configure Jenkins SMTP Server:
Navigate to your Jenkins dashboard and click on "Manage Jenkins" in the sidebar.
Select "Configure System".
Scroll down to the "E-mail Notification" section.
Enter the SMTP server details provided by your email provider. This includes the SMTP server host, port, and credentials (username and password).
You can also configure advanced settings such as SSL/TLS encryption and response timeout.
Test Email Configuration:
Scroll down to the bottom of the configuration page and click on the "Test configuration by sending test e-mail" button.
Enter an email address to which you want to send a test email.
Click on "Test configuration". If the configuration is correct, you should receive a test email.
(whenever run the pipeline , delete the previous running conatiner on terminal.)
the entire pipeline script will be
pipeline {
agent any
tools {
jdk 'jdk17'
nodejs 'node16'
}
environment {
SCANNER_HOME = tool 'sonar-scanner'
}
stages {
stage('clean workspace') {
steps {
cleanWs()
}
}
stage('Checkout from Git') {
steps {
git branch: 'main', url: 'https://github.com/Pardeep32/reddit-clone-k8s.git'
}
}
stage('Install Dependencies') {
steps {
sh "npm install"
}
}
stage("Sonarqube Analysis") {
steps {
withSonarQubeEnv('sonar-server') {
sh ''' $SCANNER_HOME/bin/sonar-scanner -Dsonar.projectName=Reddit \
-Dsonar.projectKey=Reddit '''
}
}
}
stage("quality gate") {
steps {
script {
waitForQualityGate abortPipeline: false, credentialsId: 'Sonar-token'
}
}
}
stage('TRIVY FS SCAN') {
steps {
sh "trivy fs . > trivyfs.txt"
}
}
stage("Docker Build & Push") {
steps {
script {
withDockerRegistry(credentialsId: 'docker', toolName: 'docker') {
sh "docker build -t reddit ."
sh "docker tag reddit pardeepkaur/reddit:latest "
sh "docker push pardeepkaur/reddit:latest "
}
}
}
}
stage("TRIVY") {
steps {
sh "trivy image pardeepkaur/reddit:latest > trivy.txt"
}
}
stage('Deploy to container') {
steps {
sh 'docker run -d --name reddit -p 3000:3000 pardeepkaur/reddit:latest'
}
}
stage('Deploy to Kubernetes'){
steps{
script{
withKubeConfig(credentialsId: 'k8s', serverUrl: '', caCertificate: '', contextName: '', namespace: '') {
sh 'kubectl apply -f deployment.yml'
sh 'kubectl apply -f service.yml'
sh 'kubectl apply -f ingress.yml'
}
}
}
}
}
post {
always {
emailext attachLog: true,
subject: "'${currentBuild.result}'",
body: "Project: ${env.JOB_NAME}<br/>" +
"Build Number: ${env.BUILD_NUMBER}<br/>" +
"URL: ${env.BUILD_URL}<br/>",
to: 'deepnabha20@gmail.com',
attachmentsPattern: 'trivy.txt'
}
}
}
In cluster's node's SG inbound rule add 31148 port.
Take LB's DNS name and open in new browser.
Trivy K8s Cluster scan
Give this command in Jenkins instance or If you want to add pipeline upto you
Scan a full cluster and generate a simple summary report:
trivy k8s --report=summary cluster
curl -L domain.com/test
You have successfully Deployed a Reddit Copy on Kubernetes with Ingress Enabled.
Destroy: In order to delete conatiner , load balancer. Run this pipeline:
pipeline {
agent any
stages {
stage('Deploy to Docker Container') {
steps {
script {
// Pull the latest image from Docker Hub
sh 'docker pull pardeepkaur/reddit:latest'
// Stop and remove any existing container with the same name
sh 'docker stop reddit || true'
sh 'docker rm reddit || true'
// Run a new container with the updated image
// sh 'docker run -d --name reddit -p 3000:3000 pardeepkaur/reddit:latest'
}
}
}
stage('Deploy to Kubernetes'){
steps{
script{
// Delete existing resources from Kubernetes
withKubeConfig(credentialsId: 'k8s', serverUrl: '', caCertificate: '', contextName: '', namespace: '') {
sh 'kubectl delete -f deployment.yml'
sh 'kubectl delete -f service.yml'
sh 'kubectl delete -f ingress.yml'
}
}
}
}
}
post {
always {
emailext attachLog: true,
subject: "'${currentBuild.result}'",
body: "Project: ${env.JOB_NAME}<br/>" +
"Build Number: ${env.BUILD_NUMBER}<br/>" +
"URL: ${env.BUILD_URL}<br/>",
to: 'deepnabha20@gmail.com',
attachmentsPattern: 'trivy.txt'
}
}
}
This pipeline will delete the loadbalancer.
Now delete the EKS cluster
It will delete the entire cluster.
Now terminate the reddit instance.
I hope this article serves you well.
In crafting it, my aim was to provide you with valuable insights and practical knowledge that you can apply to your endeavors. Whether you're seeking information, inspiration, or solutions to challenges, my hope is that you'll find what you need within these pages.
Should you have any questions or require further assistance, please don't hesitate to reach out. Your feedback is invaluable and helps me improve future content.
Wishing you success and fulfillment in all your endeavors.
Warm regards,
pardeep kaur
follow me on linkedin: https://www.linkedin.com/in/pardeep-kaur-kp3244/
github: https://github.com/Pardeep32
hashnode: https://hashnode.com/@PardeepKaur