코세라 강의 중에 https://www.coursera.org/learn/google-kubernetes-engine/ 강의가 있다.
거기에 https://cloud.google.com/solutions/continuous-delivery-jenkins-kubernetes-engine를 따라는 학습 내용이 있는데..jenkins-kubernetes를 언젠가는 한 번 따라해 보고 관련 내용을 참고로 올려둔다.
* 환경 설정
$ gcloud config set compute/zone <zone-name:us-central1-a>
$ git clone https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes.git -b v1
$ cd continuous-deployment-on-kubernetes
$ gcloud container clusters create bootcamp --num-nodes 5 --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"
Creating cluster bootcamp....... done.
Created [https://container.googleapis.com/v1/projects/qwiklabs-gcp-2a7478819754525a/zones/us-central1-a/clusters/bootcamp].
To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-central1-a/bootcamp?project=qwiklabs-gcp-2a747
8819754525a
kubeconfig entry generated for bootcamp.
NAME LOCATION MASTER_VERSION MASTER_IP MACHINE_TYPE NODE_VERSION NUM_NODES STATUS
bootcamp us-central1-a 1.9.7-gke.6 35.226.136.18 n1-standard-1 1.9.7-gke.6 5 RUNNING
* jenkins 볼륨 설정
디스크 이미지를 생성한다
$ gcloud compute images create jenkins-home-image --source-uri https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v3.tar.gz
Created [https://www.googleapis.com/compute/v1/projects/qwiklabs-gcp-2a7478819754525a/global/images/jenkins-home-image].
NAME PROJECT FAMILY DEPRECATED STATUS
jenkins-home-image qwiklabs-gcp-2a7478819754525a READY
디스크 이미지에서 persistent disk를 생성한다.
$ gcloud compute disks create jenkins-home --image jenkins-home-image
Created [https://www.googleapis.com/compute/v1/projects/qwiklabs-gcp-2a7478819754525a/zones/us-central1-a/disks/jenkins-home].
NAME ZONE SIZE_GB TYPE STATUS
jenkins-home us-central1-a 10 pd-standard READY
* jenkins credentials 설정
$ export PASSWORD='jenkins'
CHANGE_ME를 기본 패스워드로 변경한다.
$ vi jenkins/k8s/options
--argumentsRealm.passwd.jenkins=jenkins --argumentsRealm.roles.jenkins=admin
jenkins의 큐버네티스 네임스페이스를 생성한다.
$ kubectl create ns jenkins
namespace "jenkins" created
큐버네티스 secret을 생성한다.
$ kubectl create secret generic jenkins --from-file=jenkins/k8s/options --namespace=jenkins
secret "jenkins" created
클러스터의 RBAC에 내 계정을 클러스터 어드민 관리자로 등록한다.
$ gcloud config get-value account
Your active configuration is: [cloudshell-3833]
google1006245_student@qwiklabs.net
$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)
* jenkins를 배포한다.
kubetcl apply 커맨드의 기능은 다음과 같다.
1. jenkins를 실행할 수 있는 컨테이너와 jenkins 홈 디렉토리를 포함하는 persistent disk를 포함하는 jenkins 배치를 생성합니다.
home 디렉토리를 persistent disk에 보관하면 jenkins 마스터를 실행하는 pod가 중단되더라도 중요한 설정 데이터가 유지된다.
2. jenkins 마스터가 클러스터의 다른 pod에서 접근할 수 있도록 하는 두 가지 서비스(NodePort(8080)-외부 사용자용, ClusterIP(50000)-jenkins master와 executor간 통신)를 생성한다.
kubectl apply를 적용한다.
$ kubectl apply -f jenkins/k8s/
deployment "jenkins" created
Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply
serviceaccount "default" configured
clusterrolebinding "jenkins-admin" created
service "jenkins-ui" created
service "jenkins-discovery" created
jenkins pod가 실행 중인지 확인한다.
$ kubectl get pods -w -n jenkins
NAME READY STATUS RESTARTS AGE
jenkins-d4df5c85f-5dcxb 1/1 Running 0 9m
(ctrl+c를 줄러 종료한다)
* HTTPS 로드 밸런서를 구성한다.
우선 jenkins 네임 스페이스에 서비스가 잘 떠인지 확인한다. NodePort, ClusterIP 제대로 떠 있다.
$ kubectl get svc -n jenkins
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
jenkins-discovery ClusterIP 10.51.252.78 <none> 50000/TCP 10m
jenkins-ui NodePort 10.51.246.14 <none> 8080:31122/TCP 10m
SSL 인증서와 키를 생성한다
$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=jenkins/O=jenkins"
Generating a 2048 bit RSA private key
.............+++
....................................+++
writing new private key to '/tmp/tls.key'
-----
쿠버네티스 인증서를 secret으로 올린다.
$ kubectl create secret generic tls --from-file=/tmp/tls.crt --from-f
ile=/tmp/tls.key -n jenkins
secret "tls" created
ingress를 사용해 HTTPS 로드 밸런서를 구성한다.
$ cat jenkins/k8s/lb/ingress.yaml
# Copyright 2015 Google Inc. All rights reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
apiVersion: extensions/v1beta1
kind: Ingress
metadata:
name: jenkins
namespace: jenkins
spec:
tls:
- secretName: tls
backend:
serviceName: jenkins-ui
$ kubectl apply -f jenkins/k8s/lb/ingress.yaml
ingress "jenkins" created
* jenkins를 연결한다.
현재 로드 밸런서의 상태를 확인한다.
$ kubectl describe ingress jenkins --namespace jenkins
Name: jenkins
Namespace: jenkins
Address: 35.201.122.4
Default backend: jenkins-ui:8080 (10.48.0.5:8080)
TLS:
tls terminates
Rules:
Host Path Backends
---- ---- --------
* * jenkins-ui:8080 (10.48.0.5:8080)
Annotations:
forwarding-rule: k8s-fw-jenkins-jenkins--f71fa3ee157e1bf9
https-forwarding-rule: k8s-fws-jenkins-jenkins--f71fa3ee157e1bf9
https-target-proxy: k8s-tps-jenkins-jenkins--f71fa3ee157e1bf9
ssl-cert: k8s-ssl-jenkins-jenkins--f71fa3ee157e1bf9
static-ip: k8s-fw-jenkins-jenkins--f71fa3ee157e1bf9
url-map: k8s-um-jenkins-jenkins--f71fa3ee157e1bf9
backends: {"k8s-be-31122--f71fa3ee157e1bf9":"Unknown"}
target-proxy: k8s-tp-jenkins-jenkins--f71fa3ee157e1bf9
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ADD 2m loadbalancer-controller jenkins/jenkins
Normal CREATE 42s loadbalancer-controller ip: 35.201.122.4
Normal Service 35s (x3 over 45s) loadbalancer-controller default backend set to jenkins-ui:31122
backendsf를 살펴보면 아직 unknown이다. healthy가 될 때까지 반복해서 확인한다. (25분도 걸릴 수 있다고 함)
backends: {"k8s-be-31122--f71fa3ee157e1bf9":"Unknown"}
다음과 같이 변하는 것을 확인하면 다음 단계를 거친다.
backends: {"k8s-be-31122--f71fa3ee157e1bf9":"HEALTHY"}
jenkins 네임 스페이스의 ingress ip를 확인한다.
$ kubectl get ingress jenkins -n jenkins
NAME HOSTS ADDRESS PORTS AGE
jenkins * 35.201.122.4 80, 443 10m
$ echo "Jenkins URL: https://`kubectl get ingress jenkins -n jenkins
-o jsonpath='{.status.loadBalancer.ingress[0].ip}'`"; echo "Your username/password: jenkins/$PASSWORD"
Jenkins URL: https://35.201.122.4
Your username/password: jenkins/jenkins
https://35.201.122.4/에 접근해 본다.
jenkins/jenkins 계정과 암호를 넣는다. 저 미국 땅에 있어서 그런지 엄청 오래 걸린다.
jenkins 마스터 설정 완료했다
* ci/cd 구성하기
production/canary를 구성한다
설정 파일은 sample-app/k8s에 다 있다.
$ ls -al
total 36
drwxr-xr-x 3 google1006245_student google1006245_student 4096 Sep 7 16:07 .
drwxr-xr-x 7 google1006245_student google1006245_student 4096 Sep 7 16:07 ..
-rw-r--r-- 1 google1006245_student google1006245_student 623 Sep 7 16:07 Dockerfile
-rw-r--r-- 1 google1006245_student google1006245_student 2270 Sep 7 16:07 html.go
-rw-r--r-- 1 google1006245_student google1006245_student 2336 Sep 7 16:07 Jenkinsfile
drwxr-xr-x 6 google1006245_student google1006245_student 4096 Sep 7 16:07 k8s
-rw-r--r-- 1 google1006245_student google1006245_student 4187 Sep 7 16:07 main.go
-rw-r--r-- 1 google1006245_student google1006245_student 872 Sep 7 16:07 main_test.go
$ ls -al k8s
total 24
drwxr-xr-x 6 google1006245_student google1006245_student 4096 Sep 7 16:07 .
drwxr-xr-x 3 google1006245_student google1006245_student 4096 Sep 7 16:07 ..
drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep 7 16:07 canary
drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep 7 16:07 dev
drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep 7 16:07 production
drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep 7 16:07 services
$ ls -al k8s/production/
total 16
drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep 7 16:07 .
drwxr-xr-x 6 google1006245_student google1006245_student 4096 Sep 7 16:07 ..
-rw-r--r-- 1 google1006245_student google1006245_student 1271 Sep 7 16:07 backend-production.yaml
-rw-r--r-- 1 google1006245_student google1006245_student 1326 Sep 7 16:07 frontend-production.yaml
$ cd sample-app
$ kubectl create ns production
namespace "production" created
$ kubectl apply -f k8s/production -n production
deployment "gceme-backend-production" created
deployment "gceme-frontend-production" created
$ kubectl apply -f k8s/canary -n production
deployment "gceme-backend-canary" created
deployment "gceme-frontend-canary" created
$ kubectl apply -f k8s/services -n production
service "gceme-backend" created
service "gceme-frontend" created
kubectl scale 커맨드를 사용해 gceme-frontend-production 레플리카를 4개로 늘린다.
$ kubectl scale deployment gceme-frontend-production -n production --replicas 4
deployment "gceme-frontend-production" scaled
제대로 레플리카가 실행 중인지 확인한다.
$ kubectl get pods -n production -l app=gceme -l role=frontend
NAME READY STATUS RESTARTS AGE
gceme-frontend-canary-7ff449cd6-5zs4g 1/1 Running 0 2m
gceme-frontend-production-6f5956885f-57rjx 1/1 Running 0 1m
gceme-frontend-production-6f5956885f-85r65 1/1 Running 0 1m
gceme-frontend-production-6f5956885f-gpjq8 1/1 Running 0 2m
gceme-frontend-production-6f5956885f-mp77h 1/1 Running 0 1m
당연히 front-production만 늘렸으니 다른 것은 변동이 없어야 한다.
$ kubectl get pods -n production -l app=gceme -l role=backend
NAME READY STATUS RESTARTS AGE
gceme-backend-canary-f94986bfb-gktrj 1/1 Running 0 3m
gceme-backend-production-69d5c6c9f-8csqg 1/1 Running 0 3m
$ kubectl get service gceme-frontend -n production
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
gceme-frontend LoadBalancer 10.51.246.89 35.239.123.157 80:30795/TCP 3m
fronted service load balancer ip를 환경 변수로 설정한다.
$ kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend
35.239.123.157
$ export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend)
$ echo $FRONTEND_SERVICE_IP
35.239.123.157
http://35.239.123.157/를 요청하면 잘 떠 있는지 확인할 수 있다.
$ curl http://35.239.123.157/version
1.0.0
이제 앱 배포를 완료했다.
* jenkins 파이프라인 생성한다
gceme 샘플 앱 복사본을 cloud source repository에 보낸다
$ gcloud source repos create gceme
Created [gceme].
WARNING: You may be billed for this repository. See https://cloud.google.com/source-repositories/docs/pricing for details.
git 설정하고 코드를 push한다.
$ git init
Initialized empty Git repository in /home/google1006245_student/continuous-deployment-on-kubernetes/sample-app/.git/
$ git config credential.helper gcloud.sh
$ export PROJECT_ID=$(gcloud config get-value project)
Your active configuration is: [cloudshell-3833]
$ gcloud config get-value project
Your active configuration is: [cloudshell-3833]
qwiklabs-gcp-2a7478819754525a
$ git remote add origin https://source.developers.google.com/p/$PROJECT_ID/r/gceme
$ git config --global user.email google1006245_student@qwiklabs.net
$ git config --global user.name google1006245_student
$ git commit -m "Initial commit"
[master (root-commit) 370a2cb] Initial commit
14 files changed, 723 insertions(+)
create mode 100644 Dockerfile
create mode 100644 Jenkinsfile
create mode 100644 html.go
create mode 100644 k8s/canary/backend-canary.yaml
create mode 100644 k8s/canary/frontend-canary.yaml
create mode 100644 k8s/dev/backend-dev.yaml
create mode 100644 k8s/dev/default.yml
create mode 100644 k8s/dev/frontend-dev.yaml
create mode 100644 k8s/production/backend-production.yaml
create mode 100644 k8s/production/frontend-production.yaml
create mode 100644 k8s/services/backend.yaml
create mode 100644 k8s/services/frontend.yaml
create mode 100644 main.go
create mode 100644 main_test.go
$ git push origin master
Counting objects: 21, done.
Compressing objects: 100% (21/21), done.
Writing objects: 100% (21/21), 5.95 KiB | 0 bytes/s, done.
Total 21 (delta 9), reused 0 (delta 0)
remote: Resolving deltas: 100% (9/9)
To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme
* [new branch] master -> master
* 서비스 계정 credentials를 추가한다
아까 생성한 jenkins ui인 https://35.201.122.4/에 접속하고 jenkins/jenkins를 입력 후
Jenkins > Credentials(https://35.201.122.4/credentials/)에 접속한다
Credentials 화면의 열쇠 모양 옆의 Jenkins를 클릭한다.
그 다음 화면에서 Global Credentials를 클릭한다
왼쪽 화면에서 Add Credentials를 클릭한다.
Kind 에서 Select Google Service Account from metadata를 선택한다
Project Name에 "qwiklabs-gcp-2a7478819754525a" 계정이 나오는데. ok를 클릭한다.
credentials를 추가했다.
* jenkins job을 생성한다
jenkins new Job 화면(https://35.201.122.4/view/all/newJob)에서 sample-app 이름 넣고 multibranch pipeline을 생성한다
job화면의 add Source 의 git를 선택하고 https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme 를 추가하고 credentials로 잘 변경한다.
Build Periodically에다가 * * * * *를 추가하고 save 한다.
jenkins 화면 옆에 보면, jnlp-5b90daf3cdb (오프라인)을 추가한다.
* 개발 환경을 생성한다
$ git checkout -b new-feature
Switched to a new branch 'new-feature'
파이프라인 정의를 수정한다
$ gcloud config get-value project
Your active configuration is: [cloudshell-3833]
qwiklabs-gcp-2a7478819754525a
$ export PROJECT=$(gcloud config get-value project)
$ sed -i s/REPLACE_WITH_YOUR_PROJECT_ID/${PROJECT}/g Jenkinsfile
$ cat Jenkinsfile
node {
def project = 'qwiklabs-gcp-2a7478819754525a'
def appName = 'gceme'
def feSvcName = "${appName}-frontend"
def imageTag = "gcr.io/${project}/${appName}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"
checkout scm
stage 'Build image'
sh("docker build -t ${imageTag} .")
stage 'Run Go tests'
sh("docker run ${imageTag} go test")
stage 'Push image to registry'
sh("gcloud docker -- push ${imageTag}")
stage "Deploy Application"
switch (env.BRANCH_NAME) {
// Roll out to canary environment
case "canary":
// Change deployed image in canary to the one we just built
sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/canary/*.yaml")
sh("kubectl --namespace=production apply -f k8s/services/")
sh("kubectl --namespace=production apply -f k8s/canary/")
sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")
break
// Roll out to production
case "master":
// Change deployed image in canary to the one we just built
sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/production/*.yaml")
sh("kubectl --namespace=production apply -f k8s/services/")
sh("kubectl --namespace=production apply -f k8s/production/")
sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")
break
// Roll out a dev environment
default:
// Create namespace if it doesn't exist
sh("kubectl get ns ${env.BRANCH_NAME} || kubectl create ns ${env.BRANCH_NAME}")
// Don't use public load balancing for development branches
sh("sed -i.bak 's#LoadBalancer#ClusterIP#' ./k8s/services/frontend.yaml")
sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/dev/*.yaml")
sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/services/")
sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/dev/")
echo 'To access your environment run `kubectl proxy`'
echo "Then access your service via http://localhost:8001/api/v1/proxy/namespaces/${env.BRANCH_NAME}/services/${feSvcName}:80/"
}
}
gcme 코드를 변경한다.
$ sed -i s/blue/orange/g html.go
$ sed -i s/1.0.0/2.0.0/g main.go
$ git add Jenkinsfile html.go main.go
$ git commit -m "Version 2.0.0"
[new-feature 98a48bc] Version 2.0.0
3 files changed, 4 insertions(+), 4 deletions(-)
$ git push origin new-feature
Counting objects: 5, done.
Compressing objects: 100% (5/5), done.
Writing objects: 100% (5/5), 569 bytes | 0 bytes/s, done.
Total 5 (delta 3), reused 0 (delta 0)
remote: Resolving deltas: 100% (3/3)
To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme
* [new branch] new-feature -> new-feature
푸시하고 나서 jenkins ui(https://35.201.122.4/job/sample-app/)에 접속하면 new-feature는 성공하고 master는 에러가 난 것을 확인할 수 있다.
새로 변경한 로드 밸런서 대신 간단하게 kubectl proxy로 테스트할 수 있다.
$ kubectl proxy &
[1] 955
Starting to serve on 127.0.0.1:8001
$ curl http://localhost:8001/api/v1/proxy/namespaces/new-feature/services/gceme-frontend:80/version
2.0.0
이제 개발 환경이 구축되었다..
* canary 배포하기
$ git checkout -b canary
Switched to a new branch 'canary'
$ git push origin canary
Total 0 (delta 0), reused 0 (delta 0)
To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme
* [new branch] canary -> canary
푸시하고 나서 jenkins ui(https://35.201.122.4/job/sample-app/)에 접속하면 canary는 성공이다.
$ export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend)
$ while true; do curl http://$FRONTEND_SERVICE_IP/version; sleep 1; done
1.0.0
1.0.0
2.0.0
1.0.0
1.0.0
5번 중에 1번은 canary가 받는다.
* 상용 배포하기
$ git checkout master
Switched to branch 'master'
$ git merge canary
Updating 370a2cb..98a48bc
Fast-forward
Jenkinsfile | 2 +-
html.go | 4 ++--
main.go | 2 +-
3 files changed, 4 insertions(+), 4 deletions(-)
$ git push origin master
Total 0 (delta 0), reused 0 (delta 0)
To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme
370a2cb..98a48bc master -> master
푸시하고 나서 jenkins ui(https://35.201.122.4/job/sample-app/)에 접속하면 master는 성공이다.
$ export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend)
$ while true; do curl http://$FRONTEND_SERVICE_IP/version; sleep 1; done
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
2.0.0
제대로 상용 환경에 배포했다.