[kubernetes] kubernetes-jenkins 연동 레시피

코세라 강의 중에 https://www.coursera.org/learn/google-kubernetes-engine/ 강의가 있다.

거기에 https://cloud.google.com/solutions/continuous-delivery-jenkins-kubernetes-engine를 따라는 학습 내용이 있는데..jenkins-kubernetes를 언젠가는 한 번 따라해 보고 관련 내용을 참고로 올려둔다.

* 환경 설정

$ gcloud config set compute/zone <zone-name:us-central1-a>

$ git clone https://github.com/GoogleCloudPlatform/continuous-deployment-on-kubernetes.git -b v1

$ cd continuous-deployment-on-kubernetes

$ gcloud container clusters create bootcamp --num-nodes 5 --scopes "https://www.googleapis.com/auth/projecthosting,storage-rw"

Creating cluster bootcamp....... done.

Created [https://container.googleapis.com/v1/projects/qwiklabs-gcp-2a7478819754525a/zones/us-central1-a/clusters/bootcamp].

To inspect the contents of your cluster, go to: https://console.cloud.google.com/kubernetes/workload_/gcloud/us-central1-a/bootcamp?project=qwiklabs-gcp-2a747

8819754525a

kubeconfig entry generated for bootcamp.

NAME      LOCATION       MASTER_VERSION  MASTER_IP      MACHINE_TYPE   NODE_VERSION  NUM_NODES  STATUS

bootcamp  us-central1-a  1.9.7-gke.6     35.226.136.18  n1-standard-1  1.9.7-gke.6   5          RUNNING

* jenkins 볼륨 설정

디스크 이미지를 생성한다

$ gcloud compute images create jenkins-home-image --source-uri https://storage.googleapis.com/solutions-public-assets/jenkins-cd/jenkins-home-v3.tar.gz

Created [https://www.googleapis.com/compute/v1/projects/qwiklabs-gcp-2a7478819754525a/global/images/jenkins-home-image].

NAME                PROJECT                        FAMILY  DEPRECATED  STATUS

jenkins-home-image  qwiklabs-gcp-2a7478819754525a                      READY

디스크 이미지에서 persistent disk를 생성한다.

$ gcloud compute disks create jenkins-home --image jenkins-home-image

Created [https://www.googleapis.com/compute/v1/projects/qwiklabs-gcp-2a7478819754525a/zones/us-central1-a/disks/jenkins-home].

NAME          ZONE           SIZE_GB  TYPE         STATUS

jenkins-home  us-central1-a  10       pd-standard  READY

* jenkins credentials 설정

$ export PASSWORD='jenkins'

CHANGE_ME를 기본 패스워드로 변경한다.

$ vi jenkins/k8s/options

--argumentsRealm.passwd.jenkins=jenkins --argumentsRealm.roles.jenkins=admin

jenkins의 큐버네티스 네임스페이스를 생성한다. 

$ kubectl create ns jenkins

namespace "jenkins" created

큐버네티스 secret을 생성한다. 

$ kubectl create secret generic jenkins --from-file=jenkins/k8s/options --namespace=jenkins

secret "jenkins" created

클러스터의 RBAC에 내 계정을 클러스터 어드민 관리자로 등록한다. 

$ gcloud config get-value account

Your active configuration is: [cloudshell-3833]

google1006245_student@qwiklabs.net

$ kubectl create clusterrolebinding cluster-admin-binding --clusterrole=cluster-admin --user=$(gcloud config get-value account)

* jenkins를 배포한다. 

kubetcl apply 커맨드의 기능은 다음과 같다. 

1. jenkins를 실행할 수 있는 컨테이너와 jenkins 홈 디렉토리를 포함하는 persistent disk를 포함하는 jenkins 배치를 생성합니다. 

home 디렉토리를 persistent disk에 보관하면 jenkins 마스터를 실행하는 pod가 중단되더라도 중요한 설정 데이터가 유지된다.

2. jenkins 마스터가 클러스터의 다른 pod에서 접근할 수 있도록 하는 두 가지 서비스(NodePort(8080)-외부 사용자용,  ClusterIP(50000)-jenkins master와 executor간 통신)를 생성한다.

kubectl apply를 적용한다. 

$ kubectl apply -f jenkins/k8s/

deployment "jenkins" created

Warning: kubectl apply should be used on resource created by either kubectl create --save-config or kubectl apply

serviceaccount "default" configured

clusterrolebinding "jenkins-admin" created

service "jenkins-ui" created

service "jenkins-discovery" created

jenkins pod가 실행 중인지 확인한다.  

$ kubectl get pods -w -n jenkins

NAME                      READY     STATUS    RESTARTS   AGE

jenkins-d4df5c85f-5dcxb   1/1       Running   0          9m

(ctrl+c를 줄러 종료한다)

* HTTPS 로드 밸런서를 구성한다.

우선 jenkins 네임 스페이스에 서비스가 잘 떠인지 확인한다.  NodePort, ClusterIP 제대로 떠 있다.

$ kubectl get svc -n jenkins

NAME                TYPE        CLUSTER-IP     EXTERNAL-IP   PORT(S)          AGE

jenkins-discovery   ClusterIP   10.51.252.78   <none>        50000/TCP        10m

jenkins-ui          NodePort    10.51.246.14   <none>        8080:31122/TCP   10m

SSL 인증서와 키를 생성한다

$ openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /tmp/tls.key -out /tmp/tls.crt -subj "/CN=jenkins/O=jenkins"

Generating a 2048 bit RSA private key

.............+++

....................................+++

writing new private key to '/tmp/tls.key'

-----

쿠버네티스 인증서를 secret으로 올린다.

$ kubectl create secret generic tls --from-file=/tmp/tls.crt --from-f

ile=/tmp/tls.key -n jenkins

secret "tls" created

ingress를 사용해 HTTPS 로드 밸런서를 구성한다.

$ cat jenkins/k8s/lb/ingress.yaml

# Copyright 2015 Google Inc. All rights reserved.

#

# Licensed under the Apache License, Version 2.0 (the "License");

# you may not use this file except in compliance with the License.

# You may obtain a copy of the License at

#

#     http://www.apache.org/licenses/LICENSE-2.0

#

# Unless required by applicable law or agreed to in writing, software

# distributed under the License is distributed on an "AS IS" BASIS,

# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.

# See the License for the specific language governing permissions and

# limitations under the License.

apiVersion: extensions/v1beta1

kind: Ingress

metadata:

  name: jenkins

  namespace: jenkins

spec:

  tls:

  - secretName: tls

  backend:

    serviceName: jenkins-ui

    

    

$ kubectl apply -f jenkins/k8s/lb/ingress.yaml

ingress "jenkins" created

* jenkins를 연결한다. 

현재 로드 밸런서의 상태를 확인한다. 

$ kubectl describe ingress jenkins --namespace jenkins

Name:             jenkins

Namespace:        jenkins

Address:          35.201.122.4

Default backend:  jenkins-ui:8080 (10.48.0.5:8080)

TLS:

  tls terminates

Rules:

  Host  Path  Backends

  ----  ----  --------

  *     *     jenkins-ui:8080 (10.48.0.5:8080)

Annotations:

  forwarding-rule:        k8s-fw-jenkins-jenkins--f71fa3ee157e1bf9

  https-forwarding-rule:  k8s-fws-jenkins-jenkins--f71fa3ee157e1bf9

  https-target-proxy:     k8s-tps-jenkins-jenkins--f71fa3ee157e1bf9

  ssl-cert:               k8s-ssl-jenkins-jenkins--f71fa3ee157e1bf9

  static-ip:              k8s-fw-jenkins-jenkins--f71fa3ee157e1bf9

  url-map:                k8s-um-jenkins-jenkins--f71fa3ee157e1bf9

  backends:               {"k8s-be-31122--f71fa3ee157e1bf9":"Unknown"}

  target-proxy:           k8s-tp-jenkins-jenkins--f71fa3ee157e1bf9

Events:

  Type    Reason   Age                From                     Message

  ----    ------   ----               ----                     -------

  Normal  ADD      2m                 loadbalancer-controller  jenkins/jenkins

  Normal  CREATE   42s                loadbalancer-controller  ip: 35.201.122.4

  Normal  Service  35s (x3 over 45s)  loadbalancer-controller  default backend set to jenkins-ui:31122

  

backendsf를 살펴보면 아직 unknown이다. healthy가 될 때까지 반복해서 확인한다.   (25분도 걸릴 수 있다고 함)

backends:               {"k8s-be-31122--f71fa3ee157e1bf9":"Unknown"}

다음과 같이 변하는 것을 확인하면 다음 단계를 거친다.

backends:               {"k8s-be-31122--f71fa3ee157e1bf9":"HEALTHY"}

  

  

jenkins 네임 스페이스의 ingress ip를 확인한다. 

$ kubectl get ingress jenkins -n jenkins

NAME      HOSTS     ADDRESS        PORTS     AGE

jenkins   *         35.201.122.4   80, 443   10m

$ echo "Jenkins URL: https://`kubectl get ingress jenkins -n jenkins 

-o jsonpath='{.status.loadBalancer.ingress[0].ip}'`"; echo "Your username/password:  jenkins/$PASSWORD"

Jenkins URL: https://35.201.122.4

Your username/password:  jenkins/jenkins

https://35.201.122.4/에 접근해 본다.

jenkins/jenkins 계정과 암호를 넣는다. 저 미국 땅에 있어서 그런지 엄청 오래 걸린다. 

jenkins 마스터 설정 완료했다

* ci/cd 구성하기

production/canary를 구성한다

설정 파일은 sample-app/k8s에 다 있다.

$ ls -al

total 36

drwxr-xr-x 3 google1006245_student google1006245_student 4096 Sep  7 16:07 .

drwxr-xr-x 7 google1006245_student google1006245_student 4096 Sep  7 16:07 ..

-rw-r--r-- 1 google1006245_student google1006245_student  623 Sep  7 16:07 Dockerfile

-rw-r--r-- 1 google1006245_student google1006245_student 2270 Sep  7 16:07 html.go

-rw-r--r-- 1 google1006245_student google1006245_student 2336 Sep  7 16:07 Jenkinsfile

drwxr-xr-x 6 google1006245_student google1006245_student 4096 Sep  7 16:07 k8s

-rw-r--r-- 1 google1006245_student google1006245_student 4187 Sep  7 16:07 main.go

-rw-r--r-- 1 google1006245_student google1006245_student  872 Sep  7 16:07 main_test.go

$ ls -al k8s

total 24

drwxr-xr-x 6 google1006245_student google1006245_student 4096 Sep  7 16:07 .

drwxr-xr-x 3 google1006245_student google1006245_student 4096 Sep  7 16:07 ..

drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep  7 16:07 canary

drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep  7 16:07 dev

drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep  7 16:07 production

drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep  7 16:07 services

$ ls -al k8s/production/

total 16

drwxr-xr-x 2 google1006245_student google1006245_student 4096 Sep  7 16:07 .

drwxr-xr-x 6 google1006245_student google1006245_student 4096 Sep  7 16:07 ..

-rw-r--r-- 1 google1006245_student google1006245_student 1271 Sep  7 16:07 backend-production.yaml

-rw-r--r-- 1 google1006245_student google1006245_student 1326 Sep  7 16:07 frontend-production.yaml

$ cd sample-app

$ kubectl create ns production

namespace "production" created

$ kubectl apply -f k8s/production -n production

deployment "gceme-backend-production" created

deployment "gceme-frontend-production" created

$ kubectl apply -f k8s/canary -n production

deployment "gceme-backend-canary" created

deployment "gceme-frontend-canary" created

$ kubectl apply -f k8s/services -n production

service "gceme-backend" created

service "gceme-frontend" created

kubectl scale 커맨드를 사용해 gceme-frontend-production 레플리카를 4개로 늘린다.

$ kubectl scale deployment gceme-frontend-production -n production --replicas 4

deployment "gceme-frontend-production" scaled

제대로 레플리카가 실행 중인지 확인한다.

$ kubectl get pods -n production -l app=gceme -l role=frontend

NAME                                         READY     STATUS    RESTARTS   AGE

gceme-frontend-canary-7ff449cd6-5zs4g        1/1       Running   0          2m

gceme-frontend-production-6f5956885f-57rjx   1/1       Running   0          1m

gceme-frontend-production-6f5956885f-85r65   1/1       Running   0          1m

gceme-frontend-production-6f5956885f-gpjq8   1/1       Running   0          2m

gceme-frontend-production-6f5956885f-mp77h   1/1       Running   0          1m

당연히 front-production만 늘렸으니 다른 것은 변동이 없어야 한다. 

$ kubectl get pods -n production -l app=gceme -l role=backend

NAME                                       READY     STATUS    RESTARTS   AGE

gceme-backend-canary-f94986bfb-gktrj       1/1       Running   0          3m

gceme-backend-production-69d5c6c9f-8csqg   1/1       Running   0          3m

$ kubectl get service gceme-frontend -n production

NAME             TYPE           CLUSTER-IP     EXTERNAL-IP      PORT(S)        AGE

gceme-frontend   LoadBalancer   10.51.246.89   35.239.123.157   80:30795/TCP   3m

fronted service load balancer ip를 환경 변수로 설정한다.

$ kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}"  --namespace=production services gceme-frontend

35.239.123.157

$ export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}"  --namespace=production services gceme-frontend)

$ echo $FRONTEND_SERVICE_IP

35.239.123.157

http://35.239.123.157/를 요청하면 잘 떠 있는지 확인할 수 있다. 

$ curl http://35.239.123.157/version

1.0.0

이제 앱 배포를 완료했다.

* jenkins 파이프라인 생성한다

gceme 샘플 앱 복사본을 cloud source repository에 보낸다

$ gcloud source repos create gceme

Created [gceme].

WARNING: You may be billed for this repository. See https://cloud.google.com/source-repositories/docs/pricing for details.

git 설정하고 코드를 push한다.

$ git init

Initialized empty Git repository in /home/google1006245_student/continuous-deployment-on-kubernetes/sample-app/.git/

$ git config credential.helper gcloud.sh

$ export PROJECT_ID=$(gcloud config get-value project)

Your active configuration is: [cloudshell-3833]

$ gcloud config get-value project

Your active configuration is: [cloudshell-3833]

qwiklabs-gcp-2a7478819754525a

$ git remote add origin https://source.developers.google.com/p/$PROJECT_ID/r/gceme

$ git config --global user.email google1006245_student@qwiklabs.net

$ git config --global user.name google1006245_student

$ git commit -m "Initial commit"

[master (root-commit) 370a2cb] Initial commit

 14 files changed, 723 insertions(+)

 create mode 100644 Dockerfile

 create mode 100644 Jenkinsfile

 create mode 100644 html.go

 create mode 100644 k8s/canary/backend-canary.yaml

 create mode 100644 k8s/canary/frontend-canary.yaml

 create mode 100644 k8s/dev/backend-dev.yaml

 create mode 100644 k8s/dev/default.yml

 create mode 100644 k8s/dev/frontend-dev.yaml

 create mode 100644 k8s/production/backend-production.yaml

 create mode 100644 k8s/production/frontend-production.yaml

 create mode 100644 k8s/services/backend.yaml

 create mode 100644 k8s/services/frontend.yaml

 create mode 100644 main.go

 create mode 100644 main_test.go

$ git push origin master

Counting objects: 21, done.

Compressing objects: 100% (21/21), done.

Writing objects: 100% (21/21), 5.95 KiB | 0 bytes/s, done.

Total 21 (delta 9), reused 0 (delta 0)

remote: Resolving deltas: 100% (9/9)

To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme

 * [new branch]      master -> master

 

 

* 서비스 계정 credentials를 추가한다

 

아까 생성한 jenkins ui인 https://35.201.122.4/에 접속하고 jenkins/jenkins를 입력 후

 Jenkins > Credentials(https://35.201.122.4/credentials/)에 접속한다

 

Credentials 화면의 열쇠 모양 옆의 Jenkins를 클릭한다.

그 다음 화면에서 Global Credentials를 클릭한다 

왼쪽 화면에서 Add Credentials를 클릭한다.

Kind 에서 Select Google Service Account from metadata를 선택한다

Project Name에 "qwiklabs-gcp-2a7478819754525a" 계정이 나오는데. ok를 클릭한다.

credentials를 추가했다.

* jenkins job을 생성한다

jenkins new Job 화면(https://35.201.122.4/view/all/newJob)에서 sample-app 이름 넣고 multibranch pipeline을 생성한다

job화면의 add Source 의 git를 선택하고 https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme 를 추가하고 credentials로 잘 변경한다.

 Build Periodically에다가 * * * * *를 추가하고 save 한다. 

 

jenkins 화면 옆에 보면,  jnlp-5b90daf3cdb  (오프라인)을 추가한다.

 

* 개발 환경을 생성한다

$ git checkout -b new-feature

Switched to a new branch 'new-feature'

 

파이프라인 정의를 수정한다

$ gcloud config get-value project

Your active configuration is: [cloudshell-3833]

qwiklabs-gcp-2a7478819754525a

$ export PROJECT=$(gcloud config get-value project) 

$ sed -i s/REPLACE_WITH_YOUR_PROJECT_ID/${PROJECT}/g Jenkinsfile

$ cat Jenkinsfile

node {

  def project = 'qwiklabs-gcp-2a7478819754525a'

  def appName = 'gceme'

  def feSvcName = "${appName}-frontend"

  def imageTag = "gcr.io/${project}/${appName}:${env.BRANCH_NAME}.${env.BUILD_NUMBER}"

  checkout scm

  stage 'Build image'

  sh("docker build -t ${imageTag} .")

  stage 'Run Go tests'

  sh("docker run ${imageTag} go test")

  stage 'Push image to registry'

  sh("gcloud docker -- push ${imageTag}")

  stage "Deploy Application"

  switch (env.BRANCH_NAME) {

    // Roll out to canary environment

    case "canary":

        // Change deployed image in canary to the one we just built

        sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/canary/*.yaml")

        sh("kubectl --namespace=production apply -f k8s/services/")

        sh("kubectl --namespace=production apply -f k8s/canary/")

        sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")

        break

    // Roll out to production

    case "master":

        // Change deployed image in canary to the one we just built

        sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/production/*.yaml")

        sh("kubectl --namespace=production apply -f k8s/services/")

        sh("kubectl --namespace=production apply -f k8s/production/")

        sh("echo http://`kubectl --namespace=production get service/${feSvcName} --output=json | jq -r '.status.loadBalancer.ingress[0].ip'` > ${feSvcName}")

        break

    // Roll out a dev environment

    default:

        // Create namespace if it doesn't exist

        sh("kubectl get ns ${env.BRANCH_NAME} || kubectl create ns ${env.BRANCH_NAME}")

        // Don't use public load balancing for development branches

        sh("sed -i.bak 's#LoadBalancer#ClusterIP#' ./k8s/services/frontend.yaml")

        sh("sed -i.bak 's#gcr.io/cloud-solutions-images/gceme:1.0.0#${imageTag}#' ./k8s/dev/*.yaml")

        sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/services/")

        sh("kubectl --namespace=${env.BRANCH_NAME} apply -f k8s/dev/")

        echo 'To access your environment run `kubectl proxy`'

        echo "Then access your service via http://localhost:8001/api/v1/proxy/namespaces/${env.BRANCH_NAME}/services/${feSvcName}:80/"

  }

}

gcme 코드를 변경한다.

$ sed -i s/blue/orange/g html.go

$ sed -i s/1.0.0/2.0.0/g main.go

$ git add Jenkinsfile html.go main.go

$ git commit -m "Version 2.0.0"

[new-feature 98a48bc] Version 2.0.0

 3 files changed, 4 insertions(+), 4 deletions(-)

$ git push origin new-feature

Counting objects: 5, done.

Compressing objects: 100% (5/5), done.

Writing objects: 100% (5/5), 569 bytes | 0 bytes/s, done.

Total 5 (delta 3), reused 0 (delta 0)

remote: Resolving deltas: 100% (3/3)

To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme

 * [new branch]      new-feature -> new-feature

푸시하고 나서 jenkins ui(https://35.201.122.4/job/sample-app/)에 접속하면 new-feature는 성공하고 master는 에러가 난 것을 확인할 수 있다.

새로 변경한 로드 밸런서 대신 간단하게 kubectl proxy로 테스트할 수 있다.  

$ kubectl proxy &

[1] 955

Starting to serve on 127.0.0.1:8001

$ curl http://localhost:8001/api/v1/proxy/namespaces/new-feature/services/gceme-frontend:80/version

2.0.0

이제 개발 환경이 구축되었다..

* canary 배포하기

$ git checkout -b canary

Switched to a new branch 'canary'

$ git push origin canary

Total 0 (delta 0), reused 0 (delta 0)

To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme

 * [new branch]      canary -> canary

 

푸시하고 나서 jenkins ui(https://35.201.122.4/job/sample-app/)에 접속하면 canary는 성공이다. 

$ export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend)

$ while true; do curl http://$FRONTEND_SERVICE_IP/version; sleep 1;  done

1.0.0

1.0.0

2.0.0

1.0.0

1.0.0

5번 중에 1번은 canary가 받는다.

* 상용 배포하기

$ git checkout master

Switched to branch 'master'

$ git merge canary

Updating 370a2cb..98a48bc

Fast-forward

 Jenkinsfile | 2 +-

 html.go     | 4 ++--

 main.go     | 2 +-

 3 files changed, 4 insertions(+), 4 deletions(-)

$ git push origin master

Total 0 (delta 0), reused 0 (delta 0)

To https://source.developers.google.com/p/qwiklabs-gcp-2a7478819754525a/r/gceme

   370a2cb..98a48bc  master -> master

푸시하고 나서 jenkins ui(https://35.201.122.4/job/sample-app/)에 접속하면 master는 성공이다. 

$ export FRONTEND_SERVICE_IP=$(kubectl get -o jsonpath="{.status.loadBalancer.ingress[0].ip}" --namespace=production services gceme-frontend)

$ while true; do curl http://$FRONTEND_SERVICE_IP/version; sleep 1;  done

2.0.0

2.0.0

2.0.0

2.0.0

2.0.0

2.0.0

제대로 상용 환경에 배포했다.