基于K8S的CICD系统实现

基于K8S的CICD系统实现

前情提要

本系统仅仅是搭建好了需要的组件，尚未完成测试，还不确定哪里需要完善，可自行测试，后续会完善

系统拓扑图

K8S集群

主机名 ip1（NAT） 系统 磁盘 内存
master1 192.168.48.101 OpenEuler-22.03-LTS 100G 4G
master2 192.168.48.102 OpenEuler-22.03-LTS 100G 4G
master3 192.168.48.103 OpenEuler-22.03-LTS 100G 4G
node01 192.168.48.104 OpenEuler-22.03-LTS 100G 8G
node02 192.168.48.105 OpenEuler-22.03-LTS 100G 8G
高可用ip 192.168.48.200

harbor集群

角色 主机名 ip 系统 资源最低要求
Harbor1
nginx
Keepalived1		 harbor1 192.168.48.106 OpenEuler22.03LTS CPU：4核
内存：8G
硬盘：40G
Harbor2
nginx
Keepalived2		 harbor2 192.168.48.107 OpenEuler22.03LTS CPU：4核
内存：8G
硬盘：40G
postgresql
Redis
NFS共享		 zujian 192.168.48.108 OpenEuler22.03LTS CPU：4核
内存：8G
硬盘：40G
高可用ip 192.168.48.100

部署K8S高可用集群

OpenEuler-部署K8S高可用集群（内部etcd） - 严千屹博客 (qianyios.top)

我这里只有四台机子少了一台node02自行做完基础操作加入集群集群

部署ceph集群

基于K8S1.28.2实验rook部署ceph - 严千屹博客 (qianyios.top)

部署harbor进群

Harbor共享存储高可用 - 严千屹博客 (qianyios.top)

配置仓库地址

部署好后，需要将K8s各个节点，和harbor各个节点都要进行配置仓库地址

操作节点：[所有节点]

1
vim /etc/docker/daemon.json
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
# 客户端默认使用的是https协议，所以需要对docker做以下修改,在文件末尾添加insecure-registries
[root@qianyios ~]# vim /etc/docker/daemon.json
{
   ................
  "registry-mirrors": [],#无关紧要，不用看,
  "insecure-registries": [ "192.168.48.100" ],#重要加这行，别忘了如果他不是最后一行一定要在末尾加逗号
 ................
}

# 修改后，重启docker使其生效
systemctl daemon-reload
systemctl restart docker

# 利用docker info查看是否添加上
[root@qianyios ~]# docker info
Containers: 10
 Running: 1
 Paused: 0
 Stopped: 9
Images: 37
...
 Experimental: false
 Insecure Registries:
  192.168.48.100   ###要确保有这个才行
  127.0.0.0/8
 Registry Mirrors:

image-20240505170836267

配置免密登入

生成密钥,下面是我的账户和密码，自行替换

操作节点[harbor1]

1
2
3
4
echo -n 'admin:Harbor12345' | base64

[root@harbor1 ~]# echo -n 'admin:Harbor12345' | base64
YWRtaW46SGFyYm9yMTIzNDU=

我们需要创建一个配置文件，用于存储 Harbor 登录凭证。可以通过编辑 ~/.docker/config.json 文件来实现。如果该文件不存在，可以使用以下命令创建

1
touch ~/.docker/config.json

存在就直接编辑

1
vi ~/.docker/config.json
1
2
3
4
5
6
7
{
        "auths": {
                "192.168.48.100": {
                        "auth": "YWRtaW46SGFyYm9yMTIzNDU="
                }
        }
}

已经存在密钥文件了，记得替换auth的值。

我们需要配置 Docker 客户端以使用上一步中创建的配置文件中的凭证。首先，我们需要将配置文件复制到正确的位置。

1
2
cp ~/.docker/config.json /etc/docker/
systemctl restart docker

接着将这个文件复制到各个节点（这里自行按自己的集群提供）不做解释了

接下来测试免密登入

1
docker login 192.168.48.100

image-20240505172454889

免密登入成功！

部署jenkins

注意：以下仅实现部署了jenkins，尚未完成测试，后续有时间再进行测试，如果你有时间可以进行测试，还不确定哪里没有完善的地方

创建jenkins的cephfs

操作节点[master1]

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
cd
cd jenkins
cat >jenkins-cephfs.yaml << "EOF"
apiVersion: ceph.rook.io/v1
kind: CephFilesystem
metadata:
  name: jenkins-cephfs      #修改名字
  namespace: rook-ceph 
spec:
  metadataPool:
    replicated:
      size: 3
      requireSafeReplicaSize: true
    parameters:
      compression_mode:
        none
  dataPools:
    - name: replicated
      failureDomain: host
      replicated:
        size: 3
        requireSafeReplicaSize: true
      parameters:
        compression_mode:
          none
  preserveFilesystemOnDelete: false #当删除CephFilesystem资源时，是否保留Ceph集群中的实际文件系统。若设为true则保留，方便后续恢复使用。
  metadataServer:
    activeCount: 3
    activeStandby: true
    placement:
      podAntiAffinity:
        preferredDuringSchedulingIgnoredDuringExecution:
          - weight: 100
            podAffinityTerm:
              labelSelector:
                matchExpressions:
                  - key: app
                    operator: In
                    values:
                      - rook-ceph-mds
              topologyKey: topology.kubernetes.io/zone
    priorityClassName: system-cluster-critical
    livenessProbe:
      disabled: false
    startupProbe:
      disabled: false
EOF
kubectl create -f jenkins-cephfs.yaml
kubectl get pod -n rook-ceph | grep jenkins

可能要一分钟才会创建好

为什么我能直接在k8s直接运行ceph指令，而不进入pod，自行回去第二步看部署ceph集群的

image-20240501205306444

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
[root@master1 jenkins]# ceph fs status
jenkins-cephfs - 0 clients
==============
RANK      STATE             MDS            ACTIVITY     DNS    INOS   DIRS   CAPS
 0        active      jenkins-cephfs-f  Reqs:    0 /s    10     13     12      0
 1        active      jenkins-cephfs-e  Reqs:    0 /s    10     13     12      0
 2        active      jenkins-cephfs-d  Reqs:    0 /s     0      0      0      0
1-s   standby-replay  jenkins-cephfs-a  Evts:    0 /s     0      0      0      0
2-s   standby-replay  jenkins-cephfs-b  Evts:    0 /s     0      0      0      0
0-s   standby-replay  jenkins-cephfs-c  Evts:    0 /s     0      3      2      0
           POOL              TYPE     USED  AVAIL
 jenkins-cephfs-metadata   metadata   240k  94.9G
jenkins-cephfs-replicated    data       0   94.9G
MDS version: ceph version 17.2.6 (d7ff0d10654d2280e08f1ab989c7cdf3064446a5) quincy (stable)
[root@master1 jenkins]# ceph fs ls
name: jenkins-cephfs, metadata pool: jenkins-cephfs-metadata, data pools: [jenkins-cephfs-replicated ]

的情况下很有用，因为它确保只有在真正需要时才创建卷。

创建Storageclass

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cd 
cd jenkins
cat > jenkins-sc.yaml <<"EOF"
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: jenkins-sc
provisioner: rook-ceph.cephfs.csi.ceph.com
parameters:
  clusterID: rook-ceph #ceph集群命名空间
  fsName: jenkins-cephfs  #cephfs，刚刚创建的文件系统
  pool: jenkins-cephfs-replicated   #刚刚创建的fs所包含的pool

  csi.storage.k8s.io/provisioner-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/provisioner-secret-namespace: rook-ceph
  csi.storage.k8s.io/controller-expand-secret-name: rook-csi-cephfs-provisioner
  csi.storage.k8s.io/controller-expand-secret-namespace: rook-ceph
  csi.storage.k8s.io/node-stage-secret-name: rook-csi-cephfs-node
  csi.storage.k8s.io/node-stage-secret-namespace: rook-ceph
reclaimPolicy: Delete
allowVolumeExpansion: true
EOF
kubectl apply -f jenkins-sc.yaml
kubectl get sc

创建pvc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
#创建命名空间
kubectl create namespace jenkins
cd jenkins
cat > pvc.yaml << EOF
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: jenkins-pvc
  namespace: jenkins
spec:
  storageClassName: jenkins-sc
  resources:
    requests:
      storage: 3Gi
  accessModes:
  - ReadWriteOnce
EOF
kubectl apply -f pvc.yaml
kubectl get pvc -n jenkins

创建jenkins用户的secret

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
cd 
cd jenkins
cat >jenkins-admin-secret.yaml << "EOF"
apiVersion: v1
kind: Secret
metadata:
  name: jenkins-admin-secret
  namespace: jenkins
  annotations:
    kubernetes.io/service-account.name: jenkins-admin
type: kubernetes.io/service-account-token
EOF
kubectl apply -f jenkins-admin-secret.yaml
kubectl get secret -n jenkins

[root@master1 jenkins]# kubectl get secret -n jenkins
NAME                   TYPE                                  DATA   AGE
jenkins-admin-secret   kubernetes.io/service-account-token   3      15s

创建rabc验证

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
cd 
cd jenkins
cat > jenkins-rabc.yaml << "EOF"
apiVersion: v1
kind: ServiceAccount
metadata:
  name: jenkins-admin
  namespace: jenkins
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: jenkins-admin
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: jenkins-admin
  namespace: jenkins
EOF

kubectl apply -f jenkins-rabc.yaml

部署jenkins

自行在harbor仓库创建好cicd仓库

image-20240505231349308

构建jenkins镜像

1
docker pull jenkins/jenkins:lts-jdk17

可能会出现拉取缓慢现象，也可以用以下方法进行构建

基于阿里云容器服务构建私人docker镜像 - 严千屹博客 (qianyios.top)

构建之后就要进行拉取下载打标签，以下是我自己构建的，你可以用这个

1
2
3
4
5
6
7
8
9
10
11
12
13
14
#下载镜像
docker pull registry.cn-hangzhou.aliyuncs.com/qianyios/jenkins-lts:lts

#打标签
docker tag df4691d99e95 192.168.48.100/cicd/jenkins-lts:lts

#查看镜像
[root@master1 jenkins]# docker images | grep jenk
192.168.48.100/cicd/jenkins-lts                                        lts       df4691d99e95   2 weeks ago     469MB
registry.cn-hangzhou.aliyuncs.com/qianyios/jenkins-lts                 lts       df4691d99e95   2 weeks ago     469MB
[root@master1 jenkins]#

#推送镜像到harbor仓库
docker push 192.168.48.100/cicd/jenkins-lts:lts

image-20240505231712011

1
2
3
4
#给所有master节点打上master标签
kubectl label node master1 k8s-type=master
kubectl label node master2 k8s-type=master
kubectl label node master3 k8s-type=master
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
cd 
cd jenkins
cat >jenkins-deploy.yaml << "EOF"
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      securityContext:
            fsGroup: 995
            runAsUser: 1000
      serviceAccountName: jenkins-admin
      nodeSelector:
        k8s-type: master
      containers:
        - name: jenkins
          image: 192.168.48.100/cicd/jenkins-lts:lts #更换jenkins镜像地址#版本Jenkins 2.440.3
          imagePullPolicy: IfNotPresent
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
            - name: kubectl
              mountPath: /usr/bin/kubectl
            - name: kube-config
              mountPath: /root/.kube
            - name: docker
              mountPath: /usr/bin/docker
            - name: docker-sock
              mountPath: /var/run/docker.sock
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pvc
        - name: kubectl
          hostPath:
            path: /usr/bin/kubectl
        - name: kube-config
          hostPath:
            path: /root/.kube
        - name: docker
          hostPath:
            path: /usr/bin/docker
        - name: docker-sock
          hostPath:
            path: /var/run/docker.sock
EOF
kubectl apply -f jenkins-deploy.yaml 
kubectl get pods -n jenkins

创建svc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
cd 
cd jenkins
cat >jenkins-svc.yaml << "EOF"
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: jenkins
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
  - port: 8080
    targetPort: 8080
    nodePort: 32000
    name: httpport
  - name: jnlpport
    port: 50000
    targetPort: 50000
EOF
kubectl apply -f jenkins-svc.yaml
kubectl get svc -n jenkins

访问页面

1
http://192.168.48.200:32000/

image-20240505235348080

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
kubectl get pods -n jenkins
#查看你的pod名字替换下面的名字即可(jenkins-546cf958bd-kq5f2)

#进入容器
kubectl exec -it -n jenkins jenkins-546cf958bd-kq5f2 bash

#修改Update Center源
sed -i 's#https://updates.jenkins.io/update-center.json#http://mirrors.tuna.tsinghua.edu.cn/jenkins/updates/update-center.json#g' /var/jenkins_home/hudson.model.UpdateCenter.xml
 
#替换插件源地址： 
sed -i 's#https://updates.jenkins.io/download#http://mirrors.aliyun.com/jenkins#g' /var/jenkins_home/updates/default.json
#替换谷歌地址： 
sed -i 's#http://www.google.com#http://www.baidu.com#g' /var/jenkins_home/updates/default.json


#查看网页密码
jenkins@jenkins-546cf958bd-kq5f2:~$ cat /var/jenkins_home/secrets/initialAdminPassword
d6d07183e1e8453b8a16f507e235e5df

image-20240506000643463

一个一个搜去勾选，最后点击安装

1
安装插件：Git / Git Parameter/Pipeline/Config File Provider/Gitlab Authentication

image-20240506001148818

使用admin账户继续

image-20240506001948294

默认下一步

image-20240506002019436

自行去设置修改admin密码

image-20240506002313407

配置k8s插件和基本配置

安装插件

在首页点击系统管理然后点击插件管理—–安装Kubernetes插件

image-20240506002747172

安装之后需要等待他重启

image-20240506003015735

image-20240506003148720

配置k8s

获取k8s证书key

1
[root@master1 k8s]# cat /etc/kubernetes/pki/ca.crt

系统管理-clouds-创建cloud

image-20240506003540175

image-20240506011920607

image-20240506012135269

点击连接测试会出现版本号，表示连接成功

1
2
3
4
5
6
为什么连k8s不需要凭证：jenkins是在k8s内部搭建的，所以不需要k8s凭证，如果是在外部搭建的就需要添加k8s凭证

jenkins地址：　　
[root@master1 k8s]# kubectl get svc -n jenkins
NAME              TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
jenkins-service   NodePort   10.105.21.229   <none>        8080:32000/TCP,50000:32765/TCP   91m

image-20240506012244585

image-20240506012320549

部署gitlab

注意：以下仅实现部署了gitlab，尚未完成测试，后续有时间再进行测试，如果你有时间可以进行测试，还不确定哪里没有完善的地方

创建gitlab的secret

1
2
3
4
#长度为8个字符
[root@master1 ~]# echo -n 'qianyios' | base64
cWlhbnlpb3M=
#qianyios将会成为gitlab页面的登入密码
1
2
3
4
mkdir /root/gitlab
cd /root/gitlab
#创建namespace
kubectl create namespace gitlab
1
2
3
4
5
6
7
8
9
10
11
12
cat >gitlab-secret-pwd.yaml << "EOF"
apiVersion: v1
data:
  password: cWlhbnlpb3M=  #换成刚刚前面生成的密码
kind: Secret
metadata:
  creationTimestamp: null
  name: gitlab-pwd
  namespace: gitlab
EOF
kubectl apply -f gitlab-secret-pwd.yaml
kubectl get secret -n gitlab

创建pvc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
cat > gitlab-pvc.yaml <<"EOF"
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gitlab-pvc-logs
  namespace: gitlab
spec:
  storageClassName: jenkins-sc
  resources:
    requests:
      storage: 5Gi
  accessModes:
  - ReadWriteOnce
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gitlab-pvc-config
  namespace: gitlab
spec:
  storageClassName: jenkins-sc
  resources:
    requests:
      storage: 1Gi
  accessModes:
  - ReadWriteOnce
---
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: gitlab-pvc-data
  namespace: gitlab
spec:
  storageClassName: jenkins-sc
  resources:
    requests:
      storage: 10Gi
  accessModes:
  - ReadWriteOnce
EOF
  
kubectl apply -f gitlab-pvc.yaml 
kubectl get pvc -n gitlab

image-20240510195549565

部署gitlab

1
2
3
4
5
6
7
8
9
10
#下载gitlab-ce镜像，镜像很大要1个G多
 docker pull gitlab/gitlab-ce
# 如果下不了用我构建的镜像
docker pull registry.cn-hangzhou.aliyuncs.com/qianyios/gitlab:latest

#镜像打标签#这个里我用了自己构建的镜像，你要是用这个gitlab/gitlab-ce自行替换
docker tag registry.cn-hangzhou.aliyuncs.com/qianyios/gitlab:latest 192.168.48.100/cicd/gitlab:latest

#推送镜像
docker push 192.168.48.100/cicd/gitlab:latest
1
2
3
给node节点打上k8s-type=node标签
kubectl label nodes node01 k8s-type=node
kubectl label nodes node02 k8s-type=node
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
cat > gitlab-deploy.yaml << "EOF"
apiVersion: apps/v1
kind: Deployment
metadata:
  creationTimestamp: null
  labels:
    app: gitlab
  name: gitlab
  namespace: gitlab
spec:
  replicas: 1
  selector:
    matchLabels:
      app: gitlab
  strategy: {}
  template:
    metadata:
      creationTimestamp: null
      labels:
        app: gitlab
    spec:
      nodeSelector:
        k8s-type: node
      containers:
      - image: 192.168.48.100/cicd/gitlab:latest  #更换镜像
        imagePullPolicy: IfNotPresent
        name: gitlab-ce
        env:
        - name: GITLAB_ROOT_PASSWORD
          valueFrom:
            secretKeyRef:
              name: gitlab-pwd
              key: password
        - name: GITLAB_ROOT_MAIL
          value: xiaoohu2002@126.com
        ports:
        - name: gitlab80
          containerPort: 80
        - name: gitlab22
          containerPort: 22
        - name: gitlab443
          containerPort: 443
        volumeMounts:
        - name: gitlab-logs
          mountPath: /var/log/gitlab
        - name: gitlab-config
          mountPath: /etc/gitlab
        - name: gitlab-data
          mountPath: /var/opt/gitlab
      volumes:
      - name: gitlab-logs
        persistentVolumeClaim:
          claimName: gitlab-pvc-logs
      - name: gitlab-config
        persistentVolumeClaim:
          claimName: gitlab-pvc-config
      - name: gitlab-data
        persistentVolumeClaim:
          claimName: gitlab-pvc-data
status: {}

EOF
kubectl apply -f  gitlab-deploy.yaml
kubectl get pod -n gitlab

创建svc

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
cat > gitlab-svc.yaml << "EOF"
apiVersion: v1
kind: Service
metadata:
  creationTimestamp: null
  labels:
    app: gitlab
  name: gitlab
  namespace: gitlab
spec:
  ports:
  - name: 80-80
    port: 80
    protocol: TCP
    targetPort: 80
    nodePort: 30880
  - name: 443-443
    port: 443
    protocol: TCP
    targetPort: 443
  - name: 22-22
    port: 22
    protocol: TCP
    targetPort: 22
  selector:
    app: gitlab
  type: NodePort
status:
  loadBalancer: {}

EOF
kubectl apply -f  gitlab-svc.yaml
kubectl get svc -n gitlab

访问页面

1
http://192.168.48.200:30880/

image-20240510211753542

特别声明
千屹博客旗下的所有文章，是通过本人课堂学习和课外自学所精心整理的知识巨著
难免会有出错的地方
如果细心的你发现了小失误，可以在下方评论区告诉我，或者私信我！
非常感谢大家的热烈支持！
#Ceph #OpenEuler #Harbor #Jenkins #Gitlab
基于K8S的CICD系统实现
https://b.qianyios.top/2024/05/10/基于K8S的CICD系统实现/
作者
严千屹
发布于
2024年5月10日
更新于
2024年5月10日
许可协议