Kubernetes Persistent Volume 구성 - nfs

Kubernetes 에서 Pod에 할당 할 Persistent Volume을 nfs 구성한다.

1. nfs 서버 구성

1-1. nfs 서버 설치

$ sudo apt-get update
$ sudo apt-get install nfs-common nfs-kernel-server portmap

1-2. nfs 폴더 설정

$ sudo mkdir /home/nfs-workernode01
$ sudo chmod 777 /home/nfs-workernode01

1-3. nfs 접근 설정

Node IP를 지정하여 접근할 수 있도록 설정

$ sudo vi /etc/exports
...
/home/nfs-workernode01 192.168.0.1,192.168.0.3(rw,all_squash,async)
...

수정시 아래와 같이 재기동

$ sudo /etc/init.d/nfs-kernel-server restart

2. Persistent Volume 구성

2-1. Persistent Volume 생성

Persistent Volume 생성을 위한 yaml 작성
(pv-nfs-workernode01.yaml)

apiVersion: v1
kind: PersistentVolume
metadata:
  name: pv-nfs-workernode01
spec:
  capacity:
    storage: 100Gi
  accessModes:
    - ReadWriteMany
  nfs:
    server: 192.168.0.2
    path: /home/nfs-workernode01
  storageClassName: "nfs-workernode01"

생성

$ kubectl create -f pv-nfs-workernode01.yaml

2-2. Persistent Volume Claim 생성

Persistent Volume Claim 생성을 위한 yaml 작성
(pvc-nfs-workernode01.yaml)

apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: pvc-nfs-workernode01
spec:
  storageClassName: "nfs-workernode01"
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Gi

생성

$ kubectl create -f pvc-nfs-workernode01.yaml

Kubernetes Dashboard 설치

Kubernetes Dashboard를 설치하고, API Server를 통해 외부에서 접속할 수 있도록 설정한다.

1. yaml 수정

1-1. Kubernetes Dashboard 설치 yaml 예시 다운로드

https://raw.githubusercontent.com/kubernetes/dashboard/v2.0.0-rc5/aio/deploy/recommended.yaml

1-2. ClusterRole/ClusterRoleBinding 추가

외부에서 Dashboard 접속을 위해 anonymous role을 추가한다.

...
kind: ClusterRole
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: kubernetes-dashboard-anonymous
rules:
- apiGroups: [""]
  resources: ["services/proxy"]
  resourceNames: ["https:kubernetes-dashboard:"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- nonResourceURLs: ["/ui", "/ui/*", "/api/v1/namespaces/kube-system/services/https:kubernetes-dashboard:/proxy/*"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: kubernetes-dashboard-anonymous
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: kubernetes-dashboard-anonymous
subjects:
- kind: User
  name: system:anonymous
...

1-3. nodeSelector 수정

Dashboard를 Master Node에만 올리기 위해 설치 yaml 상 nodeSelector를 수정한다.

...
nodeSelector:
        node-role.kubernetes.io/master: ""
...

2. Kubernetes Dashboard 설치

$ kubectl create -f kubernetes_dashboard.yaml

3. 계정/RoleBinding 생성

대시보드에 접속하기 위한 service account를 생성하고, 해당 계정에 cluster-admin 권한을 부여한다.
*보안상 운영환경에서는 수정 필요

$ cat <<EOF | kubectl create -f -
apiVersion: v1
kind: ServiceAccount
metadata:
  name: admin-user
  namespace: kube-system
EOF
$ cat <<EOF | kubectl create -f -
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: admin-user
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: cluster-admin
subjects:
- kind: ServiceAccount
  name: admin-user
  namespace: kube-system
EOF

4. 접속 확인

4-1. token 확인

$ kubectl -n kube-system describe secret $(kubectl -n kube-system get secret | grep admin-user | awk '{print $1}')
AAAAAAAAAAA

4-2. url 접속

https://192.168.0.1:6443/api/v1/namespaces/kubernetes-dashboard/services/https:kubernetes-dashboard:/proxy/#/login
*'192.168.0.1'에 Master Node의 IP를 기입
*4-1에서 확인한 token을 입력하여 로그인

Kubernetes 설치 및 설정

Master Node 1대, Worker Node 다수로 구성된 Kubernetes 환경을 구축 한다.

1. 구성 환경

*각 Worker Node 구성의 수행방법은 동일

2. Node 공통 설치

2-1. 필수 설치

$ sudo apt-get install \
apt-transport-https \
ca-certificates \
curl \
gnupg-agent \
software-properties-common

2-2. Docker 설치

$ curl -fsSL [https://download.docker.com/linux/ubuntu/gpg](https://download.docker.com/linux/ubuntu/gpg) | sudo apt-key add -
$ sudo add-apt-repository \
"deb \[arch=amd64\] [https://download.docker.com/linux/ubuntu](https://download.docker.com/linux/ubuntu) \
$(lsb\_release -cs) \
stable"
$ sudo apt-get update
$ sudo apt-get install docker-ce=18.06.1~ce~3-0~ubuntu
$ sudo usermod -aG docker $USER

Docker 상태 확인

$ docker ps
CONTAINER ID IMAGE  COMMAND  CREATED  STATUS PORTS  NAMES

2-3. Kubernetes 설치

$ curl -s [https://packages.cloud.google.com/apt/doc/apt-key.gpg](https://packages.cloud.google.com/apt/doc/apt-key.gpg) | sudo apt-key add
$ sudo add-apt-repository "deb [https://apt.kubernetes.io/](https://apt.kubernetes.io/) kubernetes-$(lsb\_release -cs) main"
$ sudo apt-get update
$ sudo apt-get install kubelet=1.14.1-00 kubeadm=1.14.1-00 kubectl=1.14.1-00 

3. Master Node 구성

$ kubeadm init
$ kubectl get nodes
NAME              STATUS   ROLES    AGE    VERSION
masternode01      Ready    master   6d5h   v1.14.1
$ kubeadm token create –ttl 0
$ kubeadm token list
TOKEN                     TTL         EXPIRES   USAGES                   DESCRIPTION   EXTRA GROUPS
AAAAAAAAAAAAA   <forever>   <never>   authentication,signing   <none>        system:bootstrappers:kubeadm:default-node-token
$ openssl x509 -pubkey -in /etc/kubernetes/pki/ca.crt | openssl rsa -pubin -outform der 2>/dev/null | openssl dgst -sha256 -hex | sed 's/^.* //'
ABCABCABCABCABC

Cluster join을 위해 token 이 필요하며, 최초 생성된 token은 만료기간이 있으므로 재생성 작업을 수행한다.

4. Worker Node 구성

$ sudo kubeadm join 192.168.0.1:6443 --token AAAAAAAAAAAAA \
--discovery-token-ca-cert-hash sha256:ABCABCABCABCABC

token과 hash는 바로 위에서 생성된 값을 사용한다.
(Master Node 수행)

$ kubectl get nodes
NAME              STATUS   ROLES    AGE    VERSION
masternode01      Ready    master   6d5h   v1.14.1
workernode01      Ready    worker   6d4h   v1.14.1
workernode02      Ready    worker   5d3h   v1.14.1
workernode03      Ready    worker   2d5h   v1.14.1

5. Toubleshooting

5-1. ERROR Swap

5-1-1. 증상

(Worker Node)

$ sudo kubeadm join 192.168.0.1:6443 --token AAAAAAAAAAAAA \
--discovery-token-ca-cert-hash sha256:ABCABCABCABCABC
[preflight] Running pre-flight checks
    [WARNING IsDockerSystemdCheck]: detected "cgroupfs" as the Docker cgroup driver. The recommended driver is "systemd". Please follow the guide at https://kubernetes.io/docs/setup/cri/
error execution phase preflight: [preflight] Some fatal errors occurred:
    [ERROR Swap]: running with swap on is not supported. Please disable swap
[preflight] If you know what you are doing, you can make a check non-fatal with `--ignore-preflight-errors=...`
$ free
total        used        free      shared  buff/cache   available
Mem:       16318240      564132    14961060      106056      793048    15306024
Swap:      16671740           0    16671740
$ cat /proc/swaps
Filename                Type        Size    Used    Priority
/dev/sda3               partition    16671740    0    -2

5-1-2. 조치

Kubernetes가 swap를 관리하므로, filesystem에 적용된 swap을 off 시킨다.

$ sudo swapoff -a
$ sudo sed -i '/ swap / s/^/#/' /etc/fstab
$ sudo systemctl mask dev-sda3.swap
$ sudo reboot

fstab을 별도로 관리하였으면 swap이 기술된 UUID를 주석처리 한다.
재기동 시에도 swap off를 적용하기 위해 swap unit을 mask하여 비활성화 한다. (dev-파티션명.swap)