[Kubernetes] NetApp NFS로 Trident 연동하기 (21.07.2 버전)

※ (2021.01.03) K8S 1.23 버전에서는 Trident install 불가합니다.

[root@master trident-installer]# ./tridentctl install -n trident
INFO Created Kubernetes clients.                   namespace=default version=v1.23.1
ERRO Kubernetes version 1.23.1 is an unsupported Kubernetes version; Trident supports Kubernetes versions in the range [1.17, 1.22]. NetApp will not take Support calls or open Support tickets when using Trident with an unsupported Kubernetes version.

 

알아야 할 것: PV, PVC, CSI 관련 문서

매뉴얼: https://netapp-trident.readthedocs.io/en/latest/kubernetes/deploying/tridentctl-deploy.html

(2024.04.15) trident 매뉴얼 문서 링크 변경됨 -> https://docs.netapp.com/us-en/trident/trident-use/ontap-nas-prep.html#requirements

 

(2023.10.17) tanzu에 설치하는 경우 vserver의 nfs에 mount-rootonly 옵션을 enable해 줘야 하는 경우도 있음

https://docs.netapp.com/us-en/netapp-solutions/containers/tanzu_with_netapp/vtwn_trident_ontap_nfs.html#create-an-svm-in-ontap

---

1. Download & extract the installer

* wget 없을 시 설치 (yum install wget)

[root@master ~]# wget https://github.com/NetApp/trident/releases/download/v21.07.2/trident-installer-21.07.2.tar.gz
[root@master ~]# tar -xf trident-installer-21.07.2.tar.gz
[root@master ~]# cd trident-installer

[root@master ~]# pwd
/root/trident-installer

2. Install Trident

[root@master ~]# ./tridentctl install -n trident
....
INFO Starting Trident installation.                namespace=trident
INFO Created service account.
INFO Created cluster role.
INFO Created cluster role binding.
INFO Added finalizers to custom resource definitions.
INFO Created Trident service.
INFO Created Trident secret.
INFO Created Trident deployment.
INFO Created Trident daemonset.
INFO Waiting for Trident pod to start.
INFO Trident pod started.                          namespace=trident pod=trident-csi-679648bd45-cv2mx
INFO Waiting for Trident REST interface.
INFO Trident REST interface is up.                 version=21.07.2
INFO Trident installation succeeded.
....

[root@master ~]# kubectl get pod -n trident
NAME                           READY   STATUS    RESTARTS   AGE
trident-csi-679648bd45-cv2mx   4/4     Running   0          5m29s
trident-csi-vgc8n              2/2     Running   0          5m29s

[root@master ~]# ./tridentctl -n trident version
+----------------+----------------+
| SERVER VERSION | CLIENT VERSION |
+----------------+----------------+
| 21.07.2        | 21.07.2        |
+----------------+----------------+

3. setup directory의 yaml 수정 (스토리지 파트 참조)

매뉴얼: https://netapp-trident.readthedocs.io/en/latest/kubernetes/operations/tasks/backends/ontap/index.html

Assigning aggregates to SVMs: 연동할 Netapp SVM에 -aggr-list 속성을 지정해 줘야 한다. (link)

 

backend.json은 연동할 NetApp Storage에 대한 정보를 기입한다. 해당 부분은 스토리지 엔지니어에게 문의하면 된다.

storagePrefix는 스토리지에 생성되는 볼륨의 Prefix로, 정한 이후 수정이 불가능하다.

[root@master ~]# cp sample-input/backends-samples/ontap-nas/backend-ontap-nas.json backend.json
[root@master ~]# vi backend.json
[root@master ~]# cat backend.json
{
    "debug": true,
    "version": 1,
    "storageDriverName": "ontap-nas",
    "backendName": "ontap-nas_git",
    "managementLIF": "10.100.30.43",
    "dataLIF": "10.100.31.105",
    "svm": "<svm_name>",
    "username": "<username>",
    "password": "<password>",
    "aggregate": "<aggr_name>",
    "storagePrefix": "<k8s_>"
}

[root@master ~]# ./tridentctl -n trident create backend -f backend.json
+---------------+----------------+--------------------------------------+--------+---------+
|     NAME      | STORAGE DRIVER |                 UUID                 | STATE  | VOLUMES |
+---------------+----------------+--------------------------------------+--------+---------+
| ontap-nas_git | ontap-nas      | ********-****-****-****-************ | online |       0 |
+---------------+----------------+--------------------------------------+--------+---------+

.[root@master ~]# ./tridentctl -n trident get backend
+---------------+----------------+--------------------------------------+--------+---------+
|     NAME      | STORAGE DRIVER |                 UUID                 | STATE  | VOLUMES |
+---------------+----------------+--------------------------------------+--------+---------+
| ontap-nas_git | ontap-nas      | ********-****-****-****-************ | online |       0 |
+---------------+----------------+--------------------------------------+--------+---------+

4. storage class 적용

[root@master ~]# cp sample-input/storage-class-samples/storage-class-csi.yaml.templ sample-input/storage-class-basic-csi.yaml
[root@master ~]# vi sample-input/storage-class-basic-csi.yaml
[root@master ~]# cat sample-input/storage-class-basic-csi.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: netapp-csi
  annotations:
    storageclass.kubernetes.io/is-default-class: "true"
provisioner: csi.trident.netapp.io
parameters:
  backendType: "ontap-nas"
  provisioningType: thin
  snapshots: "True"

[root@master ~]# kubectl create -f sample-input/storage-class-basic-csi.yaml
storageclass.storage.k8s.io/netapp-csi created

[root@master ~]# kubectl get sc netapp-csi
NAME                   PROVISIONER             RECLAIMPOLICY   VOLUMEBINDINGMODE   ALLOWVOLUMEEXPANSION   AGE
netapp-csi (default)   csi.trident.netapp.io   Delete          Immediate           false                  25s

[root@master ~]# ./tridentctl -n trident get storageclass netapp-csi -o json
{
  "items": [
    {
      "Config": {
        "version": "1",
        "name": "netapp-csi",
        "attributes": {
          "backendType": "ontap-nas",
          "provisioningType": "thin",
          "snapshots": "true"
        },
        "storagePools": null,
        "additionalStoragePools": null
      },
      "storage": {
        "ontap-nas_git": [
          "n2_aggr1"
        ]
      }
    }
  ]
}

* 설치 중 문제 발생시 trident 완전 삭제

https://netapp-trident.readthedocs.io/en/stable-v20.01/kubernetes/troubleshooting.html#troubleshooting

./tridentctl -n trident obliviate crd

---

* 테스트

각 node에 nfs-util이 설치되어 있어야 한다 (yum install -y nfs-utils)

설치 후 reboot

 

1. PVC 생성

[root@master ~]# cat sample-input/pvc-basic.yaml
kind: PersistentVolumeClaim
apiVersion: v1
metadata:
  name: basic
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
  storageClassName: netapp-csi

[root@master ~]# kubectl create -f sample-input/pvc-basic.yaml
persistentvolumeclaim/basic created

[root@master ~]# kubectl get pvc
NAME    STATUS   VOLUME                                     CAPACITY   ACCESS MODES   STORAGECLASS   AGE
basic   Bound    pvc-18b6c7af-a289-4c08-a82e-28289fdc477d   1Gi        RWO            netapp-csi     2s

2. pod 생성

[root@master ~]# vi task-pv-pod.yaml
[root@master ~]# cat task-pv-pod.yaml
kind: Pod
apiVersion: v1
metadata:
  name: task-pv-pod
spec:
  volumes:
    - name: task-pv-storage
      persistentVolumeClaim:
       claimName: basic
  containers:
    - name: task-pv-container
      image: nginx
      ports:
        - containerPort: 80
          name: "http-server"
      volumeMounts:
        - mountPath: "/usr/share/nginx/html"
          name: task-pv-storage

[root@master ~]# kubectl create -f task-pv-pod.yaml
pod/task-pv-pod created

[root@master ~]# kubectl get pod
NAME          READY   STATUS    RESTARTS   AGE
task-pv-pod   1/1     Running   0          9m13s

[root@master ~]# kubectl exec -it task-pv-pod -- df -h /usr/share/nginx/html
Filesystem                                                   Size  Used Avail Use% Mounted on
10.100.31.105:/k8s_pvc_18b6c7af_a289_4c08_a82e_28289fdc477d  1.0G  320K  1.0G   1% /usr/share/nginx/html

스토리지에서도 만들어진 PV 볼륨을 확인할 수 있다.

C190::> vol show
Vserver   Volume       Aggregate    State      Type       Size  Available Used%
--------- ------------ ------------ ---------- ---- ---------- ---------- -----
k8s       k8s_pvc_18b6c7af_a289_4c08_a82e_28289fdc477d
                       n2_aggr1     online     RW          1GB     1023MB    0%