resources: # We usually recommend not to specify default resources and to leave this as a conscious # choice for the user. This also increases chances charts run on environments with little # resources, such as Minikube. If you don't want to specify resources, comment the following # lines and add the curly braces after 'resources:'. limits: cpu: 200m memory: 500Mi requests: cpu: 100m memory: 20Mi
nodeSelector: {}
tolerations: []
affinity: {}
templates/role.yaml
values.yaml 에서 "watchNamespace" 나 "watchAllNamespaces" 값이 정의될경우 Role 은 Cluster Role 로 설치가되며, admissionregistration 의 validatingwebhookconfigurations 값이 추가가 된다.
{{- if or .Values.watchNamespace .Values.watchAllNamespaces }} kind: ClusterRole {{- else }} kind: Role {{- end }} apiVersion: rbac.authorization.k8s.io/v1 metadata: name: {{ include "pxc-operator.fullname" . }} labels: {{ include "pxc-operator.labels" . | indent 4 }} rules: - apiGroups: - pxc.percona.com resources: - perconaxtradbclusters - perconaxtradbclusters/status - perconaxtradbclusterbackups - perconaxtradbclusterbackups/status - perconaxtradbclusterrestores - perconaxtradbclusterrestores/status verbs: - get - list - watch - create - update - patch - delete {{- if or .Values.watchNamespace .Values.watchAllNamespaces }} - apiGroups: - admissionregistration.k8s.io resources: - validatingwebhookconfigurations verbs: - get - list - watch - create - update - patch - delete {{- end }} - apiGroups: - "" resources: - pods - pods/exec - pods/log - configmaps - services - persistentvolumeclaims - secrets verbs: - get - list - watch - create - update - patch - delete - apiGroups: - apps resources: - deployments - replicasets - statefulsets verbs: - get - list - watch - create - update - patch - delete - apiGroups: - batch resources: - jobs - cronjobs verbs: - get - list - watch - create - update - patch - delete - apiGroups: - policy resources: - poddisruptionbudgets verbs: - get - list - watch - create - update - patch - delete - apiGroups: - certmanager.k8s.io - cert-manager.io resources: - issuers - certificates verbs: - get - list - watch - create - update - patch - delete - deletecollection
templates/role-binding.yaml
### values.yaml 에서 "watchNamespace" 나 "watchAllNamespaces" 값이 정의될경우 rolebinding 이 ClusterRolebinding 로 바인딩되며, 없을경우 Rolebinding 로 바인딩된다.
### values.yaml 에서 "watchNamespace" 값이 정의될경우 metadata 에 namespace 값이 추가된다.
### values.yaml 에서 "watchNamespace" 나 "watchAllNamespaces" 값이 정의될경우 ServiceAccount annotation 의 Label 에 namespace: 값 추가
### values.yaml 에서 "watchNamespace" 나 "watchAllNamespaces" 값이 정의될경우roleRef 값이 ClusterRole 로 정의되며, 없을경우 Role 로 정의된다.
apiVersion: v1 kind: ServiceAccount metadata: name: {{ include "pxc-operator.fullname" . }} --- apiVersion: v1 kind: ServiceAccount metadata: name: percona-xtradb-cluster-operator --- {{- if or .Values.watchNamespace .Values.watchAllNamespaces }} kind: ClusterRoleBinding {{- else }} kind: RoleBinding {{- end }} apiVersion: rbac.authorization.k8s.io/v1 metadata: name: {{ include "pxc-operator.fullname" . }} {{- if .Values.watchNamespace }} namespace: {{ .Values.watchNamespace }} {{- end }} labels: {{ include "pxc-operator.labels" . | indent 4 }} subjects: - kind: ServiceAccount name: {{ include "pxc-operator.fullname" . }} {{- if or .Values.watchNamespace .Values.watchAllNamespaces }} namespace: {{ .Release.Namespace }} {{- end }} roleRef: {{- if or .Values.watchNamespace .Values.watchAllNamespaces }} kind: ClusterRole {{- else }} kind: Role {{- end }} name: {{ include "pxc-operator.fullname" . }} apiGroup: rbac.authorization.k8s.io
templates/namespace.yaml
### watchNamespace 값이 있을경우, 해당 namespace 가 없으면 namespace 생성한다.
{{ if .Values.watchNamespace }} apiVersion: v1 kind: Namespace metadata: name: {{ .Values.watchNamespace }} {{ end }}
templates/deployment.yaml
### replicaCount 값에 따라서 replicas 갯수 반영, Default 값이 정의가 안되어 있어서 필수로 정의해서 사용해야함
### imagePullSecrets 왼쪽에 8칸 공백을 넣고나서 imagePullSecrets 값 입력
### Values.imagePullPolicy 값이 정의되어 있을경우 정책 적용
### Values.watchAllNamespaces 값이 있을경우 ENV 의 WATCH_NAMESPACE 에 Values.watchAllNamespaces 값 정의, 없을시 default .Release.Namespace .Values.watchNamespace 으로 정의
### Values.nodeSelector 값이 있을경우 nodeSelector 정의
### Values.affinity 값 이 있을경우 affinity 정의
### Values.tolerations 값이 있을경우 tolerations 값 정의
### Values.watchAllNamespaces 가 정의가되면, Pod 의 9443 Port 를 Service 443 으로 노출시키는 Service 추가 생성
Kubernetes 의 MySQL Operator 의 대표적인 Operator 둘중 하나인 Percona 에서 개발한 Operator 이다.
System Requitrements
Officially Supported Platforms
Percona XtraDB Cluster (PXC) 5.7 및 8.0 을 지원함
8.0의 Default 값인 "caching_sha2_password" 인증 플러그인은 ProxySQL 호환성 이유로 지원하지 않음
PXC 5.7 및 8.0 모두 "Default_authentication_plugin = mysql_native_password" 를 사용
PXC Operator 1.10.0
Openshift : 4.7 - 4.9
GKE : 1.19 - 1.22
EKS : 1.17 - 1.21
Minikube(kubeadm base) : 1.22
위의 k8s platform 외 다른 platform 을 테스트가 되지 않았음
Resource Limits
officially platform 에서 3개의 node 가 필요함
2GB RAM
2 CPU threads per Node for Pods provisioning
영구 볼륨을 위한 60GB 공간의 Storage
Design Overview
Percona XtraDB Cluster 는 XtraDB Storage Engine 과 함께 실행되는 Pecona Server for MySQL 과 Pecona XtraBackup 을 Galera library 와 통합하여서 Multi-primary Sync 를 가능하게함
권장 구성은 최소 3개 노드를 갖는것, Percona XtraDB Cluster 는 고가용성을 제공한다.
노드 하나가 중지되어도 계속 작동을 한다.
ProxySQL 을 통해서 Client 에서 Server 로 Loadbalancer 를 제공함
ProxySQL 를 사용하지 않아도 가능하지만, ProxySQL 이 CSP 의 내장 LB 나 Nginx Ingress Controller 를 포함한 SQL 을 인지하지 못하는 다른 LB 와 비교하여서 더 효율적인 Database Workload 관리가 가능하다.
고가용성 (High Availability) 을 제공하기 위하여서 node affinity 를 사용한다. Node 가 Fails 되면 다른 Node 에 Pod 이 re-createed 된다.
Stateful Application 을 제공하기 위해서 Kubernetes 는 영구볼륨을 제공하기 위해서 (PersistentVolumeClaim[PVC]) 를 사용하며, PVC 는 Pod 에 대해 Automatic Storage Provisioning 을 제공한다.
장애가 발생하면, CSI 가 다른 노드에 Storage 를 다시 탑재할수 있어야함 PVC StorageClass 는 이기능을 지원해야함 (Kubernetes 1.9 이상, Openshift 3.9 이상)
Operator 기능은 PerconaXtraDBCluster Object로 Kubernetes API 를 확장하며, Golang Application 으로 구현을 함
PerconaXtraDBCluster Object 는 하나의 개별 Percona XtraDB Cluster 설정에 매핑됨(CRD, CR)
$ kubectl apply -f https://raw.githubusercontent.com/percona/percona-xtradb-cluster-operator/v1.10.0/deploy/bundle.yaml customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com created customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com created customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com created customresourcedefinition.apiextensions.k8s.io/perconaxtradbbackups.pxc.percona.com created role.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created deployment.apps/percona-xtradb-cluster-operator created
$ kubectl get all -l app.kubernetes.io/name=percona-xtradb-cluster-operator NAME READY STATUS RESTARTS AGE pod/percona-xtradb-cluster-operator-5ffc5b4fc5-cdfs5 1/1 Running 0 2m29s
NAME DESIRED CURRENT READY AGE replicaset.apps/percona-xtradb-cluster-operator-5ffc5b4fc5 1 1 1 2m29s
$ kubectl get perconaxtradbcluster.pxc.percona.com/minimal-cluster NAME ENDPOINT STATUS PXC PROXYSQL HAPROXY AGE minimal-cluster minimal-cluster-haproxy.default initializing 54s
### Cluster 구성 완료
$ kubectl get perconaxtradbcluster.pxc.percona.com/minimal-cluster NAME ENDPOINT STATUS PXC PROXYSQL HAPROXY AGE minimal-cluster minimal-cluster-haproxy.default ready 1 1 2m37s
### 특정 Label 로 생성된 리소스 확인
$ kubectl get all,pvc,pv -l app.kubernetes.io/managed-by=percona-xtradb-cluster-operator NAME READY STATUS RESTARTS AGE pod/minimal-cluster-haproxy-0 2/2 Running 0 4m43s pod/minimal-cluster-pxc-0 3/3 Running 0 4m43s
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/minimal-cluster-haproxy ClusterIP 10.200.1.213 <none> 3306/TCP,3309/TCP,33062/TCP,33060/TCP 4m43s service/minimal-cluster-haproxy-replicas ClusterIP 10.200.1.32 <none> 3306/TCP 4m43s service/minimal-cluster-pxc ClusterIP None <none> 3306/TCP,33062/TCP,33060/TCP 4m43s service/minimal-cluster-pxc-unready ClusterIP None <none> 3306/TCP,33062/TCP,33060/TCP 4m43s
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS AGE persistentvolumeclaim/datadir-minimal-cluster-pxc-0 Bound pvc-12224952-03d9-4843-ad08-1c3967683dad 6G RWO nfs-client 4m43s
If you don't see a command prompt, try pressing enter. [mysql@percona-client /]$
### MySQL 접속 및 테스트 Query 수행
$ mysql -h minimal-cluster-haproxy -uroot -p9SdFVzLQpDAsAUPfWZ mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 814 Server version: 8.0.23-14.1 Percona XtraDB Cluster (GPL), Release rel14, Revision d3b9a1d, WSREP version 26.4.3
Copyright (c) 2009-2022 Percona LLC and/or its affiliates Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec)
You are in 'detached HEAD' state. You can look around, make experimental changes and commit them, and you can discard any commits you make in this state without impacting any branches by switching back to a branch.
If you want to create a new branch to retain commits you create, you may do so (now or later) by using -c with the switch command. Example:
git switch -c <new-branch-name>
Or undo this operation with:
git switch -
Turn off this advice by setting config variable advice.detachedHead to false
$ ls minimal percona-xtradb-cluster-operator $ cd percona-xtradb-cluster-operator/
### Percona MySQL Operator CRD 배포
$ kubectl apply -f deploy/crd.yaml customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusters.pxc.percona.com created customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterbackups.pxc.percona.com created customresourcedefinition.apiextensions.k8s.io/perconaxtradbclusterrestores.pxc.percona.com created customresourcedefinition.apiextensions.k8s.io/perconaxtradbbackups.pxc.percona.com created
### pxc namespace 생성
$ kubectl create namespace pxc namespace/pxc created
PS. 공식문서에서는 pxc namespace 를 default namespace 로 할당 후 작업을 진행하라고 되어있으나, context namespace 정의를 하지않고, 설치 시 namespace 를 지정하는것을 추가해서 작업진행하겠음
$ kubectl apply -f deploy/rbac.yaml role.rbac.authorization.k8s.io/percona-xtradb-cluster-operator created serviceaccount/percona-xtradb-cluster-operator created rolebinding.rbac.authorization.k8s.io/service-account-percona-xtradb-cluster-operator created
PS. RBAC 설정을 하려면, kubectl 사용자가 Cluster admin role 이 있어야 함
$ kubectl apply -f deploy/operator.yaml -n pxc deployment.apps/percona-xtradb-cluster-operator created
$ kubectl get pod -A -l app.kubernetes.io/name=percona-xtradb-cluster-operator -n pxc NAMESPACE NAME READY STATUS RESTARTS AGE pxc percona-xtradb-cluster-operator-5ffc5b4fc5-6clmq 1/1 Running 0 2m2s
If you don't see a command prompt, try pressing enter. [mysql@percona-client /]$ mysql -h cluster1-haproxy.pxc -uroot -proot_password
mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 1493 Server version: 8.0.25-15.1 Percona XtraDB Cluster (GPL), Release rel15, Revision 8638bb0, WSREP version 26.4.3
Copyright (c) 2009-2022 Percona LLC and/or its affiliates Copyright (c) 2000, 2022, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.01 sec)
$ helm repo add percona https://percona.github.io/percona-helm-charts/ $ helm repo update "percona" has been added to your repositories Hang tight while we grab the latest from your chart repositories... ...Successfully got an update from the "percona" chart repository ...Successfully got an update from the "nfs-subdir-external-provisioner" chart repository Update Complete. ⎈Happy Helming!⎈
$ helm install my-op percona/pxc-operator NAME: my-op LAST DEPLOYED: Thu Jun 2 11:11:08 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: 1. pxc-operator deployed. If you would like to deploy an pxc-cluster set cluster.enabled to true in values.yaml Check the pxc-operator logs export POD=$(kubectl get pods -l app.kubernetes.io/name=pxc-operator --namespace default --output name) kubectl logs $POD --namespace=default
$ export POD=$(kubectl get pods -l app.kubernetes.io/name=pxc-operator --namespace default --output name) $ kubectl logs $POD --namespace=default {"level":"info","ts":1654135870.9073238,"logger":"cmd","msg":"Runs on","platform":"kubernetes","version":"v1.23.6"} {"level":"info","ts":1654135870.9074643,"logger":"cmd","msg":"Git commit: 038082365e4e94cfdda40a20ce1b53fc098e5efb Git branch: release-1-10-0 Build time: 2021-11-17T16:46:03Z"} {"level":"info","ts":1654135870.9074957,"logger":"cmd","msg":"Go Version: go1.17.3"} {"level":"info","ts":1654135870.907549,"logger":"cmd","msg":"Go OS/Arch: linux/amd64"} {"level":"info","ts":1654135870.907603,"logger":"cmd","msg":"operator-sdk Version: v0.19.4"} {"level":"info","ts":1654135870.9077961,"logger":"leader","msg":"Trying to become the leader."} {"level":"info","ts":1654135871.752309,"logger":"leader","msg":"No pre-existing lock was found."} {"level":"info","ts":1654135871.7690983,"logger":"leader","msg":"Became the leader."} {"level":"info","ts":1654135872.5740485,"logger":"controller-runtime.metrics","msg":"metrics server is starting to listen","addr":":8080"} {"level":"info","ts":1654135872.5745153,"logger":"cmd","msg":"Registering Components."} {"level":"info","ts":1654135875.3557,"logger":"controller-runtime.webhook","msg":"registering webhook","path":"/validate-percona-xtradbcluster"} {"level":"info","ts":1654135875.3557413,"logger":"cmd","msg":"Starting the Cmd."} {"level":"info","ts":1654135875.3569176,"logger":"controller-runtime.manager","msg":"starting metrics server","path":"/metrics"} {"level":"info","ts":1654135875.3571787,"logger":"controller-runtime.webhook.webhooks","msg":"starting webhook server"} {"level":"info","ts":1654135875.358168,"logger":"controller-runtime.certwatcher","msg":"Updated current TLS certificate"} {"level":"info","ts":1654135875.3582811,"logger":"controller-runtime.webhook","msg":"serving webhook server","host":"","port":9443} {"level":"info","ts":1654135875.357379,"logger":"controller-runtime.manager.controller.perconaxtradbclusterbackup-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1654135875.3574393,"logger":"controller-runtime.manager.controller.perconaxtradbcluster-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1654135875.3602195,"logger":"controller-runtime.certwatcher","msg":"Starting certificate watcher"} {"level":"info","ts":1654135875.357489,"logger":"controller-runtime.manager.controller.perconaxtradbclusterrestore-controller","msg":"Starting EventSource","source":"kind source: /, Kind="} {"level":"info","ts":1654135875.4586177,"logger":"controller-runtime.manager.controller.perconaxtradbclusterbackup-controller","msg":"Starting Controller"} {"level":"info","ts":1654135875.4587562,"logger":"controller-runtime.manager.controller.perconaxtradbcluster-controller","msg":"Starting Controller"} {"level":"info","ts":1654135875.4605482,"logger":"controller-runtime.manager.controller.perconaxtradbclusterrestore-controller","msg":"Starting Controller"} {"level":"info","ts":1654135875.5589018,"logger":"controller-runtime.manager.controller.perconaxtradbclusterbackup-controller","msg":"Starting workers","worker count":1} {"level":"info","ts":1654135875.558974,"logger":"controller-runtime.manager.controller.perconaxtradbcluster-controller","msg":"Starting workers","worker count":1} {"level":"info","ts":1654135875.5611134,"logger":"controller-runtime.manager.controller.perconaxtradbclusterrestore-controller","msg":"Starting workers","worker count":1}
$ kubectl get pod NAME READY STATUS RESTARTS AGE my-op-pxc-operator-7578776fd6-wsnx9 1/1 Running 0 97s
### 설치
PS. 공식문서에 나온 설치는 가장 기본 옵션으로설치가 되는 부분이라 Helm 에서 설치가능한 옵션값에 대한 분석을 추가 로 진행하겠음
### 아래의 둘중 하나의 명령으로 설치 진행, 개인적으로 helm upgrade 으로 --install 파라미터를 추가해서 설치하는걸 즐겨한다.
Release "my-db" does not exist. Installing it now. NAME: my-db LAST DEPLOYED: Thu Jun 2 11:17:26 2022 NAMESPACE: default STATUS: deployed REVISION: 1 TEST SUITE: None NOTES: 1. To get a MySQL prompt inside your new cluster you can run:
mysql: [Warning] Using a password on the command line interface can be insecure. Welcome to the MySQL monitor. Commands end with ; or \g. Your MySQL connection id is 8295 Server version: 8.0.25-15.1 Percona XtraDB Cluster (GPL), Release rel15, Revision 8638bb0, WSREP version 26.4.3
Copyright (c) 2009-2021 Percona LLC and/or its affiliates Copyright (c) 2000, 2021, Oracle and/or its affiliates.
Oracle is a registered trademark of Oracle Corporation and/or its affiliates. Other names may be trademarks of their respective owners.
Type 'help;' or '\h' for help. Type '\c' to clear the current input statement.
mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.00 sec)
mysql> exit
### pxc haproxy service 를 통한 pxc cluster 접근 확인
$ kubectl run -i --tty --rm percona-client --image=percona --restart=Never -- mysql -h my-db-pxc-db-haproxy.default.svc.cluster.local -uroot -p"$ROOT_PASSWORD" If you don't see a command prompt, try pressing enter. mysql> show databases; +--------------------+ | Database | +--------------------+ | information_schema | | mysql | | performance_schema | | sys | +--------------------+ 4 rows in set (0.01 sec)
