[Kubernetes] Kubernetes Container runtimes 분석 (Production environment)

kubernetes.io/docs/setup/production-environment/container-runtimes/

Container runtimes

 

[Production Level Kubernetes 구성]

1. Container Runtime

=====================================================
###########
### Refer ###
###########

 

PS.1 Cgroup Drive 선택
리눅스 init 시스템이 systemd 인경우 init 프로세스는 root control group 을 생성 및 사용에 대해서 cgroup 관리자를 통해 작동한다. 
systemd 는 cgroup 와의 긴밀한 통합을 통해 Process 에 대한 cgroup 을 할당한다. 
 Container Runtime 및 kubelet 이 cgroupfs 를 사용하도록 설정할수 있으며, 이렇게 설정하는 이유는 systemd 및 cgroupfs 가 동시에 사용된다는것은 두개의 서로다른 cgroup 관리자가 존제한다는 뜻이다. 

 

kubelet 및 Docker 는 cgroupfs 를 사용하고 나머지 Process 가 systemd 에서 관리될 경우 Node 의 Resource 가 부족할 경우 시스템이 불안정해지는 사례가 있다. 

Container Runtime 및 Docker 의 cgroup 관리는 systemd 로 설정을 변경하여 운영
Docker 설정 중 "native.cgroupdriver=systemd" 옵션을 확인

 

tech.kakao.com/2020/06/29/cgroup-driver/

Cgroup Driver 선택하기

위의 블로그 내용을 확인해보면 오히려 반대되는 입장에서 이야기를 하고 있음
cgroupfs 와 systemd 관리에 대한부분은 단순 정책에 대한 부분이지, cgroup 안정성에 대한 이슈는 아니라는 내용도 존제한다.

 

하지만 docker 및 kubelet 은 동일한 cgroup managed tool 을 사용해야해야한다.
(cgroupfs or systemd)

 

PS2. Overlay Filesystem (추가 설명 작성중)

github.com/cri-o/cri-o/blob/master/install.md

 

!!! Overlay filesystem 에 대한 설명이 추가적으로 더 필요함 (Ing...)

 

PS.3 br_netfilter

ebtables.netfilter.org/documentation/bridge-nf.html

ebtables – Bridge-netfilter

docs.oracle.com/en/operating-systems/olcne/start/netfilter.html

2.3.5.3 br_netfilter Module

medium.com/finda-tech/kubernetes-%EB%84%A4%ED%8A%B8%EC%9B%8C%ED%81%AC-%EC%A0%95%EB%A6%AC-fccd4fd0ae6

쿠버네티스(Kubernetes) 네트워크 정리

ebtables.netfilter.org/misc/brnf-faq.html

Bridge-nf Frequently Asked Questions

net.bridge.bridge-nf-call-iptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

==> Bringe 된 IPV4 및 IPV6 Triffic 을 Iptables 의 Chains 으로 전달

 

PS.4 Kernal Parameter 필수 적용값

www.kernel.org/doc/Documentation/networking/ip-sysctl.txt

access.redhat.com/documentation/ko-kr/red_hat_enterprise_linux/5/html/virtual_server_administration/s1-lvs-forwarding-vsa

2.5. 패킷 포워딩 활성화 Red Hat Enterprise Linux 5 | Red Hat Customer Portal

www.joinc.co.kr/w/man/12/ipforwarding

Linux IP forwarding

net.ipv4.ip_forward = 1

==>  Interface 간의 Packets 을 전달, 0 = disabled, 0 이외의 값은 enabled
==> 패킷 포워딩, 혹은 Routing 으로 표현하기도 함

===================================================== 

 

===================================================== 

##########
###수행 ###
##########

 

1-1 Docker 설치
###Ubuntu 16.04 + ###

docs.docker.com/engine/install/ubuntu/

Install Docker Engine on Ubuntu

# (도커 CE 설치)
## 리포지터리 설정
### apt가 HTTPS 리포지터리를 사용할 수 있도록 해주는 패키지 설치
sudo apt-get update && sudo apt-get install -y \

  apt-transport-https ca-certificates curl software-properties-common gnupg2

 

# 도커 공식 GPG 키 추가

curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

 

# 도커 apt 리포지터리 추가.

sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \

  $(lsb_release -cs) \
  stable"

 

# 도커 CE 설치.

sudo apt-get update && sudo apt-get install -y \

  containerd.io=1.2.13-2 \

  docker-ce=5:19.03.11~3-0~ubuntu-$(lsb_release -cs) \
  docker-ce-cli=5:19.03.11~3-0~ubuntu-$(lsb_release -cs)

 

# 도커 데몬 설정

cat <<EOF | sudo tee /etc/docker/daemon.json

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2"
}
EOF

 

sudo mkdir -p /etc/systemd/system/docker.service.d

 

# 도커 재시작.
sudo systemctl daemon-reload
sudo systemctl restart docker

# 부팅 시 도커 서비스를 시작
sudo systemctl enable docker

 

###CentOS/RHEL 7.4+ ###

docs.docker.com/engine/install/centos/

Install Docker Engine on CentOS

# (도커 CE 설치)
## 리포지터리 설정
### 필요한 패키지 설치
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

 

## 도커 리포지터리 추가
sudo yum-config-manager --add-repo \

  https://download.docker.com/linux/centos/docker-ce.repo

 

# 도커 CE 설치.

sudo yum update -y && sudo yum install -y \

  containerd.io-1.2.13 \

  docker-ce-19.03.11 \
  docker-ce-cli-19.03.11

 

## /etc/docker 생성.
sudo mkdir /etc/docker

 

# 도커 데몬 설정.
cat <<EOF | sudo tee /etc/docker/daemon.json

{
  "exec-opts": ["native.cgroupdriver=systemd"],
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "100m"
  },
  "storage-driver": "overlay2",
  "storage-opts": [
    "overlay2.override_kernel_check=true"
  ]
}
EOF

 

sudo mkdir -p /etc/systemd/system/docker.service.d

 

# 도커 재시작.
sudo systemctl daemon-reload
sudo systemctl restart docker

 

# 부팅 시 도커 서비스를 시작
sudo systemctl enable docker

 

1-2 CRI-O

github.com/cri-o/cri-o

cri-o/cri-o

참고: CRI-O 메이저와 마이너 버전은 쿠버네티스 메이저와 마이너 버전이 일치해야 한다. 더 자세한 정보는 CRI-O 호환 매트릭스를 본다

 

# 선행 조건
sudo modprobe overlay
sudo modprobe br_netfilter

 

# 요구되는 sysctl 파라미터 설정, 이 설정은 재부팅 간에도 유지된다.
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf

net.bridge.bridge-nf-call-iptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.ipv4.ip_forward = 1

EOF

sudo sysctl --system

 

# Containerd 설치

###Ubuntu ###
다음의 운영 체제에서 CRI-O를 설치하려면, 환경 변수 $OS를 아래의 표에서 적절한 필드로 설정한다.

운영 체제 $OS
Ubuntu 20.04 xUbuntu_20.04
Ubuntu 19.10 xUbuntu_19.10
Ubuntu 19.04 xUbuntu_19.04
Ubuntu 18.04 xUbuntu_18.04

 

그런 다음, $VERSION 을 사용자의 쿠버네티스 버전과 일치하는 CRI-O 버전으로 설정한다. 예를 들어, CRI-O 1.18을 설치하려면, VERSION=1.18 로 설정한다. 사용자의 설치를 특정 릴리스에 고정할 수 있다. 버전 1.18.3을 설치하려면, VERSION=1.18:1.18.3 을 설정한다.

 

그런 다음, 아래를 실행한다.

 

echo "deb https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable.list

echo "deb http://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable:/cri-o:/$VERSION/$OS/ /" > /etc/apt/sources.list.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.list

 

curl -L https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/Release.key | apt-key add -

curl -L https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/Release.key | apt-key add -

apt-get update
apt-get install cri-o cri-o-runc

 

### CentOS ###
다음의 운영 체제에서 CRI-O를 설치하려면, 환경 변수 $OS를 아래의 표에서 적절한 필드로 설정한다.

운영 체제 $OS
Centos 8 CentOS_8
Centos 8 Stream CentOS_8_Stream
Centos 7 CentOS_7

그런 다음, $VERSION 을 사용자의 쿠버네티스 버전과 일치하는 CRI-O 버전으로 설정한다. 예를 들어, CRI-O 1.18을 설치하려면, VERSION=1.18 로 설정한다. 사용자의 설치를 특정 릴리스에 고정할 수 있다. 버전 1.18.3을 설치하려면, VERSION=1.18:1.18.3 을 설정한다.

그런 다음, 아래를 실행한다.

 

sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable.repo https://download.opensuse.org/repositories/devel:/kubic:/libcontainers:/stable/$OS/devel:kubic:libcontainers:stable.repo

sudo curl -L -o /etc/yum.repos.d/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo https://download.opensuse.org/repositories/devel:kubic:libcontainers:stable:cri-o:$VERSION/$OS/devel:kubic:libcontainers:stable:cri-o:$VERSION.repo 

sudo yum install cri-o

# CRI-O 시작
sudo systemctl daemon-reload
sudo systemctl start crio

 

1-3 Containerd 설치

 

# 선행 조건 
sudo modprobe overlay 
sudo modprobe br_netfilter 

 

# 요구되는 sysctl 파라미터 설정, 이 설정은 재부팅 간에도 유지된다. 
cat <<EOF | sudo tee /etc/sysctl.d/99-kubernetes-cri.conf

net.bridge.bridge-nf-call-iptables = 1

net.bridge.bridge-nf-call-ip6tables = 1

net.ipv4.ip_forward = 1

EOF

sudo sysctl --system

 

# Containerd 설치
###Ubuntu 16.04 ###

# (containerd 설치)
## 리포지터리 설정
### apt가 HTTPS로 리포지터리를 사용하는 것을 허용하기 위한 패키지 설치
sudo apt-get update && sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common

 

## 도커 공식 GPG 키 추가
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add -

 

## 도커 apt 리포지터리 추가.

sudo add-apt-repository \
    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \

    $(lsb_release -cs) \
    stable"

 

## containerd 설치
sudo apt-get update && sudo apt-get install -y containerd.io

 

# containerd 설정
sudo mkdir -p /etc/containerd
sudo containerd config default > /etc/containerd/config.toml

 

# containerd 재시작
sudo systemctl restart containerd

 

###CentOS/RHEL 7.4+ ###
# (containerd 설치)
## 리포지터리 설정
### 필요한 패키지 설치
sudo yum install -y yum-utils device-mapper-persistent-data lvm2

## 도커 리포지터리 추가
sudo yum-config-manager \
    --add-repo \
    https://download.docker.com/linux/centos/docker-ce.repo

 

## containerd 설치
sudo yum update -y && yum install -y containerd.io

 

## containerd 설정
sudo mkdir -p /etc/containerd
sudo containerd config default > /etc/containerd/config.toml

 

# containerd 재시작
sudo systemctl restart containerd

 

# systemd cgoup driver 를 사용하기 위한 방법
systemd cgroup driver를 사용하려면, /etc/containerd/config.toml에 다음을 설정한다.

 

[plugins.cri]

systemd_cgroup = true