Enable management of Ceph tolerations and resources#

Warning

This document does not provide any specific recommendations on requests and limits for Ceph resources. The document stands for a native Ceph resources configuration.

You can configure Pelagia to manage Ceph resources by specifying their requirements and constraints. To configure the resource consumption for Ceph nodes, consider the following options that are based on different Helm release configuration values:

Configuring tolerations for taint nodes for the Ceph Monitor, Ceph Manager, and Ceph OSD daemons. For details, see Kubernetes documentation: Taints and Tolerations.
Configuring node resource requests or limits for the Ceph daemons and for each Ceph OSD device class such as HDD, SSD, or NVMe. For details, see Kubernetes documentation: Managing Resources for Containers.

Warning

Affinity placement settings specified in the CephDeployment spec for Ceph daemons will be ignored in favor of using node roles.

To enable management of Ceph tolerations and resources:

To avoid Ceph cluster health issues during daemon configuration changes, set Ceph noout, nobackfill, norebalance, and norecover flags through the pelagia-ceph-toolbox pod before editing Ceph tolerations and resources:
```
kubectl -n rook-ceph exec deploy/pelagia-ceph-toolbox -- bash
ceph osd set noout
ceph osd set nobackfill
ceph osd set norebalance
ceph osd set norecover
exit
```
Note

Skip this step if you are only configuring the PG rebalance timeout and replicas count parameters.
Open the CephDeployment custom resource (CR) for editing:
```
kubectl -n pelagia edit cephdpl
```

In the cluster.placement section, specify the placement parameters. For reference, see Rook API documentation: Placement Configuration Settings.

Example configuration

spec:
  cluster:
    placement:
      tolerations:
        mon:
          rules:
          - effect: NoSchedule
            key: node-role.kubernetes.io/controlplane
            operator: Exists
        mgr:
          rules:
          - effect: NoSchedule
            key: node-role.kubernetes.io/controlplane
            operator: Exists
        osd:
          rules:
          - effect: NoSchedule
            key: node-role.kubernetes.io/controlplane
            operator: Exists

In the cluster.resources section, specify the resource requirement parameters. For reference, see Rook API documentation: Resource Requirements/Limits.

Example configuration

spec:
  cluster:
    resources:
      mon:
        requests:
          memory: 1Gi
          cpu: 2
        limits:
          memory: 2Gi
          cpu: 3
      mgr:
        requests:
          memory: 1Gi
          cpu: 2
        limits:
          memory: 2Gi
          cpu: 3
      osd:
        requests:
          memory: 1Gi
          cpu: 2
        limits:
          memory: 2Gi
          cpu: 3
      osd-hdd:
        requests:
          memory: 1Gi
          cpu: 2
        limits:
          memory: 2Gi
          cpu: 3
      osd-ssd:
        requests:
          memory: 1Gi
          cpu: 2
        limits:
          memory: 2Gi
          cpu: 3
      osd-nvme:
        requests:
          memory: 1Gi
          cpu: 2
        limits:
          memory: 2Gi
          cpu: 3

In the spec.nodes section, specify the resources parameters for the Ceph node-specific resources:

spec:
  nodes:
  - name: <nodeName>
    resources:
      requests: <kubernetes valid spec of daemon resource requests>
      limits: <kubernetes valid spec of daemon resource limits>

Substitute <nodeName> with the node requested for specific resources. For example:

spec:
  nodes:
  - name: kaas-node-worker-1
    resources:
      requests:
        memory: 1Gi
        cpu: 2
      limits:
        memory: 2Gi
        cpu: 3

In the spec.objectStorage section, specify the resources and placement parameters for the RADOS Gateway instances:

spec:
  objectStorage:
    objectStore:
    - name: rgw-store
      spec:
        ...
        gateway:
          resources:
            requests: <kubernetes valid spec of daemon resource requests>
            limits: <kubernetes valid spec of daemon resource limits>
          placement:
            tolerations:
            - effect: NoSchedule
              key: node-role.kubernetes.io/control-plane
              operator: Exists
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
              operator: Exists

For reference, see Rook API documentation: Gateway Settings.

In the spec.sharedFilesystems section, specify the resources and placement parameters for the CephFS daemons:

spec:
  sharedFilesystem:
    cephFilesystems:
    - name: cephfs-store
      spec:
        ...
        metadataServer:
          placement:
            tolerations:
            - effect: NoSchedule
              key: node-role.kubernetes.io/control-plane
              operator: Exists
            - key: node-role.kubernetes.io/master
              effect: NoSchedule
              operator: Exists
          resources: # example, non-prod values
            requests:
              memory: 1Gi
              cpu: 1
            limits:
              memory: 2Gi
              cpu: 2

For reference, see Rook API documentation: CephFilesystem.

Save the reconfigured CephDeployment CR and wait for Pelagia Deployment Controller to apply the updated Ceph configuration to Rook. Rook will recreate Ceph Monitors, Ceph Managers, and Ceph OSDs according to the specified configuration.
Specify tolerations for different Rook resources using Pelagia Helm chart values. For details, see Specify Rook daemons placement.
After a successful Ceph reconfiguration, unset the flags set in step 1 through the pelagia-ceph-toolbox pod:
```
kubectl -n rook-ceph exec -it deploy/pelagia-ceph-toolbox -- bash
ceph osd unset
ceph osd unset noout
ceph osd unset nobackfill
ceph osd unset norebalance
ceph osd unset norecover
exit
```
Note

Skip this step if you have only configured the PG rebalance timeout and replicas count parameters.

Once done, proceed to Verify Ceph tolerations and resources.