Configure Ceph Shared File System (CephFS)#

The Ceph Shared File System, or CephFS, provides the ability to create read/write shared file system Persistent Volumes (PVs). These PVs support the ReadWriteMany access mode for the FileSystem volume mode. CephFS deploys its own daemons called MetaData Servers or Ceph MDS. For details, see Ceph Documentation: Ceph File System.

Note

By design, the default CephFS data and metadata pools must be replicated only.

CephFS specification parameters#

The CephDeployment custom resource (CR) spec includes the sharedFilesystem.cephFilesystems section with the following CephFS parameters:

name - CephFS instance name.
spec - CephFS spec definition based on the Rook API CephFS specification.

For configuration details, refer to the CephDeployment API.

Example of the CephFS specification:

spec:
  sharedFilesystem:
    cephFilesystems:
    - name: cephfs-store
      spec:
        dataPools:
        - name: default-pool
          deviceClass: ssd
          replicated:
            size: 3
          failureDomain: host
        - name: second-pool
          deviceClass: hdd
          failureDomain: rack
          erasureCoded:
            dataChunks: 2
            codingChunks: 1
        metadataPool:
          deviceClass: nvme
          replicated:
            size: 3
          failureDomain: host
        metadataServer:
          activeCount: 1
          activeStandby: false
          resources: # example, non-prod values
            requests:
              memory: 1Gi
              cpu: 1
            limits:
              memory: 2Gi
              cpu: 2

Warning

When using the non-recommended Ceph pools replicated.size of less than 3, Ceph OSD removal cannot be performed. The minimum replica size equals a rounded up half of the specified replicated.size.

For example, if replicated.size is 2, the minimal replica size is 1, and if replicated.size is 3, then the minimal replica size is 2. The replica size of 1 allows Ceph having PGs with only one Ceph OSD in the acting state, which may cause a PG_TOO_DEGRADED health warning that blocks Ceph OSD removal. Mirantis recommends setting replicated.size to 3 for each Ceph pool.

Warning

Modifying dataPools or metadataPool on a deployed CephFS has no effect. You can manually adjust pool settings through the Ceph CLI. However, for any changes in dataPools or metadataPool, we recommend re-creating CephFS.

Configure CephFS#

Optional. Override the CSI CephFS gRPC and liveness metrics port. For example, if an application is already using the default CephFS ports 9092 and 9082, which may cause conflicts on the node. Upgrade Pelagia Helm release values with desired port numbers:
```
helm upgrade --install pelagia-ceph oci://registry.mirantis.com/pelagia/pelagia-ceph --version 1.0.0 -n pelagia \
     --set rookConfig.csiCephFsGPCMetricsPort=<desiredPort>,rookConfig.csiCephFsLivenessMetricsPort=<desiredPort>
```
Rook will enable the CephFS CSI plugin and provisioner.
Open the CephDeployment CR for editing:
```
kubectl -n pelagia edit cephdpl
```

Update the sharedFilesystem section specification as required using the configuration reference above. For example:

spec:
  sharedFilesystem:
    cephFilesystems:
    - name: cephfs-store
      spec:
        dataPools:
        - name: cephfs-pool-1
          deviceClass: hdd
          replicated:
            size: 3
          failureDomain: host
        metadataPool:
          deviceClass: nvme
          replicated:
            size: 3
          failureDomain: host
        metadataServer:
          activeCount: 1
          activeStandby: false

Define the mds role for the corresponding nodes where Ceph MDS daemons should be deployed. We recommend labeling only one node with the mds role. For example:
```
spec:
  nodes:
    ...
    worker-1:
      roles:
      ...
      - mds
```

Once CephFS is specified in the CephDeployment CR, Pelagia Deployment Controller will validate it and request Rook to create CephFS. Then Pelagia Deployment Controller will create a Kubernetes StorageClass, required to start provisioning the storage, which will operate the CephFS CSI driver to create Kubernetes PVs.