Skip to content

Enable Multisite for Ceph Object Storage#

Technical Preview

This feature is in Technical Preview, use it on own risk.

The Ceph Object Storage Multisite feature allows object storage to replicate its data over multiple Ceph clusters. Using multisite, such object storage is independent and isolated from another object storage in the cluster. Only the multi-zone multisite setup is currently supported. For more details, see Ceph documentation: Multisite.

Multisite parameters #

  • realms - Required. List of realms to use, represents the realm namespaces. Includes the following parameters:

    • name - required, the realm name.

    • pullEndpoint - optional, required only when the master zone is in a different storage cluster. The endpoint, access key, and system key of the system user from the realm to pull from. Includes the following parameters:

      • endpoint - the endpoint of the master zone in the master zone group.
      • accessKey - the access key of the system user from the realm to pull from.
      • secretKey - the system key of the system user from the realm to pull from.

  • zoneGroups - Required. The list of zone groups for realms. Includes the following parameters:

    • name - required, the zone group name.
    • realmName - required, the realm namespace name to which the zone group belongs to.

  • zones - Required. The list of zones used within one zone group. Includes the following parameters:

    • name - required, the zone name.
    • metadataPool - required, the settings used to create the Object Storage metadata pools. Must use replication. For details, see description of Pool parameters.
    • dataPool - required, the settings used to create the Object Storage data pool. Can use replication or erasure coding. For details, see Pool parameters.
    • zoneGroupName - required, the zone group name.
    • endpointsForZone - optional. The list of all endpoints in the zone group. If you use ingress proxy for RGW, the list of endpoints must contain that FQDN/IP address to access RGW. By default, if no ingress proxy is used, the list of endpoints is set to the IP address of the RGW external service. Endpoints must follow the HTTP URL format.

Enable the multisite RGW Object Storage #

  1. Open the CephDeployment custom resource for editing: 
    kubectl -n pelagia edit cephdpl <name>

  2. Using the parameters from Multisite parameters, update the spec.objectStorage.multiSite section specification as required.

  3. Select from the following options:

    • If you do not need to replicate data from a different storage cluster, and the current cluster represents the master zone, modify the current objectStorage section to use the multisite mode:

      1. Configure the zone RADOS Gateway (RGW) parameter by setting it to the RGW Object Storage name.

        Note

        Leave dataPool and metadataPool empty. These parameters are ignored because the zone block in the multisite configuration specifies the pools parameters. Other RGW parameters do not require changes.

        For example: 

        spec:
 objectStorage:
   rgw:
     gateway:
       allNodes: false
       instances: 2
       port: 80
       securePort: 8443
     name: openstack-store
     preservePoolsOnDelete: false
     zone:
       name: openstack-store

      2. Create the multiSite section where the names of realm, zone group, and zone must match the current RGW name.

        Specify the endpointsForZone parameter according to your configuration:

        • If you use ingress proxy, which is defined in the spec.ingressConfig section, add the FQDN endpoint.
        • If you do not use any ingress proxy and access the RGW API using the default RGW external service, add the IP address of the external service or leave this parameter empty.

        The following example illustrates a complete objectStorage section: 

        objectStorage:
  multiSite:
    realms:
    - name: openstack-store
    zoneGroups:
    - name: openstack-store
      realmName: openstack-store
    zones:
    - name: openstack-store
      zoneGroupName: openstack-store
      endpointsForZone: http://10.11.0.75:8080
      metadataPool:
        failureDomain: host
          replicated:
            size: 3
      dataPool:
        erasureCoded:
          codingChunks: 1
          dataChunks: 2
        failureDomain: host
  rgw:
    gateway:
      allNodes: false
      instances: 2
      port: 80
      securePort: 8443
    name: openstack-store
    preservePoolsOnDelete: false
    zone:
      name: openstack-store

    • If you use a different storage cluster, and its object storage data must be replicated, specify the realm and zone group names along with the pullEndpoint parameter. Additionally, specify the endpoint, access key, and system keys of the system user of the realm from which you need to replicate data. For details, see step 2 of this procedure.

      Note

      All commands below should be executed inside pelagia-ceph-toolbox pod.

      • To obtain the endpoint of the cluster zone that must be replicated, run the following command by specifying the zone group name of the required master zone on the master zone side: 

        radosgw-admin zonegroup get --rgw-zonegroup=<ZONE_GROUP_NAME> | jq -r '.endpoints'
        The endpoint is located in the endpoints field.

      • To obtain the access key and the secret key of the system user, run the following command on the required Ceph cluster: 

        radosgw-admin user list

      • To obtain the system user name, which has your RGW ObjectStorage name as prefix: 
        radosgw-admin user info --uid="<USER_NAME>" | jq -r '.keys'

      For example: 

      spec:
  objectStorage:
    multiSite:
      realms:
      - name: openstack-store
        pullEndpoint:
          endpoint: http://10.11.0.75:8080
          accessKey: DRND5J2SVC9O6FQGEJJF
          secretKey: qpjIjY4lRFOWh5IAnbrgL5O6RTA1rigvmsqRGSJk
      zoneGroups:
      - name: openstack-store
        realmName: openstack-store
      zones:
      - name: openstack-store-backup
        zoneGroupName: openstack-store
        metadataPool:
          failureDomain: host
          replicated:
            size: 3
        dataPool:
          erasureCoded:
            codingChunks: 1
            dataChunks: 2
          failureDomain: host

      Note

      We recommend using the same metadataPool and dataPool settings as you use in the master zone.

  4. Configure the zone RGW parameter and leave dataPool and metadataPool empty. These parameters are ignored because the zone section in the multisite configuration specifies the pool parameters.

    Also, you can split the RGW daemon on daemons serving clients and daemons running synchronization. To enable this option, specify splitDaemonForMultisiteTrafficSync in the gateway section.

    For example: 

    spec:
  objectStorage:
    multiSite:
       realms:
       - name: openstack-store
         pullEndpoint:
           endpoint: http://10.11.0.75:8080
           accessKey: DRND5J2SVC9O6FQGEJJF
           secretKey: qpjIjY4lRFOWh5IAnbrgL5O6RTA1rigvmsqRGSJk
       zoneGroups:
       - name: openstack-store
         realmName: openstack-store
       zones:
       - name: openstack-store-backup
         zoneGroupName: openstack-store
         metadataPool:
           failureDomain: host
           replicated:
             size: 3
         dataPool:
           erasureCoded:
             codingChunks: 1
             dataChunks: 2
           failureDomain: host
    rgw:
      dataPool: {}
      gateway:
        allNodes: false
        instances: 2
        splitDaemonForMultisiteTrafficSync: true
        port: 80
        securePort: 8443
      healthCheck: {}
      metadataPool: {}
      name: openstack-store-backup
      preservePoolsOnDelete: false
      zone:
        name: openstack-store-backup

  5. Verify the multisite status: 

    radosgw-admin sync status

Once done, Pelagia Deployment Controller will create the required resources and Rook will handle the multisite configuration. For details, see: Rook documentation: Object Multisite.

Configure and clean up a multisite configuration#

Warning

Rook does not handle multisite configuration changes and cleanup. Therefore, once you enable multisite for Ceph RGW Object Storage, perform these operations manually in the pelagia-ceph-toolbox pod. For details, see Rook documentation: Multisite cleanup.

Automatic update of zonegroup hostnames is disabled in CephDeployment CR if RADOS Gateway Multisite is enabled or External Ceph cluster mode is enabled, therefore, manually specify all required hostnames and update the zone group. In the pelagia-ceph-toolbox pod, run the following script:

Note

The script is actual for Rook resources deployed by Pelagia Helm chart. If you're using Rook which is not deployed by Pelagia Helm chart, update zonegroup configuration manually.

/usr/local/bin/zonegroup_hostnames_update.sh --rgw-zonegroup <ZONEGROUP_NAME> --hostnames fqdn1[,fqdn2]

If the multisite setup is completely cleaned up, manually execute the following steps on the pelagia-ceph-toolbox pod:

  1. Due to the Rook issue #16328, verify that .rgw.root pool is removed:

    • Verify .rgw.root pool does not exist: 

      ceph osd pool ls | grep .rgw.root

    • If the pool .rgw.root is not removed, remove it manually: 

      ceph osd pool rm .rgw.root .rgw.root --yes-i-really-really-mean-it

    Some other RGW pools may also require a removal after cleanup.

  2. Remove the related RGW crush rules: 

    ceph osd crush rule ls | grep rgw | xargs -I% ceph osd crush rule rm %