rgwadmin¶

rgwadmin is a Python library to access the Ceph Object Storage Admin API.

http://docs.ceph.com/docs/master/radosgw/adminops/

Contents:¶

Installation¶

Via Python Package Index¶

Install the latest version from PyPI:

pip install rgwadmin

Via Source¶

The project is hosted at https://github.com/UMIACS/rgwadmin and can be installed using git:

git clone https://github.com/umiacs/rgwadmin.git
cd rgwadmin
python setup.py install

User Guide¶

Connecting¶

To get started, you’ll want to connect to your Ceph instance by instantiating a RGWAdmin object.

from rgwadmin import RGWAdmin

rgw = RGWAdmin(
    access_key='MY_ACCESS_KEY',
    secret_key='MY_SECRET_KEY',
    server='ceph.example.com')

Note

rgwadmin connects over https by default. To connect over http, pass secure=False, verify=False

You can also use get_environment_creds() to pull in environment variables to form the connection.

from rgwadmin import RGWAdmin
from rgwadmin.utils import get_environment_creds

rgw = RGWAdmin(**get_environment_creds())

The environment variables that it is looking for are OBJ_ACCESS_KEY_ID, OBJ_SECRET_ACCESS_KEY, and OBJ_SERVER.

For a full list of options that can be passed when establishing a connection, see RGWAdmin().

Connection Pooling¶

The pool_connections option can be passed to enable a tcp connection to be reused on every request. There are performance improvements to be had by enabling connection persistence. By default this is disabled and a new connection is formed every time.

The downside to enabling connection pooling is that there is no built-in connection resumption capability in the library. If you enable connection pooling you will need to handle connection resets yourself. Check your frontend loadbalancer to see what sort of keepalive settings your rgw server is negotiating.

User Management¶

Creating Users¶

Let’s create a new user for ourselves with create_user(). This will generate an access/secret keypair for the new user by default.

rgw.create_user(
    uid='liam',
    display_name='Liam Monahan',
    email='liam@umiacs.umd.edu',
    user_caps='usage=read, write; users=read',
    max_buckets=1000)

Get User Details¶

We can fetch a user back by calling get_user().

>>> rgw.get_user('liam', stats=True)
{   'admin': 'false',
 'bucket_quota': {   'check_on_raw': False,
                     'enabled': False,
                     'max_objects': -1,
                     'max_size': -1,
                     'max_size_kb': 0},
 'caps': [],
 'default_placement': '',
 'default_storage_class': '',
 'display_name': 'Liam Monahan',
 'email': 'liam@umiacs.umd.edu',
 'keys': [   {   'access_key': '91C3KDI66JG9ILSJRU5S',
                 'secret_key': '********************',
                 'user': 'liam'}],
 'max_buckets': 1000,
 'mfa_ids': [],
 'op_mask': 'read, write, delete',
 'placement_tags': [],
 'stats': {   'num_objects': 6,
              'size': 1163924507,
              'size_actual': 1163931648,
              'size_kb': 1136646,
              'size_kb_actual': 1136652,
              'size_kb_utilized': 0,
              'size_utilized': 0},
 'subusers': [],
 'suspended': 0,
 'swift_keys': [],
 'system': 'false',
 'temp_url_keys': [],
 'tenant': '',
 'type': 'rgw',
 'user_id': 'liam',
 'user_quota': {   'check_on_raw': False,
                   'enabled': True,
                   'max_objects': -1,
                   'max_size': 1168400384,
                   'max_size_kb': 1141016}}

Modifying Users¶

Most attributes about a user can be modified later.

For example, let’s suspend our example user:

rgw.modify_user(uid='liam', suspended=True)

All of the modifications are idempotent. See modify_user() for a list of kwargs that the function accepts.

Listing Users¶

We can get a list of all users with get_users().

>>> rgw.get_users()
['liam', 'bob', 'alice']

Removing Users¶

Delete a user from rgw. The operation will fail if the user owns data and purge_data is not set to True.

rgw.remove_user(uid='liam', purge_data=True)

Add A User Capability¶

Add an administrative capability to a specified user.

>>> rgw.add_capability(
    uid='liam',
    user_caps='usage=read,write;user=write'
)
[{'type': 'usage', 'perm': '*'}]

This will return a dict of the user’s full set of capabilities.

Remove A User Capability¶

Remove an administrative capability from a specified user.

>>> rgw.remove_capability(
    uid='liam',
    user_caps='usage=read,write;user=write'
)
[{'type': 'usage', 'perm': '*'}]

Similarly to adding caps, this call will also return the user’s new set of capabilities as they are after the operation has completed.

Bandwidth Usage¶

Request bandwidth usage information.

Note

This feature is disabled by default. It can be enabled by setting rgw enable usage log = true in the appropriate section of ceph.conf.

Trim Usage¶

Remove usage information. With no dates specified, removes all usage information. More info here: trim_usage().

rgw.trim_usage(uid='liam')

Subusers¶

Subusers can be managed using key types of either S3 or Swift.

Creating Subusers¶

Let’s say that we have a webadmin user and we want to create a subuser for liam to have FULL_CONTROL.

rgw.create_subuser(
    uid='webadmin',
    subuser='liam',
    access='full',
    key_type='s3',
    generate_secret=True,
)

See the full documentation for create_subuser().

Modifying Subusers¶

The level of access granted to a subuser can be changed after they are created with modify_subuser(). The secret key can be regerated here, too.

rgw.modify_subuser(
    uid='webadmin',
    subuser='liam',
    access='read',
)

Removing Subusers¶

We can remove a subuser by calling remove_subuser().

Note

By default this will purge keys. This is usually what you want.

rgw.remove_subuser(
    uid='webadmin',
    subuser='liam',
)

Keys¶

Create Key¶

To create a key on a user:

rgw.create_key(
    uid='liam',
    key_type='s3',
    generate_key=True)

If you are creating a key for a subuser, you will need to pass in the subuser.

rgw.create_key(
    uid='webadmin',
    subuser='liam',
    key_type='s3',
    generate_key=True)

Note

The function gen_secret_key() can help generate a random string of characters that can be used as a SECRET_KEY.

Remove Key¶

To remove a key using remove_key() it is required to provide the access-key. If you are removing a swift key, it is also necessary to pass the key_type='swift'

rgw.remove_key(access_key='81C3KDI66FG9ILSJRU5S')

More details in the API docs: https://docs.ceph.com/docs/master/radosgw/adminops/#remove-key

Buckets and Objects¶

List Buckets¶

A list of all buckets can be returned with get_buckets().

>>> rgw.get_buckets()
['foo', 'bar', 'baz']

Get Bucket¶

This can be used to return specific info on a bucket:

>>> rgw.get_bucket(bucket='liam-www')
{'bucket': 'liam-www',
 'num_shards': 0,
 'tenant': '',
 'zonegroup': '29946069-33ce-49b7-b93d-de8c95a0c344',
 'placement_rule': 'default-placement',
 'explicit_placement': {'data_pool': '.rgw.buckets',
 'data_extra_pool': '',
 'index_pool': '.rgw.buckets.index'},
 'id': 'default.7519.1',
 'marker': 'default.7519.1',
 'index_type': 'Normal',
 'owner': 'liam',
 'ver': '0#6701',
 'master_ver': '0#0',
 'mtime': '2018-01-26 16:07:10.290779Z',
 'max_marker': '0#00000006690.7081727.5',
 'usage': {'rgw.main': {'size': 1091085,
   'size_actual': 1130496,
   'size_utilized': 1091085,
   'size_kb': 1066,
   'size_kb_actual': 1104,
   'size_kb_utilized': 1066,
   'num_objects': 18},
 'rgw.multimeta': {'size': 0,
   'size_actual': 0,
   'size_utilized': 0,
   'size_kb': 0,
   'size_kb_actual': 0,
   'size_kb_utilized': 0,
   'num_objects': 0}},
 'bucket_quota': {'enabled': False,
 'check_on_raw': False,
 'max_size': -1024,
 'max_size_kb': 0,
 'max_objects': -1}}

get_bucket() can also be used to return a list of buckets for a given user:

>>> rgw.get_bucket(uid='liam')
['bucket1', 'bucket2', 'bucket3']

Check Bucket Index¶

Check the index of an existing bucket.

Note

to check multipart object accounting with check_objects, fix must be set to True.

This will return the status of the bucket index, if any.

>>> rgw.check_bucket_index(bucket='liampriv')
[]

Remove Bucket¶

Remove a bucket using remove_bucket(). You must pass purge_objects to delete a non-empty bucket. If called on a non-empty bucket when purge_data=False, this will throw a BucketNotEmpty exception.

>>> rgw.remove_bucket(bucket='bucket1')
BucketNotEmpty: {'Code': 'BucketNotEmpty',
                 'RequestId': 'tx0000000000000002a6446-005ed13a10-ad92-default',
                 'HostId': 'ad92-default-default'}

>>> rgw.remove_bucket(bucket='bucket1', purge_data=True)

Unlink Bucket¶

Unlink a bucket from a specified user using unlink_bucket().

rgw.unlink_bucket(bucket='launch-codes', uid='liam')

Link Bucket¶

Link the bucket to a new owner using link_bucket().

# first you have to find the bucket id
metadata = rgw.get_metadata(metadata_type='bucket', key=bucket)
bucket_id = metadata['data']['bucket']['bucket_id']

# then do the linking
rgw.link_bucket(
    bucket=bucket,
    bucket_id=bucket_id,
    uid='newowner',
)

Remove Object¶

Ceph Docs: https://docs.ceph.com/docs/master/radosgw/adminops/#remove-object

Objects can be removed from a bucket with remove_object().

rgw.remove_object(bucket='bkt1', object_name='index.html')

Get Bucket or Object Policy¶

Read the policy of an object or bucket.

>>> rgw.get_policy(bucket='liampub')
{'acl': {'acl_user_map': [{'user': 'liam', 'acl': 15}],
  'acl_group_map': [{'group': 1, 'acl': 1}],
  'grant_map': [{'id': '',
    'grant': {'type': {'type': 2},
     'id': '',
     'email': '',
     'permission': {'flags': 1},
     'name': '',
     'group': 1,
     'url_spec': ''}},
   {'id': 'liam',
    'grant': {'type': {'type': 0},
     'id': 'liam',
     'email': '',
     'permission': {'flags': 15},
     'name': 'Liam Monahan',
     'group': 0,
     'url_spec': ''}}]},
 'owner': {'id': 'liam', 'display_name': 'Liam Monahan'}}

And similarly for an object policy:

>>> rgw.get_policy(bucket='liampub', object_name='index.html')
{'acl': {'acl_user_map': [{'user': 'liam', 'acl': 15}],
  'acl_group_map': [],
  'grant_map': [{'id': 'liam',
    'grant': {'type': {'type': 0},
     'id': 'liam',
     'email': '',
     'permission': {'flags': 15},
     'name': 'Liam Monahan',
     'group': 0,
     'url_spec': ''}}]},
 'owner': {'id': 'liam', 'display_name': 'Liam Monahan'}}

Quotas¶

There are two main types of quotas: user quotas and bucket quotas. The API allows you to work with both.

Set User Quota¶

Maybe this user needs a 1GB quota. Set a quota with set_user_quota().

rgw.set_user_quota(
    uid='liam',
    quota_type='user',
    max_size_kb=1024*1024,  # 1GB
    enabled=True,
)

Set Bucket Quota¶

Set the quota on an individual bucket:

rgw.set_bucket_quota(
    uid='liam',
    bucket='launch-codes',
    max_size_kb=1024*1024,  # 1GB
    enabled=True)

Get User Quota¶

Return the quota set for the user.

>>> rgw.get_user_quota(uid='liam')
{'check_on_raw': False,
 'enabled': True,
 'max_objects': -1,
 'max_size': 1168400384,
 'max_size_kb': 1141016}

This user has a size quota enabled.

Get User Bucket Quota¶

Return the quota set on every bucket owned/created by a user. We can see that our example user does not have this quota enabled.

>>> rgw.get_user_bucket_quota(uid='liam')
{'check_on_raw': False,
 'enabled': False,
 'max_objects': -1,
 'max_size': -1,
 'max_size_kb': 0}

Get Quota¶

As a lower-level function you can get quotas and specify the quota type:

rgw.get_quota(uid='liam', quota_type='user')

Metadata Ops¶

Get Bucket Instances¶

There is a convenience method called get_bucket_instances() to get all the bucket instance metadata.

rgw.get_bucket_instances()

API Documentation¶

This documentation is generated directly from the source code.

rgwadmin.exceptions¶

This module contains the rgwadmin exceptions.

exception rgwadmin.exceptions.RGWAdminException(code, raw=None)[source]¶: There was a unlabeled exception that was raised during your request

exception rgwadmin.exceptions.AccessDenied(code, raw=None)[source]¶: Access was denied for the request.

exception rgwadmin.exceptions.UserExists(code, raw=None)[source]¶: Attempt to create existing user.

exception rgwadmin.exceptions.InvalidAccessKey(code, raw=None)[source]¶: Invalid access key specified.

exception rgwadmin.exceptions.InvalidArgument(code, raw=None)[source]¶: Invalid argument specified.

exception rgwadmin.exceptions.InvalidKeyType(code, raw=None)[source]¶: Invalid key type specified.

exception rgwadmin.exceptions.InvalidSecretKey(code, raw=None)[source]¶: Invalid secret key specified.

exception rgwadmin.exceptions.KeyExists(code, raw=None)[source]¶: Provided access key exists and belongs to another user.

exception rgwadmin.exceptions.EmailExists(code, raw=None)[source]¶: Provided email address exists.

exception rgwadmin.exceptions.SubuserExists(code, raw=None)[source]¶: Specified subuser exists.

exception rgwadmin.exceptions.InvalidAccess(code, raw=None)[source]¶: Invalid subuser access specified.

exception rgwadmin.exceptions.IndexRepairFailed(code, raw=None)[source]¶: Bucket index repair failed.

exception rgwadmin.exceptions.BucketNotEmpty(code, raw=None)[source]¶: Attempted to delete non-empty bucket.

exception rgwadmin.exceptions.ObjectRemovalFailed(code, raw=None)[source]¶: Unable to remove objects.

exception rgwadmin.exceptions.BucketUnlinkFailed(code, raw=None)[source]¶: Unable to unlink bucket from specified user.

exception rgwadmin.exceptions.BucketLinkFailed(code, raw=None)[source]¶: Unable to link bucket to specified user.

exception rgwadmin.exceptions.NoSuchObject(code, raw=None)[source]¶: Specified object does not exist.

exception rgwadmin.exceptions.IncompleteBody(code, raw=None)[source]¶: Either bucket was not specified for a bucket policy request or bucket and object were not specified for an object policy request.

exception rgwadmin.exceptions.InvalidCap(code, raw=None)[source]¶: Attempt to grant invalid admin capability.

exception rgwadmin.exceptions.NoSuchCap(code, raw=None)[source]¶: User does not possess specified capability.

exception rgwadmin.exceptions.InternalError(code, raw=None)[source]¶: Internal server error.

exception rgwadmin.exceptions.NoSuchUser(code, raw=None)[source]¶: User does not exist.

exception rgwadmin.exceptions.NoSuchBucket(code, raw=None)[source]¶: Bucket does not exist.

exception rgwadmin.exceptions.NoSuchKey(code, raw=None)[source]¶: No such access key.

exception rgwadmin.exceptions.ServerDown(code, raw=None)[source]¶: The backing server is not available.

exception rgwadmin.exceptions.InvalidQuotaType(code, raw=None)[source]¶: You must specify either a ‘user’ or ‘bucket’ quota type

exception rgwadmin.exceptions.BucketAlreadyExists(code, raw=None)[source]¶: The bucket already exists

rgwadmin.rgw¶

class rgwadmin.rgw.RGWAdmin(access_key, secret_key, server, admin='admin', response='json', ca_bundle=None, secure=True, verify=True, timeout=None, pool_connections=False)[source]¶

metadata_types = ['user', 'bucket', 'bucket.instance']¶

classmethod connect(**kwargs)[source]¶

Establish a new connection to RGWAdmin

Only one connection can be active in any single process

classmethod set_connection(connection: rgwadmin.rgw.RGWAdmin)[source]¶: Set a connection for the RGWAdmin session to use.

classmethod get_connection() → rgwadmin.rgw.RGWAdmin[source]¶: Return the RGWAdmin connection that was set

get_base_url() → str[source]¶: Return a base URL. I.e. https://ceph.server

request(method, request, headers: Optional[Dict[KT, VT]] = None, data=None)[source]¶

get_metadata(metadata_type, key=None, max_entries=None, marker=None, headers=None)[source]¶: Returns a JSON object representation of the metadata

put_metadata(metadata_type, key, json_string)[source]¶

set_metadata(metadata_type, key, json_string)¶

delete_metadata(metadata_type, key)[source]¶

lock_metadata(metadata_type, key, lock_id, length)[source]¶

unlock_metadata(metadata_type: str, key, lock_id)[source]¶

get_user(uid: Optional[str] = None, access_key: Optional[str] = None, stats=False, sync=False)[source]¶

get_users()[source]¶

create_user(uid, display_name, email=None, key_type='s3', access_key=None, secret_key=None, user_caps=None, generate_key=True, max_buckets=None, suspended=False)[source]¶

get_usage(uid=None, start=None, end=None, show_entries=False, show_summary=False)[source]¶

trim_usage(uid=None, start=None, end=None, remove_all=False)[source]¶

modify_user(uid, display_name=None, email=None, key_type='s3', access_key=None, secret_key=None, user_caps=None, generate_key=False, max_buckets=None, suspended=None)[source]¶

get_quota(uid, quota_type)[source]¶

get_user_quota(uid)[source]¶

get_user_bucket_quota(uid)[source]¶: Return the quota set on every bucket owned/created by a user

set_user_quota(uid, quota_type, max_size_kb=None, max_objects=None, enabled=None)[source]¶

Set quotas on users and buckets owned by users

If quota_type is user, then the quota applies to the user. If quota_type is bucket, then the quota applies to buckets owned by the specified uid.

If you want to set a quota on an individual bucket, then use set_bucket_quota() instead.

set_bucket_quota(uid, bucket, max_size_kb=None, max_objects=None, enabled=None)[source]¶: Set the quota on an individual bucket

remove_user(uid, purge_data=False)[source]¶

create_subuser(uid, subuser=None, secret_key=None, access_key=None, key_type=None, access=None, generate_secret=False)[source]¶

modify_subuser(uid, subuser, secret=None, key_type='swift', access=None, generate_secret=False)[source]¶

remove_subuser(uid, subuser, purge_keys=True)[source]¶

create_key(uid, subuser=None, key_type='s3', access_key=None, secret_key=None, generate_key=True)[source]¶

remove_key(access_key, key_type=None, uid=None, subuser=None)[source]¶

get_buckets()[source]¶: Returns a list of all buckets in the radosgw

get_bucket(bucket=None, uid=None, stats=False)[source]¶

check_bucket_index(bucket, check_objects=False, fix=False)[source]¶

remove_bucket(bucket, purge_objects=False)[source]¶

unlink_bucket(bucket, uid)[source]¶

link_bucket(bucket, uid, bucket_id=None, new_bucket_name=None)[source]¶

remove_object(bucket, object_name)[source]¶

get_policy(bucket, object_name=None)[source]¶

add_capability(uid, user_caps)[source]¶

remove_capability(uid, user_caps)[source]¶

get_bucket_instances()[source]¶: Returns a list of all bucket instances in the radosgw

static parse_rados_datestring(s)[source]¶

static gen_secret_key(size=40, chars='abcdefghijklmnopqrstuvwxyzABCDEFGHIJKLMNOPQRSTUVWXYZ0123456789')[source]¶

rgwadmin.user¶

class rgwadmin.user.AttributeMixin[source]¶

attrs = []¶

to_tuples()[source]¶

to_dict()[source]¶

class rgwadmin.user.RGWCap(type, perm)[source]¶

attrs = ['type', 'perm']¶

class rgwadmin.user.RGWKey(user, access_key, secret_key)[source]¶

attrs = ['user', 'access_key', 'secret_key']¶

static generate(user, access_size=20, secret_size=40)[source]¶

class rgwadmin.user.RGWSubuser(id, permissions)[source]¶

attrs = ['id', 'permissions']¶

permission_list = ['read', 'write', 'read-write', 'full-control']¶

permissions¶

class rgwadmin.user.RGWSwiftKey(user, secret_key)[source]¶

attrs = ['user', 'secret_key']¶

static generate(user, secret_size=40)[source]¶

class rgwadmin.user.RGWQuota(**kwargs)[source]¶

attrs = ['enabled', 'max_objects', 'max_size_kb']¶

defaults = {'enabled': False, 'max_objects': -1, 'max_size_kb': -1}¶

string_size()[source]¶

classmethod default()[source]¶

size¶

class rgwadmin.user.RGWUser(**kwargs)[source]¶

attrs = ['user_id', 'display_name', 'email', 'caps', 'keys', 'max_buckets', 'suspended', 'swift_keys', 'subusers', 'placement_tags', 'bucket_quota', 'user_quota', 'default_placement', 'op_mask', 'temp_url_keys']¶

modify_attrs_mask = ['placement_tags', 'default_placement', 'op_mask', 'temp_url_keys']¶

sub_attrs = {'bucket_quota': <class 'rgwadmin.user.RGWQuota'>, 'caps': <class 'rgwadmin.user.RGWCap'>, 'keys': <class 'rgwadmin.user.RGWKey'>, 'subusers': <class 'rgwadmin.user.RGWSubuser'>, 'swift_keys': <class 'rgwadmin.user.RGWSwiftKey'>, 'user_quota': <class 'rgwadmin.user.RGWQuota'>}¶: Representation of a RadosGW User

classmethod create(user_id, display_name, **kwargs)[source]¶

diff()[source]¶

exists()[source]¶: Return True if the user exists. False otherwise.

save()[source]¶

delete()[source]¶

classmethod list()[source]¶

classmethod fetch(user)[source]¶

to_dict()[source]¶: Return the dict representation of the object

rgwadmin¶

Contents:¶

Installation¶

Via Python Package Index¶

Via Source¶

User Guide¶

Connecting¶

Connection Pooling¶

User Management¶

Creating Users¶

Get User Details¶

Modifying Users¶

Listing Users¶

Removing Users¶

Add A User Capability¶

Remove A User Capability¶

Bandwidth Usage¶

Trim Usage¶

Subusers¶

Creating Subusers¶

Modifying Subusers¶

Removing Subusers¶

Keys¶

Create Key¶

Remove Key¶

Buckets and Objects¶

List Buckets¶

Get Bucket¶

Check Bucket Index¶

Remove Bucket¶

Unlink Bucket¶

Link Bucket¶

Remove Object¶

Get Bucket or Object Policy¶

Quotas¶

Set User Quota¶

Set Bucket Quota¶

Get User Quota¶

Get User Bucket Quota¶

Get Quota¶

Metadata Ops¶

Get Metadata¶

Put Metadata¶

Delete Metadata¶

Lock Metadata¶

Unlock Metadata¶

Get Bucket Instances¶

API Documentation¶

rgwadmin.exceptions¶

rgwadmin.exceptions¶

rgwadmin.rgw¶

rgwadmin.user¶

Indices and tables¶