Utilization¶
Types of Utilization Data¶
Utilization is tracked for two types of data: storage and transfer. Each type of utilization data has a separate set of API endpoints and slightly different characteristics.
Note
The accounts exposed by the Utilization API are Swift storage accounts
and will not usually match the username for authentication. For example,
cluster usage by a user "bob" authenticating via SwiftStack Auth middleware
would appear in the Swift account AUTH_bob.
Storage Data¶
Storage data is tracked and accessed per Swift storage policy index. Storage values are the maximum single observed value for a given time interval and consist of:
container_count: the count of containers in the Swift account.object_count: the total count of objects stored in the Swift account.bytes_used: the total number of bytes stored in the Swift account.
Storage utilization data comes from the Swift Account databases and is therefore subject to eventual consistency. So the data returned by this API may slightly "lag" Swift API operations for large clusters.
Transfer Data¶
Transfer data values are the sum (total) over the given time interval and consist of:
bytes_in: the number of object bytes transferred into the Swift cluster (uploaded by clients).bytes_out: the number of object bytes transferred out of the Swift cluster (downloaded by clients).req_count: the number of Swift API requests made to the account.
Time Ranges¶
All Utilization API endpoints require a query parameter, start, which defines the
beginning of the time range for the request. A query parameter, end
defines the end of the time range and is optional. If end is not specified the
current time ("now") is used.
The start parameter is "rounded down" to
the nearest half-hour. The end parameter is then "rounded up" to the
nearest even multiple of 1 hour from start. See the examples below.
When times appear in the output of this API, they will be in UTC and formatted
according to ISO 8601.
Specifically, the date and time portions will be separated with a T,
microseconds will not be included, and
the time string will end with a Z to indicate it is in UTC.
After rounding, the start value must be at least one hour before end.
The start and end query parameters must be in
ISO 8601 format. To avoid
confusion, start and end should be in UTC, but the hour/minute offset
syntax of ISO 8601 is fully supported. Here are some example valid input times
and how they would be rounded:
- A
starttime of2013-08-29T19:24:44.22739-07:00would be rounded to2013-08-29T19:00:00Z. A correspondingendtime of2013-08-30t06:21:44zwould be rounded up to2013-08-30T07:00:00Z. - A
starttime of2013-08-30 02:37:44is assumed to be in UTC and would be rounded to2013-08-30T02:30:00Z. A correspondingendtime of2013-08-30 14:02:17+01:15would be rounded up to2013-08-30T13:30:00Z.
When a piece of data returned by this API is the result of summarizing one or
more hourly record, the number of hourly records which were consolidated is
reported in the hourly_row_count field.
Correctness¶
Your Organization has an associated configuration parameter called "Utilization finalization days" (See https://scswiftcontroller.tatamediaservices.com/organization/edit/.) If a SwiftStack node cannot reach the controller for more than this number of days in a row, Utilization data will be lost and that time period will have "missing data". If a SwiftStack node is powered off or otherwise unable to actually gather Utilization data, that time period will have "missing data". If a SwiftStack node cannot talk to the controller for a period of time less than the "Utilization finalization days", utilization data for that period will still get processed when the node-to-controller communication is restored. Because missing data up to "Utilization finalization days" old will still get processed, we recommend you only request Utilization data via this API for time periods older than your Organization's configured "Utilization finalization days."
Missing data is reflected in the API results in the pct_complete field.
For any given piece of utilization
data, the associated pct_complete value is the percentage of expected data
points which were received.
Per-Account Storage Utilization¶
For a given cluster and Swift storage policy index, lists summarized storage utilization data for all accounts during the specified time range.
-
GET/api/v1/clusters/<cluster_id>/utilization/storage/<policy_idx>/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
Example response:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 | {
"meta": {
"start": "2013-08-31 06:30:00Z",
"end": "2013-09-01 01:30:00Z",
"policy_idx": 0,
"total_count": 2,
"limit": 20,
"offset": 0,
"previous": null,
"next": null
},
"objects": [
{
"account": "AUTH_bob",
"bytes_used": 519000,
"container_count": 5190,
"hourly_row_count": 19,
"object_count": 51900,
"pct_complete": 100.0,
"resource_uri": "/api/v1/clusters/6/utilization/storage/0/AUTH_bob/"
},
{
"account": "AUTH_sally",
"bytes_used": 520000,
"container_count": 5200,
"hourly_row_count": 19,
"object_count": 52000,
"pct_complete": 100.0,
"resource_uri": "/api/v1/clusters/6/utilization/storage/0/AUTH_sally/"
}
]
}
|
Per-Account Transfer Utilization¶
For a given cluster, lists summarized storage utilization data for all accounts
during the specified time range. Note that there is no policy_idx for
this data type.
-
GET/api/v1/clusters/<cluster_id>/utilization/transfer/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
Example response:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 | {
"meta": {
"start": "2013-08-31 06:30:00Z",
"end": "2013-09-01 01:30:00Z",
"total_count": 2,
"limit": 20,
"offset": 0,
"previous": null,
"next": null
},
"objects": [
{
"account": "AUTH_bob",
"bytes_in": 126920,
"req_count": 25384,
"bytes_out": 76152,
"hourly_row_count": 19,
"pct_complete": 100.0,
"resource_uri": "/api/v1/clusters/6/utilization/transfer/AUTH_bob/"
},
{
"account": "AUTH_sally",
"bytes_in": 127300,
"bytes_out": 76380,
"req_count": 25460,
"hourly_row_count": 19,
"pct_complete": 100.0,
"resource_uri": "/api/v1/clusters/6/utilization/transfer/AUTH_sally/"
}
]
}
|
Per-Cluster Total Storage Utilization¶
For a given cluster and Swift storage policy index, aggregates storage
utilization data for all accounts over the specified time range. The
pct_complete value will be the average (mean) of each account's
pct_complete value.
-
GET/api/v1/clusters/<cluster_id>/utilization/storage/<policy_idx>/total/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
Example response:
1 2 3 4 5 6 7 8 9 10 | {
"start": "2013-08-31 06:30:00Z",
"end": "2013-09-01 01:30:00Z",
"policy_idx": 0,
"bytes_used": 1039000,
"container_count": 10390,
"hourly_row_count": 38,
"object_count": 103900,
"pct_complete": 100.0,
}
|
Per-Cluster Total Transfer Utilization¶
For a given cluster, aggregates transfer utilization data for all accounts in the
cluster over the specified time range. Note that there is no policy_idx for
this data type. The pct_complete value will be the average (mean) of each
account's pct_complete value.
-
GET/api/v1/clusters/<cluster_id>/utilization/transfer/total/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
Example response:
1 2 3 4 5 6 7 8 9 | {
"start": "2013-08-31 06:30:00Z",
"end": "2013-09-01 01:30:00Z",
"bytes_in": 254220,
"req_count": 50844,
"bytes_out": 152532,
"hourly_row_count": 38,
"pct_complete": 100.0
}
|
Single-Account Storage Utilization¶
For a given cluster, Swift storage policy index, and account name, returns summarized storage utilization data during the specified time range.
-
GET/api/v1/clusters/<cluster_id>/utilization/storage/<policy_idx>/<account>/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
curl -1 "https://scswiftcontroller.tatamediaservices.com/api/v1/clusters/<cluster_id>/utilization/storage/<policy_idx>/AUTH_sally/?start=<start_time>&end=<end_time>&format=json" -H 'Authorization: apikey <user>:<api_key>'
Example response:
1 2 3 4 5 6 7 8 9 10 11 12 | {
"start": "2013-08-31 06:30:00Z",
"end": "2013-09-01 01:30:00Z",
"policy_idx": 0,
"account": "AUTH_sally",
"container_count": 5200,
"object_count": 52000,
"bytes_used": 520000,
"hourly_row_count": 19,
"pct_complete": 100.0,
"resource_uri": "/api/v1/clusters/6/utilization/storage/0/AUTH_sally/"
}
|
Single-Account Transfer Utilization¶
For a given cluster and account name, returns summarized transfer utilization data during the specified time range.
-
GET/api/v1/clusters/<cluster_id>/utilization/transfer/<account>/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
curl -1 "https://scswiftcontroller.tatamediaservices.com/api/v1/clusters/<cluster_id>/utilization/transfer/AUTH_bob/?start=<start_time>&end=<end_time>&format=json" -H 'Authorization: apikey <user>:<api_key>'
Example response:
1 2 3 4 5 6 7 8 9 10 11 | {
"start": "2013-08-31 06:30:00Z",
"end": "2013-09-01 01:30:00Z",
"account": "AUTH_bob",
"bytes_in": 126920,
"bytes_out": 76152,
"req_count": 25384,
"hourly_row_count": 19,
"pct_complete": 100.0,
"resource_uri": "/api/v1/clusters/6/utilization/transfer/AUTH_bob/"
}
|
Single-Account Storage Utilization Details¶
For a given cluster, Swift storage policy index, and account name, returns a list of each hourly data point during the specified time range. Hourly chunks which contain no data will be skipped.
-
GET/api/v1/clusters/<cluster_id>/utilization/storage/<policy_idx>/<account>/detail/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
curl -1 "https://scswiftcontroller.tatamediaservices.com/api/v1/clusters/<cluster_id>/utilization/storage/<policy_idx>/AUTH_sally/detail/?start=<start_time>&end=<end_time>&format=json" -H 'Authorization: apikey <user>:<api_key>'
Example response:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 37 | {
"meta": {
"account": "AUTH_sally",
"policy_idx": 0,
"total_count": 3,
"limit": 20,
"offset": 0,
"previous": null,
"next": null
},
"objects": [
{
"start": "2013-08-31 23:30:00Z",
"end": "2013-09-01 00:30:00Z",
"container_count": 5000,
"object_count": 50000,
"bytes_used": 500000,
"pct_complete": 100.0
},
{
"start": "2013-09-01 00:30:00Z",
"end": "2013-09-01 01:30:00Z",
"container_count": 5200,
"object_count": 52000,
"bytes_used": 520000,
"pct_complete": 100.0
},
{
"start": "2013-09-01 01:30:00Z",
"end": "2013-09-01 02:30:00Z",
"container_count": 5400,
"object_count": 54000,
"bytes_used": 540000,
"pct_complete": 100.0
}
]
}
|
Single-Account Transfer Utilization Details¶
For a given cluster and account name, returns a list of hourly data points for the specified time range. Hourly chunks which contain no data will be skipped.
-
GET/api/v1/clusters/<cluster_id>/utilization/transfer/<account>/detail/¶ Query Parameters: - start -- ISO 8601 time; Required
- end -- ISO 8601 time; Optional
Status: Status: Status:
Example request:
curl -1 "https://scswiftcontroller.tatamediaservices.com/api/v1/clusters/<cluster_id>/utilization/transfer/AUTH_bob/detail/?start=<start_time>&end=<end_time>&format=json" -H 'Authorization: apikey <user>:<api_key>'
Example response:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31 32 33 34 35 36 | {
"meta": {
"account": "AUTH_bob",
"total_count": 3,
"limit": 20,
"offset": 0,
"previous": null,
"next": null
},
"objects": [
{
"start": "2013-08-31 23:30:00Z",
"end": "2013-09-01 00:30:00Z",
"bytes_in": 9880,
"bytes_out": 5928,
"req_count": 1976,
"pct_complete": 100.0
},
{
"start": "2013-09-01 00:30:00Z",
"end": "2013-09-01 01:30:00Z",
"bytes_in": 10280,
"bytes_out": 6168,
"req_count": 2056,
"pct_complete": 100.0
},
{
"start": "2013-09-01 01:30:00Z",
"end": "2013-09-01 02:30:00Z",
"bytes_in": 10680,
"bytes_out": 6408,
"req_count": 2136,
"pct_complete": 100.0
}
]
}
|
Troubleshooting¶
These API endpoints require the trailing slash. If not provided, you will receive a 301 redirect instead of data. If using curl without -v you just get no response.
Otherwise valid queries which result in no data will return a 404.
