SwiftStack Metadata Search

Metadata search integrates Swift object metadata with Elasticsearch. Elasticsearch allows for quick look-up of objects by any of the metadata associated with them. The feature operates on individual Swift containers. For each indexed container, an Elasticsearch endpoint and an index need to be specified.

The processes doing the indexing reside on the Swift container nodes. These nodes need network access to the Elasticsearch cluster. The indexing work happens in the background and does not affect any of the requests against Swift.

Warning

Beware of using the same index and Elasticsearch cluster for multiple SwiftStack clusters. In that case, if the same account and container exist on two clusters, one cluster may overwrite documents from the other.

Configuring mappings

To create a metadata search mapping, from the Data Index tab follow these steps:

1. Click Add a metadata mapping

2. Enter a Swift Account, Container, one or multiple endpoints for the Elasticsearch cluster (separated by commas), and specify the Index to use. You may also opt to have all containers in the account indexed. However, beware that they should store similar data, otherwise the index can become wide.

   

Afterward, each mapping will appear in the table on the Date Index page.

Resetting a mapping

In case documents are removed from the index either by accident or if you are attempting to re-index the container, you can reset the metadata mapping for a given container. This will restart the metadata search daemons and all of the objects in the container will be indexed.

Elasticsearch document mappings

The metadata search daemons will not create a metadata index. You should do so before enabling the feature. When creating the index, pay careful attention to the document mapping. The daemons will attempt to ensure that the Swift object fields such as X-Timestamp and Content-Length have proper mappings, but cannot ensure that the user-supplied metadata is correctly parsed. Please refer to the Elasticsearch documentation for more information on this.

For example, to create a simple index to automatically detect strings, dates and numeric values, the following command could be used.

curl -X PUT -H 'content-type:application/json' http://localhost:9200/my_index  --data-binary '{
    "mappings": {
     "_doc": {
      "numeric_detection": true
    }
  }
}'

Swift Encryption at rest

If Swift encryption at rest is enabled, the metadata indexing daemon will decrypt the metadata and supply it to Elasticsearch.

Searching for Objects

To search for objects indexed using metadata search, see SwiftStack Client Metadata Search.