Alcion helps their multi-tenant platform with Amazon OpenSearch Serverless

This can be a visitor weblog put up co-written with Zack Rossman from Alcion.

Alcion, a security-first, AI-driven backup-as-a-service (BaaS) platform, helps Microsoft 365 directors rapidly and intuitively defend information from cyber threats and unintended information loss. Within the occasion of knowledge loss, Alcion clients want to look metadata for the backed-up objects (information, emails, contacts, occasions, and so forth) to pick particular merchandise variations for restore. Alcion makes use of Amazon OpenSearch Service to offer their clients with correct, environment friendly, and dependable search functionality throughout this backup catalog. The platform is multi-tenant, which implies that Alcion requires information isolation and robust safety in order to make sure that tenants can solely search their very own information.

OpenSearch Service is a totally managed service that makes it straightforward to deploy, scale, and function OpenSearch within the AWS Cloud. OpenSearch is an Apache-2.0-licensed, open-source search and analytics suite, comprising OpenSearch (a search, analytics engine, and vector database), OpenSearch Dashboards (a visualization and utility person interface), and plugins that present superior capabilities like enterprise-grade safety, anomaly detection, observability, alerting, and far more. Amazon OpenSearch Serverless is a serverless deployment choice that makes it easy to make use of OpenSearch with out configuring, managing, and scaling OpenSearch Service domains.

On this put up, we share how adopting OpenSearch Serverless enabled Alcion to fulfill their scale necessities, cut back their operational overhead, and safe their tenants’ information by implementing tenant isolation inside their multi-tenant surroundings.

OpenSearch Service managed domains

For the primary iteration of their search structure, Alcion selected the managed domains deployment choice in OpenSearch Service and was capable of launch their search performance in manufacturing in lower than a month. To satisfy their safety, scale, and tenancy necessities, they saved information for every tenant in a devoted index and used fine-grained entry management in OpenSearch Service to stop cross-tenant information leaks. As their workload advanced, Alcion engineers tracked OpenSearch area utilization by way of the offered Amazon CloudWatch metrics, making modifications to extend storage and optimize their compute assets.

The crew at Alcion used a number of options of OpenSearch Service managed domains to enhance their operational stance. They launched index aliases, which offer a single alias identify to entry (learn and write) a number of underlying indexes. In addition they configured Index State Administration (ISM) insurance policies to assist them management their information lifecycle by rolling indexes over based mostly on index measurement. Collectively, the ISM insurance policies and index aliases have been essential to scale indexes for giant tenants. Alcion additionally used index templates to outline the shards per index (partitioning) of their information in order to automate their information lifecycle and enhance the efficiency and stability of their domains.

The next structure diagram exhibits how Alcion configured their OpenSearch managed domains.

The next diagram exhibits how Microsoft 365 information was listed to and queried from tenant-specific indexes. Alcion carried out request authentication by offering the OpenSearch major person credentials with every API request.

OpenSearch Serverless overview and tenancy mannequin choices

OpenSearch Service managed domains offered a steady basis for Alcion’s search performance, however the crew wanted to manually provision assets to the domains for his or her peak workload. This left room for value optimizations as a result of Alcion’s workload is bursty—there are massive variations within the variety of search and indexing transactions per second, each for a single buyer and brought as a complete. To cut back prices and operational burden, the crew turned to OpenSearch Serverless, which gives auto-scaling functionality.

To make use of OpenSearch Serverless, step one is to create a group. A assortment is a bunch of OpenSearch indexes that work collectively to assist a selected workload or use case. The compute assets for a group, referred to as OpenSearch Compute Models (OCUs), are shared throughout all collections in an account that share an encryption key. The pool of OCUs is robotically scaled up and down to fulfill the calls for of indexing and search visitors.

The extent of effort required emigrate from an OpenSearch Service managed area to OpenSearch Serverless was manageable due to the truth that OpenSearch Serverless collections assist the identical OpenSearch APIs and libraries as OpenSearch Service managed domains. This allowed Alcion to give attention to optimizing the tenancy mannequin for the brand new search structure. Particularly, the crew wanted to determine learn how to partition tenant information inside collections and indexes whereas balancing safety and whole value of possession. Alcion engineers, in collaboration with the OpenSearch Serverless crew, thought of three tenancy fashions:

• Silo mannequin: Create a group for every tenant
• Pool mannequin: Create a single assortment and use a single index for a number of tenants
• Bridge mannequin: Create a single assortment and use a single index per tenant

All three design decisions had advantages and trade-offs that needed to be thought of for designing the ultimate answer.

Silo mannequin: Create a group for every tenant

On this mannequin, Alcion would create a brand new assortment at any time when a brand new buyer onboarded to their platform. Though tenant information can be cleanly separated between collections, this selection was disqualified as a result of the gathering creation time meant that clients wouldn’t have the ability to again up and search information instantly after registration.

Pool mannequin: Create a single assortment and use a single index for a number of tenants

On this mannequin, Alcion would create a single assortment per AWS account and index tenant-specific information in one in every of many shared indexes belonging to that assortment. Initially, pooling tenant information into shared indexes was enticing from a scale perspective as a result of this led to probably the most environment friendly use of index assets. However after additional evaluation, Alcion discovered that they might be properly inside the per-collection index quota even when they allotted one index for every tenant. With that scalability concern resolved, Alcion pursued the third choice as a result of siloing tenant information into devoted indexes ends in stronger tenant isolation than the shared index mannequin.

Bridge mannequin: Create a single assortment and use a single index per tenant

On this mannequin, Alcion would create a single assortment per AWS account and create an index for every of the lots of of tenants managed by that account. By assigning every tenant to a devoted index and pooling these indexes in a single assortment, Alcion diminished onboarding time for brand new tenants and siloed tenant information into cleanly separated buckets.

Implementing role-based entry management for supporting multi-tenancy

OpenSearch Serverless gives a multi-point, inheritable set of safety controls, overlaying information entry, community entry, and encryption. Alcion took full benefit of OpenSearch Serverless information entry insurance policies to implement role-based entry management (RBAC) for every tenant-specific index with the next particulars:

• Allocate an index with a typical prefix and the tenant ID (for instance, index-v1-<tenantID>)
• Create a devoted AWS Id and Entry Administration (IAM) function that’s used to signal requests to the OpenSearch Serverless assortment
• Create an OpenSearch Serverless information entry coverage that grants doc learn/write permissions inside a devoted tenant index to the IAM function for that tenant
• OpenSearch API requests to a tenant index are signed with non permanent credentials belonging to the tenant-specific IAM function

The next is an instance OpenSearch Serverless information entry coverage for a mock tenant with ID t-eca0acc1-12345678910. This coverage grants the IAM function doc learn/write entry to the devoted tenant entry.

[
    {
        "Rules": [
            {
                "Resource": [
                    "index/collection-searchable-entities/index-v1-t-eca0acc1-12345678910"
                ],
                "Permission": [
                    "aoss:ReadDocument",
                    "aoss:WriteDocument",
                ],
                "ResourceType": "index"
            }
        ],
        "Principal": [
            "arn:aws:iam::12345678910:role/OpenSearchAccess-t-eca0acc1-1b9f-4b3f-95d6-12345678910"
        ],
        "Description": "Enable doc learn/write entry to OpenSearch index belonging to tenant t-eca0acc1-1b9f-4b3f-95d6-12345678910"
    }
] 

The next structure diagram depicts how Alcion carried out indexing and trying to find Microsoft 365 assets utilizing the OpenSearch Serverless shared assortment strategy.

The next is the pattern code snippet for sending an API request to an OpenSearch Serverless assortment. Discover how the API consumer is initialized with a signer object that indicators requests with the identical IAM principal that’s linked to the OpenSearch Serverless information entry coverage from the earlier code snippet.

bundle alcion

import (
	"context"
	"encoding/json"
	"strings"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/credentials/stscreds"
	"github.com/aws/aws-sdk-go-v2/service/sts"
	"github.com/opensearch-project/opensearch-go/v2"
	"github.com/opensearch-project/opensearch-go/v2/opensearchapi"
	"github.com/opensearch-project/opensearch-go/v2/signer"
	awssignerv2 "github.com/opensearch-project/opensearch-go/v2/signer/awsv2"
	"github.com/pkg/errors"
)

const (
	// Scope the API request to the AWS OpenSearch Serverless service
	aossService = "aoss"

	// Mock values
	indexPrefix        = "index-v1-"
	collectionEndpoint = "<https://kfbr3928z4y6vot2mbpb.us-east-1.aoss.amazonaws.com>"
	tenantID           = "t-eca0acc1-1b9f-4b3f-95d6-b0b96b8c03d0"
	roleARN            = "arn:aws:iam::1234567890:function/OpenSearchAccess-t-eca0acc1-1b9f-4b3f-95d6-b0b96b8c03d0"
)

func CreateIndex(ctx context.Context, tenantID string) (*opensearchapi.Response, error) {

	sig, err := createRequestSigner(ctx)
	if err != nil {
		return nil, errors.Wrapf(err, "error creating new signer for AWS OSS")
	}

	cfg := opensearch.Config{
		Addresses: []string{collectionEndpoint},
		Signer:    sig,
	}

	aossClient, err := opensearch.NewClient(cfg)
	if err != nil {
		return nil, errors.Wrapf(err, "error creating new OpenSearch API consumer")
	}

  physique, err := getSearchBody()
  if err != nil {
    return nil, errors.Wrapf(err, "error getting search physique")
  }

	req := opensearchapi.SearchRequest{
		Index: []string{indexPrefix + tenantID},
		Physique:  physique,
	}

	return req.Do(ctx, aossClient)
}

func createRequestSigner(ctx context.Context) (signer.Signer, error) {

	awsCfg, err := config.LoadDefaultConfig(ctx)
	if err != nil {
		return nil, errors.Wrapf(err, "error loading default config")
	}

	stsClient := sts.NewFromConfig(awsCfg)
	supplier := stscreds.NewAssumeRoleProvider(stsClient, roleARN)

	awsCfg.Credentials = aws.NewCredentialsCache(supplier)
	return awssignerv2.NewSignerWithService(awsCfg, aossService)
}

func getSearchBody() (*strings.Reader, error) {
	// Match all paperwork, web page measurement = 10
	question := map[string]interface{}{
		"measurement": 10,
	}

	queryJson, err := json.Marshal(question)
  if err != nil {
    return nil, err
  }

	return strings.NewReader(string(queryJson)), nil
} 

Conclusion

In Could of 2023, Alcion rolled out its search structure based mostly on the shared assortment and devoted index-per-tenant mannequin in all manufacturing and pre-production environments. The crew was capable of tear out complicated code and operational processes that had been devoted to scaling OpenSearch Service managed domains. Moreover, due to the auto scaling capabilities of OpenSearch Serverless, Alcion has diminished their OpenSearch prices by 30% and expects the fee profile to scale favorably.

Of their journey from managed to serverless OpenSearch Service, Alcion benefited of their preliminary alternative of OpenSearch Service managed domains. In migrating ahead, they have been capable of reuse the identical OpenSearch APIs and libraries for his or her OpenSearch Serverless collections that they used for his or her OpenSearch Service managed area. Moreover, they up to date their tenancy mannequin to reap the benefits of OpenSearch Serverless information entry insurance policies. With OpenSearch Serverless, they have been capable of effortlessly adapt to their clients’ scale wants whereas making certain tenant isolation.

For extra details about Alcion, go to their web site.

In regards to the Authors

Zack Rossman is a Member of Technical Employees at Alcion. He’s the tech lead for the search and AI platforms. Previous to Alcion, Zack was a Senior Software program Engineer at Okta, creating core workforce id and entry administration merchandise for the Directories crew.

Niraj Jetly is a Software program Growth Supervisor for Amazon OpenSearch Serverless. Niraj leads a number of information airplane groups answerable for launching Amazon OpenSearch Serverless. Previous to AWS, Niraj led a number of product and engineering groups as CTO, VP of Engineering, and Head of Product Administration for over 15 years. Niraj is a recipient of over 15 innovation awards, together with being named CIO of the 12 months in 2014 and high 100 CIO in 2013 and 2016. A frequent speaker at a number of conferences, he has been quoted in NPR, WSJ, and The Boston Globe.

Jon Handler is a Senior Principal Options Architect at Amazon Internet Companies based mostly in Palo Alto, CA. Jon works intently with OpenSearch and Amazon OpenSearch Service, offering assist and steerage to a broad vary of consumers who’ve search and log analytics workloads that they need to transfer to the AWS Cloud. Previous to becoming a member of AWS, Jon’s profession as a software program developer included 4 years of coding a large-scale, ecommerce search engine. Jon holds a Bachelor of the Arts from the College of Pennsylvania and a Grasp of Science and a PhD in Pc Science and Synthetic Intelligence from Northwestern College.