Entry Amazon OpenSearch Serverless collections utilizing a VPC endpoint

Amazon OpenSearch Serverless helps you index, analyze, and search your logs and knowledge utilizing OpenSearch APIs and dashboards. The OpenSearch Serverless assortment is a gaggle of indexes. API and dashboard purchasers can entry the collections from public networks or a number of VPCs. For VPC entry to collections and dashboards, you may create VPC endpoints. On this submit, we show how one can create and use VPC endpoints and OpenSearch Serverless community insurance policies to manage entry to your collections and OpenSearch dashboards from a number of community areas.

The demo on this submit makes use of an AWS Lambda-based consumer in a VPC to ingest knowledge into a group through a VPC endpoint and a browser in a public community accessing the identical assortment.

Resolution overview

As an instance how one can ingest knowledge into an OpenSearch Serverless assortment from inside a VPC, we use a Lambda operate. We use a VPC-hosted Lambda operate to create an index in an OpenSearch Serverless assortment and add paperwork to the index utilizing a VPC endpoint. We then use a publicly accessible OpenSearch Serverless dashboard to see the paperwork ingested from Lambda operate.

The next sections element the steps to ingest knowledge into the gathering utilizing Lambda and entry the OpenSearch Serverless dashboard.

Conditions

This setup assumes that you’ve got already created a VPC with personal subnets.

Ingest knowledge utilizing Lambda and entry the OpenSearch Serverless dashboard

To arrange your answer, full the next steps:

1. On the OpenSearch Service console, create a non-public connection between your VPC and OpenSearch Serverless utilizing a VPC endpoint. Use the personal subnets and a safety group out of your VPC.
2. Create an OpenSearch assortment utilizing the VPC endpoint created within the earlier step.
   
3. Create a community coverage to allow VPC entry to the OpenSearch endpoint so the Lambda operate can ingest paperwork to the gathering. You must also allow public entry to the OpenSearch dashboard endpoint so we are able to see the paperwork ingested.
4. After you create the gathering, create an information entry coverage to grant ingestion entry to the Lambda operate’s AWS Id and Entry Administration (IAM) function.
5. Moreover, grant learn entry to the dashboard consumer’s IAM function.
6. Add IAM permissions to the Lambda operate’s IAM function and the dashboard consumer’s IAM function for the OpenSearch Serverless assortment.
7. Create a Lambda operate in the identical VPC and subnet that we used for the OpenSearch endpoint (see the next code). This operate creates an index referred to as sitcoms-eighties within the OpenSearch Serverless assortment and provides a pattern doc to the index:

import datetime
import time
from opensearchpy import OpenSearch, RequestsHttpConnection
from requests_aws4auth import AWS4Auth
import boto3

host="<Insert-OpenSearch-Serverless-Endpoint>"
area = 'us-east-1'
service="aoss"
credentials = boto3.Session().get_credentials()
awsauth = AWS4Auth(credentials.access_key, credentials.secret_key, area, service,session_token=credentials.token)

# Construct the OpenSearch consumer
consumer = OpenSearch(
    hosts=[{'host': host, 'port': 443}],
    http_auth=awsauth,
    use_ssl=True,
    verify_certs=True,
    connection_class=RequestsHttpConnection,
    timeout=300
)

def lambda_handler (occasion, context):

    # Create index
    response = consumer.indices.create('sitcoms-eighties')
    print('nCreating index:')
    print(response)
    time.sleep(5)
    
    dt = datetime.datetime.now()
    # Add a doc to the index.
    response = consumer.index(
        index='sitcoms-eighties',
        physique={
            'title': 'Seinfeld',
            'creator': 'Larry David',
            '12 months': 1989,
            'createtime': dt
        },
        id='1',
    )
    print('nDocument added:')
    print(response)

8. Run the Lambda operate, and you need to see the output as proven within the following screenshot.
9. Now you can see the paperwork from this index via your publicly accessible OpenSearch Dashboards URL.
10. Create the index sample in OpenSearch Dashboards, after which you may see the paperwork as proven within the following screenshot.

Use a VPC DNS resolver out of your community

A consumer in your VPN community can hook up with the gathering or dashboards over a VPC endpoint. The consumer wants to search out the VPC endpoint’s IP deal with utilizing an Amazon Route 53 inbound resolver endpoint. To be taught extra about Route 53 inbound resolver endpoints, seek advice from Resolving DNS queries between VPCs and your community. The next diagram reveals a pattern setup.

The circulation for this structure is as follows:

1. The DNS question for the OpenSearch Serverless consumer is routed to a domestically configured on-premises DNS server.
2. The on-premises DNS as configured performs conditional forwarding for the zone us-east-1.aoss.amazonaws.com to a Route 53 inbound resolver endpoint IP deal with. You should change your Area identify within the previous zone identify.
3. The inbound resolver endpoint performs DNS decision by forwarding the question to the personal hosted zone that was created together with the OpenSearch Serverless VPC endpoint.
4. The IP addresses returned by the DNS question are the personal IP addresses of the interface VPC endpoint, which permit your on-premises host to ascertain personal connectivity over AWS Web site-to-Web site VPN.
5. The interface endpoint is a group of a number of elastic community interfaces with a non-public IP deal with in your account that serves as an entry level for visitors going to an OpenSearch Serverless endpoint.

Abstract

OpenSearch Serverless permits you to arrange and management entry to the service utilizing VPC endpoints and community insurance policies. On this submit, we explored how one can entry an OpenSearch Serverless assortment API and dashboard from inside a VPC, on premises, and public networks. When you have any questions or solutions, please write to us within the feedback part.

Concerning the Authors

Raj Ramasubbu is a Senior Analytics Specialist Options Architect centered on large knowledge and analytics and AI/ML with Amazon Internet Companies. He helps prospects architect and construct extremely scalable, performant, and safe cloud-based options on AWS. Raj offered technical experience and management in constructing knowledge engineering, large knowledge analytics, enterprise intelligence, and knowledge science options for over 18 years previous to becoming a member of AWS. He helped prospects in numerous business verticals like healthcare, medical gadgets, life science, retail, asset administration, automotive insurance coverage, residential REIT, agriculture, title insurance coverage, provide chain, doc administration, and actual property.

Vivek Kansal works with the Amazon OpenSearch group. In his function as Principal Software program Engineer, he makes use of his expertise within the areas of safety, coverage engines, cloud-native options, and networking to assist safe buyer knowledge in OpenSearch Service and OpenSearch Serverless in an evolving menace panorama.