Construct a serverless log analytics pipeline utilizing Amazon OpenSearch Ingestion with managed Amazon OpenSearch Service

On this submit, we present the way to construct a log ingestion pipeline utilizing the brand new Amazon OpenSearch Ingestion, a completely managed information collector that delivers real-time log and hint information to Amazon OpenSearch Service domains. OpenSearch Ingestion is powered by the open-source information collector Information Prepper. Information Prepper is a part of the open-source OpenSearch venture. With OpenSearch Ingestion, you possibly can filter, enrich, rework, and ship your information for downstream evaluation and visualization. OpenSearch Ingestion is serverless, so that you don’t want to fret about scaling your infrastructure, working your ingestion fleet, and patching or updating the software program.

For a complete overview of OpenSearch Ingestion, go to Amazon OpenSearch Ingestion, and for extra details about the Information Prepper open-source venture, go to Information Prepper.

On this submit, we discover the logging infrastructure for a fictitious firm, AnyCompany. We discover the parts of the end-to-end resolution after which present the way to configure OpenSearch Ingestion’s foremost parameters and the way the logs come out and in of OpenSearch Ingestion.

Resolution overview

Take into account a situation by which AnyCompany collects Apache net logs. They use OpenSearch Service to watch net entry and establish attainable root causes to error logs of kind 4xx and 5xx. The next structure diagram outlines using each element used within the log analytics pipeline: Fluent Bit collects and forwards logs; OpenSearch Ingestion processes, routes, and ingests logs; and OpenSearch Service analyzes the logs.

The workflow comprises the next levels:

1. Generate and accumulate – Fluent Bit collects the generated logs and forwards them to OpenSearch Ingestion. On this submit, you create faux logs that Fluent Bit forwards to OpenSearch Ingestion. Test the record of supported purchasers to assessment the required configuration for every consumer supported by OpenSearch Ingestion.
2. Course of and ingest – OpenSearch Ingestion filters the logs primarily based on response worth, processes the logs utilizing a grok processor, and applies conditional routing to ingest the error logs to an OpenSearch Service index.
3. Retailer and analyze – We will analyze the Apache httpd error logs utilizing OpenSearch Dashboards.

Conditions

To implement this resolution, be sure to have the next conditions:

Configure OpenSearch Ingestion

First, you outline the suitable AWS Identification and Entry Administration (IAM) permissions to write down to and from OpenSearch Ingestion. You then arrange the pipeline configuration within the OpenSearch Ingestion. Let’s discover every step in additional element.

Configure IAM permissions

OpenSearch Ingestion works with IAM to safe communications into and out of OpenSearch Ingestion. You want two roles, authenticated utilizing AWS Signature V4 (SigV4) signed requests. The originating entity requires permissions to write down to OpenSearch Ingestion. OpenSearch Ingestion requires permissions to write down to your OpenSearch Service area. Lastly, you could create an entry coverage utilizing OpenSearch Service’s fine-grained entry management, which permits OpenSearch Ingestion to create indexes and write to them in your area.

The next diagram illustrates the IAM permissions to permit OpenSearch Ingestion to write down to an OpenSearch Service area. Check with Organising roles and customers in Amazon OpenSearch Ingestion to get extra particulars on roles and permissions required to make use of OpenSearch Ingestion.

Within the demo, you employ the AWS Cloud9 EC2 occasion profile’s credentials to signal requests despatched to OpenSearch Ingestion. You employ Fluent Bit to fetch the credentials and assume the function you cross within the aws_role_arn you configure later.

1. Create an ingestion function (known as IngestionRole) to permit Fluent Bit to ingest the logs into your pipeline.

Create a belief relationship to permit Fluent Bit to imagine the ingestion function, as proven within the following code. Fluent Bit makes an attempt to fetch the credentials within the following order. In configuring the entry coverage for this function, you grant permission for the osis:Ingest.

{
    "Model": "2012-10-17",
    "Assertion": [
        {
            "Effect": "Allow",
            "Principal": {
                "AWS": "{your-account-id}"
            },
            "Action": "sts:AssumeRole"
        }
    ]
}

2. Create a pipeline function (known as PipelineRole) with a belief relationship for OpenSearch Ingestion to imagine that function. The domain-level entry coverage of the OpenSearch area grants the pipeline function entry to the area.

3. Lastly, configure your area’s safety plugin to allow OpenSearch Ingestion’s assumed function to create indexes and write information to the area.

On this demo, the OpenSearch Service area makes use of fine-grained entry management for authentication, so you want to map the OpenSearch Ingestion pipeline function to the OpenSearch backend function all_access. For directions, check with Step 2: Embrace the pipeline function within the area entry coverage web page.

Create the pipeline in OpenSearch Ingestion

To create an OpenSearch Ingestion pipeline, full the next steps:

1. On the OpenSearch Service console, select Pipelines within the navigation pane.
2. Select Create pipeline.
3. For Pipeline identify, enter a reputation.

4. Enter the minimal and most Ingestion OpenSearch Compute Items (Ingestion OCUs). On this instance, we use the default pipeline capability settings of minimal 1 Ingestion OCU and most 4 Ingestion OCUs.

Every OCU is a mix of roughly 8 GB of reminiscence and a couple of vCPUs that may deal with an estimated 8 GiB per hour. OpenSearch Ingestion helps as much as 96 OCUs, and it robotically scales up and down primarily based in your ingest workload demand.

5. Within the Pipeline configuration part, configure Information Prepper to course of your information by selecting the suitable blueprint configuration template on the Configuration blueprints menu. For this submit, we select AWS-LogAggregationWithConditionalRouting.

The OpenSearch Ingestion pipeline configuration consists of 4 sections:

• Supply – That is the enter element of a pipeline. It defines the mechanism by which a pipeline consumes information. On this submit, you employ the http_source plugin and supply the Fluent Bit output URI worth inside the path attribute.
• Processors – This represents an intermediate processing to filter, rework, and enrich your enter information. Check with Supported plugins for extra particulars on the record of operations supported in OpenSearch Ingestion. On this submit, we use the grok processor COMMONAPACHELOG, which matches enter logs in opposition to the frequent Apache log sample and makes it simple to question in OpenSearch Service.
• Sink – That is the output element of a pipeline. It defines a number of locations to which a pipeline publishes information. On this submit, you outline an OpenSearch Service area and index as sink.
• Route – That is the a part of a processor that enables the pipeline to route the information into totally different sinks primarily based on particular circumstances. On this instance, you create 4 routes primarily based within the response discipline worth of the log. If the response discipline worth of the log line matches 2xx or 3xx, the log is distributed to the OpenSearch Service index aggregated_2xx_3xx. If the response discipline worth matches 4xx, the log is distributed to the index aggregated_4xx. If the response discipline worth matches 5xx, the log is distributed to the index aggregated_5xx.

6. Replace the blueprint primarily based in your use case. The next code exhibits an instance of the pipeline configuration YAML file:

model: "2"
log-aggregate-pipeline:
  supply:
    http:
      # Present the FluentBit output URI worth.
      path: "/log/ingest"
  processor:
    - date:
        from_time_received: true
        vacation spot: "@timestamp"
    - grok:
        match:
          log: [ "%{COMMONAPACHELOG_DATATYPED}" ]
  route:
    - 2xx_status: "/response >= 200 and /response < 300"
    - 3xx_status: "/response >= 300 and /response < 400"
    - 4xx_status: "/response >= 400 and /response < 500"
    - 5xx_status: "/response >= 500 and /response < 600"
  sink:
    - opensearch:
        # Present an AWS OpenSearch Service area endpoint
        hosts: [ "{your-domain-endpoint}" ]
        aws:
          # Present a Function ARN with entry to the area. This function ought to have a belief relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "arn:aws:iam::{your-account-id}:function/PipelineRole"
          # Present the area of the area.
          area: "{AWS_Region}"
        index: "aggregated_2xx_3xx"
        routes:
          - 2xx_status
          - 3xx_status
    - opensearch:
        # Present an AWS OpenSearch Service area endpoint
        hosts: [ "{your-domain-endpoint}"  ]
        aws:
          # Present a Function ARN with entry to the area. This function ought to have a belief relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "arn:aws:iam::{your-account-id}:function/PipelineRole"
          # Present the area of the area.
          area: "{AWS_Region}"
        index: "aggregated_4xx"
        routes:
          - 4xx_status
    - opensearch:
        # Present an AWS OpenSearch Service area endpoint
        hosts: [ "{your-domain-endpoint}"  ]
        aws:
          # Present a Function ARN with entry to the area. This function ought to have a belief relationship with osis-pipelines.amazonaws.com
          sts_role_arn: "arn:aws:iam::{your-account-id}:function/PipelineRole"
          # Present the area of the area.
          area: "{AWS_Region}"
        index: "aggregated_5xx"
        routes:
          - 5xx_status

Present the related values on your area endpoint, account ID, and Area associated to your configuration.

7. Test the well being of your configuration setup by selecting Validate pipeline if you end the replace.

When designing a manufacturing workload, deploy your pipeline inside a VPC. For directions, check with Securing Amazon OpenSearch Ingestion pipelines inside a VPC.

8. For this submit, choose Public entry underneath Community.

9. Within the Log publishing choices part, choose Publish to CloudWatch logs and Create new group.

OpenSearch Ingestion makes use of the log ranges of INFO, WARN, ERROR, and FATAL. Enabling log publishing helps you monitor your pipelines in manufacturing.

10. Select Subsequent and Create pipeline.
11. Choose the pipeline and select View particulars to see the progress of the pipeline creation.

Wait till the standing adjustments to Lively to start out utilizing the pipeline.

Ship logs to the OpenSearch Ingestion pipeline

To begin sending logs to the OpenSearch Ingestion pipeline, full the next steps:

1. On the AWS Cloud9 console, create a Fluent Bit configuration file and replace the next attributes:
   • Host – Enter the ingestion URL of your OpenSearch Ingestion pipeline.
   • aws_service – Enter osis.
   • aws_role_arn – Enter the ARN of the IAM function IngestionRole.

The next code exhibits an instance of the Fluent-bit.conf file:

[SERVICE]
    parsers_file          ./parsers.conf
    
[INPUT]
    identify                  tail
    refresh_interval      5
    path                  /var/log/*.log
    read_from_head        true
[FILTER]
    Identify parser
    Key_Name log
    Parser apache
[OUTPUT]
    Identify http
    Match *
    Host {Ingestion URL}
    Port 443
    URI /log/ingest
    format json
    aws_auth true
    aws_region {AWS_region}
    aws_role_arn arn:aws:iam::{your-account-id}:function/IngestionRole
    aws_service osis
    Log_Level hint
    tls On

2. Within the AWS Cloud9 setting, create a docker-compose YAML file to deploy Fluent Bit and Flog containers:

model: '3'
companies:
  fluent-bit:
    container_name: fluent-bit
    picture: docker.io/amazon/aws-for-fluent-bit
    volumes:
      - ./fluent-bit.conf:/fluent-bit/and many others/fluent-bit.conf
      - ./apache-logs:/var/log
  flog:
    container_name: flog
    picture: mingrammer/flog
    command: flog -t log -f apache_common -o net/log/check.log -w -n 100000 -d 1ms -p 1000
    volumes:
      - ./apache-logs:/net/log

Earlier than you begin the Docker containers, you want to replace the IAM EC2 occasion function in AWS Cloud9 so it could possibly signal the requests despatched to OpenSearch Ingestion.

3. For demo functions, create an IAM service-linked function and select EC2 underneath Use case to permit the AWS Cloud9 EC2 occasion to name OpenSearch Ingestion in your behalf.
4. Add the OpenSearch Ingestion coverage, which is similar coverage you used with IngestionRole.
5. Add the AdministratorAccess permission coverage to the function as nicely.

Your function definition ought to appear to be the next screenshot.

6. After you create the function, return to AWS Cloud9, choose your demo setting, and select View particulars.
7. On the EC2 occasion tab, select Handle EC2 occasion to view the main points of the EC2 occasion hooked up to your AWS Cloud9 setting.

8. On the Amazon EC2 console, change the IAM function of your AWS Cloud9 EC2 occasion with the brand new function.
9. Open a terminal in AWS Cloud9 and run the command docker-compose up.

Test the output within the terminal—if every thing is working accurately, you get standing 200.

Fluent Bit collects logs from the /var/log repository within the container and pushes the information to the OpenSearch Ingestion pipeline.

10. Open OpenSearch Dashboards, navigate to Dev Instruments, and run the command GET _cat/indices to validate that the information has been delivered by OpenSearch Ingestion to your OpenSearch Service area.

You must see the three indexes created: aggregated_2xx_3xx, aggregated_4xx, and aggregated_5xx.

Now you possibly can concentrate on analyzing your log information and reinvent your enterprise with out having to fret about any operational overhead concerning your ingestion pipeline.

Greatest practices for monitoring

You possibly can monitor the Amazon CloudWatch metrics made obtainable to you to take care of the proper efficiency and availability of your pipeline. Test the record of accessible pipeline metrics associated to the supply, buffer, processor, and sink plugins.

Navigate to the Metrics tab on your particular OpenSearch Ingestion pipeline to discover the graphs obtainable to every metric, as proven within the following screenshot.

In your manufacturing workloads, be certain that to configure the next CloudWatch alarms to inform you when the pipeline metrics breach a particular threshold so you possibly can promptly remediate every concern.

Managing value

Whereas OpenSearch Ingestion robotically provisions and scales the OCUs on your spiky workloads, you solely pay for the compute sources actively utilized by your pipeline to ingest, course of, and route information. Due to this fact, establishing a most capability of Ingestion OCUs permits you to deal with your workload peak demand whereas controlling value.

For manufacturing workloads, be certain that to configure a minimal of two Ingestion OCUs to make sure 99.9% availability for the ingestion pipeline. Test the sizing suggestions and find out how OpenSearch Ingestion responds to workload spikes.

Clear up

Be sure to clear up undesirable AWS sources created throughout this submit so as to stop extra billing for these sources. Comply with these steps to scrub up your AWS account:

1. On the AWS Cloud9 console, select Environments within the navigation pane.
2. Choose the setting you wish to delete and select Delete.
3. On the OpenSearch Service console, select Domains underneath Managed clusters within the navigation pane.
4. Choose the area you wish to delete and select Delete.
5. Choose Pipelines underneath Ingestion within the navigation pane.
6. Choose the pipeline you wish to delete and on the Actions menu, select Delete.

Conclusion

On this submit, you discovered the way to create a serverless ingestion pipeline to ship Apache entry logs to an OpenSearch Service area utilizing OpenSearch Ingestion. You discovered the IAM permissions required to start out utilizing OpenSearch Ingestion and the way to use a pipeline blueprint as a substitute of making a pipeline configuration from scratch.

You used Fluent Bit to gather and ahead Apache logs, and used OpenSearch Ingestion to course of and conditionally route the log information to totally different indexes in OpenSearch Service. For extra examples about writing to OpenSearch Ingestion pipelines, check with Sending information to Amazon OpenSearch Ingestion pipelines.

Lastly, the submit offered you with suggestions and greatest practices to deploy OpenSearch Ingestion pipelines in a manufacturing setting whereas controlling value.

Comply with this submit to construct your serverless log analytics pipeline, and check with Prime methods for prime quantity tracing with Amazon OpenSearch Ingestion to study extra about excessive quantity tracing with OpenSearch Ingestion.

Concerning the authors

Hajer Bouafif is an Analytics Specialist Options Architect at Amazon Internet Providers. She focuses on OpenSearch Service and helps prospects design and construct well-architected analytics workloads in numerous industries. Hajer enjoys spending time outside and discovering new cultures.

Francisco Losada is an Analytics Specialist Options Architect primarily based out of Madrid, Spain. He works with prospects throughout EMEA to architect, implement, and evolve analytics options at AWS. He advocates for OpenSearch, the open-source search and analytics suite, and helps the neighborhood by sharing code samples, writing content material, and talking at conferences. In his spare time, Francisco enjoys enjoying tennis and working.

Muthu Pitchaimani is a Search Specialist with Amazon OpenSearch Service. He builds large-scale search purposes and options. Muthu is within the matters of networking and safety, and is predicated out of Austin, Texas.