Importing historic tools knowledge into AWS IoT SiteWise

Introduction

AWS IoT SiteWise is a managed service that helps clients accumulate, retailer, arrange and monitor knowledge from their industrial tools at scale. Prospects usually have to carry their historic tools measurement knowledge from current techniques corresponding to knowledge historians and time collection databases into AWS IoT SiteWise for guaranteeing knowledge continuity, coaching synthetic intelligence (AI) & machine studying (ML) fashions that may predict tools failures, and deriving actionable insights.

On this weblog submit, we are going to present how one can get began with the BulkImportJob API and import historic tools knowledge into AWS IoT SiteWise utilizing a code pattern.

You need to use this imported knowledge to achieve insights via AWS IoT SiteWise Monitor and Amazon Managed Grafana, practice ML fashions on Amazon Lookout for Tools and Amazon SageMaker, and energy analytical purposes.

To start a bulk import, clients have to add a CSV file to Amazon Easy Storage Service (Amazon S3) containing their historic knowledge in a predefined format. After importing the CSV file, clients can provoke the asynchronous import to AWS IoT SiteWise utilizing the CreateBulkImportJob operation, and monitor the progress utilizing the DescribeBulkImportJob and ListBulkImportJob operations.

Stipulations

To comply with via this weblog submit, you have to an AWS account and an AWS IoT SiteWise supported area. If you’re already utilizing AWS IoT SiteWise, select a unique area for an remoted surroundings. You might be additionally anticipated to have some familiarity with Python.

Setup the surroundings

1. Create an AWS Cloud9 surroundings utilizing Amazon Linux 2 platform
2. Utilizing the terminal in your Cloud9 surroundings, set up Git and clone the sitewise-bulk-import-example repository from Github

   sudo yum set up git
   git clone https://github.com/aws-samples/aws-iot-sitewise-bulk-import-example.git
   cd aws-iot-sitewise-bulk-import-example
   pip3 set up -r necessities.txt

Walkthrough

For the demonstration on this submit, we are going to use an AWS Cloud9 occasion to signify an on-premises developer workstation and simulate two months of historic knowledge for just a few manufacturing strains in an car manufacturing facility.

We are going to then put together the info and import it into AWS IoT SiteWise at scale, leveraging a number of bulk import jobs. Lastly, we are going to confirm whether or not the info was imported efficiently.

A bulk import job can import knowledge into the 2 storage tiers supplied by AWS IoT SiteWise, relying on how the storage is configured. Earlier than we proceed, allow us to first outline these two storage tiers.

Scorching tier: Shops regularly accessed knowledge with decrease write-to-read latency. This makes the recent tier ultimate for operational dashboards, alarm administration techniques, and some other purposes that require quick entry to the current measurement values from tools.

Chilly tier: Shops less-frequently accessed knowledge with larger learn latency, making it ultimate for purposes that require entry to historic knowledge. As an example, it may be utilized in enterprise intelligence (BI) dashboards, synthetic intelligence (AI), and machine studying (ML) coaching. To retailer knowledge within the chilly tier, AWS IoT SiteWise makes use of an S3 bucket within the buyer’s account.

Retention Interval: Determines how lengthy your knowledge is saved within the sizzling tier earlier than it’s deleted.

Now that we realized concerning the storage tiers, allow us to perceive how a bulk import job handles writes for various eventualities. Check with the desk beneath:

Within the subsequent part, we are going to comply with step-by-step directions to import historic tools knowledge into AWS IoT SiteWise.

Steps to import historic knowledge

Step 1: Create a pattern asset hierarchy

For the aim of this demonstration, we are going to create a pattern asset hierarchy for a fictitious car producer with operations throughout 4 totally different cities. In a real-world situation, it’s possible you’ll have already got an current asset hierarchy in AWS IoT SiteWise, during which case this step is non-compulsory.

Step 1.1: Evaluate the configuration

1. From terminal, navigate to the foundation of the Git repo.
2. Evaluate the configuration for asset fashions and property.
   cat config/assets_models.yml
3. Evaluate the schema for asset properties.
   cat schema/sample_stamping_press_properties.json

Step 1.2: Create asset fashions and property

1. Run python3 src/create_asset_hierarchy.py to routinely create asset fashions, hierarchy definitions, property, asset associations.
2. Within the AWS Console, navigate to AWS IoT SiteWise, and confirm the newly created Fashions and Belongings.
3. Confirm that you just see the asset hierarchy much like the one beneath.

Step 2: Put together historic knowledge

Step 2.1: Simulate historic knowledge

On this step, for demonstration objective, we are going to simulate two months of historic knowledge for 4 stamping presses throughout two manufacturing strains. In a real-world situation, this knowledge would sometimes come from supply techniques corresponding to knowledge historians and time collection databases.

The CreateBulkImportJob API has the next key necessities:

• To determine an asset property, you have to to specify both an ASSET_ID + PROPERTY_ID mixture or the ALIAS.On this weblog, we will probably be utilizing the previous.
• The information must be in CSV format.

Observe the steps beneath to generate knowledge in line with these expectations. For extra particulars concerning the schema, seek advice from Ingesting knowledge utilizing the CreateBulkImportJob API.

1. Evaluate the configuration for knowledge simulation.
   cat config/data_simulation.yml
2. Run python3 src/simulate_historical_data.py to generate simulated historic knowledge for the chosen properties and time interval. If the entire rows exceed rows_per_job as configured in bulk_import.yml, a number of knowledge recordsdata will probably be created to help parallel processing. On this pattern, about 700,000+ knowledge factors are simulated for the 4 stamping presses (A-D) throughout two manufacturing strains (Sample_Line 1 and Sample_Line 2). Since we configured rows_per_job as 20,000, a complete of 36 knowledge recordsdata will probably be created.
3. Confirm the generated knowledge recordsdata beneath knowledge listing.
4. The information schema will comply with the column_names configured in bulk_import.yml config file.

   79817017-bb13-4611-b4b2-8094913cd287,c487c0d7-a9f2-4fe7-b4bc-92bf6f4f697b,DOUBLE,1667275200,0,GOOD,78.76
   79817017-bb13-4611-b4b2-8094913cd287,c487c0d7-a9f2-4fe7-b4bc-92bf6f4f697b,DOUBLE,1667275260,0,GOOD,67.33
   79817017-bb13-4611-b4b2-8094913cd287,c487c0d7-a9f2-4fe7-b4bc-92bf6f4f697b,DOUBLE,1667275320,0,GOOD,82.13
   79817017-bb13-4611-b4b2-8094913cd287,c487c0d7-a9f2-4fe7-b4bc-92bf6f4f697b,DOUBLE,1667275380,0,GOOD,72.72
   79817017-bb13-4611-b4b2-8094913cd287,c487c0d7-a9f2-4fe7-b4bc-92bf6f4f697b,DOUBLE,1667275440,0,GOOD,61.45

Step 2.2: Add historic knowledge to Amazon S3

As AWS IoT SiteWise requires the historic knowledge to be out there in Amazon S3, we are going to add the simulated knowledge to the chosen S3 bucket.

1. Replace the info bucket beneath bulk_import.yml with any current non permanent S3 bucket that may be deleted later.
2. Run python3 src/upload_to_s3.py to add the simulated historic knowledge to the configured S3 bucket.
3. Navigate to Amazon S3 and confirm the objects had been uploaded efficiently.

Step 3: Import historic knowledge into AWS IoT SiteWise

Earlier than you possibly can import historic knowledge, AWS IoT SiteWise requires that you just allow Chilly tier storage. For added particulars, seek advice from Configuring storage settings.

You probably have already activated chilly tier storage, think about modifying the S3 bucket to a brief one which will be later deleted whereas cleansing up the pattern sources.

Observe that by altering the S3 bucket, not one of the knowledge from current chilly tier S3 bucket is copied to the brand new bucket. When modifying S3 bucket location, make sure the IAM function configured beneath S3 entry function has permissions to entry the brand new S3 bucket.

Step 3.1: Configure storage settings

1. Navigate to AWS IoT SiteWise, choose Storage, then choose Activate chilly tier storage.
2. Decide an S3 bucket location of your selection.
3. Choose Create a job from an AWS managed template.
4. Test Activate retention interval, enter 30 days, and save.

Step 3.2: Present permissions for AWS IoT SiteWise to learn knowledge from Amazon S3

1. Navigate to AWS IAM, choose Insurance policies beneath Entry administration, and Create coverage.
2. Swap to JSON tab and change the content material with the next. Replace <bucket-name> with the identify of knowledge S3 bucket configured in bulk_import.yml.

   {
     "Model": "2012-10-17",
     "Assertion": [
       {
         "Effect": "Allow",
         "Action": [
           "s3:*"
         ],
         "Useful resource": ["arn:aws:s3:::<bucket-name>"]
       }
     ]
   }

3. Save the coverage with Identify as SiteWiseBulkImportPolicy.
4. Choose Roles beneath Entry administration, and Create function.
5. Choose Customized belief coverage and change the content material with the next.

   {
     "Model": "2012-10-17",
     "Assertion": [
       {
         "Sid": "",
         "Effect": "Allow",
         "Principal": {
           "Service": "iotsitewise.amazonaws.com"
          },
       "Action": "sts:AssumeRole"
       }
     ]
   }

6. Click on Subsequent and choose the SiteWiseBulkImportPolicy IAM coverage created within the earlier steps.
7. Click on Subsequent and create the function with Position identify as SiteWiseBulkImportRole.
8. Choose Roles beneath Entry administration, seek for the newly created IAM function SiteWiseBulkImportRole, and click on on its identify.
9. Copy the ARN of the IAM function utilizing the copy icon.

Step 3.3: Create AWS IoT SiteWise bulk import jobs

1. Substitute the role_arn discipline in config/bulk_import.yml with the ARN of SiteWiseBulkImportRole IAM function copied in earlier steps.
2. Replace the config/bulk_import.yml file:
   • Substitute the role_arn with the ARN of SiteWiseBulkImportRole IAM function.
   • Substitute the error_bucket with any current non permanent S3 bucket that may be deleted later.
3. Run python3 src/create_bulk_import_job.py to import historic knowledge from the S3 bucket into AWS IoT SiteWise:
4. The script will create a number of jobs to concurrently import all the info recordsdata created into AWS IoT SiteWise. In a real-world situation, a number of terabytes of knowledge will be rapidly imported into AWS IoT SiteWise utilizing concurrently operating jobs.
5. Test the standing of jobs from the output:

   Complete S3 objects: 36
   Variety of bulk import jobs to create: 36
           Created job: 03e75fb2-1275-487f-a011-5ae6717e0c2e for importing knowledge from knowledge/historical_data_1.csv S3 object
           Created job: 7938c0d2-f177-4979-8959-2536b46f91b3 for importing knowledge from knowledge/historical_data_10.csv S3 object
           …
   Checking job standing each 5 secs till completion.
           Job id: 03e75fb2-1275-487f-a011-5ae6717e0c2e, standing: COMPLETED
           Job id: 7938c0d2-f177-4979-8959-2536b46f91b3, standing: COMPLETED
           …

6. Should you see the standing of any job as COMPLETED_WITH_FAILURES or FAILED, seek advice from Troubleshoot frequent points part.

Step 4: Confirm the imported knowledge

As soon as the majority import jobs are accomplished, we have to confirm if the historic knowledge is efficiently imported into AWS IoT SiteWise. You may confirm the info both by straight wanting on the chilly tier storage or by visually inspecting the charts out there in AWS IoT SiteWise Monitor.

Step 4.1: Utilizing the chilly tier storage

On this step, we are going to examine if new S3 objects have been created within the bucket that was configured for chilly tier.

1. Navigate to Amazon S3 and find the S3 bucket configured beneath AWS IoT SiteWise → Storage → S3 bucket location (in Step 3) for chilly tier storage.
2. Confirm the partitions and objects beneath the uncooked/ prefix.

Step 4.2: Utilizing AWS IoT SiteWise Monitor

On this step, we are going to visually examine if the charts present knowledge for the imported date vary.

1. Navigate to AWS IoT SiteWise and find Monitor.
2. Create a portal to entry knowledge saved in AWS IoT SiteWise.
   • Present AnyCompany Motor because the Portal identify.
   • Select IAM for Person authentication.
   • Present your e-mail tackle for Assist contact e-mail, and click on Subsequent.
   • Go away the default configuration for Extra options, and click on Create.
   • Beneath Invite directors, choose your IAM person or IAM Position, and click on Subsequent.
   • Click on on Assign Customers.
3. Navigate to Portals and open the newly created portal.
4. Navigate to Belongings and choose an asset, for instance, AnyCompany_Motor → Sample_Arlington → Sample_Stamping → Sample_Line 1 → Sample_Stamping Press A.
5. Use Customized vary to match the date vary for the info uploaded.
6. Confirm the info rendered within the time collection line chart.

Troubleshoot frequent points

On this part, we are going to cowl the frequent points encountered whereas importing knowledge utilizing bulk import jobs and spotlight some doable causes.

If a bulk import job isn’t efficiently accomplished, it’s best follow to seek advice from logs within the error S3 bucket configured in bulk_import.yml and perceive the foundation trigger.

No knowledge imported

• Incorrect schema: dataType doesn't match dataType tied to the asset-property
  The schema supplied at Ingesting knowledge utilizing the CreateBulkImportJob API needs to be adopted precisely. Utilizing the console, confirm the supplied DATA_TYPE supplied matches with the info sort within the corresponding asset mannequin property.
• Incorrect ASSET_ID or PROPERTY_ID: Entry isn't modeled
  Utilizing the console, confirm the corresponding asset and property exists.
• Duplicate knowledge: A worth for this timestamp already exists
  AWS IoT SiteWise detects and routinely discards any duplicate. Utilizing console, confirm if the info already exists.

Lacking solely sure components of knowledge

• Lacking current knowledge: BulkImportJob API imports the current knowledge (that falls inside the sizzling tier retention interval) into AWS IoT SiteWise sizzling tier and doesn’t switch it instantly to Amazon S3 (chilly tier). Chances are you’ll want to attend for the following sizzling to chilly tier switch cycle, which is at present set to six hours.

Clear Up

To keep away from any recurring costs, take away the sources created on this weblog. Observe the steps to delete these sources:

1. Navigate to AWS Cloud9 and delete your surroundings.
2. Run python3 src/clean_up_asset_hierarchy.py to delete the next sources, so as, from AWS IoT SiteWise:
   • Asset associations
   • Belongings
   • Hierarchy definitions from asset fashions
   • Asset fashions
3. From AWS IoT SiteWise console, navigate to Monitor → Portals, choose the beforehand created portal, and delete.
4. Navigate to Amazon S3 and carry out the next:
   • Delete the S3 bucket location configured beneath the Storage part of AWS IoT SiteWise
   • Delete the info and error buckets configured within the /config/bulk_import.yml of Git repo

Conclusion

On this submit, you’ve realized learn how to use the AWS IoT SiteWise BulkImportJob API to import historic tools knowledge into AWS IoT SiteWise utilizing AWS Python SDK (Boto3). You too can use the AWS CLI or SDKs for different programming languages to carry out the identical operation. To study extra about all supported ingestion mechanisms for AWS IoT SiteWise, go to the documentation.

Concerning the authors

Raju Gottumukkala is an IoT Specialist Options Architect at AWS, serving to industrial producers of their good manufacturing journey. Raju has helped main enterprises throughout the vitality, life sciences, and automotive industries enhance operational effectivity and income progress by unlocking true potential of IoT knowledge. Previous to AWS, he labored for Siemens and co-founded dDriven, an Trade 4.0 Knowledge Platform firm.

Avik Ghosh is a Senior Product Supervisor on the AWS Industrial IoT staff, specializing in the AWS IoT SiteWise service. With over 18 years of expertise in expertise innovation and product supply, he focuses on Industrial IoT, MES, Historian, and large-scale Trade 4.0 options. Avik contributes to the conceptualization, analysis, definition, and validation of Amazon IoT service choices.