Step-by-Step Guide To Store AWS IoT data into S3 Bucket
TABLE OF CONTENT
1. Introduction2. High-Level Architectural Diagram3. Step-by-Step Guide4. Use Cases5. Conclusion 6. CloudThat 7. FAQs1. Introduction
Real-time IoT applications like home automation, CCTV Camera, asset tracking, agricultural production climate monitoring and vehicle tracking have all increased in popularity. These IoT applications for commercial and industrial purposes generate huge amounts of data. The downside is that storing the data in a database for a long time can be costly and have a negative impact on your business.
These massive amounts of data can be stored in AWS S3 (Simple Storage Service) to reduce costs. AWS S3 (Object Storage Service) is an object storage service that offers industrial data storage at a low cost, high availability, scalability and security.
Here are some references to AWS Cloud and AWS IoT Services.
Step-by-Step Guide to Register a New IoT Device in AWS
AWS IoT 1 Click: AWS IoT New Service To Enable Simple Devices
Uploading Files to AWS S3 Bucket Using Html-Form Using A Python CGI script
I will demonstrate how to use AWS IoT Rules to store IoT device information in AWS S3.
2. High-Level Architectural Diagram
3. Step-by-Step Guide:
1. Make a S3 bucket:
Log in to your AWS account and type S3 in the search box. Next, select S3 from services menu.
Click the Create bucket option in the left-hand menu of the AWS S3 console.
Enter the Bucket name as s3-bucket-for-iot-data and select an AWS region as you prefeed to ensure that AWS IoT services and S3 are both in the same region. I have chosen US East (N. Virginia), us-east-1 as my area
Don’t bother with the rest of these options and just click Create bucket at bottom
We can see that the bucket has been successfully created with the name s3-bucket-for-iot-data
2. To send data to the S3 using IoT rules, create an IoT rule
Type IoT Core in the search box and then select IoT Core from your services selection.
Click on the left blade of the IoT Core console to click Act. Then click on the right blade to click rule. To transfer IoT data from the IoT Core console to the S3 bucket, click Create in Rules.
Enter IoT_data_rule_for_S3 in the name field and describe the rule in section description.
Select 2016-03-23 as the SQL version in the Rule query statement and input the Rule Query statement as SELECT * FROM ‘iotdevice/+/datas3’, where * denotes reading the whole data sequence from the IoT topic iotdevice/+/datas3 and + denotes a wildcard character.
Select Add action to add an action to a Rule
Click on Add an Action, then select an action blade and click Configure action at bottom of page.
In the section “Configure action” Choose the name of the S3 generated in the previous step, which is s3-bucket-for-iot-data, for S3 bucket name. In the key, type $cast(topic(2) AS DECIMAL)/$timestamp() to help differentiate the data over devices id and time. $cast(2) AS DECIMAL is the device id that is in the IoT topic. $timestamp() is the current time of decimal value.
To send data to S3 using AWS IoT rules, we need to give role click Create Role
Enter Iotdata_S3_role in the Create a New Role console and choose the Create Role option
Short period Iotdata_S3_role created with attached policy. Click Add action to add the position
To create a rule, click Create rule at the bottom on the Create a Rule page. As an example, we can see the IoT_data_rule_for_S3 is created.
3. Test IoT rule & S3:
To test an IoT Rule, go to AWS IoT Core dashboard. Next, scroll to the left blade and select Test. Then, click the MQTT Test Client option. Click the publish button on the MQTT Client page. Enter iotdevice/55/datas