Internet of Things Event Data Analytics
Perform real-time analytics on the streaming data from variety of IoT devices on cloud
- Build a proof of concept to ingest, combine, process and analyze massive amounts of data from variety of devices such as the following:
- AWS IoT button – single, double and long click event.
- Globalscale starter kit for temperature control.
- Custom mobile app that tracks activity of the user such as latitude, longitude, call log details, installed apps, mobile system usage, etc
- Sensor programs that generates events based on server resource usage, and consumers connected to PubNub streaming data.
- Manage relationships and data between devices, and securely connect devices and cloud applications.
- Perform analytics on streaming data with different rolling windows and persist in AWS storage resources for further analysis.
- Continuously look for outliers by triggering EMR jobs on the data stored in S3 and perform machine learning algorithms to train models and patterns.
- The complete stack was set up on AWS cloud by using different IoT service offerings from AWS.
- Registered things and associated policies and certificates to each thing to exchange information with AWS IoT ecosystem.
- Created multiple IoT Rules to pass inbound data to filter, transform and then pass it to other Kinesis Firehose and Streaming.
- Configured necessary ElasticSearch cluster and create respective index with fields mapping to persist sensor data into the index.
- Created Kinesis Fireshose to capture, transform, and load streaming data into S3 and ElasticSearch.
- Implemented Kinesis analytics by consuming data from Kinesis Firehose and performed transformation and windowing queries such as tumbling, sliding, and other custom analytics.
- Data from Kinesis analytics is streamed into Kinesis streams and Lambda expression consumes that stream and store it in DynamoDB, Aurora, and S3.
- EMR with Spark clusters are launched and connected to the S3 that contains sensor data and performed some machine learning algorithms such as outlier detection, classifying streaming data into different categories, and created ML model from the data.
- Note:This work was done as a proof of concept for a customer business proposal to their investors.
THE SOLUTION DIAGRAM: