The internet of things is one of my favorite topics. IOT enables low-powered connected devices that opens gateways from the digital to the real world. While I love tinkering away with an Arduino sketch and the latest Espressif or Arduino board, there is always an air of frustration when trying to build out what at first seems like simple functionality using one of these “smart devices” because of the limited view we have into their operations.
This is where observability and tracing can alleviate some of these woes and help identify issues with how data is flowing into the application. Particularly, once the devices are connected and MQTT broker messages disappear into the cloud ether (or sometimes don’t arrive at all). In this blog, I’ll use an IOT device built with AWS Lambda (come see it at Lumigo’s booth at this year’s AWS re:Invent) to demonstrate how to create a connected clicker that triggers different Lambda functions from multiple fast board clicks, and monitor it using Lumigo.
Let’s talk briefly about IOT boards
The first step with any IOT adventure is choosing a board, making sure it works and that you can upload code to it. For this I opted for an ESP32D devkit that I had left over from a past project.
Starting with the Hello world of IOT, the blinking light, I was up and running with sketch and serial code being passed back and forth. To add some complexity to that, the default Arduino Blink sketch uses `LED_BUILTIN` as a LED reference. This also assumes that the board the sketch goes onto has pinout mapping for it, which does not work on many of the Espressif devkit boards.
But it’s nothing a breadboard, a LED and some wires can’t sort out.
All good IOT adventures begin with a blinky light.
Once the wifi was connected and returning a network IP address to serial monitor–always a good sign that it is connected– the next step is to authenticate it with the AWS gateway, using the 3 certificate authentication method.
One of the frustrations I’ve always found with connecting IOT devices is that you don’t always know what they are doing. This was one of those moments, and while I could get some serial output on the device, there were no errors or issues displaying from the connection itself nor on the server side.
Stepping into trace and debug mode to investigate further, I set up a Lambda function and routed the IOT device messaging using the AWS IOT message routing functionality. I used Lumigo’s LiveTail I was then able to see that traces were not being invoked.
The next troubleshooting step was to verify the connection using the AWS Device Advisor test suite. I was able to identify that there was a SSL handshake issue in the connection, which is a known issue with older ESP32 devices.
Switching boards out for an Arduino MKR quickly sorted this out. Many newer IOT boards, like the Arduino MKR series, now have built in cryptography chips to generate X.509 certificates which then get uploaded to AWS.
Observing the IOT flow
With the IOT board now connected, it was time to start triggering some Lambda’s via the IOT device. Functionality wise, I want to have a single button trigger a number of different functions, using a sequence of clicks to invoke lambda functions as needed.
One thing I want to mention here is the way the AWS IOT message routing rules work using AWS IOT SQL. Using this, I was able to route specific requests to specific services, based on received IOT broker messages.
Handy Tip: remember to put single quotes around the broker topic in the AWS IOT SQL query.
Publishing from device to the `clicker` IOT topic I can then use select * from ‘clicker’, along with any specific message filtering as part of the where clause. Then define the service that it needed to go to.
Using that approach there were a few ways to then tackle the clicker task:
IOT clicks via SQS
From the board, this method would involve just sending a click through to the IOT broker and then have the logic be handled on the cloud side. This would mean multiple click logic could be handled server side, which seems simple in theory.
The data structure from the board itself was super simple, with the broker sequence number keeping track of the number of click messages coming in.
Setting up a SQS queue as part of a IOT routing rule that passed all incoming traffic straight into the SQS. The approach here was to use SQS as a Lambda trigger to batch process from the queue, determine the highest sequence and then send a message into a SNS queue to be handled by other functions set up on a SNS queue.
I wasn’t expecting more than 3 clicks from the device at a time so I set the batch size to 5 (just to be sure) and the batch window to 3 to allow enough time to be processed.
With everything in place, I switched over to Lumigo Live Tail to see the results and triggered a few clicks. The first few times it worked with two messages arriving as one invocation into the two-click flow.
However, upon further testing using Lumigo, it wasn’t long before I was seeing batches processing only one record from the device per invocation. This was due to a few issues in latency between the queues, and one of the quirky features around Lambda batching using five parallel polling.
So with that ruled out, it was time for plan B to handle the click logic.
IOT clicks with the board
While somewhat of a hard-coded approach, moving the click logic back to the board itself did resolve the above issue. From the Arduino sketch side, this is just using a timer to manage the click number that was sent server side, then sending that number as part of the broker message.
The rest of the flow could be handled a few ways. I decided to use the IOT routing based on the message click number, and just have it route to different Lambdas depending on the number using this as the IOT SQL.
SELECT message, sequence
FROM 'clicker'
WHERE message.click = 2
Because I’m sending the data using the Arduino Json Library it is available as a query filter item within Amazon’s IOT SQL. Any matching records that arrive via the broker into the clicker topic will be passed on to the nominated Lambda.
Even better is that as I build out each click function further, I can end-to-end trace through each service as needed.
Start monitoring your IOT Lambdas
Building out IOT devices can be painstaking, especially beyond the board and into the cloud. Tracing messages through the broker and beyond can leave you wrangling logs and hunting connections between services.
Sign up for Lumigo now and to monitor your AWS services.