Posted in raspberrypi, aws

header_lq.jpg

Did you ever have a lot of data in your code that sits better in a file? Why not trying something else?

If I can, I always avoid storing information locally, especially when I need to change something remotely. Recently I made Conway's Game of Life and I want the patterns to be accessible by other devices. This will allow me to create new patterns, store them and reuse them as they are added.

For any thought/question/problem, use the comment section and tell me your thoughts on this. 

What do I need to get started?

If you never had any contact with AWS (Amazon Web Services), you can create an account for free at aws.amazon.com. 

DynamoDb it's a NoSQL database where you can store and retrieve data along with other capabilities. The best part, you can use 25Gb where you can perform 25 reads and writes per second for free. If these values get deprecated, just take a look at what free tier has to offer.

In other to be able to save and get items from DynamoDb we need to have the right permissions to access the AWS from Raspberry Pi. Go to the AWS Console and create (or use an existing user ) and attach the following policy to it, DynamoDbFullAccess. If you create a new user, don't forget to save the access key id and secret access key, we need them later. That's it the only thing we need to do from the AWS Console.

Take me to the code

Before jumping directly into the code, I will shortly describe what we will do. First, I will integrate DynamoDB into one of my old project, Game of Life. However, you can apply the concepts to any other project. 

I divided my code into two parts. The first one is responsible for seeding the table with data. The second one, as you suspect, it is responsible for reading the saved information.

Below, don't forget to replace the values for AWS_ACCESS_KEY and AWS_SECRET_ACCESS s

Seeding the table

This part can happen from any device you want, your PC or any other board that comes in handy.

This part is very simple, and you can see it from the code. We just need to create a client for DynamoDB using boto3. After we managed to do this, we have to create the table and save the data in it.

Because of my project, I need to save matrixes. Because DynamoDb does not support matrixes (I don't know if any database supports them), I need a serializer/deserializer. I didn't want to pollute the code with my own version, so I used pickle instead. 

I created three patterns for Game Of Life and stored them GameOfLifePatterns table.

import boto3
import pickle

AWS_ACCESS_KEY = 'REPLACE_ME'
AWS_SECRET_ACCESS = 'REPLACE_ME'
TABLE_NAME = 'GameOfLifePatterns' def createTable(client): table = client.create_table( TableName=TABLE_NAME, KeySchema=[ { 'AttributeName': 'pattern-id', 'KeyType': 'HASH' } ], AttributeDefinitions=[ { 'AttributeName': 'pattern-id', 'AttributeType': 'N' } ], ProvisionedThroughput={ 'ReadCapacityUnits': 5, 'WriteCapacityUnits': 5 } ) # Wait until the table exists. client.get_waiter('table_exists').wait(TableName=TABLE_NAME) print "Table with name %s created.\n" % TABLE_NAME def putPattern(client, id, pattern): response = client.put_item( TableName=TABLE_NAME, Item={ 'pattern-id': { 'N': id }, 'pattern': { 'S': pickle.dumps(pattern) } } ) print "Item with id %s and pattern %s saved.\n" % (id, pattern) if __name__ == '__main__': try: dynamodbClient = boto3.client( 'dynamodb', aws_access_key_id=AWS_ACCESS_KEY, aws_secret_access_key=AWS_SECRET_ACCESS, region_name='eu-west-1' ) # just delete/comment this line if you alredy have the tabel # and you need to just save new things createTable(dynamodbClient) pulsy = [ [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 0, 1, 0, 0], [0, 0, 0, 0, 0, 1, 1, 0], [0, 0, 0, 0, 0, 0, 0, 0], [0, 1, 1, 1, 0, 0, 1, 1], [0, 0, 0, 1, 0, 1, 0, 1], [0, 0, 0, 0, 0, 1, 1, 0] ] toad = [ [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 1, 1, 1, 0, 0], [0, 0, 1, 1, 1, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0] ] beacon = [ [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 1, 1, 0, 0, 0, 0], [0, 0, 1, 1, 0, 0, 0, 0], [0, 0, 0, 0, 1, 1, 0, 0], [0, 0, 0, 0, 1, 1, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0], [0, 0, 0, 0, 0, 0, 0, 0] ] # and many others putPattern(dynamodbClient, "0", pulsy) putPattern(dynamodbClient, "1", toad) putPattern(dynamodbClient, "2", beacon) # close execution by pressing CTRL + C except KeyboardInterrupt: print("Intrerrupted by user") pass finally: print("Program stopped")

I saved the above code in a save-patterns.py file, run it to save the patterns:

python save-patterns.py

Let's use the stored data

The data can be used for almost any device, as long it supports a connection to the internet. 

For my project, I just replaced the hard coded pattern (stored in a variable) with a method that retrieves it from the DynamoDB table. I chose to randomly use one pattern from the DynamoDb.

import time
import boto3
import pickle
import random

from luma.led_matrix.device import max7219
from luma.core.interface.serial import spi, noop
from luma.core.render import canvas

AWS_ACCESS_KEY = 'REPLACE_ME'
AWS_SECRET_ACCESS = 'REPLACE_ME'
TABLE_NAME = 'GameOfLifePatterns' UNIVERSE_WIDTH = 8 UNIVERSE_HEIGHT = 8 def isAlive(cell): return cell == 1 def drawMatrix(device, universe): with canvas(device) as draw: for y in range(UNIVERSE_WIDTH): for x in range(UNIVERSE_HEIGHT): fill = 'white' if isAlive(universe[x][y]) else 'black' draw.point((x,y), fill=fill) def countNeighbors(universe, cellX, cellY): count = 0 neighborsRadius =[-1, 0, 1] for x in neighborsRadius: neighbordPositionX = cellX + x if neighbordPositionX >= UNIVERSE_WIDTH or neighbordPositionX < 0: continue for y in neighborsRadius: neighbordPositionY = cellY + y if neighbordPositionY >= UNIVERSE_HEIGHT or neighbordPositionY < 0: continue if cellX != neighbordPositionX or cellY != neighbordPositionY: count += universe[neighbordPositionX][neighbordPositionY] return count def gameOfLife(universe): # copy the universe so that we can modify it without worry newUniverse = [row[:] for row in universe] for cellX in range(UNIVERSE_WIDTH): for cellY in range(UNIVERSE_HEIGHT): neighbors = countNeighbors(universe, cellX, cellY) if isAlive(newUniverse[cellX][cellY]): # Any live cell with fewer than two live neighbors dies, as if by under population. # Any live cell with more than three live neighbors dies, as if by overpopulation. if neighbors < 2 or neighbors > 3: newUniverse[cellX][cellY] = 0 # Any live cell with two or three live neighbors lives on to the next generation. elif neighbors == 2 or neighbors == 3: newUniverse[cellX][cellY] = 1 # Any dead cell with exactly three live neighbors becomes a live cell, as if by reproduction. elif neighbors == 3: newUniverse[cellX][cellY] = 1 return newUniverse def getDevice(n, block_orientation, rotate): # create matrix device serial = spi(port=0, device=0, gpio=noop()) device = max7219(serial, cascaded=n or 1, block_orientation=block_orientation, rotate=rotate or 0) print("Created device") return device def getPattern(client, id): response = client.get_item( TableName=TABLE_NAME, Key={ 'pattern-id': { 'N': str(id) } } ) pattern = response['Item']['pattern'][u'S'] return pickle.loads(pattern) def getNumberOfItems(client): response = client.scan( TableName=TABLE_NAME ) itemCount = response['Count'] return itemCount if __name__ == '__main__': try: dynamodbClient = boto3.client( 'dynamodb', aws_access_key_id=AWS_ACCESS_KEY, aws_secret_access_key=AWS_SECRET_ACCESS, region_name='eu-west-1' ) # replace 1 with the number of devices device = getDevice(1, 0, 0) # get a random pattern from DynamoDb numberOfPatterns = getNumberOfItems(dynamodbClient) universe = getPattern(dynamodbClient, random.randint(0, numberOfPatterns)) while 1: drawMatrix(device, universe) time.sleep(1) universe = gameOfLife(universe) # close execution by pressing CTRL + C except KeyboardInterrupt: print("Intrerrupted by user") pass finally: print("Program stopped")

I saved the above code in a game-of-life.py file, run it using the command below to play a random pattern from DynamoDb. 

python game-of-life.py

Conclusion

Storing data in a different storage than the local one is very useful. Especially when you have multiple devices access the same source of data. 

This can get very useful when you have devices that stores data from the sensor How-to be used later by other devices or even by an interface.

You managed to go through my poor writing skills, brave one. 

Posted in raspberrypi, aws

How to send notifications from Raspberry Pi using AWS SNS for FREE

Have you felt the need to be notified by your Rasp when something is happening in your home (and not only)?

Well, I certainly did. I always forgot a lot of things that can be friendly reminded. Like watering the plants, closing the door, leaving the light open, although I don't have a cat or dog, I would definitely have needed to know if they are low on food or water. (and many other situations)

If you want to be notified on your phone, this can get very expensive and complicated. Why spending money on a GSM modem and a prepaid phone card when you can get the same benefits for free using AWS SNS? Luckily, there is another way.

Use the comment section and tell me your thoughts about this. 

AWS prerequisite

If you never had any contact with AWS (Amazon Web Services), you can create an account for free at aws.amazon.com

But, what is AWS SNS? AWS Simple Notification Service (SNS) makes easier to send notifications to your phone or email address. It allows you to send 100 SMS and 100.000 emails to be sent each month for free (at the moment of writing this). I proved that it’s easy to work with AWS on how to publish temperature and humidity to CloudWatch. You can compare this service with the Observer design pattern.

The first thing that we need it's a way to give permissions to our Rasp to publish SNS topics. In the post mentioned above, I showed how to give permission to the CloudWatch service. The process is very similar, but we need to give access to AWS SNS instead. Please give the following permission policy to your IAM user, AmazonSNSFullAccess. Don't forget to save the AWS access key id and secret access key in a safe place.

After we have the access keys we need to go to AWS SNS Console and follow this tutorial created by AWS to create a topic. After you successfully created the topic, save the topic ARN, we will need it later.

I said that we will send the notifications to our phones and emails, we need to add them as subscribers to the topic that we just created. Use the guidance provided by the AWS to register them. For email addresses don't forget that you have to confirm your registrations, strangely, for a phone, you don't have to confirm that you want to receive notifications.

What do I need to get started?

It's very easy to integrate the AWS SNS functionality in your project. In order to prove this, I will use a project that detects if the door is open or closed. I will modify this project to send notifications with the door status (open/close).

To give you a little context about the project, it uses an ultrasonic sensor to read the distance from the nearest object. When the door is opened/closed the distance increase/decrease as well. To determine if the door status it uses an if/else code block using a fixed distance to compare against. At a first glance this doesn't seem a problem, but if the door is opened, the program will output:

Door Open # 1 notification
Door Open # 2 notifications Door Open # 3 notifications
...
Door Open # n notifications

Every time this will run, it will send a notification. Because the number of notifications from the free tier is limited we need to make some changes in order to send only 1 message. Even if the money wasn't a problem, it would be very annoying and not cost effective to send so many alerts in a short span of time.

How can I send the messages only once?

After searching a little, the solution that I stumbled upon is to observe the percentage change between 2 consecutive distances (also known as Delta).

\text{Percentage change} = \frac{\Delta V}{V_1} = \frac{V_2 - V_1}{V_1} \times100 .

We put the sensor at 4cm from the door, with the door closed the output is ~4cm. When the door opens, the distance will significantly increase. The reverse applies equally when the door closes, the distance significantly decrease. In this case, significantly means that the distance increase/decrease by more than 100% (you can increase or decrease this value as you like).

Let's suppose for simplicity that we read 4cm and for the next point we read 20cm. You can notice even if from 4cm to 20cm it's not much it will give a huge output (20cm - 4cm)/4cm * 100 = 400%. Because 400 breaches our threshold of 100 we can say that the door is Open or Close.

Note: If the door opens (or closes) very slowly this algorithm won't catch the status change, before implementing this make sure that you don't have any ninja or cats in the house in order to be effective.

Take me to the code

In the below snipped I focused mainly on the AWS part of the code, the part that is focusing on the ultrasonic sensor can be found here

For Python 2 (if you use Python 3 please search for the equivalent commands), in order to be able to run the code, you will need to install the AWS SDK in Python, who is called boto3.

pip install boto3

If you don't have pip you can install it using the following command

sudo apt-get install python-pip

Let's do the most important things first so that we don't forget, replace INSER_AWS_ACCESS_KEY,  INSER_AWS_SECRET_ACCESS, and INSER_TOPIC_ARN with your own.

In order to publish the message to our listeners, we need to use the publish method from the SNS client API. This method has the following parameters that we will use:

  • TopicArn - this is the topic identification (ARN) 
  • Message - the message that will be sent in the email/SMS, be careful with the text size because an SMS allows only 153 characters
  • Subject - (optional) this is used only in the emails (an SMS doesn't have a subject)

For a comprehensive documentation about SNS you can check the following page, you can find details to build more sophisticated systems using this service.

import RPi.GPIO as GPIO
import time
import boto3

AWS_ACCESS_KEY = 'INSERT_AWS_ACCESS_KEY'
AWS_SECRET_ACCESS = 'INSERT_AWS_SECRET_ACCESS'
TOPIC_ARN = 'INSERT_TOPIC_ARN' # should similar with this arn:aws:sns:us-east-1:1234567890:door-status

# ....
# missing functions/constants can be found here 
# https://thehumblecode.com/blog/how-to-detect-your-door-activity-using-raspberry-pi-and-hc-sr04/
# ....

def publish(snsClient, message):
# replace the TOPIC_ARN with your own
response = snsClient.publish( TopicArn=TOPIC_ARN,
Message=message, Subject='Door status' ) print ("Published with id %s " % response) def percentChange(current, previous, difference): increase = current - previous increasePercentage = increase / previous * 100 if increasePercentage > difference: return True return False if __name__ == '__main__': try: snsClient = boto3.client( 'sns', aws_access_key_id=AWS_ACCESS_KEY, aws_secret_access_key=AWS_SECRET_ACCESS, region_name='us-east-1' ) currentDistance=distance() previousDistance=currentDistance while 1: currentDistance = distance() print ("Distance = %.1f cm" % currentDistance) if percentChange(currentDistance, previousDistance, 100): message = "Door opened. Distance greatly increased from %.1f cm to %.1f" % (previousDistance, currentDistance) print (message) publish(snsClient, message) elif percentChange(previousDistance, currentDistance, 100): message = "Door closed. Distance greatly decreased from %.1f cm to %.1f" % (previousDistance, currentDistance) print (message)
publish(snsClient, message) previousDistance = currentDistance time.sleep(1) # close execution by pressing CTRL + C except KeyboardInterrupt: print("Intrerrupted by user") pass finally: print("Program stopped") GPIO.cleanup()

Run the code using the following command on the terminal and press CTRL + C to cancel anytime.

python distance.py

terminal.jpg

Final thoughts

I hope that you find this useful, AWS SNS is a tool that can come in handy in some situations and has a lot of potential for future projects. 

If you encounter any trouble or have any curiosity, feel free to use the comment section to share your feelings about this.

But most importantly, don't stop learning!

Posted in raspberrypi, aws, cloudwatch

header image

Did you ever wonder how to bring your mini computer to the cloud?

I always was under the impression that the cloud services are very expensive and it would be impossible for people like me to use them. But I was wrong, cloud giants offer several services for free. Even though there are more limited, they can provide value on a personal/home and even startup project. There are many cloud services providers out there: Azure, Amazon Web Services (AWS), Google Cloud, etc. Because of the competition, most of them offer something for free or for a limited period (12 months).

What’s the best service to learn? It should be fun, interesting and easy to use. From so many out there to chose, the only that matched the description was AWS CloudWatch. This product allows us to publish a value at a point in time. If we publish enough points we will end up with a time-series. We can have up to 10 time-series.

For any thought/question/problem, use the comment section and tell me your thoughts on this. 

What's the hardware prerequisite?

raspberry pi dht11 sensor with aws inside

I always start with the hardware, because the only thing that remains is to focus on the code.

For this how-to you are free to choose/imagine any sensor you would like. I have chosen the DHT11 sensor (it’s the only sensor that I have, not much of a choice) that will collect the humidity and temperature. If you are going this way, you will also need 3 wires and a resistor of 6KΩ - 10KΩ (ohm). I know that are components out there which already contain the resistor, if you own one of them, you don't need a resistor.

I the image below shows how the components are connected between them. The good part of this sensor is that it can be connected at 3.3V or 5V (red wire), your choice! The blue wire is the data wire, used to read the values of interest. The resistor needs to be connected between this two pins (red and blue). The black wire is connected to GND. 

Use this table for writing representation of the cable connection that I made. 

DHT11SignalRaspberry Pi Zero
13.3V ~ 5.5V 4 (VCC)
2Data 7 (BCM 4)
3not used 
4Ground 6 (GND)

How do I setup my AWS account?

Although this part looks a little bit scary if you never had any contact with AWS. You don’t have to worry about this, you are in good hands.

The first thing that you have to do is creating your account for free at https://aws.amazon.com/free .

After you log in to the console, you might feel intimidated by the number of services, luckily, we need to use just one, CloudWatch. Maybe I lied, we need to use 2 of them (IAM – Identity and Access Management). Before going on, you can go on an adventure on your own by exploring the console.

For what is IAM used for? Simply put, we need a way to authenticate and publish the metrics. In this service, we will create a user that will have the rights to publish metrics in CloudWatch.

Go to IAM service in the AWS console and click on the Add user button, give it a name and make sure that you select the Programmatic access

aws iam add user

 

In the next step, we will grant permissions to the user in order to be able to publish metrics in CloudWatch. To keep things simple, we will choose an existing policy, CloudWatchFullAccess. Make sure that you check the checkbox in order to give the rights. 

aws iam attach policy for cloudwatch full access

One last step and we are done. Hit next until you stumble on to the Access key ID and Secret access keyDon’t close the window, the Secret access key will be shown only in this place. Save them in a safe place, we need them in the next phase.

aws iam save credentials step

I am all set; how do I publish metrics in CloudWatch?

Remember what I said at the beginning? We need to publish one point at a time. 

The first thing that should be done is creating the CloudWatch client. Replace the INSERT_YOUR_ACCESS_KEY_ID_HERE and INSERT_YOUR_SECRET_ACCESS_KEY_HERE with the values saved above. Make sure that you are in the right region. In the region that you configure, you will be able to see the metrics.

In order to publish metrics to cloudwatch, we need to use the putMetricData method from the API. This method requires some parameters to be set and are explained below:

  • MetricName - the name of the metric, in this case, Temperature and Humidity
  • Dimensions - it's used to identify the metric, as a best practice I recommend to include the source of the metric (SensorType: DHT11)
  • TimeStamp - the moment in time when I read the metric. Using new Date() will assure that I will log the metrics at the current time.
  • Unit - unfortunately CloudWatch doesn't have the "Custom" option, here should have been the °C for temperature. Humidity uses percent and it already exists.
  • Value - the value that is read from the sensor. 
  • Namespace - this can be anything you like

The most important fields from the above are TimeStamp and Value. This is what is really building or time-series. For full details, you can check the official documentation for CloudWatch.

We use setInterval to read the values from DHT11 every five seconds and insert them in the cloud.

Don't forget that you have to make some preparation before using the node-dht-sensor library. If you are stumbling in any problem, I described in a previous project how to make the setup.

const sensor = require('node-dht-sensor');
const AWS = require('aws-sdk');

var cloudWatch = new AWS.CloudWatch({
    accessKeyId: "INSERT_YOUR_ACCESS_KEY_ID_HERE",
    secretAccessKey: "INSERT_YOUR_SECRET_ACCESS_KEY_HERE",
    region: "us-east-1" 
}); const DHT = 11; const SENSOR_PIN = 4; publishMetric = (metricName, value) => { var params = { MetricData: [ /* required */ { MetricName: metricName, /* required */ Dimensions: [ { Name: 'SensorType', /* required */ Value: 'DHT11' /* required */ } /* more dimensions */ ], Timestamp: new Date(), Unit: "None", Value: value } /* more items */ ], Namespace: 'TheHumbleCode' /* required */ }; cloudWatch.putMetricData(params, function(err, data) { if (err) console.log(err, err.stack); // an error occurred else console.log(data); // successful response }); } setInterval(() => { const sensorResult = sensor.read(DHT, SENSOR_PIN); console.log(`The current temperature is ${sensorResult.temperature.toFixed(2)} °C and humidity ${sensorResult.humidity.toFixed(2)} %`); // publish sensor values in Cloud Watch publishMetric("Temperature", sensorResult.temperature) publishMetric("Humidity", sensorResult.humidity) }, 5000)

In case you need it, I'm using the following package.json file.

{
  "name": "cloudwatch-sensor",
  "version": "0.0.1",
  "description": "A tool that publishes metrics to cloudwatch (e.g. temperature and humidity).",
  "main": "publisher.js",
  "author": "TheHumbleCode",
  "dependencies": {
    "aws-sdk": "^2.228.1",
    "node-dht-sensor": "^0.0.34"
  }
}

First thing, install all the dependencies for this project and run it by executing the following commands:

npm install
node publisher

The metrics should start publishing and in ~15 minutes can be seen in AWS CloudWatch console.

If you connect to your Raspberry Pi by SSH and plan to leave the terminal open for several days, you need to use the screen command, otherwise, it will close after a while. If you don't have the command, you can install it using the following command:

sudo apt-get install screen
screen #open a new screen and start the publisher

After you start the program on the screen, you can detach from it by pressing Ctrl+a d. If you wish to reattach to the screen to make sure that everything works fine, run this command:

screen -r #reattach to the screen

if you stumble on any problem/curiosity/question use the comment section to address them, I'm more then glad to help/answer you. 

Final thoughts

Showing temperature and humidity metrics in CloudWatch

In the above results, I monitored my room for more than 1 day. From the time-series, ti can be seen a drop in temperature and humidity. That happened because I opened the window very early in the morning all the values dropped but recovered after during the day.

CloudWatch allows setting alarms on the metrics that you publish. If you left your AC ON while you are out, you can receive an email that the temperature dropped/increased to a certain level. And other handy situations that can be built with different sensors. 

But most importantly, stay curious!