We have been writing user’s feed data to both Cassandra 3.3 and PostgreSQL 9.5 for the past couple weeks as the first step in transitioning fully to Cassandra as the primary data store for feed data. We will still be using Postgres as our primary database for relational data, but it should be much easier to scale Cassandra horizontally as the large amount of logged data increases.

Here’s the current write performance data from Cassandra’s nodetool tablestats:

Write Count: 59051081
Write Latency: 0.012758596713919598 ms.

That’s a little over 59 million writes in the past ~17 days, with the load on a m4.large barely ever passing 0.05.

load average: 0.01, 0.01, 0.05

Today, we have also started testing read performance in production. For now users are still receiving their data from Postgres, but we are also hitting Cassandra to see if our configuration can handle production loads. We’ll post an update once we have more info about how reads are performing.

Here’s some early info after the deploy:

Read Count: 314
Read Latency: 0.6644044585987261 ms.

We moved Adafruit IO’s node.js services to a group of t2.nano EC2 instances, and I wanted to document my dsh setup. I used it reguarly a couple years ago for managing the backend workers when working on other services, but I had forgot how it was configured. It came in really handy when I needed to quickly check or kick anything on the servers.

First, you will need to install dsh using brew, apt-get, yum, etc:

$ brew install dsh

You will need to make a couple config directories for your user:

$ mkdir -p ~/.dsh/group

Create a new dsh.conf file using your favorite text editor:

$ vim ~/.dsh/dsh.conf

Paste the following config into ~/.dsh/dsh.conf, and save the file:

showmachinenames = 1
remoteshell = ssh

Now, you can create a dsh group for the set of servers you would like to send commands to. Create a new file called example in the ~/.dsh/group folder using your favorite text editor:

$ vim ~/.dsh/group/example

Enter the host names for your servers, and save the file:

example-1.adafruit.com
example-2.adafruit.com
example-3.adafruit.com
example-4.adafruit.com

Now you can send commands (like uptime) to all servers in the group using the following command:

$ dsh -g example -w 'uptime'
example-1.adafruit.com:  14:17:44 up 17:15,  0 users,  load average: 0.08, 0.03, 0.05
example-2.adafruit.com:  14:17:43 up 16:36,  0 users,  load average: 0.00, 0.01, 0.05
example-3.adafruit.com:  14:17:45 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05
example-4.adafruit.com:  14:17:45 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05

Using -w will send the command to each server in the order they appear in the group config, and will wait for a response before continuing. If you want to send the commands concurrently to all servers in the list, use the -c option to send the command:

$ dsh -g example -c 'uptime'
example-1.adafruit.com:  14:22:42 up 17:20,  0 users,  load average: 0.00, 0.01, 0.05
example-3.adafruit.com:  14:22:42 up 16:46,  0 users,  load average: 0.00, 0.01, 0.05
example-2.adafruit.com:  14:22:41 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05
example-4.adafruit.com:  14:22:42 up 16:46,  0 users,  load average: 0.10, 0.05, 0.05

You can get an interactive shell for all servers in the group by using the -c and -i commands together. Use CTRL-D to exit the session:

$ dsh -g -c -i

Adafruit IO has been growing steadily, and we decided to add a changelog so it’s easier for users to see what we are up to. We will be posting here reguarly whenever we add or update features, and we will also be posting stats so you have more information about the growth of IO.

We have moved IO to a handful of Amazon EC2 instances, and we are finding it much easier to deal with increased load. The individual services are now split out into separate instances, which will allow us to scale horizontally.

Here’s a screenshot of what it looks like when we are monitoring the state of the new EC2 instances:

We have added IO to the Adafruit Status Page, which will allow you to check and see if any of IO’s components are currently down. You can also follow @adafruitstatus if you would like to be notified of outages.

Here are the stats for the past week:

* 24,290,428 inserts in the last 7 days
* 8,997 users
* ~6,700 online feeds (~18,200 feeds total)
* ~50 inserts per second via MQTT
* ~5 inserts per second via REST API