Hello IO community! Over the last few weeks we’ve released some changes to Adafruit IO that should be useful to everyone using the platform.

IO+ Yearly Billing

We’ve added the option to pay for IO+ on an annual basis at a reduced rate of $99 per year. So instead of paying $10 each month–$120 total for a year of service–you can pay $99 once and get the same year of IO+ features.

If you’re already an IO+ subscriber (thanks for your support!) and you want to switch, you can change to yearly billing on your profile page when you’re logged in to Adafruit IO. We’ve also added an Adafruit IO+ 1 Year Subscription Pass to the Adafruit shop, so you can buy one for yourself and gift a subscription to a friend!

MQTT /get and the case of the missing retain flag

MQTT is a tremendously useful protocol for building small connected devices and is relatively simple to understand and implement (if implementing networking protocols is your thing). Unfortunately, a few features of the Adafruit IO platform make it difficult for us to support the entire MQTT 3.1+ protocol specification in our broker. I’m going to talk briefly about one particular feature: the publish retain flag.

In the MQTT protocol, setting the retain flag on a published message asks the MQTT broker (server) to store that message. Then any new clients which connect and subscribe to that topic will immediately receive the retained message. Retain makes writing basic MQTT-only Internet of Things clients easier, without it, a client that connects and subscribes to a feed topic has to wait until a new value is published on the feed to know what state it should be in. In the case of slowly updated feeds, there could be hours between updates which means a device that disconnects and reconnects (for example, due to power cycling or sleeping) might lose the current value for a long time between updates.

Among other factors, our scale, our mix of MQTT & HTTP APIs, the speed at which we’re taking in new data, and the fact that we’re already storing almost every message that is sent mean that a “simple” feature like retain becomes difficult to support without making MQTT service performance worse for everyone.

Since we don’t actually store data in the broker but at a lower level and can’t support PUBLISH retain directly, we’re proposing a different solution for the retaining problem: the /get topic modifier.

The way it works is that for any given Adafruit IO MQTT feed topic (groups coming soon), subscribe to the feed topic, then add /get to the topic and publish anything to that new topic (our Arduino library uses the null character: '\0') . IO will immediately publish, just for that client, the most recent value received on the feed.

For example, if I have a Feather HUZZAH subscribed to a counter feed: abachman/f/counter, and I want to get the latest value when I connect, I could publish to abachman/f/counter/get and immediately receive a message on abachman/f/counter with the last value that feed received.

The feature is live on Adafruit IO and in version 2.7.3 of the Adafruit IO Arduino library which was released today. If you’re already using the library, you can add /get support to your project in one line of code.

// ... from the adafruitio_01_subscribe example sketch
AdafruitIO_Feed *counter = io.feed("counter");

void setup() {

  // 1. start IO connection
  io.connect();

  // 2. prepare MQTT subscription with handler function
  counter->onMessage(handleMessage);

  // 3. wait for successful connection
  while(io.mqttStatus() < AIO_CONNECTED) {
    delay(500);
  }

  // 4. send /get message, requesting last value
  counter->get(); // ask Adafruit IO to resend the last value

}
// ....

You can see the get() function in action in the example sketches in the updated Adafruit IO Arduino library.

Platform Upgrades

Early in the afternoon of April 18, we swapped out about 60% of the servers that are running IO and deployed an upgrade one of the primary software frameworks IO is built on (Ruby on Rails). Because framework upgrades are part of the boring, behind the scenes parts of software development and maintenance, you shouldn’t notice any change to IO except that data is processed about 4% faster on average. :D

No data in transit was lost, but there was a period of about 5 minutes of delayed MQTT data processing. A few early bugs were caught and fixed, so there may be some devices that were sending data successfully that failed for a brief time.

Get in touch with us on the Adafruit IO forums or ping @adam.io on Discord if you have any questions.

Here are the stats for the past week:

* 97.8 million inserts of logged data in the last 7 days
* 37,663 online feeds (140,620 feeds total)

We’re responding to issues you may run into on our Adafruit IO Forum. Please post any issues with as much detail as you can. It definitely helps us out when fixing the bugs.

Also, if you’re still using version 1 of our API, you may want to upgrade to the latest version 2. It has more features, and is generally better designed and easier to use than our previous API. The default for our docs is V2 now as well.

Hello everyone and happy Friday! It’s been a busy few weeks and there are a couple new features at Adafruit IO we’d like to mention.

First off, we’ve added a new feedback form directly into Adafruit IO, so you can inform us quickly about any trouble with the site or ask any questions about the Adafruit IO+ plan without having to go through the forums. We included a few options for feedback and while we will do our best to address any issues with the site or our client libraries quickly and directly via email, we aren’t able to respond to every feedback request. If you’re having trouble with your personal code or hardware projects, the Adafruit IO forum is still the best place to go for help.

You can find a link to the Feedback form at the bottom of the left hand sidebar on most pages.

Second, and something I’m personally excited about, we’re introducing public pages for every user. You can see mine here. There are a few pieces of Adafruit IO you can make visible to the public as read-only data. Right now that’s Dashboards and Feeds. With the public pages, we now have a one-stop location for listing everything you have shared so you can see what’s public and you can share the link with anyone who might be interested.

Our intention isn’t to make Adafruit IO yet-another-social-media platform, but we firmly believe that things can turn out better when we work together, so we try to make that easier when we can. Stop by the forums or our Discord chat server and talk about what you’re working on!

December 2017 Throttle Updates

MQTT Group Publish Data Rate Correction

Hello IO friends!

We’ve found and corrected a problem with our data rate tracking system that was allowing MQTT group publish events to be counted as a single event for the purpose of data rate monitoring instead of tracking each feed as a separate event. This means publishing to a group with five feed values was counted as 1 data point, even though we were storing 5 data points.

The correction updates our MQTT broker to count a group message with 5 feeds as 5 data points against your accounts total available data rate.

Free Plan Changes in 2018

Speaking of IO+, IO Free, and permitted data rates: starting in January 2018 we will begin lowering the current allowed IO Free data rate from 60 points-per-minute to the advertised 30 points-per-minute. For the vast majority of people using Adafruit IO Free accounts, this shouldn’t be a problem, but we’ll be making the change gradually to avoid disabling a vast swathe of devices all at once.

Each week we’ll be dropping the data rate by 5 points-per minute until it’s down to 30. That’s plenty of useable bandwidth and storage for any of the Adafruit IO projects on the Adafruit Learning System.

Why so many data rate changes?

Our ultimate goal with data rate monitoring and limiting is to make sure we provide a good balance of data storage and response speed across the entire Adafruit IO platform for every user whether they have a free account or an IO+ account.

While IO+ has a higher data rate and more data storage, every message should still move through IO from publish to IO back out to subscribers at the same speed. By our measurements that speed is around 35ms on average for a round-trip from a device to IO and back out to a device. The way we keep that speed steady and available for everyone is making sure no account is able to take an unfair amount of it.

When we make changes like this that can feel restrictive, it’s to make sure resources are shared fairly. The good news is that over a longer amount of time, improving the efficiency of Adafruit IO means we’ll be able to give more away. So, some pain now for more good stuff in the future. If or when we can find a way to double the amount of data we can process at the same cost-per-user we’re at right now, we’re probably going double the data rate for everyone!

Just for comparison and since I was curious about the numbers, Adafruit IO Free provides a steady 30-data-points-per-second data rate which over 30 days comes out to 1296000 messages possible: 30 messages-per-minute * 60 minutes-in-an-hour * 24 hours-in-a-day * 30 days. To use a lower-level Internet of Things service provider as an example, in this case Amazon, you can achieve a much higher data rate (up to 100 messages-per-second per-client, 9000 per-second per-account), but for the same amount of data we provide for free you’d pay about $5 a month at their $5 per million messages price (retrieved December 2017). Your dashboards would also be way less pretty, but that’s another story :D

Stop by the Adafruit IO forum or the adafruit-io room on our Discord server if you have any questions or comments. We’d love to hear what you’re making with Adafruit IO!

Groups, Feeds, and Proper MQTT Topics

We’re deploying a fix to a subtle bug in our MQTT subscription topics today and wanted to write a quick post describing the problem and the fix for people who might currently be using workarounds. MQTT subscription behavior is going to change slightly (for the better) in a way that may break existing systems.

Publishing to “composite” feed keys through our HTTP or MQTT APIs has worked since we introduced that feature with API version 2–that’s anything in the form of group.feed.

You can see an example of those keys here:

But, due to the way our internal MQTT message routing system worked, subscribing to a composite key hasn’t worked until today. Publishing to a feed at username/feeds/group_key.feed_key only sent data to subscribers that used topics that looked like username/feeds/feed_key. From this point in time forwards, however, data published to username/feeds/group_key.feed_key will always only be sent to subscribers to username/feeds/group_key.feed_key.

We are considering this a bug fix, but it introduces some subtle effects for people who add feeds to multiple groups. Here are the three things you need to keep in mind with Adafruit IO and MQTT. The

MQTT Feed Subscription Data Routing

First and most critically, MQTT subscriptions to username/feeds/feed_key for all feeds that are not in your default group will stop working. Some users found that using the bare feed key worked for MQTT subscriptions, but it’s a workaround and not something we want to continue supporting now that we have a fix for it.

Second, subscriptions to feeds that belong to multiple groups will only produce data that is published over MQTT to that group + feed combination. However, all data sent to that feed will still be shown on that feed on the Adafruit IO feed page and any dashboard blocks that use the feed, regardless of the group or groups it belongs to. A practical example of that would be a feed belonging to two groups.

For example, in the case of a group named “Signal” belonging to the “Garage” and “Office” groups, I would end up with something that looks like this on my feeds page:

“Signal” is a single Feed object, but shared between two different groups. If publish to one of those feed keys:

PUBLISH abachman/feeds/garage.signal "1"

Then only MQTT subscribers to abachman/feeds/garage.signal will see the value “1” come through. If I publish to both feed keys:

PUBLISH abachman/feeds/garage.signal "1"
PUBLISH abachman/feeds/office.signal "2"
PUBLISH abachman/feeds/garage.signal "3"
PUBLISH abachman/feeds/office.signal "4"
...

Then the abachman/feeds/garage.signal subscription will only see:

SUBSCRIBE abachman/feeds/garage.signal
>>> "1"
>>> "3"
...

Likewise, group subscriptions will only see the messages published to that feed in the context of that group. So if instead of abachman/feeds/garage.signal I subscribed to abachman/groups/garage I would see:

SUBSCRIBE abachman/groups/garage
>>> {"feeds":{"signal":"1"}}
>>> {"feeds":{"signal":"3"}}
...

Over MQTT, feed subscriptions in the context of a given group will only produce messages that were published to that group + feed.

HTTP Feed Data Requests

Finally, all HTTP data queries for a given feed, regardless of the group context (group.feed key), will produce all data published to that feed. In the case of the example above, that means HTTP GET requests to /api/v2/abachman/feeds/garage.signal and /api/v2/abachman/feeds/office.signal will both produce the same data.

$ export KEY='860b36ff6537fa01e9ee755be3ae18d4fake'
$ export API_URL='https://io.adafruit.com/api/v2'
$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/garage.signal/data.csv?limit=4

id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,garage.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,garage.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,garage.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,garage.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z

$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/office.signal/data.csv?limit=4

id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,office.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,office.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,office.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,office.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z

This is because although the feed belongs to two different groups, it’s still the same underlying Feed record.

Receiving ALL Messages

If you need to see all data published to a specific Feed, you can use the raw feed id value. You can find the corresponding MQTT topic on the “Feed Information” form associated with the Feed you want to track.

Alternatively, every user account has a global data feed at the MQTT topic :username/dashboard/stream/create where :username is your account username. This topic produces a data record for every new data point stored:

SUBSCRIBE abachman/dashboard/stream/create
>>> {
  "id": "0DQBM1EPCDA0J6Z89CE1YC26D1",
  "value": "5",
  "feed_id": 30,
  "feed_key": "garage.signal",
  "created_at": "2017-11-02T16:41:32Z",
  "location": null,
  "lat": null,
  "lon": null,
  "ele": null,
  "created_epoch": 1509640892,
  "expiration": "2017-12-02T16:41:32Z"
}

With Feeds + Groups, we’ve created a powerful tool for flexibly routing and storing messages over MQTT. A single feed can be used multiple ways by adding it to different groups, which can be helpful if you’re still using a feed-limited plan like IO Basic. IO Plus gives you an unlimited number of feeds, so reuse is less critical, but we expect the flexibility described to be useful in certain cases or else we wouldn’t have built it :D

I’ve published a short GitHub Gist that includes code demonstrating the topic described in this post, you can find that here: https://gist.github.com/abachman/a04694748ad887d5aa7e644e3292fa81.

Thank you for for joining us as we continue to extend and improve Adafruit IO. As always, stop by the Adafruit IO forum or our Discord server in the #adafruit-io room to share any questions or comments, especially when it comes to details of our MQTT support. We’d love to hear about what you’re building!