Here are the stats for the past week:

* 97.8 million inserts of logged data in the last 7 days
* 37,663 online feeds (140,620 feeds total)

We’re responding to issues you may run into on our Adafruit IO Forum. Please post any issues with as much detail as you can. It definitely helps us out when fixing the bugs.

Also, if you’re still using version 1 of our API, you may want to upgrade to the latest version 2. It has more features, and is generally better designed and easier to use than our previous API. The default for our docs is V2 now as well.

Hello everyone and happy Friday! It’s been a busy few weeks and there are a couple new features at Adafruit IO we’d like to mention.

First off, we’ve added a new feedback form directly into Adafruit IO, so you can inform us quickly about any trouble with the site or ask any questions about the Adafruit IO+ plan without having to go through the forums. We included a few options for feedback and while we will do our best to address any issues with the site or our client libraries quickly and directly via email, we aren’t able to respond to every feedback request. If you’re having trouble with your personal code or hardware projects, the Adafruit IO forum is still the best place to go for help.

You can find a link to the Feedback form at the bottom of the left hand sidebar on most pages.

Second, and something I’m personally excited about, we’re introducing public pages for every user. You can see mine here. There are a few pieces of Adafruit IO you can make visible to the public as read-only data. Right now that’s Dashboards and Feeds. With the public pages, we now have a one-stop location for listing everything you have shared so you can see what’s public and you can share the link with anyone who might be interested.

Our intention isn’t to make Adafruit IO yet-another-social-media platform, but we firmly believe that things can turn out better when we work together, so we try to make that easier when we can. Stop by the forums or our Discord chat server and talk about what you’re working on!

December 2017 Throttle Updates

MQTT Group Publish Data Rate Correction

Hello IO friends!

We’ve found and corrected a problem with our data rate tracking system that was allowing MQTT group publish events to be counted as a single event for the purpose of data rate monitoring instead of tracking each feed as a separate event. This means publishing to a group with five feed values was counted as 1 data point, even though we were storing 5 data points.

The correction updates our MQTT broker to count a group message with 5 feeds as 5 data points against your accounts total available data rate.

Free Plan Changes in 2018

Speaking of IO+, IO Free, and permitted data rates: starting in January 2018 we will begin lowering the current allowed IO Free data rate from 60 points-per-minute to the advertised 30 points-per-minute. For the vast majority of people using Adafruit IO Free accounts, this shouldn’t be a problem, but we’ll be making the change gradually to avoid disabling a vast swathe of devices all at once.

Each week we’ll be dropping the data rate by 5 points-per minute until it’s down to 30. That’s plenty of useable bandwidth and storage for any of the Adafruit IO projects on the Adafruit Learning System.

Why so many data rate changes?

Our ultimate goal with data rate monitoring and limiting is to make sure we provide a good balance of data storage and response speed across the entire Adafruit IO platform for every user whether they have a free account or an IO+ account.

While IO+ has a higher data rate and more data storage, every message should still move through IO from publish to IO back out to subscribers at the same speed. By our measurements that speed is around 35ms on average for a round-trip from a device to IO and back out to a device. The way we keep that speed steady and available for everyone is making sure no account is able to take an unfair amount of it.

When we make changes like this that can feel restrictive, it’s to make sure resources are shared fairly. The good news is that over a longer amount of time, improving the efficiency of Adafruit IO means we’ll be able to give more away. So, some pain now for more good stuff in the future. If or when we can find a way to double the amount of data we can process at the same cost-per-user we’re at right now, we’re probably going double the data rate for everyone!

Just for comparison and since I was curious about the numbers, Adafruit IO Free provides a steady 30-data-points-per-second data rate which over 30 days comes out to 1296000 messages possible: 30 messages-per-minute * 60 minutes-in-an-hour * 24 hours-in-a-day * 30 days. To use a lower-level Internet of Things service provider as an example, in this case Amazon, you can achieve a much higher data rate (up to 100 messages-per-second per-client, 9000 per-second per-account), but for the same amount of data we provide for free you’d pay about $5 a month at their $5 per million messages price (retrieved December 2017). Your dashboards would also be way less pretty, but that’s another story :D

Stop by the Adafruit IO forum or the adafruit-io room on our Discord server if you have any questions or comments. We’d love to hear what you’re making with Adafruit IO!

Groups, Feeds, and Proper MQTT Topics

We’re deploying a fix to a subtle bug in our MQTT subscription topics today and wanted to write a quick post describing the problem and the fix for people who might currently be using workarounds. MQTT subscription behavior is going to change slightly (for the better) in a way that may break existing systems.

Publishing to “composite” feed keys through our HTTP or MQTT APIs has worked since we introduced that feature with API version 2–that’s anything in the form of group.feed.

You can see an example of those keys here:

But, due to the way our internal MQTT message routing system worked, subscribing to a composite key hasn’t worked until today. Publishing to a feed at username/feeds/group_key.feed_key only sent data to subscribers that used topics that looked like username/feeds/feed_key. From this point in time forwards, however, data published to username/feeds/group_key.feed_key will always only be sent to subscribers to username/feeds/group_key.feed_key.

We are considering this a bug fix, but it introduces some subtle effects for people who add feeds to multiple groups. Here are the three things you need to keep in mind with Adafruit IO and MQTT. The

MQTT Feed Subscription Data Routing

First and most critically, MQTT subscriptions to username/feeds/feed_key for all feeds that are not in your default group will stop working. Some users found that using the bare feed key worked for MQTT subscriptions, but it’s a workaround and not something we want to continue supporting now that we have a fix for it.

Second, subscriptions to feeds that belong to multiple groups will only produce data that is published over MQTT to that group + feed combination. However, all data sent to that feed will still be shown on that feed on the Adafruit IO feed page and any dashboard blocks that use the feed, regardless of the group or groups it belongs to. A practical example of that would be a feed belonging to two groups.

For example, in the case of a group named “Signal” belonging to the “Garage” and “Office” groups, I would end up with something that looks like this on my feeds page:

“Signal” is a single Feed object, but shared between two different groups. If publish to one of those feed keys:

PUBLISH abachman/feeds/garage.signal "1"

Then only MQTT subscribers to abachman/feeds/garage.signal will see the value “1” come through. If I publish to both feed keys:

PUBLISH abachman/feeds/garage.signal "1"
PUBLISH abachman/feeds/office.signal "2"
PUBLISH abachman/feeds/garage.signal "3"
PUBLISH abachman/feeds/office.signal "4"
...

Then the abachman/feeds/garage.signal subscription will only see:

SUBSCRIBE abachman/feeds/garage.signal
>>> "1"
>>> "3"
...

Likewise, group subscriptions will only see the messages published to that feed in the context of that group. So if instead of abachman/feeds/garage.signal I subscribed to abachman/groups/garage I would see:

SUBSCRIBE abachman/groups/garage
>>> {"feeds":{"signal":"1"}}
>>> {"feeds":{"signal":"3"}}
...

Over MQTT, feed subscriptions in the context of a given group will only produce messages that were published to that group + feed.

HTTP Feed Data Requests

Finally, all HTTP data queries for a given feed, regardless of the group context (group.feed key), will produce all data published to that feed. In the case of the example above, that means HTTP GET requests to /api/v2/abachman/feeds/garage.signal and /api/v2/abachman/feeds/office.signal will both produce the same data.

$ export KEY='860b36ff6537fa01e9ee755be3ae18d4fake'
$ export API_URL='https://io.adafruit.com/api/v2'
$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/garage.signal/data.csv?limit=4

id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,garage.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,garage.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,garage.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,garage.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z

$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/office.signal/data.csv?limit=4

id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,office.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,office.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,office.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,office.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z

This is because although the feed belongs to two different groups, it’s still the same underlying Feed record.

Receiving ALL Messages

If you need to see all data published to a specific Feed, you can use the raw feed id value. You can find the corresponding MQTT topic on the “Feed Information” form associated with the Feed you want to track.

Alternatively, every user account has a global data feed at the MQTT topic :username/dashboard/stream/create where :username is your account username. This topic produces a data record for every new data point stored:

SUBSCRIBE abachman/dashboard/stream/create
>>> {
  "id": "0DQBM1EPCDA0J6Z89CE1YC26D1",
  "value": "5",
  "feed_id": 30,
  "feed_key": "garage.signal",
  "created_at": "2017-11-02T16:41:32Z",
  "location": null,
  "lat": null,
  "lon": null,
  "ele": null,
  "created_epoch": 1509640892,
  "expiration": "2017-12-02T16:41:32Z"
}

With Feeds + Groups, we’ve created a powerful tool for flexibly routing and storing messages over MQTT. A single feed can be used multiple ways by adding it to different groups, which can be helpful if you’re still using a feed-limited plan like IO Basic. IO Plus gives you an unlimited number of feeds, so reuse is less critical, but we expect the flexibility described to be useful in certain cases or else we wouldn’t have built it :D

I’ve published a short GitHub Gist that includes code demonstrating the topic described in this post, you can find that here: https://gist.github.com/abachman/a04694748ad887d5aa7e644e3292fa81.

Thank you for for joining us as we continue to extend and improve Adafruit IO. As always, stop by the Adafruit IO forum or our Discord server in the #adafruit-io room to share any questions or comments, especially when it comes to details of our MQTT support. We’d love to hear about what you’re building!

NEW FEATURE: Tracking MQTT Client Connections

New feature!

We’ve added a new MQTT topic at :username/clients (where :username is replaced with your username) and a section on the IO Monitor page to track live MQTT client connections. We aren’t doing anything fancy with the data (yet!), so it’s “stateless” / streaming only for now.

What you’ll see if you subscribe to the topic is a JSON object with the connection status, time of event (at), and an object containing information about the client. At this moment it’s only the client’s self-reported MQTT id. For example:

    // on connection
    {"status":"connected","at":"2017-10-24T16:08:38.552Z","client":{"id":"io-mqtt-39dc5a71"}}

    // on disconnectiohn
    {"status":"disconnected","at":"2017-10-24T16:08:40.691Z","client":{"id":"io-mqtt-39dc5a71"}}

We use that bit of data on your IO monitor page to show when clients are connecting and disconnecting in real time.

Using the /clients topic

We’re not yet storing detailed client connection state, we’re just reporting on it live. So how is this useful? The first and easiest way to use the topic is just to track clients as they connect and disconnect and catch funny business before you get throttled. The feed only shows authenticated client connections, so you won’t see failed attempts, but you will be able to track any hardware, software, or browser sessions as they connect and disconnect.

Since the only piece of identifying information that’s sent along with connection updates is the id, we can use that to record which specific clients is doing the [dis]connecting. Most MQTT clients generate a random identifier if one is not specified. So, by setting a custom client ID, we can pass some information through the system to see which device is acting up.

For example, on my home environmental monitor, which is running on an Arduino MKR1000, I can use the Adafruit MQTT Library to manually specify the MQTT client ID I use. Using the Adafruit_MQTT_Client constructor that takes a clientid argument, my code looks like this:

Which shows up on my monitor page with the custom clientid value!

Choosing a client ID value

Now, not just any client ID will do, there are a few guidelines:

1. Stick with printable bytes, anything else won’t be helpful for tracking.
2. Only the first 32 bytes of your clientid will be displayed, so keep the IDs at or under 32 bytes in length. The MQTT v3.1 spec requires that we allow at least 23, but lets us handle more. Internally IO can use more than 32 bytes to track clients, but only the first 32 will be published on the /clients topic.
3. Make sure it’s unique! Only one client with the given ID can be present on the network at a time. A good pattern for creating a client ID might be: your username + a device name + something random. Then you’ll know whether it’s the lamp controller or the weather station that keeps dropping its connection.

This is just a step towards a more complete device tracking / monitoring system, but gives a lot more insight into what’s happening on your IO account. Let us know how you’re using IO or IO+ on the forums or in our Discord chat room