Groups, Feeds, and Proper MQTT Topics

We’re deploying a fix to a subtle bug in our MQTT subscription topics today and wanted to write a quick post describing the problem and the fix for people who might currently be using workarounds. MQTT subscription behavior is going to change slightly (for the better) in a way that may break existing systems.

Publishing to “composite” feed keys through our HTTP or MQTT APIs has worked since we introduced that feature with API version 2–that’s anything in the form of group.feed.

You can see an example of those keys here:

But, due to the way our internal MQTT message routing system worked, subscribing to a composite key hasn’t worked until today. Publishing to a feed at username/feeds/group_key.feed_key only sent data to subscribers that used topics that looked like username/feeds/feed_key. From this point in time forwards, however, data published to username/feeds/group_key.feed_key will always only be sent to subscribers to username/feeds/group_key.feed_key.

We are considering this a bug fix, but it introduces some subtle effects for people who add feeds to multiple groups. Here are the three things you need to keep in mind with Adafruit IO and MQTT. The

MQTT Feed Subscription Data Routing

First and most critically, MQTT subscriptions to username/feeds/feed_key for all feeds that are not in your default group will stop working. Some users found that using the bare feed key worked for MQTT subscriptions, but it’s a workaround and not something we want to continue supporting now that we have a fix for it.

Second, subscriptions to feeds that belong to multiple groups will only produce data that is published over MQTT to that group + feed combination. However, all data sent to that feed will still be shown on that feed on the Adafruit IO feed page and any dashboard blocks that use the feed, regardless of the group or groups it belongs to. A practical example of that would be a feed belonging to two groups.

For example, in the case of a group named “Signal” belonging to the “Garage” and “Office” groups, I would end up with something that looks like this on my feeds page:

“Signal” is a single Feed object, but shared between two different groups. If publish to one of those feed keys:

PUBLISH abachman/feeds/garage.signal "1"

Then only MQTT subscribers to abachman/feeds/garage.signal will see the value “1” come through. If I publish to both feed keys:

PUBLISH abachman/feeds/garage.signal "1"
PUBLISH abachman/feeds/office.signal "2"
PUBLISH abachman/feeds/garage.signal "3"
PUBLISH abachman/feeds/office.signal "4"
...

Then the abachman/feeds/garage.signal subscription will only see:

SUBSCRIBE abachman/feeds/garage.signal
>>> "1"
>>> "3"
...

Likewise, group subscriptions will only see the messages published to that feed in the context of that group. So if instead of abachman/feeds/garage.signal I subscribed to abachman/groups/garage I would see:

SUBSCRIBE abachman/groups/garage
>>> {"feeds":{"signal":"1"}}
>>> {"feeds":{"signal":"3"}}
...

Over MQTT, feed subscriptions in the context of a given group will only produce messages that were published to that group + feed.

HTTP Feed Data Requests

Finally, all HTTP data queries for a given feed, regardless of the group context (group.feed key), will produce all data published to that feed. In the case of the example above, that means HTTP GET requests to /api/v2/abachman/feeds/garage.signal and /api/v2/abachman/feeds/office.signal will both produce the same data.

$ export KEY='860b36ff6537fa01e9ee755be3ae18d4fake'
$ export API_URL='https://io.adafruit.com/api/v2'
$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/garage.signal/data.csv?limit=4

id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,garage.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,garage.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,garage.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,garage.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z

$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/office.signal/data.csv?limit=4

id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,office.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,office.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,office.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,office.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z

This is because although the feed belongs to two different groups, it’s still the same underlying Feed record.

Receiving ALL Messages

If you need to see all data published to a specific Feed, you can use the raw feed id value. You can find the corresponding MQTT topic on the “Feed Information” form associated with the Feed you want to track.

Alternatively, every user account has a global data feed at the MQTT topic :username/dashboard/stream/create where :username is your account username. This topic produces a data record for every new data point stored:

SUBSCRIBE abachman/dashboard/stream/create
>>> {
  "id": "0DQBM1EPCDA0J6Z89CE1YC26D1",
  "value": "5",
  "feed_id": 30,
  "feed_key": "garage.signal",
  "created_at": "2017-11-02T16:41:32Z",
  "location": null,
  "lat": null,
  "lon": null,
  "ele": null,
  "created_epoch": 1509640892,
  "expiration": "2017-12-02T16:41:32Z"
}

With Feeds + Groups, we’ve created a powerful tool for flexibly routing and storing messages over MQTT. A single feed can be used multiple ways by adding it to different groups, which can be helpful if you’re still using a feed-limited plan like IO Basic. IO Plus gives you an unlimited number of feeds, so reuse is less critical, but we expect the flexibility described to be useful in certain cases or else we wouldn’t have built it :D

I’ve published a short GitHub Gist that includes code demonstrating the topic described in this post, you can find that here: https://gist.github.com/abachman/a04694748ad887d5aa7e644e3292fa81.

Thank you for for joining us as we continue to extend and improve Adafruit IO. As always, stop by the Adafruit IO forum or our Discord server in the #adafruit-io room to share any questions or comments, especially when it comes to details of our MQTT support. We’d love to hear about what you’re building!

NEW FEATURE: Tracking MQTT Client Connections

New feature!

We’ve added a new MQTT topic at :username/clients (where :username is replaced with your username) and a section on the IO Monitor page to track live MQTT client connections. We aren’t doing anything fancy with the data (yet!), so it’s “stateless” / streaming only for now.

What you’ll see if you subscribe to the topic is a JSON object with the connection status, time of event (at), and an object containing information about the client. At this moment it’s only the client’s self-reported MQTT id. For example:

    // on connection
    {"status":"connected","at":"2017-10-24T16:08:38.552Z","client":{"id":"io-mqtt-39dc5a71"}}

    // on disconnectiohn
    {"status":"disconnected","at":"2017-10-24T16:08:40.691Z","client":{"id":"io-mqtt-39dc5a71"}}

We use that bit of data on your IO monitor page to show when clients are connecting and disconnecting in real time.

Using the /clients topic

We’re not yet storing detailed client connection state, we’re just reporting on it live. So how is this useful? The first and easiest way to use the topic is just to track clients as they connect and disconnect and catch funny business before you get throttled. The feed only shows authenticated client connections, so you won’t see failed attempts, but you will be able to track any hardware, software, or browser sessions as they connect and disconnect.

Since the only piece of identifying information that’s sent along with connection updates is the id, we can use that to record which specific clients is doing the [dis]connecting. Most MQTT clients generate a random identifier if one is not specified. So, by setting a custom client ID, we can pass some information through the system to see which device is acting up.

For example, on my home environmental monitor, which is running on an Arduino MKR1000, I can use the Adafruit MQTT Library to manually specify the MQTT client ID I use. Using the Adafruit_MQTT_Client constructor that takes a clientid argument, my code looks like this:

Which shows up on my monitor page with the custom clientid value!

Choosing a client ID value

Now, not just any client ID will do, there are a few guidelines:

1. Stick with printable bytes, anything else won’t be helpful for tracking.
2. Only the first 32 bytes of your clientid will be displayed, so keep the IDs at or under 32 bytes in length. The MQTT v3.1 spec requires that we allow at least 23, but lets us handle more. Internally IO can use more than 32 bytes to track clients, but only the first 32 will be published on the /clients topic.
3. Make sure it’s unique! Only one client with the given ID can be present on the network at a time. A good pattern for creating a client ID might be: your username + a device name + something random. Then you’ll know whether it’s the lamp controller or the weather station that keeps dropping its connection.

This is just a step towards a more complete device tracking / monitoring system, but gives a lot more insight into what’s happening on your IO account. Let us know how you’re using IO or IO+ on the forums or in our Discord chat room

IO Plus, The Adafruit IO Subscription Plan

LIVE NOW We announced it was coming, and now it’s finally here. You can sign up for IO+ from your profile page and check out more details, and screenshots from our previous announcement!

Thank you to all of the beta testers that have been providing feedback to us while we built out a system that you hopefully find easy, useful and fun to use.

Here are the stats for the past week:

* 82.1 million inserts of logged data in the last 7 days
* 30,816 online feeds (105,563 feeds total)

We’re responding to issues you may run into on our Adafruit IO Forum. Please post any issues with as much detail as you can. It definitely helps us out when fixing the bugs.

Also, if you’re still using version 1 of our API, you may want to upgrade to the latest version 2. It has more features, and is generally better designed and easier to use than our previous API. The default for our docs is V2 now as well.

IO Plus, The Adafruit IO Subscription Plan

COMING SOON After a long time in beta testing mode, IO is getting a subscription plan, IO Plus! It’s been in our minds since we started this project a few years ago and “How can I get more feeds?” has been the most frequent and persistent question from people using and interested in IO, which is GREAT! The good news is that we finally have an answer for you. Say hello to IO+!

Here is what’s coming (we have not decided on a date yet!):

1. For a flat $10 per month, you can subscribe to IO+ and get unlimited feeds, unlimited dashboards, and two months of data storage.
2. We will always continue to provide a free plan, but for new subscribers, the free plan will be limited to 5 feeds and everyone on the free plan will be limited to 30 new data points per minute.
3. As a thank you to everyone who’s been with us this far, we’re giving away three months of IO+ free when you subscribe¹. You won’t pay anything unless you keep your subscription after the third month. Additionally, if you have more than 5 feeds when we move to the new account limits, you can keep the extras².
4. No more “Beta” ;)

It’s always been our goal with Adafruit IO to make the best designed Internet of Things platform for makers of all ages and skill levels. We want it to work well with the connected devices and kits we sell and we built it with an open API to make sure it can work with any device that can speak HTTP or MQTT. Additionally, as a business, we need a way to make the service self-sustaining financially, without making it inaccessible to the makers we are trying to serve.

IO+ will provide more functionality at a single flat rate to the people who need it while our continuing free plan will let anyone build a project on IO, whether it’s their first time working with the Internet of Things or their hundredth.

We will keep standing up for everything we’ve said before in our Internet of Things Bill of Rights, whether you’re using IO+ or IO Free. Our platform is built on open technologies and you control your clients and devices. You own your data and you will always be able to decide what is public and what is private. We want to keep building a system that is easier to use for good and harder to use maliciously. We hope you’ll join us!

So that’s the big news, IO+ is coming in the beginning of October. If all goes as planned, it will be available Monday, October 2. Please stop by the IO Forum or our Discord chat server to ask questions or communicate with us. We’d love to hear from you.

Other changes

We’ll also be launching some changes to the IO interface to clean up navigation and make sure you know what’s up with your plan. Since those changes will go live with the IO+ launch, here are some screenshots to give you a preview:

An updated landing page. Way more information and an attractive new design.

We’ve cleaned up the site navigation and made it easier to tell when you’re looking at Adafruit IO and not www.adafruit.com.

All Feeds and Groups are now on a single page. This should make it easier to see how Groups and Feeds are related, and also easier to manage them.

A new account profile page. This one is going to see more work in the coming months, but for now it’s the one-stop screen to see which plan you’re running and to get access to your account settings.

The account download feature has been rebuilt to be more robust and reliable. (this feature has already launched)

There will, of course, be design tweaks as we get closer to launch and we’re always working on the site, so small things will continue to get better.

That’s all … for now!

We’re still hard at work on IO and there’s lots more to come. If you’re already using Adafruit IO, thank you for continuing on this journey with us, and if you haven’t yet signed up, now is a great time :D

¹: The “IO Beta Thank You” coupon is for any user who has had an account since before September 1, 2017 and will be applied automatically at checkout.

²: If you keep your free account after IO+ launches, we’re not going to delete your feeds, but you won’t be able to create new feeds if you already have more than 5, the IO Free plan limit. If you choose to subscribe to IO+ and then cancel, however, your feeds will be dropped back down to 5 after your subscription ends but you will have plenty of time to download all your data before that happens.

We’ll continue to clarify details as we get closer to launching and if you have any questions please stop by the forums and get in touch.

Just remember this is coming soon, we have not set a date yet 🙂