RSS Feed
Groups, Feeds, and Proper MQTT Topics
Groups, Feeds, and Proper MQTT Topics
We’re deploying a fix to a subtle bug in our MQTT subscription topics today and wanted to write a quick post describing the problem and the fix for people who might currently be using workarounds. MQTT subscription behavior is going to change slightly (for the better) in a way that may break existing systems.
Publishing to “composite” feed keys through our HTTP or MQTT APIs has worked since we introduced that feature with API version 2–that’s anything in the form of group.feed.
You can see an example of those keys here:
But, due to the way our internal MQTT message routing system worked, subscribing to a composite key hasn’t worked until today. Publishing to a feed at username/feeds/group_key.feed_key only sent data to subscribers that used topics that looked like username/feeds/feed_key. From this point in time forwards, however, data published to username/feeds/group_key.feed_key will always only be sent to subscribers to username/feeds/group_key.feed_key.
We are considering this a bug fix, but it introduces some subtle effects for people who add feeds to multiple groups. Here are the three things you need to keep in mind with Adafruit IO and MQTT. The
MQTT Feed Subscription Data Routing
First and most critically, MQTT subscriptions to username/feeds/feed_key for all feeds that are not in your default group will stop working. Some users found that using the bare feed key worked for MQTT subscriptions, but it’s a workaround and not something we want to continue supporting now that we have a fix for it.
Second, subscriptions to feeds that belong to multiple groups will only produce data that is published over MQTT to that group + feed combination. However, all data sent to that feed will still be shown on that feed on the Adafruit IO feed page and any dashboard blocks that use the feed, regardless of the group or groups it belongs to. A practical example of that would be a feed belonging to two groups.
For example, in the case of a group named “Signal” belonging to the “Garage” and “Office” groups, I would end up with something that looks like this on my feeds page:
“Signal” is a single Feed object, but shared between two different groups. If publish to one of those feed keys:
PUBLISH abachman/feeds/garage.signal "1"
Then only MQTT subscribers to abachman/feeds/garage.signal will see the value “1” come through. If I publish to both feed keys:
PUBLISH abachman/feeds/garage.signal "1"
PUBLISH abachman/feeds/office.signal "2"
PUBLISH abachman/feeds/garage.signal "3"
PUBLISH abachman/feeds/office.signal "4"
...
Then the abachman/feeds/garage.signal subscription will only see:
SUBSCRIBE abachman/feeds/garage.signal
>>> "1"
>>> "3"
...
Likewise, group subscriptions will only see the messages published to that feed in the context of that group. So if instead of abachman/feeds/garage.signal I subscribed to abachman/groups/garage I would see:
SUBSCRIBE abachman/groups/garage
>>> {"feeds":{"signal":"1"}}
>>> {"feeds":{"signal":"3"}}
...
Over MQTT, feed subscriptions in the context of a given group will only produce messages that were published to that group + feed.
HTTP Feed Data Requests
Finally, all HTTP data queries for a given feed, regardless of the group context (group.feed key), will produce all data published to that feed. In the case of the example above, that means HTTP GET requests to /api/v2/abachman/feeds/garage.signal and /api/v2/abachman/feeds/office.signal will both produce the same data.
$ export KEY='860b36ff6537fa01e9ee755be3ae18d4fake'
$ export API_URL='https://io.adafruit.com/api/v2'
$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/garage.signal/data.csv?limit=4
id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,garage.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,garage.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,garage.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,garage.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z
$ curl -H "X-AIO-Key: $KEY" $API_URL/abachman/feeds/office.signal/data.csv?limit=4
id,value,feed_id,feed_key,created_at,location,lat,lon,ele,created_epoch,expiration
0DQAZP2V6NZ37PA1662SG30R7G,4,30,office.signal,2017-11-01 21:43:11 UTC,,,,,1509572591,2017-12-01T21:43:11Z
0DQAZP1ADTNFJ990QGSZ6HG9F4,3,30,office.signal,2017-11-01 21:43:06 UTC,,,,,1509572586,2017-12-01T21:43:06Z
0DQAZNZSJJR9ZP5NSS1R6YV56Q,2,30,office.signal,2017-11-01 21:43:01 UTC,,,,,1509572581,2017-12-01T21:43:01Z
0DQAZNY8VC0FS8KMA9TZWBV77Z,1,30,office.signal,2017-11-01 21:42:56 UTC,,,,,1509572576,2017-12-01T21:42:56Z
This is because although the feed belongs to two different groups, it’s still the same underlying Feed record.
Receiving ALL Messages
If you need to see all data published to a specific Feed, you can use the raw feed id value. You can find the corresponding MQTT topic on the “Feed Information” form associated with the Feed you want to track.
Alternatively, every user account has a global data feed at the MQTT topic :username/dashboard/stream/create where :username is your account username. This topic produces a data record for every new data point stored:
SUBSCRIBE abachman/dashboard/stream/create
>>> {
"id": "0DQBM1EPCDA0J6Z89CE1YC26D1",
"value": "5",
"feed_id": 30,
"feed_key": "garage.signal",
"created_at": "2017-11-02T16:41:32Z",
"location": null,
"lat": null,
"lon": null,
"ele": null,
"created_epoch": 1509640892,
"expiration": "2017-12-02T16:41:32Z"
}
With Feeds + Groups, we’ve created a powerful tool for flexibly routing and storing messages over MQTT. A single feed can be used multiple ways by adding it to different groups, which can be helpful if you’re still using a feed-limited plan like IO Basic. IO Plus gives you an unlimited number of feeds, so reuse is less critical, but we expect the flexibility described to be useful in certain cases or else we wouldn’t have built it :D
I’ve published a short GitHub Gist that includes code demonstrating the topic described in this post, you can find that here: https://gist.github.com/abachman/a04694748ad887d5aa7e644e3292fa81.
Thank you for for joining us as we continue to extend and improve Adafruit IO. As always, stop by the Adafruit IO forum or our Discord server in the #adafruit-io room to share any questions or comments, especially when it comes to details of our MQTT support. We’d love to hear about what you’re building!