There’s an old joke that in computer science, there are only two hard problems: cache invalidation, naming things, and off-by-one errors.

It’s taken us a bit of time to get a handle on how we give things inside Adafruit IO names, but we’re making progress. Today we deployed an update to the way Feeds are identified within Adafruit IO that may have short-term negative effects, but should, in the long run, make the whole MQTT / HTTP API easier to understand, anticipate, and use.

I’d like to talk briefly about how we identify Feeds in Adafruit IO and how the rules we’ve set up will effect your code. You can also find this guide at https://learn.adafruit.com/naming-things-in-adafruit-io.

The Two Feed Identifiers

Feeds have two properties that we regularly interact with when we’re talking to the Adafruit IO API: Name and Key.

Name

The Name is the user provided name for this Feed. It should be a “human readable” descriptive term that helps you find your Feed in the web-based user interface and keep track of which code is talking to what feed.

The rules for Feed names are:

• A Feed Name MUST include at least one ASCII letter.
• A Name MAY include any upper or lower case ASCII letters, numbers, spaces, dashes, or underscores (“ “, “-“, or “_”, respectively).
• A new Feed Name MAY NOT be the same as an existing Feed’s Key if the Feeds are in the same group. By default, all newly created Feeds are part of the Group whose name matches your username.
• Names are case-sensitive. This means “Blender” and “BLENDER” are treated as two different Names.

Names are helpful because they let us associate a human friendly word with our data. We see names when we browse io.adafruit.com on the web and when we get Feed records from the HTTP API. We use names when subscribing or publishing to a Feed with the MQTT API.

Some examples of valid, possibly useful names are:

• Temperature
• door one
• 99 Red Balloons
• books_I_would_like_to_read_before_2022

Key

The Key is a system-generated, URL-safe identifier based on the given Feed Name that can be used in API requests to refer to a particular Feed. Keys are generated based on the Name given when the Feed is created and follows strict rules. The rules for Feed keys are simple:

• A Feed Key MAY ONLY contain lower case ASCII letters, numbers, and the dash character (“-“).
• Two Feeds in the same Group may not have the same Key.

These rules in combination with the default Group all Feeds are added to means means a new Feed cannot be created if it will use a duplicate Key and a Feed’s Name cannot be modified if the new Name will produce a Key that conflicts with another Feed in any of the Feed’s Groups.

The rules Adafruit IO uses to generate Keys from Names are roughly:

1. Remove formatting. This step requires a lot of discrete operations, but boils down to transliterating Unicode to ASCII and replacing any non-URL safe characters with “-“.
2. Collapse whitespace and replace with “-“.
3. Collapse all instances of “-“ into a single “-“ and remove them from the beginning and end of the string.
4. Make the whole thing lowercase.

It’s also important to note that when you change a Feed’s Name the Key will also update. We keep Keys in sync with Names whenever a Feed is updated.

Keys are handy because they let us use a human friendly URL when communicating with the AIO API. For example, https://io.adafruit.com/abachman/feeds/beta-test and abachman/f/beta-test are nicer and easier to remember than https://io.adafruit.com/abachman/feeds/588995 or abachman/f/588995.

Aside: Naming things in MQTT

MQTT has its own rules for naming things and in MQTT the things we’re concerned with are called “topics”. If you read Todd’s recent post on MQTT in Adafruit IO, you know we are like an MQTT broker, but we’ve got some extra guidelines. Anyhow, here are the official rules: (you don’t need to memorize these, we handle it for you. They’re just included here for illustration)

• All Topic Names and Topic Filters MUST be at least one character long
• Topic Names and Topic Filters are case sensitive
• Topic Names and Topic Filters can include the space character
• A leading or trailing ‘/’ creates a distinct Topic Name or Topic Filter
• A Topic Name or Topic Filter consisting only of the ‘/’ character is valid
• Topic Names and Topic Filters MUST NOT include the null character (Unicode U+0000)
• Topic Names and Topic Filters are UTF-8 encoded strings, they MUST NOT encode to more than 65535 bytes

Retrieved from the MQTT Version 3.1.1 OASIS Standard, July 8, 2016.

The full MQTT topic used to describe a Feed in Adafruit IO is in the form: username/feeds/identifier where username should be replaced with the username of the account that owns the Feed and identifier should be replaced with the Name or Key that uniquely identifies the Feed you’re talking about.

So, MQTT considers the whole topic test_username/feeds/identifier when validating names but for the purposes of describing Feeds, we’re only considering the identifier portion.

Naming and Accessing Feeds From the io.adafruit.com MQTT API

Naming a Feed on the fly and then referring to it reliably can be tricky. Here are the rules we’re using right now to generate new Feeds and provide continuing access to them from the MQTT interface. For the purposes of demonstration, we’ll be using the example code provided here, but any MQTT publisher or subscriber code should work the same.

1. Listening

Start an MQTT subscription to topic in the form username/f/identifier, for the purpose of the following examples I’ll be using, test_username/f/Test Mode. A Feed with the name “Test Mode” doesn’t exist yet, but that’s okay with the MQTT API. The subscription will remain active and start receiving whenever you start publishing to a Feed whose Name or Key matches the given identifier value exactly.

NOTE: no new Feeds are created in this step.

$ AIO_FEED_NAME='Test Mode' ruby adafruit-errors-sub.rb
CONNECT TO mqtts://test_username:12345@io.adafruit.com
SUB test_username/f/Test Mode

We’ll also start an MQTT subscriber listening to test_username/errors. This will let us see when there are problems with publishing or subscribing to Feeds.

$ ruby adafruit-errors-sub.rb
CONNECT TO mqtts://test_username:12345@io.adafruit.com

2. Initial MQTT publish / creating a new Feed

To create the Feed in Adafruit IO and to start populating it with data, we’ll need to publish and MQTT message to the appropriate topic. In this case, we’re subscribing to a Feed named “Test Mode”, so we’ll need to publish on a Feed with the same name.

Using the example script provided, we’ll publish a simple MQTT message with the topic test_username/f/Test Mode:

$ AIO_FEED_NAME='Test Mode' ruby adafruit-pub.rb
CONNECT TO mqtts://test_username:12345@io.adafruit.com
PUBLISHING TO test_username/f/Test Mode
PUB 2609815 to test_username/f/Test Mode at 2016-07-11 12:53:23 -0400

If this is your first time publishing to the Feed, the subscriber that’s listing to test_username/f/Test Mode should receive its first message:

[test_username/f/Test Mode 2016-07-11 12:53:23 -0400] 2609815

This first is a Feed created message and the second is the actual data received message.

3. Tweaking Names: Publish to a Feed by name with capitalization changed

Once the Feed has been established, publishing to any named Feed whose Key is the same as an existing Feed will add Data to the existing Feeds stream.

PUB 3124870 to test_username/f/test mode at 2016-07-11 12:39:34 -0400

And the original Feed subscriber, which is still watching test_username/f/Test Mode, receives:

[test_username/f/Test Mode 2016-07-11 12:39:34 -0400] 3124870

4. Tweaking Names: Publish to a Feed by key

Once the Feed has been established, publishing to an existing Feed’s Key will add Data to the existing Feeds stream.

PUB 1181702 to test_username/f/test-mode at 2016-07-11 12:42:28 -0400

The Feed subscriber, still watching test_username/f/Test Mode, receives:

[test_username/f/Test Mode 2016-07-11 12:42:28 -0400] 1181702

5. Valid name variations for publishing

When publishing, the method Adafruit IO uses internally to convert a given topic in the form username/feeds/identifier to a specific, existing Feed works like this:

1. Find the Feed belonging to username whose Key is exactly the same as identifier.
2. If no Feed is found, convert the given identifier using the Name-to-Key translation (described above) and find the Feed belonging to username whose Key is exactly the same as the converted value.
3. If no Feed is found, find the Feed belonging to username whose Name is exactly the same as identifier.

Thanks to the Name-to-Key conversion rules, the following topics will all publish to the original Feed created in step 2 and be received by the subscriber at test_username/f/Test Mode:

• test_username/f/Test_Mode
• test_username/f/Test-Mode
• test_username/f/Test Mode
• test_username/f/ Test Mode
• test_username/f/Test Mode
• test_username/f/Test -Mode
• test_username/f/ Test - Mode

And so on, including any variation of modified capitalization.

Some variations that include symbols will be converted to URL-safe Keys when looking up the requested Feed:

• test_username/f/Test(Mode
• test_username/f/Test\[Mode
• test_username/f/Test{Mode
• test_username/f/test modé
• test_username/f/test' mode

6. Valid name variations for subscribing

Subscriptions, on the other hand, must use an exact Name or Key. So, for the given examples, the only topics that will produce the Feed we care about are:

• test_username/f/Test Mode
• test_username/f/test-mode

Naming and Accessing Feeds From the io.adafruit.com HTTP API

The HTTP API follows the same Feed identifying and Name-to-Key conversion rules as the MQTT API because under the hood they’re talking to the same backend. This means if you’re using the Ruby IO client library, the following will produce publications to the same feed as the MQTT examples given above.

require 'rubygems'
require 'adafruit/io'

client = Adafruit::IO::Client.new(key: ENV['AIO_KEY'])

[
  'Test Mode',
  'test mode',
  'test-mode',
  '44'
].each do |feed_ident|
  client.feeds(feed_ident).data.send_data(feed_ident)
end

Potential Problems With Naming

It really stinks to get taken by surprise in a negative way when working with code. Reducing surprise of the unpleasant sort and increasing predictability and stability are the primary motivating factors for the subtle changes this guide introduces.

Publishing to an invalid name

While the Name-to-Key converter keeps things feeling pretty loose and improvisational in terms of referring to Feeds once they exist, if your initial publish is to a Feed that can’t be found it will be rejected if it doesn’t match the rules for valid Feed names.

In the case of our MQTT example, a publish that looks like this:

PUB 2948554 to test_username/f/Test Modes[ at 2016-07-11 15:42:31 -0400

would trigger a message on the error feed that looks like this:

[test_username/errors 2016-07-11 15:42:31 -0400] "Validation failed: Name may contain only letters, digits, underscores, spaces, or dashes"

If the Feed named “Test Modes” already existed, then the publish would work fine, but because it doesn’t Adafruit IO tries to create a Feed with the given identifier, “Test Modes[” as a Name. Since “Test Modes[” is an invalid name, Adafruit IO rejects it :(

Publishing to the wrong identifier

If you set up your MQTT subscription first, it’s important to note that no feed will be created, so the Name-to-Key rules laid out above won’t have the effect you may have anticipated. This happens when the Feed you eventually publish to doesn’t end up with the Key or Name your subscriber has requested. The end result is a subscriber that’s silent while your device is merrily publishing away. Maddening! This is a common error of subscription and publishing when trying to juggle the different identifiers that point to a given Feed.

The safest way to avoid this situation is to make sure that your subscribing topic and your publishing topic are exactly the same. If you want to switch to a different identifier–for example, using a Key instead of a Name–copy the value directly from Adafruit IO. When in doubt, use the Name value.

The Feed we’re publishing to in the MQTT examples above has the following identifiers:

key:  test-mode
name: "Test Mode"

The Name-to-Key translator is how all the “valid name variations” shown above for publishing work, but they only after the Feed already exists. The only way to create this Feed from the MQTT interface is to publish to test_username/f/Test Mode.

Keeping a browser open to your Feeds page while setting up or programming your Adafruit IO devices is recommended.

Modifying a name or key

Remember, changing a Feed’s name will automatically update its Key. This is a change to existing behavior and will require modifications to any systems you’re running that refer to Feeds by Key, but it prevents more confusing situations from occurring.

Here’s a non-hypothetical scenario that illustrates the trouble when we don’t keep Keys and Names in sync:

• I make a new Feed and call it, “Light Switch” (IoT light switch, low risk). In JSON, the Feed looks like:

  {
    "name": "Light Switch",
    "key": "light-switch"
  }
  

• I have a Feather Huzzah controlling a relay, acting as a subscriber listening to username/f/light-switch and a publisher sending data to username/f/Light Switch. Things talk, everything is great with the world.
• I move the hardware over to a new spot and rename the Feed, “Blender Toggle” (IoT blender, high risk). In a non-sync world, the Feed’s new Name is “Blender Toggle” and it’s Key is still “light-switch. The Feed is now:

  {
    "name": "Blender Toggle",
    "key": "light-switch"
  }
  

• My subscriber is still listening to username/f/light-switch, so it still gets all the Feed’s messages.
• I build a new remote control and have it publish to username/f/Blender Toggle and it works, because there is a Feed with that exact name. Everything just keeps working, which is okay for now.
• Later on, I decide I’d like to build another remote control light switch, so I put together an Arduino MKR1000 publishing to username/f/Light Switch.
• My new light switch controlling, MKR1000-powered, motion sensor publishes a message to username/f/Light Switch and my blender turns on! What the heck! Hope you’re wearing Kevlar gloves!

The MQTT Feed routing rules described above mean that a message received on the topic username/f/Light Switch gets routed to the Feed whose Key matches light-switch, which already exists. And controls the blender. Which should not turn on unexpectedly when the room gets dark :P

This is why we keep all Feed Keys updated to match to their respective Feed Names. In Adafruit IO, renaming the Feed to “Blender Toggle” changes the Key to “blender-toggle”.

Old Feed:

{
    "name": "Light Switch",
    "key": "light-switch"
}

Change Name to “Blender Toggle” and the Feed now looks like:

{
    "name": "Blender Toggle",
    "key": "blender-toggle"
}

My existing subscriber would immediately stop working because none of the messages sent to username/f/Blender Toggle (new Name) will get routed to the subscriber listening to username/f/light-switch (old Key). This doesn’t mean the subscriber breaks or shuts down, only that it stops getting messages for now. This is my chance to realize something is wrong and debug my system.

Here’s the tricky bit, if I go in and make a new Feed and name it “Light Switch”, it’ll get the Key “light-switch”. If I didn’t update my subscriber when it stopped working, that means it’s still listening to username/f/light-switch. When I start posting to my new Feed at username/f/Light Switch, the old subscriber at username/f/light-switch will start getting messages again.

The best defense against confusion is to refer to the Adafruit IO web interface to double check what the Feed you’re working with has for Name and Key values. And, when you make changes in Adafruit IO always make sure the cooperating systems are updated, especially when dealing with control systems that interact with the world.

Summary

If you got this far, I hope it’s clear that this is an area of Adafruit IO we’ve put a particular amount of thought into. Our intention continues to be building a clear, simple, powerful platform for connecting the things you build and we think this refinement supports that intention.

Please join us at the Adafruit IO forum and share your thoughts, projects, questions, or requests. We’d love to talk to you about what we’re building!

We introduced time utilities to Adafruit IO about a month ago, but we haven’t provided any examples of how to use the feature. To correct this oversight, we added an example to v0.14.2 of the Adafruit MQTT Arduino Library. This example was a response to a feature request from @phlemoine in the io-issues GitHub repo:

I am looking for a way to trigger at a specific time in the day… how can I set up the time the trigger starts ? Same for a 12 hours or even 4 hours …

Here is the relevant code that allows code to be run at 4 hour intervals (midnight, 4am, 8am, etc):

Adafruit_MQTT_Subscribe timefeed = Adafruit_MQTT_Subscribe(&mqtt, "time/seconds");

// set timezone offset from UTC
int timeZone = -4; // UTC - 4 eastern daylight time (nyc)
int interval = 4; // trigger every X hours
int hour = 0; // current hour

void timecallback(uint32_t current) {

  // stash previous hour
  int previous = hour;

  // adjust to local time zone
  current += (timeZone * 60 * 60);

  // calculate current hour
  hour = (current / 60 / 60) % 24;

  // only trigger on interval
  if((hour != previous) && (hour % interval) == 0) {
    Serial.println("Run your code here");
  }

}

The full example can be found in the Adafruit_MQTT Arduino Library on GitHub. You will need to install or update the Adafruit MQTT Library to version 0.14.2 in the Arduino Library Manager, and open the adafruitio_time_esp8266 example to get started.

Are these examples helpful? Please visit our IO forum and share your thoughts.

v0.14.1 of the Adafruit MQTT Library for Arduino introduces subscription callbacks. This change will allow users to organize their code into separate blocks by attaching callbacks to feed and group subscriptions. Here’s a simplified example of what the change looks like:

setup() {

  // add the rest of setup code here

  // register callback for feed
  onoffbutton.setCallback(onoffcallback);

}

loop() {

  // Ensure the connection to the MQTT server is alive (this will make the first
  // connection and automatically reconnect when disconnected).
  // See the MQTT_connect function in the full example for more info.
  MQTT_connect();

  // wait 10 seconds for subscription messages
  // since we have no other tasks in this example.
  mqtt.processPackets(10000);

  // keep the connection alive
  mqtt.ping();

}

void onoffcallback(char *data, uint16_t len) {
  Serial.print("Hey we're in a onoff callback, the button value is: ");
  Serial.println(data);
}

In the example above, you can see that we added the call to mqtt.processPackets(10000);. This tells the library to wait 10 seconds (10000ms) for incoming packets before moving on to the next task, and it will block all other code execution. If your sketch is only waiting for subscription messages, then 10 seconds might be a good timeout, but if your sketch handles other tasks, you may want to reduce the timeout to 1 second (1000ms).

The full example with multiple feed subscriptions can be found in the Adafruit_MQTT Arduino Library on GitHub. You will need to install or update the Adafruit MQTT Library to version 0.14.1 in the Arduino Library Manager, and open the mqtt_esp8266_callback example to get started.

Are these examples helpful? Please visit our IO forum and share your thoughts.

Unlike HTTP, the MQTT protocol does not provide a standard way of reporting errors to connected clients. This poses a problem for Adafruit IO, because there are multiple reasons why sending data to Adafruit IO might fail.

Here are a few common examples:

• Publishing data to an invalid topic
• Publishing data to an unauthorized topic
• Publishing data too fast (rate limit)
• Exceeding the number of feeds for your account (10 currently)
• Sending too large of a payload

Adafruit IO is not a standard MQTT broker. Normally MQTT brokers will rebroadcast published data to any authorized subscribed client as soon as the data is received, so errors are rarely encountered.

The standard MQTT pub/sub workflow looks something like this:

Adafruit IO’s message flow is a bit more complicated, and looks something like this:

In the simplified example above, you can see that there are multiple steps that need to be completed before data is rebroadcasted to subscribed clients. If any of these steps fail, data will not make it back to subscribed clients. The reasons for failure are often unclear to users, so we added two special MQTT topics that will help inform users of any issues related to publishing data.

The first topic you can subscribe to is the error topic for your user. This topic will receive any errors related to data validation and database saves. The topic path looks like this:

{username}/errors

For example, if your username was ladyada, you would subscribe to the following topic using your favorite MQTT client:

ladyada/errors

The second topic you can subscribe to is the throttle topic for your user. This topic contains any messages related to rate limits on your account. You might see these if you are publishing too fast. The topic path looks like this:

{username}/throttle

Again, if your username was ladyada, you would subscribe to the following topic:

ladyada/throttle

If you are using the Adafruit MQTT Library, you can define the throttle and error subscriptions at the top of your sketch like this:

/*************************** Error Reporting *********************************/

const char ERRORS[] PROGMEM = AIO_USERNAME "/errors";
Adafruit_MQTT_Subscribe errors = Adafruit_MQTT_Subscribe(&mqtt, ERRORS);

const char THROTTLE[] PROGMEM = AIO_USERNAME "/throttle";
Adafruit_MQTT_Subscribe throttle = Adafruit_MQTT_Subscribe(&mqtt, THROTTLE);

Then, subscribe to the topics at the bottom of the setup function:

void setup() {

  // ...network connection setup code here

  // MQTT subscriptions for throttle & error messages
  mqtt.subscribe(&throttle);
  mqtt.subscribe(&errors);

}

Finally, inside of the main loop, print any errors to the serial monitor:

void loop() {

  // Ensure the connection to the MQTT server is alive (this will make the first
  // connection and automatically reconnect when disconnected).  See the MQTT_connect
  // function definition further below.
  MQTT_connect();

  // this is our 'wait for incoming subscription packets' busy subloop
  // try to spend your time here
  Adafruit_MQTT_Subscribe *subscription;
  while ((subscription = mqtt.readSubscription(5000))) {
    if(subscription == &errors) {
      Serial.print(F("ERROR: "));
      Serial.println((char *)errors.lastread);
    } else if(subscription == &throttle) {
      Serial.print(F("THROTTLED: "));
      Serial.println((char *)throttle.lastread);
    }
  }

  mqtt.ping();

}

You can see the full example for the ESP8266 in the Adafruit_MQTT Arduino Library on GitHub. Install or update the Adafruit MQTT Library to version 0.13.3 in the Arduino Library Manager, and open the adafruitio_errors_esp8266 example to get started.

Are these examples helpful? Please visit our IO forum and share your thoughts.

The ESP8266 based Adafruit HUZZAH breakout and the Adafruit Feather HUZZAH are both popular options to use with Adafruit IO. Because ESP8266 SSL/TLS support is fairly new, most of our Adafruit IO examples use the insecure MQTT port 1883.

Why is this a problem? The MQTT protocol is an insecure protocol on it’s own. All data (including username & password) are sent in the clear, so SSL/TLS is required to protect any sensitive information.

What are SSL & TLS? Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), are cryptographic protocols that you use whenever you browse the internet. Have you ever noticed the lock symbol 🔒 in your browser’s URL bar? That lock symbol represents a secure connection between your browser & a web site over SSL/TLS. We use those same protocols to secure traffic between your ESP8266 and Adafruit IO.

We have added an example to the Adafruit_MQTT Arduino Library that you can use to secure communication between your ESP8266 and Adafruit IO. Install or update the Adafruit MQTT Library to version 0.13.2 in the Arduino Library Manager, and open the adafruitio_secure_esp8266 example to get started.

The main changes to the standard ESP8266 example are that WiFiClientSecure is used in place of WiFiClient, and port 8883 is used instead of MQTT port 1883. The sketch also checks the fingerprint for the io.adafruit.com certificate using the verifyFingerprint function.

A simplified summary of the changes can be seen below:

#define AIO_SERVER      "io.adafruit.com"
#define AIO_SERVERPORT  8883                   // 8883 for MQTTS

// WiFiFlientSecure for SSL/TLS support
WiFiClientSecure client;

// io.adafruit.com SHA1 fingerprint
const char* fingerprint = "26 96 1C 2A 51 07 FD 15 80 96 93 AE F7 32 CE B9 0D 01 55 C4";

void verifyFingerprint() {

  const char* host = AIO_SERVER;

  Serial.print("Connecting to ");
  Serial.println(host);

  if (! client.connect(host, AIO_SERVERPORT)) {
    Serial.println("Connection failed. Halting execution.");
    while(1);
  }

  if (client.verify(fingerprint, host)) {
    Serial.println("Connection secure.");
  } else {
    Serial.println("Connection insecure! Halting execution.");
    while(1);
  }

}