• State of IO 03.22.17

    Here are the stats for the past week:

    * 60.7 million inserts of logged data in the last 7 days
    * 24,942 users
    * 19,888 online feeds (68,525 feeds total)
    * ~95 inserts per second via MQTT
    * ~12 inserts per second via REST API
    

    This past week we’ve been focusing on mostly behind the scenes functionality as we prepare to migrate data to a new backend. Most of our focus right now is on maintaining a high uptime. New features are in the works as well.

    We’re also responding to issues you may run into on our Adafruit IO Forum. Please post any issues with as much detail as you can. It definitely helps us out when fixing the bugs.

    Also, if you’re still using version 1 of our API, you may want to upgrade to the latest version 2. It has more features, and is generally better designed and easier to use than our previous API. The default for our docs is V2 now as well.

  • State of IO 02.21.17

    Here are the stats for the past week:

    * 63.5 million inserts of logged data in the last 7 days
    * 23,365 users
    * 18,096 online feeds (63,209 feeds total)
    * ~100 inserts per second via MQTT
    * ~15 inserts per second via REST API
    

    This past week we’ve increased the size of at least one server, and made a fairly major database migration that is the first step towards long-term stability and speed. We have more changes in the pipeline, and will make announcements here and on the forums when they are ready to go.

    We’re also responding to issues you may run into on our Adafruit IO Forum. Please post any issues with as much detail as you can. It definitely helps us out when fixing the bugs.

    Also, if you’re still using version 1 of our API, you may want to upgrade to the latest version 2. It has more features, and is generally better designed and easier to use than our previous API. The default for our docs is V2 now as well.

  • User Submitted Tutorial - Guide for using Adafruit IO with Raspberry Pi and IFTTT

    A user on a reddit post has put together a great guide on general connectivity with IO from a Raspberry Pi and IFTTT. They cover the end-to-end setup from signing up for IO, IFTTT and example sketches.

    I’ll describe the basic steps for sending data from the RPi to Adafruit.io and then getting IFTTT to take an action based on it, but you can go in the other direction too: have IFTTT post information to Adafruit.io then have the RPi take some action based on that.

    We also have a guide that explains how to setup a door sensor that triggers through IFTTT as well.

    If you would like your Adafruit IO tutorial to be featured, share it with us via Twitter (@adafruit) or Instagram (@adafruit) by using the hashtag #adafruitio.

  • State of IO 01.18.17

    Here are the stats for the past week:

    * 70.2 million inserts of logged data in the last 7 days
    * 21,682 users
    * 16,420 online feeds (57,551 feeds total)
    * ~110 inserts per second via MQTT
    * ~12 inserts per second via REST API
    
  • Adafruit IO User Interface Update

    We’ve deployed an all-new Adafruit IO user interface that should be faster, more responsive and streamlined than our previous UI. Beyond being faster and generally nicer to look at, another benefit of this change is it’s going to allow us to develop new features and fix existing bugs much more quickly than before. The high level changelog of the new features is below.

    This should only impact the front-end browser UI. There should be no impact to your device connectivity or API/MQTT calls.

    The UI is now directly using our API V2. It’s recommended that you also utilize the API V2 going forward. We’re working on updating our documentation and tutorials to indicate this as well. Also, if you’re using one of the many compatible devices, we highly recommend using our new client library that significantly simplifies and wraps the boilerplate needed to get a project going: https://github.com/adafruit/Adafruit_IO_Arduino

    IO Changelog (high level):

    • Updated design across the site
    • Faster, more responsive and streamlined than before
    • New charting features
    • New Dashboard layout engine
    • More granular permissions (in API V2) for your data
    • New Activity page which should be more useful than before
    • New functional sidebar in feeds page
    • New dashboard block creation experience
    • Bug fixes and performance optimizations
  • State of IO 01.10.17

    Here are the stats for the past week:

    * 70.0 million inserts of logged data in the last 7 days
    * 21,259 users
    * 16,063 online feeds (56,401 feeds total)
    * ~110 inserts per second via MQTT
    * ~12 inserts per second via REST API
    
  • State of IO 12.14.16

    Here are the stats for the past week:

    * 65.3 million inserts of logged data in the last 7 days
    * 19,940 users
    * 15,004 online feeds (52,527 feeds total)
    * ~100 inserts per second via MQTT
    * ~12 inserts per second via REST API
    
  • Quick Tip: Adding a Power Switch to your Feather

    Here’s a quick tip from @ladyada that will show you how you can use the EN (enable) pin to control the power on your Adafruit Feather with a SPDT slide switch.

    Power Switch

    Here is how to connect the switch to your Feather:

    • Feather GND -> Center pin on the switch
    • Feather EN -> One of the outer pins on the switch

    You can now turn your Feather off by flipping the switch to the side that is connected to the EN pin.

    For more info about the EN (enable) pin, check out this guide.

  • User Submitted Tutorial - Magnetic Door Spy

  • State of IO 12.06.16

    Here are the stats for the past week:

    * 63.6 million inserts of logged data in the last 7 days
    * 19,537 users
    * 14,583 online feeds (51,303 feeds total)
    * ~100 inserts per second via MQTT
    * ~10 inserts per second via REST API
    
  • Loading Data from Adafruit IO into Pandas

    Pandas

    If you are logging sensor data to Adafruit IO, you may have a need to do a deeper examination of your data. One popular option is the pandas data analysis library for Python.

    Here is a description from the pandas website:

    pandas is an open source, BSD-licensed library providing high-performance, easy-to-use data structures and data analysis tools for the Python programming language.

    Let’s take a quick look at how to load data into pandas from a public Adafruit IO feed.

    Setup

    You will need Python 3.5 installed, and you will also need to install pip if it wasn’t included with your Python install.

    Once Python and pip are installed, you can install pandas by running:

    $ pip install pandas
    

    For this example, we will be pointing pandas at a public Adafruit IO feed. If your feed is currently private, you will need to make it public.

    Public Feed

    Importing Feed Data

    To get started, you will need to open up a new Python file in your favorite editor, and start by importing pandas:

    import pandas as pd
    

    Next, we need to pull in feed data via the read_json method. Replace USERNAME and FEED_KEY with your Adafruit IO username, and the key of the feed you wish to load.

    feed_data = pd.read_json('https://io.adafruit.com/api/v2/USERNAME/feeds/FEED_KEY/data')
    

    If you are not sure where to find your feed’s key, you can find it below the feed name.

    Feed Key

    Date Conversion

    Next, we will convert the created_at column from a string to datetime, and set it as the index for the data frame:

    feed_data['created_at'] =  pd.to_datetime(feed_data['created_at'], infer_datetime_format=True)
    feed_data = feed_data.set_index('created_at')
    

    Plot Values on a Chart

    Now that the data has a valid date column, we can load the feed values into a time series, and plot the values:

    values = pd.Series(feed_data['value'])
    values.plot()
    

    Pandas Line

    Another simple example is to resample the data into daily averages, and plot them on a bar chart:

    daily = values.resample('1d').mean()
    daily.plot(kind='barh')
    

    Pandas Bar

    You can do a wide range of useful operations with Time Series, and you should refer to the documentation for more details.

  • State of IO 11.23.16

    Here are the stats for the past week:

    * 56.1 million inserts of logged data in the last 7 days
    * 18,941 users
    * 14,043 online feeds (49,321 feeds total)
    * ~90 inserts per second via MQTT
    * ~10 inserts per second via REST API
    
  • State of IO 11.07.16

    Here are the stats for the past week:

    * 54.7 million inserts of logged data in the last 7 days
    * 18,147 users
    * 13,309 online feeds (46,968 feeds total)
    * ~77 inserts per second via MQTT
    * ~10 inserts per second via REST API
    
  • State of IO 10.05.16

    Here are the stats for the past week:

    * 48.2 million inserts of logged data in the last 7 days
    * 16,798 users
    * 12,383 online feeds (42,857 feeds total)
    * ~75 inserts per second via MQTT
    * ~10 inserts per second via REST API
    
  • SSH Config Includes

    OpenSSH version 7.3 introduced a very handy Include feature, which is great for people who have to manage connection info for multiple servers. This makes it easy for us to generate updated SSH configs via AWS CLI for the multiple EC2 instances that serve Adafruit IO.

    Here is how you can use Include to pull in separate SSH config files from your main ~/.ssh/config. First, you will need to install OpenSSH version 7.3 or higher. If you are using Linux, you will need to install version 7.3+ via your package manager (yum,apt-get, etc), or build it from source.

    On OS X, you can do this via homebrew:

    $ brew install homebrew/dupes/openssh
    

    Confirm that you are now running 7.3 or higher by running ssh -V:

    $ ssh -V
    OpenSSH_7.3p1, OpenSSL 1.0.2j  26 Sep 2016
    

    Now you can create a new child config file in ~/.ssh using a text editor. For example, we can create an example child config at ~/.ssh/pi_config and add configuration info just as we would in the main SSH config file:

    Host pi-1
      HostName 10.0.0.10
      User pi
      IdentityFile ~/.ssh/pi_cluster
    
    Host pi-2
      HostName 10.0.0.11
      User pi
      IdentityFile ~/.ssh/pi_cluster
    
    Host pi-3
      HostName 10.0.0.12
      User pi
      IdentityFile ~/.ssh/pi_cluster
    

    From your main ~/.ssh/config, add the following line at the top:

    Include ~/.ssh/pi_config
    

    You should now be able to connect to your servers as you normally would:

    $ ssh pi-1
    

    That’s it! Check out this OpenSSH feature request if you would like more info.

  • State of IO 09.20.16

    Here are the stats for the past week:

    * 45.3 million inserts of logged data in the last 7 days
    * 16,227 users
    * 11,985 online feeds (41,024 feeds total)
    * ~75 inserts per second via MQTT
    * ~10 inserts per second via REST API
    
  • Exporting Data from Adafruit IO

    Here’s a quick tip to help you export one of your Adafruit IO feeds as CSV, Excel, JSON, or XML. First, you will need to navigate to io.adafruit.com/feeds. Once your feed list has loaded, select the feed you wish to export:

    Feed Select

    Then, click the download button below the chart:

    Download Feed

    A modal will pop up and allow you to download your feed as CSV, Excel, JSON, or XML:

    Feed Format

    If you would like to download all a copy of all of your Adafruit IO data, you can do this by navigating to io.adafruit.com/settings, and clicking the Download All Data button. This button will currently only allow you to download your data in JSON format.

    Download All

    Was this tip helpful? Please visit our IO forum to ask questions or share your thoughts.

  • State of IO 09.06.16

    Here are the stats for the past week:

    * 44.40 million inserts of logged data in the last 7 days
    * 15,671 users
    * 11,503 online feeds (39,364 feeds total)
    * ~70 inserts per second via MQTT
    * ~10 inserts per second via REST API
    
  • Hardware Testing with Jenkins & TAP

    Jenkins CI

    At Adafruit, when we push or merge changes to one of our hundreds of Arduino libraries on GitHub, we have to ask ourselves “Does this change work?”. In a lot of cases, the best answer we can give, even after extensive manual testing, is “Maybe?”. For Adafruit IO, it is important that we are able to easily answer “Yes.” to that question. A large portion of Adafruit IO users could be impacted by a bug in a library such as the Adafruit IO Arduino Library, and we want to make sure our client code is reliable.

    The hard truth is that there aren’t enough hours in the day to test each change to every library on all of the supported platforms, so most of the time we have to settle with testing one or two platforms manually, and hoping for the best on the rest of the supported platforms.

    What do we mean by platform? In this case, we are talking about the multiple microcontrollers that are supported in the Arduino IDE via the Arduino 1.6.x Third Party Hardware Specification. Arduino Uno, ESP8266, Feather M0 (ATSAMD21), and the Feather WICED (STM32F205) are all good examples different platforms that require separate testing.

    First Attempt: Build Verification using Travis CI

    We had previously attempted automated build checking using Travis CI, and that method works great for checking if sketches build on multiple platforms.

    Travis CI

    As you may have guessed, this leaves us with a huge blind spot. A passing build does not mean the code will do what you expect it to do.

    Second Attempt: Manual Compatibility Testing

    Our next attempt involved building a ‘compatibility matrix’ web app that allowed for easy logging of the results of manual tests on supported platforms. The app pushes test results to a GitHub repo as JSON (in case we need to access it programatically). It also automatically adds/updates a compatibility table in the README of the target library.

    Compatibility Matrix

    Here’s an example of the README output:

    Compatibility README Example

    This method is very effective for answering “Does this change work?”, but it is very impractical to manually test the daily changes to libraries on every supported platform.

    Latest Attempt: Running Tests on Hardware

    Our latest approach to this problem uses Jenkins CI running on an AWS EC2 instance, and small fleet of local Raspberry Pi test nodes. Each node uploads and runs unit & integration tests on hardware attached to USB ports.

    Hardware

    Each Raspberry Pi can support multiple platforms, and each platform can be targeted by the test. The tests themselves output results using the TAP protocol over a serial connection, and the Raspberry Pi sends the results back to Jenkins so we are aware when builds fail.

    Here’s a visual overview of how it works:

    Jenkins Workflow

    Test status can be monitored in real time from anywhere, and we are able to view the raw console output from the test nodes when needed.

    Jenkins Stage View

    TAP results are also parsed and attached to each build:

    Jenkins TAP View

    We will be sharing the details of this in a tutorial soon, but if you would like to take a look at our early experiments with Jenkins, head to jenkins.adafruit.com or check out our Jenkins Pipeline Library on GitHub.

  • State of IO 08.23.16

    Here are the stats for the past week:

    * 37.3 million inserts of logged data in the last 7 days
    * 15,121 users
    * 11,080 online feeds (37,600 feeds total)
    
  • Adafruit IO Arduino v2.0 Beta Release

    We have been working hard on a new version of the Adafruit IO Arduino Library. We are releasing it in ‘beta’ form today for existing IO users to try, and we will be adding new tutorials and updating existing tutorials with usage examples very soon.

    Device Independence

    v2.0.0 of the library provides a simple device independent interface for interacting with Adafruit IO. This allows you to switch beween WiFi (ESP8266, M0 WINC1500, & WICED), Cellular (32u4 FONA), and Ethernet (Ethernet FeatherWing) with only a two line change in your sketch. No changes are required to the sketch to switch between any of the supported Feather WiFi boards.

    The included examples focus on specific concepts without extra boilerplate related to setting up the specific WiFi, cell, or ethernet hardware being used. This will allow you to prototype your sketch on WiFi hardware, and easily move to cellular or ethernet with a very small change to your config file.

    Location

    The library makes it simple to publish GPS location info with your data. For example, if you wanted to publish your location with your current speed, you would send it like this:

    car->save(speed, lat, lon, ele);
    

    You could then use the map block on io.adafruit.com to display your location info. The included adafruitio_04_location example demonstrates how to send and receive location info.

    Type Conversion

    The library adds type conversion helpers for both publishing and receiving data. You can publish any data type to the save(value) method of your feed, and you can use helpers like toInt(), toBool(), & toFloat() to easily convert received messages to the appropriate data type. The included adafruitio_05_type_conversion example demonstrates converting to and from all available types.

    Updating to v2.0.0

    IO Library

    The README for the library has more info about required dependencies, and you should check there to make sure you have all of the requirements for the hardware you are using. Make sure you install v2.0.0 or higher of the Adafruit IO Arduino Library, and v0.16.0 of the Adafruit MQTT Library using the Arduino IDE’s Library Manager.

    MQTT Library

  • State of IO 08.10.16

    Here are the stats for the past week:

    * 37.6 million inserts of logged data in the last 7 days
    * 14,701 users
    * 10,713 online feeds (36,320 feeds total)
    * ~60 inserts per second via MQTT
    * ~10 inserts per second via REST API
    
  • State of IO 07.19.16

    Here are the stats for the past week:

    * 37.3 million inserts of logged data in the last 7 days
    * 13,840 users
    * 10,138 online feeds (33,776 feeds total)
    * ~60 inserts per second via MQTT
    * ~10 inserts per second via REST API
    

    To help with the increased load, we have added an additional sidekiq server to help process jobs, and it has helped with ensuring that messages from users are processed as soon as they arrive. The image below shows how adding the new server has helped smooth out job processing over the last day.

    MQTT Graph

    This week Adafruit IO passed the 10,000 online feeds milestone!

    10k Online Feeds

    We have some exciting new things coming soon, and we are always looking for feedback about Adafruit IO. Please post any questions, feature requests, or show off your project in the Adafruit IO forum.

  • Naming Feeds

    There’s an old joke that in computer science, there are only two hard problems: cache invalidation, naming things, and off-by-one errors.

    It’s taken us a bit of time to get a handle on how we give things inside Adafruit IO names, but we’re making progress. Today we deployed an update to the way Feeds are identified within Adafruit IO that may have short-term negative effects, but should, in the long run, make the whole MQTT / HTTP API easier to understand, anticipate, and use.

    I’d like to talk briefly about how we identify Feeds in Adafruit IO and how the rules we’ve set up will effect your code. You can also find this guide at https://learn.adafruit.com/naming-things-in-adafruit-io.

    The Two Feed Identifiers

    Feeds have two properties that we regularly interact with when we’re talking to the Adafruit IO API: Name and Key.

    Name

    The Name is the user provided name for this Feed. It should be a “human readable” descriptive term that helps you find your Feed in the web-based user interface and keep track of which code is talking to what feed.

    The rules for Feed names are:

    • A Feed Name MUST include at least one ASCII letter.
    • A Name MAY include any upper or lower case ASCII letters, numbers, spaces, dashes, or underscores (“ “, “-“, or “_”, respectively).
    • A new Feed Name MAY NOT be the same as an existing Feed’s Key if the Feeds are in the same group. By default, all newly created Feeds are part of the Group whose name matches your username.
    • Names are case-sensitive. This means “Blender” and “BLENDER” are treated as two different Names.

    Names are helpful because they let us associate a human friendly word with our data. We see names when we browse io.adafruit.com on the web and when we get Feed records from the HTTP API. We use names when subscribing or publishing to a Feed with the MQTT API.

    Some examples of valid, possibly useful names are:

    • Temperature
    • door one
    • 99 Red Balloons
    • books_I_would_like_to_read_before_2022

    Key

    The Key is a system-generated, URL-safe identifier based on the given Feed Name that can be used in API requests to refer to a particular Feed. Keys are generated based on the Name given when the Feed is created and follows strict rules. The rules for Feed keys are simple:

    • A Feed Key MAY ONLY contain lower case ASCII letters, numbers, and the dash character (“-“).
    • Two Feeds in the same Group may not have the same Key.

    These rules in combination with the default Group all Feeds are added to means means a new Feed cannot be created if it will use a duplicate Key and a Feed’s Name cannot be modified if the new Name will produce a Key that conflicts with another Feed in any of the Feed’s Groups.

    The rules Adafruit IO uses to generate Keys from Names are roughly:

    1. Remove formatting. This step requires a lot of discrete operations, but boils down to transliterating Unicode to ASCII and replacing any non-URL safe characters with “-“.
    2. Collapse whitespace and replace with “-“.
    3. Collapse all instances of “-“ into a single “-“ and remove them from the beginning and end of the string.
    4. Make the whole thing lowercase.

    It’s also important to note that when you change a Feed’s Name the Key will also update. We keep Keys in sync with Names whenever a Feed is updated.

    Keys are handy because they let us use a human friendly URL when communicating with the AIO API. For example, https://io.adafruit.com/abachman/feeds/beta-test and abachman/f/beta-test are nicer and easier to remember than https://io.adafruit.com/abachman/feeds/588995 or abachman/f/588995.

    Aside: Naming things in MQTT

    MQTT has its own rules for naming things and in MQTT the things we’re concerned with are called “topics”. If you read Todd’s recent post on MQTT in Adafruit IO, you know we are like an MQTT broker, but we’ve got some extra guidelines. Anyhow, here are the official rules: (you don’t need to memorize these, we handle it for you. They’re just included here for illustration)

    • All Topic Names and Topic Filters MUST be at least one character long
    • Topic Names and Topic Filters are case sensitive
    • Topic Names and Topic Filters can include the space character
    • A leading or trailing ‘/’ creates a distinct Topic Name or Topic Filter
    • A Topic Name or Topic Filter consisting only of the ‘/’ character is valid
    • Topic Names and Topic Filters MUST NOT include the null character (Unicode U+0000)
    • Topic Names and Topic Filters are UTF-8 encoded strings, they MUST NOT encode to more than 65535 bytes

    Retrieved from the MQTT Version 3.1.1 OASIS Standard, July 8, 2016.

    The full MQTT topic used to describe a Feed in Adafruit IO is in the form: username/feeds/identifier where username should be replaced with the username of the account that owns the Feed and identifier should be replaced with the Name or Key that uniquely identifies the Feed you’re talking about.

    So, MQTT considers the whole topic test_username/feeds/identifier when validating names but for the purposes of describing Feeds, we’re only considering the identifier portion.

    Naming and Accessing Feeds From the io.adafruit.com MQTT API

    Naming a Feed on the fly and then referring to it reliably can be tricky. Here are the rules we’re using right now to generate new Feeds and provide continuing access to them from the MQTT interface. For the purposes of demonstration, we’ll be using the example code provided here, but any MQTT publisher or subscriber code should work the same.

    1. Listening

    Start an MQTT subscription to topic in the form username/f/identifier, for the purpose of the following examples I’ll be using, test_username/f/Test Mode. A Feed with the name “Test Mode” doesn’t exist yet, but that’s okay with the MQTT API. The subscription will remain active and start receiving whenever you start publishing to a Feed whose Name or Key matches the given identifier value exactly.

    NOTE: no new Feeds are created in this step.

    $ AIO_FEED_NAME='Test Mode' ruby adafruit-errors-sub.rb
    CONNECT TO mqtts://test_username:12345@io.adafruit.com
    SUB test_username/f/Test Mode
    

    We’ll also start an MQTT subscriber listening to test_username/errors. This will let us see when there are problems with publishing or subscribing to Feeds.

    $ ruby adafruit-errors-sub.rb
    CONNECT TO mqtts://test_username:12345@io.adafruit.com
    

    2. Initial MQTT publish / creating a new Feed

    To create the Feed in Adafruit IO and to start populating it with data, we’ll need to publish and MQTT message to the appropriate topic. In this case, we’re subscribing to a Feed named “Test Mode”, so we’ll need to publish on a Feed with the same name.

    Using the example script provided, we’ll publish a simple MQTT message with the topic test_username/f/Test Mode:

    $ AIO_FEED_NAME='Test Mode' ruby adafruit-pub.rb
    CONNECT TO mqtts://test_username:12345@io.adafruit.com
    PUBLISHING TO test_username/f/Test Mode
    PUB 2609815 to test_username/f/Test Mode at 2016-07-11 12:53:23 -0400
    

    If this is your first time publishing to the Feed, the subscriber that’s listing to test_username/f/Test Mode should receive its first message:

    [test_username/f/Test Mode 2016-07-11 12:53:23 -0400] 2609815
    

    This first is a Feed created message and the second is the actual data received message.

    3. Tweaking Names: Publish to a Feed by name with capitalization changed

    Once the Feed has been established, publishing to any named Feed whose Key is the same as an existing Feed will add Data to the existing Feeds stream.

    PUB 3124870 to test_username/f/test mode at 2016-07-11 12:39:34 -0400
    

    And the original Feed subscriber, which is still watching test_username/f/Test Mode, receives:

    [test_username/f/Test Mode 2016-07-11 12:39:34 -0400] 3124870
    

    4. Tweaking Names: Publish to a Feed by key

    Once the Feed has been established, publishing to an existing Feed’s Key will add Data to the existing Feeds stream.

    PUB 1181702 to test_username/f/test-mode at 2016-07-11 12:42:28 -0400
    

    The Feed subscriber, still watching test_username/f/Test Mode, receives:

    [test_username/f/Test Mode 2016-07-11 12:42:28 -0400] 1181702
    

    5. Valid name variations for publishing

    When publishing, the method Adafruit IO uses internally to convert a given topic in the form username/feeds/identifier to a specific, existing Feed works like this:

    1. Find the Feed belonging to username whose Key is exactly the same as identifier.
    2. If no Feed is found, convert the given identifier using the Name-to-Key translation (described above) and find the Feed belonging to username whose Key is exactly the same as the converted value.
    3. If no Feed is found, find the Feed belonging to username whose Name is exactly the same as identifier.

    Thanks to the Name-to-Key conversion rules, the following topics will all publish to the original Feed created in step 2 and be received by the subscriber at test_username/f/Test Mode:

    • test_username/f/Test_Mode
    • test_username/f/Test-Mode
    • test_username/f/Test Mode
    • test_username/f/ Test Mode
    • test_username/f/Test Mode
    • test_username/f/Test -Mode
    • test_username/f/ Test - Mode

    And so on, including any variation of modified capitalization.

    Some variations that include symbols will be converted to URL-safe Keys when looking up the requested Feed:

    • test_username/f/Test(Mode
    • test_username/f/Test\[Mode
    • test_username/f/Test{Mode
    • test_username/f/test modé
    • test_username/f/test' mode

    6. Valid name variations for subscribing

    Subscriptions, on the other hand, must use an exact Name or Key. So, for the given examples, the only topics that will produce the Feed we care about are:

    • test_username/f/Test Mode
    • test_username/f/test-mode

    Naming and Accessing Feeds From the io.adafruit.com HTTP API

    The HTTP API follows the same Feed identifying and Name-to-Key conversion rules as the MQTT API because under the hood they’re talking to the same backend. This means if you’re using the Ruby IO client library, the following will produce publications to the same feed as the MQTT examples given above.

    require 'rubygems'
    require 'adafruit/io'
    
    client = Adafruit::IO::Client.new(key: ENV['AIO_KEY'])
    
    [
      'Test Mode',
      'test mode',
      'test-mode',
      '44'
    ].each do |feed_ident|
      client.feeds(feed_ident).data.send_data(feed_ident)
    end
    

    Potential Problems With Naming

    It really stinks to get taken by surprise in a negative way when working with code. Reducing surprise of the unpleasant sort and increasing predictability and stability are the primary motivating factors for the subtle changes this guide introduces.

    Publishing to an invalid name

    While the Name-to-Key converter keeps things feeling pretty loose and improvisational in terms of referring to Feeds once they exist, if your initial publish is to a Feed that can’t be found it will be rejected if it doesn’t match the rules for valid Feed names.

    In the case of our MQTT example, a publish that looks like this:

    PUB 2948554 to test_username/f/Test Modes[ at 2016-07-11 15:42:31 -0400
    

    would trigger a message on the error feed that looks like this:

    [test_username/errors 2016-07-11 15:42:31 -0400] "Validation failed: Name may contain only letters, digits, underscores, spaces, or dashes"
    

    If the Feed named “Test Modes” already existed, then the publish would work fine, but because it doesn’t Adafruit IO tries to create a Feed with the given identifier, “Test Modes[” as a Name. Since “Test Modes[” is an invalid name, Adafruit IO rejects it :(

    Publishing to the wrong identifier

    If you set up your MQTT subscription first, it’s important to note that no feed will be created, so the Name-to-Key rules laid out above won’t have the effect you may have anticipated. This happens when the Feed you eventually publish to doesn’t end up with the Key or Name your subscriber has requested. The end result is a subscriber that’s silent while your device is merrily publishing away. Maddening! This is a common error of subscription and publishing when trying to juggle the different identifiers that point to a given Feed.

    The safest way to avoid this situation is to make sure that your subscribing topic and your publishing topic are exactly the same. If you want to switch to a different identifier–for example, using a Key instead of a Name–copy the value directly from Adafruit IO. When in doubt, use the Name value.

    The Feed we’re publishing to in the MQTT examples above has the following identifiers:

    key:  test-mode
    name: "Test Mode"
    

    The Name-to-Key translator is how all the “valid name variations” shown above for publishing work, but they only after the Feed already exists. The only way to create this Feed from the MQTT interface is to publish to test_username/f/Test Mode.

    Keeping a browser open to your Feeds page while setting up or programming your Adafruit IO devices is recommended.

    Modifying a name or key

    Remember, changing a Feed’s name will automatically update its Key. This is a change to existing behavior and will require modifications to any systems you’re running that refer to Feeds by Key, but it prevents more confusing situations from occurring.

    Here’s a non-hypothetical scenario that illustrates the trouble when we don’t keep Keys and Names in sync:

    • I make a new Feed and call it, “Light Switch” (IoT light switch, low risk). In JSON, the Feed looks like:
      {
        "name": "Light Switch",
        "key": "light-switch"
      }
      
    • I have a Feather Huzzah controlling a relay, acting as a subscriber listening to username/f/light-switch and a publisher sending data to username/f/Light Switch. Things talk, everything is great with the world.
    • I move the hardware over to a new spot and rename the Feed, “Blender Toggle” (IoT blender, high risk). In a non-sync world, the Feed’s new Name is “Blender Toggle” and it’s Key is still “light-switch. The Feed is now:
      {
        "name": "Blender Toggle",
        "key": "light-switch"
      }
      
    • My subscriber is still listening to username/f/light-switch, so it still gets all the Feed’s messages.
    • I build a new remote control and have it publish to username/f/Blender Toggle and it works, because there is a Feed with that exact name. Everything just keeps working, which is okay for now.
    • Later on, I decide I’d like to build another remote control light switch, so I put together an Arduino MKR1000 publishing to username/f/Light Switch.
    • My new light switch controlling, MKR1000-powered, motion sensor publishes a message to username/f/Light Switch and my blender turns on! What the heck! Hope you’re wearing Kevlar gloves!

    The MQTT Feed routing rules described above mean that a message received on the topic username/f/Light Switch gets routed to the Feed whose Key matches light-switch, which already exists. And controls the blender. Which should not turn on unexpectedly when the room gets dark :P

    This is why we keep all Feed Keys updated to match to their respective Feed Names. In Adafruit IO, renaming the Feed to “Blender Toggle” changes the Key to “blender-toggle”.

    Old Feed:

    {
        "name": "Light Switch",
        "key": "light-switch"
    }
    

    Change Name to “Blender Toggle” and the Feed now looks like:

    {
        "name": "Blender Toggle",
        "key": "blender-toggle"
    }
    

    My existing subscriber would immediately stop working because none of the messages sent to username/f/Blender Toggle (new Name) will get routed to the subscriber listening to username/f/light-switch (old Key). This doesn’t mean the subscriber breaks or shuts down, only that it stops getting messages for now. This is my chance to realize something is wrong and debug my system.

    Here’s the tricky bit, if I go in and make a new Feed and name it “Light Switch”, it’ll get the Key “light-switch”. If I didn’t update my subscriber when it stopped working, that means it’s still listening to username/f/light-switch. When I start posting to my new Feed at username/f/Light Switch, the old subscriber at username/f/light-switch will start getting messages again.

    The best defense against confusion is to refer to the Adafruit IO web interface to double check what the Feed you’re working with has for Name and Key values. And, when you make changes in Adafruit IO always make sure the cooperating systems are updated, especially when dealing with control systems that interact with the world.

    Summary

    If you got this far, I hope it’s clear that this is an area of Adafruit IO we’ve put a particular amount of thought into. Our intention continues to be building a clear, simple, powerful platform for connecting the things you build and we think this refinement supports that intention.

    Please join us at the Adafruit IO forum and share your thoughts, projects, questions, or requests. We’d love to talk to you about what we’re building!

  • Running Code at Intervals Using Adafruit IO

    We introduced time utilities to Adafruit IO about a month ago, but we haven’t provided any examples of how to use the feature. To correct this oversight, we added an example to v0.14.2 of the Adafruit MQTT Arduino Library. This example was a response to a feature request from @phlemoine in the io-issues GitHub repo:

    I am looking for a way to trigger at a specific time in the day… how can I set up the time the trigger starts ? Same for a 12 hours or even 4 hours …

    Here is the relevant code that allows code to be run at 4 hour intervals (midnight, 4am, 8am, etc):

    
    Adafruit_MQTT_Subscribe timefeed = Adafruit_MQTT_Subscribe(&mqtt, "time/seconds");
    
    // set timezone offset from UTC
    int timeZone = -4; // UTC - 4 eastern daylight time (nyc)
    int interval = 4; // trigger every X hours
    int hour = 0; // current hour
    
    void timecallback(uint32_t current) {
    
      // stash previous hour
      int previous = hour;
    
      // adjust to local time zone
      current += (timeZone * 60 * 60);
    
      // calculate current hour
      hour = (current / 60 / 60) % 24;
    
      // only trigger on interval
      if((hour != previous) && (hour % interval) == 0) {
        Serial.println("Run your code here");
      }
    
    }
    

    The full example can be found in the Adafruit_MQTT Arduino Library on GitHub. You will need to install or update the Adafruit MQTT Library to version 0.14.2 in the Arduino Library Manager, and open the adafruitio_time_esp8266 example to get started.

    Are these examples helpful? Please visit our IO forum and share your thoughts.

  • Callbacks Added to the Adafruit MQTT Library for Arduino

    v0.14.1 of the Adafruit MQTT Library for Arduino introduces subscription callbacks. This change will allow users to organize their code into separate blocks by attaching callbacks to feed and group subscriptions. Here’s a simplified example of what the change looks like:

    setup() {
    
      // add the rest of setup code here
    
      // register callback for feed
      onoffbutton.setCallback(onoffcallback);
    
    }
    
    loop() {
    
      // Ensure the connection to the MQTT server is alive (this will make the first
      // connection and automatically reconnect when disconnected).
      // See the MQTT_connect function in the full example for more info.
      MQTT_connect();
    
      // wait 10 seconds for subscription messages
      // since we have no other tasks in this example.
      mqtt.processPackets(10000);
    
      // keep the connection alive
      mqtt.ping();
    
    }
    
    void onoffcallback(char *data, uint16_t len) {
      Serial.print("Hey we're in a onoff callback, the button value is: ");
      Serial.println(data);
    }
    

    In the example above, you can see that we added the call to mqtt.processPackets(10000);. This tells the library to wait 10 seconds (10000ms) for incoming packets before moving on to the next task, and it will block all other code execution. If your sketch is only waiting for subscription messages, then 10 seconds might be a good timeout, but if your sketch handles other tasks, you may want to reduce the timeout to 1 second (1000ms).

    The full example with multiple feed subscriptions can be found in the Adafruit_MQTT Arduino Library on GitHub. You will need to install or update the Adafruit MQTT Library to version 0.14.1 in the Arduino Library Manager, and open the mqtt_esp8266_callback example to get started.

    Are these examples helpful? Please visit our IO forum and share your thoughts.

  • MQTT Error Reporting

    Unlike HTTP, the MQTT protocol does not provide a standard way of reporting errors to connected clients. This poses a problem for Adafruit IO, because there are multiple reasons why sending data to Adafruit IO might fail.

    Here are a few common examples:

    • Publishing data to an invalid topic
    • Publishing data to an unauthorized topic
    • Publishing data too fast (rate limit)
    • Exceeding the number of feeds for your account (10 currently)
    • Sending too large of a payload

    Adafruit IO is not a standard MQTT broker. Normally MQTT brokers will rebroadcast published data to any authorized subscribed client as soon as the data is received, so errors are rarely encountered.

    The standard MQTT pub/sub workflow looks something like this:

    standard mqtt flow

    Adafruit IO’s message flow is a bit more complicated, and looks something like this:

    adafruit io mqtt flow

    In the simplified example above, you can see that there are multiple steps that need to be completed before data is rebroadcasted to subscribed clients. If any of these steps fail, data will not make it back to subscribed clients. The reasons for failure are often unclear to users, so we added two special MQTT topics that will help inform users of any issues related to publishing data.

    The first topic you can subscribe to is the error topic for your user. This topic will receive any errors related to data validation and database saves. The topic path looks like this:

    {username}/errors
    

    For example, if your username was ladyada, you would subscribe to the following topic using your favorite MQTT client:

    ladyada/errors
    

    The second topic you can subscribe to is the throttle topic for your user. This topic contains any messages related to rate limits on your account. You might see these if you are publishing too fast. The topic path looks like this:

    {username}/throttle
    

    Again, if your username was ladyada, you would subscribe to the following topic:

    ladyada/throttle
    

    If you are using the Adafruit MQTT Library, you can define the throttle and error subscriptions at the top of your sketch like this:

    /*************************** Error Reporting *********************************/
    
    const char ERRORS[] PROGMEM = AIO_USERNAME "/errors";
    Adafruit_MQTT_Subscribe errors = Adafruit_MQTT_Subscribe(&mqtt, ERRORS);
    
    const char THROTTLE[] PROGMEM = AIO_USERNAME "/throttle";
    Adafruit_MQTT_Subscribe throttle = Adafruit_MQTT_Subscribe(&mqtt, THROTTLE);
    

    Then, subscribe to the topics at the bottom of the setup function:

    void setup() {
    
      // ...network connection setup code here
    
      // MQTT subscriptions for throttle & error messages
      mqtt.subscribe(&throttle);
      mqtt.subscribe(&errors);
    
    }
    
    

    Finally, inside of the main loop, print any errors to the serial monitor:

    void loop() {
    
      // Ensure the connection to the MQTT server is alive (this will make the first
      // connection and automatically reconnect when disconnected).  See the MQTT_connect
      // function definition further below.
      MQTT_connect();
    
      // this is our 'wait for incoming subscription packets' busy subloop
      // try to spend your time here
      Adafruit_MQTT_Subscribe *subscription;
      while ((subscription = mqtt.readSubscription(5000))) {
        if(subscription == &errors) {
          Serial.print(F("ERROR: "));
          Serial.println((char *)errors.lastread);
        } else if(subscription == &throttle) {
          Serial.print(F("THROTTLED: "));
          Serial.println((char *)throttle.lastread);
        }
      }
    
      mqtt.ping();
    
    }
    

    You can see the full example for the ESP8266 in the Adafruit_MQTT Arduino Library on GitHub. Install or update the Adafruit MQTT Library to version 0.13.3 in the Arduino Library Manager, and open the adafruitio_errors_esp8266 example to get started.

    Are these examples helpful? Please visit our IO forum and share your thoughts.

  • IoT Security: Connecting Your ESP8266 to Adafruit IO with SSL/TLS

    The ESP8266 based Adafruit HUZZAH breakout and the Adafruit Feather HUZZAH are both popular options to use with Adafruit IO. Because ESP8266 SSL/TLS support is fairly new, most of our Adafruit IO examples use the insecure MQTT port 1883.

    Why is this a problem? The MQTT protocol is an insecure protocol on it’s own. All data (including username & password) are sent in the clear, so SSL/TLS is required to protect any sensitive information.

    What are SSL & TLS? Transport Layer Security (TLS) and its predecessor, Secure Sockets Layer (SSL), are cryptographic protocols that you use whenever you browse the internet. Have you ever noticed the lock symbol 🔒 in your browser’s URL bar? That lock symbol represents a secure connection between your browser & a web site over SSL/TLS. We use those same protocols to secure traffic between your ESP8266 and Adafruit IO.

    We have added an example to the Adafruit_MQTT Arduino Library that you can use to secure communication between your ESP8266 and Adafruit IO. Install or update the Adafruit MQTT Library to version 0.13.2 in the Arduino Library Manager, and open the adafruitio_secure_esp8266 example to get started.

    example

    The main changes to the standard ESP8266 example are that WiFiClientSecure is used in place of WiFiClient, and port 8883 is used instead of MQTT port 1883. The sketch also checks the fingerprint for the io.adafruit.com certificate using the verifyFingerprint function.

    A simplified summary of the changes can be seen below:

    #define AIO_SERVER      "io.adafruit.com"
    #define AIO_SERVERPORT  8883                   // 8883 for MQTTS
    
    // WiFiFlientSecure for SSL/TLS support
    WiFiClientSecure client;
    
    // io.adafruit.com SHA1 fingerprint
    const char* fingerprint = "26 96 1C 2A 51 07 FD 15 80 96 93 AE F7 32 CE B9 0D 01 55 C4";
    
    void verifyFingerprint() {
    
      const char* host = AIO_SERVER;
    
      Serial.print("Connecting to ");
      Serial.println(host);
    
      if (! client.connect(host, AIO_SERVERPORT)) {
        Serial.println("Connection failed. Halting execution.");
        while(1);
      }
    
      if (client.verify(fingerprint, host)) {
        Serial.println("Connection secure.");
      } else {
        Serial.println("Connection insecure! Halting execution.");
        while(1);
      }
    
    }
    
    
  • From the Forums: Two Buttons & Two LEDs

    User smcculley posted a useful example in the IO Forum:

    I have had some requests from people to share my code to talk to Adafruit IO using MQTT and being able to trigger two LED’s.

    Note that I am using the latest AdafruitIO MQTT code from GitHub, as it has the latest fixes available: https://github.com/adafruit/Adafruit_MQTT_Library

    In Adafruit IO, I created two feeds, called ledone and ledtwo, and added them to a group called leds. I then created a dashboard with two toggle switches, and tied them to the feeds (one to each, obviously) using the default ON and OFF button text.

    Check out the post for more details. Thanks for sharing!

  • Extending the MQTT Protocol

    Last month we deployed a change to Adafruit IO’s MQTT broker that allowed us to rate limit connection attempts. Although we don’t like to throttle or ban users, the change was necessary to protect Adafruit IO from abuse, and ensure the overall health of the service.

    This presented a problem: How do we inform users when they have been throttled or banned? Ideally the MQTT protocol would allow for us to inform the user using a standard response, but currently the connection acknowledgement packet (CONNACK) has a limited set of response codes.

    The MQTT v3.1.1 protocol has the following values defined for the CONNACK packet response code:

    Value

    Return Code Response

    Description

    0

    0x00 Connection Accepted

    Connection accepted

    1

    0x01 Connection Refused, unacceptable protocol version

    The Server does not support the level of the MQTT protocol requested by the Client

    2

    0x02 Connection Refused, identifier rejected

    The Client identifier is correct UTF-8 but not allowed by the Server

    3

    0x03 Connection Refused, Server unavailable

    The Network Connection has been made but the MQTT service is unavailable

    4

    0x04 Connection Refused, bad user name or password

    The data in the user name or password is malformed

    5

    0x05 Connection Refused, not authorized

    The Client is not authorized to connect

    6-255

     

    Reserved for future use

    The current set of codes didn’t allow us to clearly communicate the reason for disconnects to our users. We have extended the list of codes in the CONNACK packet to include connection throttle (0x06) and ban (0x07).

    Value

    Return Code Response

    Description

    6

    0x06 Connection Refused, throttled

    Client has exceeded connection rate limit. Please try again later.

    7

    0x07 Connection Refused, banned

    The client has been banned by the server administrator.

    We have submitted these two new codes to the OASIS MQTT Technical Committee for comment, and we are adding support to our MQTT client libraries. If you would like to comment on these changes, please visit our IO forum and share your thoughts.

  • Using MQTT Last Will with Adafruit IO

    MQTT’s last will feature allows you to ask the Adafruit IO servers to publish a message to a feed or group on your behalf. This is helpful if you would like to track when your device unexpectedly disconnects due to issues like network or power outages.

    Here are some helpful tips from the IO forum that might help you if you are looking to use MQTT’s last will feature with Adafruit IO:

    • The last will topic must match the normal IO MQTT topic format for feeds or group publishes.
    • Last will is only published by the MQTT broker if the client fails to disconnect cleanly by sending the MQTT disconnect packet.
    • Last will is only published by the MQTT broker if the the keep alive timeout expires, and the last will is not sent if your device reconnects within the timeout window. The Adafruit MQTT Library for Arduino has a default keep alive timeout of 5 minutes.

    Here’s an example of setting a last will message using the Adafruit MQTT Library for Arduino:

    // always set the last will message before calling connect.
    // if the device unexpectedly disconnects, the 'disconnect'
    // feed will receive the 'water monitor disconnected' message.
    mqtt.will("your_username_here/feeds/disconnect", "water monitor disconnected");
    mqtt.connect();
    
  • State of IO 6.14.16

    We’re continuing forward with refactoring our UI so that we can get to a point where it’s easier and quicker to build out new features. The front-end is also going to use the same API V2 that will be the future default API. As another reminder, it would be a good idea to explicitly set ‘/api/v1’ in your paths until you’re ready to upgrade to the ‘/api/v2’.

    Stability of the entire system has been an ongoing project for us. We’re getting there, and continue to add more monitoring and fixes to reduce any future downtime.

    Also, we’re updating our client libraries with new features and bug fixes. If you haven’t tried out the Go client library, now is a good time! If you have any suggestions for our libraries, please let us know in the forums.

    Here are the stats for the past week:

    * 35.6 million inserts of logged data in the last 7 days
    * 12578 users
    * 8,900 online feeds (29554 feeds total)
    
  • Postmortem 06.07.16

    Issue

    We had a brief outage this morning due to a disk space issue on our Sidekiq EC2 instance. One set of log files grew to almost 20GB, which is strange because they are setup to be rotated using logrotate. The files should have been limited to 700MB max. According to logrotate status, it last ran on 6-1-16, when it should run daily via cron.

    Fix

    logrotate is now setup to run hourly, and we are going have monit monitor disk space on all of our EC2 instances so we can catch this issue in the future.

  • New Client Library in Go

    We are pleased to announce the release of a new officially supported client library for the Adafruit IO API! We now have an client library for Go. This client meets a few goals for us, and hopefully provides a useful hook into Adafruit IO for curious and interested web developers.

    It’s my (Adam) first project with the IO team, and so is a good starting place for exploring both the external API and the internal systems that drive it. It’s also a good chance for me to introduce my teammates to Go which is a very pleasant language to write web-connected code in. I’m a Rubyist by trade but exploring Go for lower level and Machine-to-Machine web services. I like it.

    More client libraries means broader coverage of the Adafruit IO API. As web developers, we aren’t often forced to think of the systems we build from the outside in. By making a public API and building the client libraries for it, we have a chance to see what works and doesn’t work. That’s both from a design perspective–is this API “friendly”?–and from a very practical perspective. For example, I committed two bug fixes to Adafruit IO’s core codebase while building io-client-go, huge success! Today, this library covers version 1 of the Adafruit IO API, but we’re hard at work on version 2.

    More client libraries means more open doors for new developers. Adafruit IO’s primary goal is to be the easiest way to get your Internet of Things project online. That involves a big team of people at Adafruit working at every part of the stack: engineers building new hardware and writing new firmware, makers coming up with new projects and spending a lot of time on clearly communicating what they’re doing, web developers inventing new ways of building an accessible Internet of Things platform, and, MOST IMPORTANTLY, an awesome crowd of people who can think of things to do with the tools we build that we cannot think of ourselves.

    This library isn’t the ending place for anyone’s project, but it could be the starting place for all kinds of interesting work. We don’t know! That’s the point! Here, take this, we made it for you, enjoy! :D

    This isn’t the first client library we’ve built and it won’t be the last. If Go isn’t your style yet, make sure to check out similar tools in Python, Javascript, and Ruby. And when you’re using the things we’ve built in the Things you’re building, feel free to talk to us and tell us what’s good, what’s bad, or what you’d like to see. Leave a note in the bug tracker, on the forum, or in the respective client library’s GitHub repository.

    And please, show us what you’re making! We would LOVE to see it almost as much as (I hope) you loved to build it. Also, speaking as a new member of the Adafruit team, I have never met a group of people more genuinely committed to supporting people at every level of experience than here at Adafruit.

    As always, Adafruit’s open source work is supported by the clever, creative folks who shop at Adafruit and by the community that contributes in code to any of our open source projects.

    Current status:
  • MQTT Time Utilities

    We will be deploying a new feature to our MQTT broker tomorrow morning. We will start broadcasting the current time in two formats at time/ISO-8601 and time/seconds at one second intervals. This should allow users to easily keep track of the current time from devices without real time clocks.

    Here are a couple examples of the payloads:

    ISO-8601:

    topic: time/ISO-8601
    payload: 2016-06-01T21:34:12.629Z
    

    Seconds since Unix epoch:

    topic: time/seconds
    payload: 1464816851
    
  • MQTT Connection Throttle

    We did some profiling of our node MQTT workers using the --prof flag today, and we found that our workers were spending an excessive amount of time handling MQTT connection authorization. It turns out that a few users were hammering our servers with failed auth attempts, so we decided to implement a rate limit for MQTT connection attempts.

    The new connection rate limit allows for 50 connection attempts in a 15 minute window, and attempts past the limit will be rejected for the remainder of the window. If you would like to comment on this change, please visit our IO forum.

  • Please welcome Adam to the Adafruit team!

    Hello all you lovely makers! I am very excited to introduce myself: my name is Adam Bachman, I am here to write code and chew bubblegum, and I am all out of bubblegum! As of May 23, 2016, I am the newest member of the Adafruit team and will mainly be working with Justin, Todd, Tyler, and the rest of the incredibly talented folks making Adafruit IO the best it can be.

    I’ve been a resident of Baltimore City, making my way through the world as a software developer for the last 10 years. By day building software for Fortune 500’s, startups, bootstrapped artist-loving theater makers, and a few in between. On my own time I’ve been an Adafruit customer since 2007, started a hackerspace in 2009, and I’m super into Processing and creative code in general.

    You can follow me on Github or Twitter if you like, though I can’t promise to be half as interesting as the people at the middle of this incredible enterprise :D

    ENOUGH ABOUT ME! I look forward to connecting with anyone crossing boundaries from software to hardware and back again. Artists, makers, hobbyists, professionals, first-timers, long-timers, indie, corporate, all of you, everyone, cats, maybe not so much dogs, whatever! I am psyched to get started, let’s make amazing things!

  • AIO Keys Backend Updates

    We’ve made some more backend changes to how the AIO Keys are generated and maintained. If you run into any authentication or strange AIO Key issues please let us know as soon as possible in the IO forum or Issues Tracker.

    Most of the changes were security enhancements, but we’re laying the groundwork for more features specifically related to the AIO Keys, including added administration functionality.

  • Multicolumn Index Order in Postgres

    We started to notice some speed issues yesterday with certain queries in one of the largest tables (~50 million rows) in our PostgresSQL database, and we wanted to share some of the results of the investigation.

    During our tests today, we noticed that the slow queries ordered by id DESC were running in about 5 seconds, and queries that were ordered by created_at DESC were running in 0.1 milliseconds. Multicolumn indexes were present for both queries to use, and EXPLAIN ANALYZE showed that the queries hitting those indexes with an index scan.

    This led to the obvious question: wtf?

    After some investigation, we found that the order of columns in the index were reversed. One had the columns defined as (feed_id, created_at DESC) and the other had the columns defined as (id DESC, feed_id). After looking at the PostgresSQL docs for multicolumn indexes, the issue was clear.

    It turns out that you need to define your indexes with a constraint (feed_id) first, so that you are only scanning a portion of the index. By defining id first, the full index was scanned, which resulted in very slow queries. We switched the order of the columns in the index to (feed_id, id DESC) and the queries are now running in the ~0.1 ms range, so you should notice that things are now much more responsive.

    Please check out the relevant docs in the PostgresSQL 9.5 manual if you would like more info about multicolumn indexes.

  • State of IO 05.01.16

    We are considering a change to the way IO handles the retain flag on MQTT publish packets. The retain flag is used to allow the MQTT broker to retain the sent value, and will tell the server to pass it to any new subscribers as soon as they connect. We found that this feature was a little hard for people to understand at first, so we decided to force the retain flag to true for users by default. This change ensured that users always got the current state of their IO feeds as soon as they connected, without having to send a HTTP request to the REST API for the current value.

    The decision to force the retention of values caused issues with some users, because they lost the option to decide which values were cached. We are planning to revert this change, and give people the option to cache the values using the retain flag.

    In addition to this change, we are thinking of extending the meaning of the retain flag, and also allowing users to decide which values are logged to the database. So for example, if you wanted to push out temperature values every second to your connected devices, but only wanted to log the temperature value every minute, you could do so by only setting the retain flag to true for the values you wish to log. This should allow for greater flexibility when logging, and will result in increased speed for feeds that are focused on realtime interaction. We welcome your feedback in the IO forum.

    If you would like to read more about the MQTT retain flag, please check out this article on HiveMQ’s blog.

    Here are the stats for the past week:

    * 29.03 million inserts of logged data in the last 7 days
    * 10,969 users
    * 8,675 online feeds (24,277 feeds total)
    
  • REST API v2 Deployment on April 21st

    On Thursday, April 21st we will be deploying Version 2 of our API. This version is quite different from our existing API V1.

    IMPORTANT: Your code or sketches will need to be updated if you are using the REST API.

    From now until Thursday, you will want to update your code or sketches to change ‘/api/…’ to ‘/api/v1/’ so that you continue using v1 of the API until you are ready to upgrade to the latest version.

    Here are some of the highlights (not including bug fixes):

    NEW

    • New permission system that will allow for sharing read and write access to your feeds, groups, & dashboards with other users.
    • New AWS inspired HTTP request signing to help avoid exposing the user’s AIO key over insecure connections.
    • Dashboards, blocks, & triggers will be able to be modified via the REST API.

    CHANGES

    • Username will be a required component of the URL.
    • You will no longer be able to access feeds & groups via numeric ID. Feed key & group keys will be used as the unique identifier in API v2.
    • Feeds will be able to be added to many groups, and the feed’s data will be namespaced to the group the data is pushed to. You will also be able to access all of the feed’s data by accessing the feed directly.
  • Trigger Emails Temporarily Disabled

    We’re temporarily disabling the email triggers until we can make them more robust in a near future update. In the meantime, the alternative would be to use our IFTTT channel, which can do pretty much the same thing as our internal trigger emails.

    We want to build a much more robust system. As is, there are some weaknesses in our current design that can trigger emails to be sent far too often, or without a way for them to reset after the trigger threshold has been hit.

    If you have any suggestions on what you’d like to have implemented with this feature, please let us know at the our IO forum.

  • State of IO 4.17.16

    This week we will be deploying two major changes to IO. We will be deploying v2 of the REST API, and we will be moving our MQTT services from mosca to aedes. This should allow us to squeeze even more performance out of our MQTT workers, and will also allow us to support MQTT QoS 2.

    Here are the stats for the past week:

    * 28.69 million inserts of logged data in the last 7 days
    * 10,512 users
    * 8,077 online feeds (22,695 feeds total)
    
  • State of IO 4.10.16

    Here are the stats for the past week:

    * 30.21 million inserts of logged data in the last 7 days
    * 10,086 users
    * 7,766 online feeds (21,498 feeds total)
    * ~50 inserts per second via MQTT
    * ~5 inserts per second via REST API
    

    After some load testing, it seems that our current server configuration is able to handle more traffic. Our current theory is that the inserts per second aren’t rising evenly with the increased online feed count because users are using IO for real-time interaction, and not data logging. Inserts per second only seem to rise with heavy and persistent data logging projects.

    LoRa IO

    We have been testing the new Feather LoRa Radios with IO, and they seem to be a fantastic pair. We were able to send messages to IO through a LoRa gateway from 1.27 miles away in the rolling hills of central Maryland using a small wire antenna.

    LoRa IO

    Pretty impressive for not having line of sight to the gateway!

    LoRa Mesurement

  • State of IO 4.3.16

    We added support for AWS v4 inspired request signing to IO’s v2 REST API this week. This change should allow devices that don’t support TLS/SSL to communicate with Adafruit IO without sending their AIO Key in the clear. Because the requests are signed using a HMAC, users also will be able to ensure that their requests were not manipulated by things like man-in-the-middle attacks. This change will also reduce the risk of replay attacks by including a timestamp in the signature, so the request will only be valid within a 15 minute window.

    We are considering requiring signed requests for any API v2 requests that don’t use a TLS/SSL connection. If you think this is a bad idea, please let us know in the forums. We will be updating the Adafruit IO client libraries this week to support the new request signatures.

    Here are the stats for the past week:

    * 27.4 million inserts of logged data in the last 7 days
    * 9,744 users
    * 7,537 online feeds (20,474 feeds total)
    * ~50 inserts per second via MQTT
    * ~5 inserts per second via REST API
    

    Inserts per second seem to be holding steady, despite the increase in online feeds. We are going to add another queue worker and monitor the inserts per second to see if that fixes the bottleneck.

  • State of IO 3.25.16

    We have been working on finishing up v2 of the REST API this past week, and you can read more about that here.

    We are also in the process of finishing up transitioning our frontend javascript to React from jQuery and Backbone. When combined with API v2, this change should increase performance, and also will make development easier and faster. We will post more about this change in the next few days.

    Here are the stats for the past week:

    * 25,951,672 inserts of logged data in the last 7 days
    * 9,296 users
    * 6,949 online feeds (19,084 feeds total)
    * ~50 inserts per second via MQTT
    * ~5 inserts per second via REST API
    
  • REST API v2 Development

    We are currently in the process of developing version 2.0 of the Adafruit IO REST API, which should address a large portion of user requests. The current version of the REST API (v1.0) will still be available at io.adafruit.com/api/v1/.

    Here are some of the highlights (not including bug fixes):

    NEW

    • New permission system that will allow for sharing read and write access to your feeds, groups, & dashboards with other users.
    • New AWS inspired HTTP request signing to help avoid exposing the user’s AIO key over insecure connections.
    • Dashboards, blocks, & triggers will be able to be modified via the REST API.

    CHANGES

    • Username will be a required component of the URL.
    • You will no longer be able to access feeds & groups via numeric ID. Feed key & group keys will be used as the unique identifier in API v2.
    • Feeds will be able to be added to many groups, and the feed’s data will be namespaced to the group the data is pushed to. You will also be able to access all of the feed’s data by accessing the feed directly.
  • AIO Key Length Changes

    We had a request in the forums to reduce the length of the AIO Key to 32 characters, and we decided to use UUID v4 (with the dashes stripped) to generate keys. Version 4 UUIDs are randomly generated, so they should work well as AIO Keys:

    …after generating 1 billion UUIDs every second for the next 100 years, the probability of creating just one duplicate would be about 50%.

  • Testing Cassandra Reads

    We have been writing user’s feed data to both Cassandra 3.3 and PostgreSQL 9.5 for the past couple weeks as the first step in transitioning fully to Cassandra as the primary data store for feed data. We will still be using Postgres as our primary database for relational data, but it should be much easier to scale Cassandra horizontally as the large amount of logged data increases.

    Here’s the current write performance data from Cassandra’s nodetool tablestats:

    Write Count: 59051081
    Write Latency: 0.012758596713919598 ms.
    

    That’s a little over 59 million writes in the past ~17 days, with the load on a m4.large barely ever passing 0.05.

    load average: 0.01, 0.01, 0.05
    

    Today, we have also started testing read performance in production. For now users are still receiving their data from Postgres, but we are also hitting Cassandra to see if our configuration can handle production loads. We’ll post an update once we have more info about how reads are performing.

    Here’s some early info after the deploy:

    Read Count: 314
    Read Latency: 0.6644044585987261 ms.
    
  • Sending Commands to Multiple EC2 Instances Using dsh

    We moved Adafruit IO’s node.js services to a group of t2.nano EC2 instances, and I wanted to document my dsh setup. I used it reguarly a couple years ago for managing the backend workers when working on other services, but I had forgot how it was configured. It came in really handy when I needed to quickly check or kick anything on the servers.

    First, you will need to install dsh using brew, apt-get, yum, etc:

    $ brew install dsh
    

    You will need to make a couple config directories for your user:

    $ mkdir -p ~/.dsh/group
    

    Create a new dsh.conf file using your favorite text editor:

    $ vim ~/.dsh/dsh.conf
    

    Paste the following config into ~/.dsh/dsh.conf, and save the file:

    showmachinenames = 1
    remoteshell = ssh
    

    Now, you can create a dsh group for the set of servers you would like to send commands to. Create a new file called example in the ~/.dsh/group folder using your favorite text editor:

    $ vim ~/.dsh/group/example
    

    Enter the host names for your servers, and save the file:

    example-1.adafruit.com
    example-2.adafruit.com
    example-3.adafruit.com
    example-4.adafruit.com
    

    Now you can send commands (like uptime) to all servers in the group using the following command:

    $ dsh -g example -w 'uptime'
    example-1.adafruit.com:  14:17:44 up 17:15,  0 users,  load average: 0.08, 0.03, 0.05
    example-2.adafruit.com:  14:17:43 up 16:36,  0 users,  load average: 0.00, 0.01, 0.05
    example-3.adafruit.com:  14:17:45 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05
    example-4.adafruit.com:  14:17:45 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05
    

    Using -w will send the command to each server in the order they appear in the group config, and will wait for a response before continuing. If you want to send the commands concurrently to all servers in the list, use the -c option to send the command:

    $ dsh -g example -c 'uptime'
    example-1.adafruit.com:  14:22:42 up 17:20,  0 users,  load average: 0.00, 0.01, 0.05
    example-3.adafruit.com:  14:22:42 up 16:46,  0 users,  load average: 0.00, 0.01, 0.05
    example-2.adafruit.com:  14:22:41 up 16:41,  0 users,  load average: 0.00, 0.01, 0.05
    example-4.adafruit.com:  14:22:42 up 16:46,  0 users,  load average: 0.10, 0.05, 0.05
    

    You can get an interactive shell for all servers in the group by using the -c and -i commands together. Use CTRL-D to exit the session:

    $ dsh -g -c -i
    
  • State of IO 3.18.16

    Adafruit IO has been growing steadily, and we decided to add a changelog so it’s easier for users to see what we are up to. We will be posting here reguarly whenever we add or update features, and we will also be posting stats so you have more information about the growth of IO.

    We have moved IO to a handful of Amazon EC2 instances, and we are finding it much easier to deal with increased load. The individual services are now split out into separate instances, which will allow us to scale horizontally.

    Here’s a screenshot of what it looks like when we are monitoring the state of the new EC2 instances:

    IO on EC2

    We have added IO to the Adafruit Status Page, which will allow you to check and see if any of IO’s components are currently down. You can also follow @adafruitstatus if you would like to be notified of outages.

    Adafruit Status Page

    Here are the stats for the past week:

    * 24,290,428 inserts in the last 7 days
    * 8,997 users
    * ~6,700 online feeds (~18,200 feeds total)
    * ~50 inserts per second via MQTT
    * ~5 inserts per second via REST API