Attempting to access the data points on your Adafruit IO feed and only receiving 1000 data points back? kevinljxljx on the Adafruit IO Forum was running into the same issue:

I am trying to use the data from adafruit.io with my project, but no matter what I try, adafruit.io only gives 1000 data points to my javascript program. ( I am using XMLHttpRequest). I tried to set limit to 2000 but without luck. ( I have 1200 or so data points under my feed). Is the maximum data points I can get is 1000? or am I missing something?

Adafruit IO’s API only returns 1000 data points at a time. To get the rest of the data (if your feed is over 1000 data points), you’ll need to “paginate” the data.

What does this mean?

When you perform a data query, the results are always sorted newest-to-oldest and include *x-pagination-** headers. For example:

X-Pagination-Limit: 1000
X-Pagination-Total: 84548
X-Pagination-Start: 2019-02-11T22:52:18.103+0000
X-Pagination-End: 2019-02-12T16:03:00.694+0000
X-Pagination-Count: 1000

Limit (X-Pagination-Limit) is either the requested limit or 1000, whichever is less; Total (X-Pagination-Total) is the total number of data points in the feed. Note, this value may be up to 5 minutes behind real time.

Start (X-Pagination-Start) is the timestamp on the oldest value; End ( X-Pagination-End) is the timestamp on the newest value; and Count ( X-Pagination-Count) is the number of data points in the current request.

Whenever Limit and Count are both 1000 and Total is more than 1000, that’s evidence that more data is available.

You can get the next 1000 data points by using the X-Pagination-Start value OR the created_at value of the oldest data point in the API response as the end_time parameter in your next request to the data API.

When visualized on a timeline, the concept of pagination looks like this:

Note that long running, frequently updated feeds could have more than a hundred “pages” of data. If you make requests without a delay in between, you could hit a rate limit. To avoid this, watch for 429 HTTP error responses and handle them in code by adding a 30 second timeout between requests.

Regarding the data storage and feed history, storage size in this instance is the per-data point value size limit.

With history on, meaning we preserve every data point, each data point value can be at most 1KB. With history off, meaning we only preserve the most recent data point, each value can be at most 100KB.

Post originally written by AdamB on the Adafruit IO Forum, read the entire thread here…

Need help getting started with Adafruit IO? Not sure what feeds are? Need a place to ask questions, but there are too many links on the internet? The Adafruit IO Team has added a new help menu directly into Adafruit IO with mini-guides designed to help you through any questions you could have about Adafruit IO, such as… How do I get started with Python? Follow the new Getting Started with Adafruit IO Python mini-guide… or “What is metadata, and how does it relate to my Adafruit IO Project?” Still have more questions? Having an issue with your code? We’ve also added links to our Discord Chatroom and the Adafruit IO Forum to the Quick Help menu - no googling necessary!

To check it out, log into Adafruit IO (if you don’t have an account, create one - it’s free!) and click the Get Help button on the sidebar.

Want to see anything else added to the Quick Help box? Let us know in the comments below!

Hackaday posted up about a beta for a new cloud service, which at this time requires Arduino-only hardware: MKR1000 or MKR WiFi 1010. There’s a “passionate” debate in the comments about this direction, & the MKR1000 is “non-commercial” etc… more choices, the better, it’s a great time to make things and do IoT projects… Anyway, we saw a really nice mini-review of our adafruit.io service that can be used with any device…

“I gave adafruit IO a try recently – super easy, just works, no fuss, super fast to get a device online, compatible with generic devices, open standards and protocols such as MQTT, working example code – nothing locked-in or proprietary.” - Luke.

That’s why we made adafruit.io - We play nice with any device. The internet of things for everyone. The easiest way to stream, log, and interact with your data. We have, and will always have, a free version which can be used by any hardware, and we have a plus version for folks who want to run a biz and more on it, also - any hardware. In the last 30 days there are 12,459 makers, hackers, artists, and engineers, making something on adafruit.io - try it out today!

Tons of guides to get you started with the Internet of Things - learn.adafruit.com

Adafruit IO ToS: No spam, no mining your personal data, easy to read and understand terms of service/use, etc.

Want to export the Data from your account ASAP? It’s your data.

An Internet of Things Bill of Rights - Adafruit.

Adafruit’s blog of all the projects, resources, and community projects, tagged with #adafruitio & the adafruit.io changelog’ blog’

Join us on Discord for the adafruit.io channel!

Built from the ground up to be easy to use. We do the hard work so you can focus on the fun stuff. Powerful API The same API that drives our user interface is available to you. We provide documentation so you can build a library in your favorite language to talk to IO, or use one of ours. It’s Your Data The data you store with IO is yours to manage and control. You can download it all anytime and we will never sell or give it away to another company. We feel strongly enough about this that we put it in our IoT Bill of Rights. Beautiful Dashboards Data won’t help you make better decisions or understand the world you’re living in unless you can see it. Charts, graphs, gauges, logging, and more are availble from anywhere in our carefully designed web-based dashboards. Private & Secure We won’t share your account information with anyone without your permission and everything you store on Adafruit IO is private by default, whether you’re a paying customer or not. Documentation We’ve written a lot more about Adafruit IO than just our web API. Teaching is at the heart of Adafruit, and our Learning Systemhas all the projects and guides you need to get started or to level up! Triggers Keep yourself and your systems up to date by automatically responding to changes. Triggers monitor your data to notify you when your temperature sensor gets too hot or when your DIY security system goes offline. Hardware We have been building Internet of Things devices and kits since long before Adafruit IO ever existed. Start your project with the right equipment or pick up the one last thing you need to finish. We’ve got hardware for all types of makers. Amazing Community We Love Makers! Join us on the forums, on our Discord chat server, stop by the weekly Show and Tell live stream to share your project, and subscribe to our YouTube channel to see what we’re up to.

Welcome to the Adafruit IOT Monthly for the cold month of February (anyone out there making an internet-enabled hot-cocoa machine?). This newsletter (in blog form) highlights projects and updates about Adafruit IO – our Internet-of-Things Service for makers, news, stories, and advances in the world of IoT.

Adafruit IO News: Features and Updates

IO News highlights what’s new on Adafruit IO, our internet-of-things service for makers. There’s been some new updates to the platform since we’ve blogged an IoT Monthly.

New Feature: Feed Webhooks

For Adafruit IO, webhooks are unique URLs that you can use to send data to your feeds from anywhere on the internet, without having to share you IO API secret keys. They’re kind of like one-off API endpoints that you can manage separately from your security credentials and feed privacy. There have long been requests for “shared write access” to feeds, which can be done using the feed sharing feature already built into IO, but that requires everyone sending data to have an Adafruit IO account and use their credentials to send data. That, unfortunately, limits you from, for example, creating a simple web form to publish data directly to a feed.

Read more about this feature on the Adafruit IO Development Blog

Library Update: Adafruit IO Python

We’ve renamed the Python library for Adafruit IO to reflect the naming scheme for our other library - Adafruit IO Arduino. So, IO Client Python is now known as Adafruit IO Python. The IO Team has recently added a few services to Adafruit IO - Dark Sky for weather forecasting and a random data service to create random colors, words, numbers. Until now, you haven’t been able to use these services with the Adafruit IO Python library.

The latest version of Adafruit IO Python (2.1) includes updated API and MQTT clients for using Adafruit IO’s random data and weather services.

Powered by Adafruit IO: Projects from the Community

Each month, we select our favorite projects from around the internet which use Adafruit IO. Here are some of our favorites:

How to Automate your Home with Adafruit IO

https://www.youtube.com/watch?v=JHmK6N-dhvM MakerIO published a new video on home automation using Adafruit IO. For the sake of simplicity, Robin uses two Huzzah boards to create a ping-pong application between the two of them. One board houses a button, which changes the state of the other board’s LED based on the first button. This simple example can be applied to home automation – one board could be used to control the lights in your home, the garage door, or anything wired up to a relay.

Are you working on a project which uses Adafruit IO and want to show it off? Add the #adafruitio hashtag to your tweets, join our Discord channel (http://adafru.it/discord), or join the SHOW-AND-TELL every Wednesday night at 7:30pm ET on Google+ Hangouts.

New Adafruit Learning System Guides

Have you heard about making, cosplay and electronics, but don’t know where to start? Visit the Adafruit Learning System for over 1500 tutorials for electronics projects, ideas and techniques! We’ve selected two, internet-of-things-based guides from the learning system:

LoRa/LoRaWAN Weather Logging Network

Want to build your own network? This new guide uses the Adafruit LoRa radio bonnet with OLED and two Feathers with BME280 sensors to build a Weather Monitoring Network.

Read more..

Adding a WiFi Co-Processor to CircuitPython

This guide will show you how to connect your CircuitPython board to the Internet by using an ESP8266 or ESP32 as the ‘Wireless modem’ – we’ll even show you how to upload the required AT command firmware to the chip, all from your CircuitPython board

See this guide today - there’s even an example for posting to Adafruit IO!

#IOTuesday: IOT News and Projects from around the ‘net

We publish the latest and most interesting IoT news and projects from around the `net every Tuesday. Here’s some of our favorite recent stories.

The SweRV RISC-V Core from Western Digital

Western Digital announced at the RISC-V Summit three new open-source innovations designed to support Western Digital’s internal RISC-V development efforts and those of the growing RISC-V ecosystem. In his keynote address, Western Digital’s Chief Technology Officer Martin Fink unveiled plans to release a new open source RISC-V core, an open standard initiative for cache coherent memory over a network and an open source RISC-V instruction set simulator.

Read more on Business Wire

TWIST is DIY Tweeting Weather Station

Ever wanted to monitor your city’s Current Weather Conditions, Carbon Footprint, Noise and Pollution levels? Do you want be a Climate Change Crusader or set-up your own Tweeting Weather Station and share your local weather conditions with the world? Meet Tweeting Weather IoT Station aka TWIST – a DIY, Open-Source Environmental Monitoring and Meteorological Data Acquisition Platform.

Read more

IoT Camera Mover

In making videos for my projects I have often wanted to get a smooth shot of a part, such as a pan of an Arduino Nano. I have seen about 3 projects where people made devices that could pan a camera, but they were either not internet-connected or they were very expensive. So I set out to improve them.

Read more

The Internet of Unprofitable Things (never hardcode what you don’t own into an IoT device)

Andy posts on The Ongoing Struggle a story that anyone developing an electronic product but especially an Internet of Things (IoT) product should heed. The subtitle is the NTPmare shortly after Christmas.

Read more..

How to control a device with a Kronaby smartwatch with IFTTT

Via Mike Diamond on What I Made Today, a look at a special breed of watch that incorporates the characteristics of a regular watch (digital or analog), with select functions of the Smart Watch.

The hybrid smartwatch that caught my attention was the small Swedish company, Kronaby. They specialize in beautifully-designed analog (hybrid) smart watches that radiate prestige, while quietly synchronizing with your phone. They offer multiple functions – applied through their simple, easy-to-use and equally well-designed app. This post is for general interest, but may be most useful to the Kronaby owner who wants to know what their IFTTT function is, and how to use it with their watch.

Read more…

Let us know how you’re using IO or IO+ on the forums or in our Discord chat room.

New Feature: Feed Webhooks

What’s a webhook? According to Wikipedia:

A webhook in web development is a method of augmenting or altering the behaviour of a web page, or web application, with custom callbacks. These callbacks may be maintained, modified, and managed by third-party users and developers who may not necessarily be affiliated with the originating website or application.

For Adafruit IO, webhooks are unique URLs that you can use to send data to your feeds from anywhere on the internet, without having to share you IO API secret keys. They’re kind of like one-off API endpoints that you can manage separately from your security credentials and feed privacy.

There have long been requests for “shared write access” to feeds, which can be done using the feed sharing feature already built into IO, but that requires everyone sending data to have an Adafruit IO account and use their credentials to send data. That, unfortunately, limits you from, for example, creating a simple web form to publish data directly to a feed.

There are a few use cases we’ve come up with so far, but we hope to be surprised with what the community can come up with. Here’s two things you can try.

One-off IFTTT Applets

First, other web services that can publish to Webhook URLs like IFTTT’s maker webhooks can be used to directly send data to a feed without entering Adafruit credentials or signing into your Adafruit IO account through IFTTT.

We already support IFTTT directly, but we can’t support every service that exists on the web. We can work with any service that knows how to send webhook-style requests, though :D

Public write-only connections to IO feeds

Second, webhook URLs can be used directly in simple HTML forms, to create a deploy-anywhere application that sends data directly to your feed. For example:

<form target="_blank" action="https://io.adafruit.com/api/v2/webhooks/feed/dQTc61soU5S4tMv8jSuSjY9uSqVz" method="post">
    <label>
        Make a choice:
        <select name="value">
            <option value="YES" selected>Yes</option>
            <option value="NO">No</option>
        </select>
    </label>
    <button type="submit">Save my vote!</button>
</form>

And the actual form in action:

After clicking “Save my vote!” you should see a standard Adafruit IO data record on a new tab, which means your vote was recorded. If you see a “request failed - Throttle error” message instead, that’s because this particular webhook URL’s data rate limit of 3 events per minute has already been reached.

By setting a very low data rate for the webhook, I can share the URL publicly without risking my account, unlike sharing my Adafruit IO Keys. Even better, if the webhook ends up flooded with data, I can just delete it from my account to shut off the stream. And because webhook URLs are write-only, I don’t need to make the feed public.

Publicly controllable Christmas light displays? Crowd-sourced weather monitoring system? GitHub -> Discord -> Adafruit IO cross-posting webhook circle of fun? Let us know what you come up with. Join us in the forum or on Discord in the adafruit-io channel with questions, comments, or suggestions!