Attempting to access the data points on your Adafruit IO feed and only receiving 1000 data points back? kevinljxljx on the Adafruit IO Forum was running into the same issue:

I am trying to use the data from adafruit.io with my project, but no matter what I try, adafruit.io only gives 1000 data points to my javascript program. ( I am using XMLHttpRequest). I tried to set limit to 2000 but without luck. ( I have 1200 or so data points under my feed). Is the maximum data points I can get is 1000? or am I missing something?

Adafruit IO’s API only returns 1000 data points at a time. To get the rest of the data (if your feed is over 1000 data points), you’ll need to “paginate” the data.

What does this mean?

When you perform a data query, the results are always sorted newest-to-oldest and include *x-pagination-** headers. For example:

X-Pagination-Limit: 1000
X-Pagination-Total: 84548
X-Pagination-Start: 2019-02-11T22:52:18.103+0000
X-Pagination-End: 2019-02-12T16:03:00.694+0000
X-Pagination-Count: 1000

Limit (X-Pagination-Limit) is either the requested limit or 1000, whichever is less; Total (X-Pagination-Total) is the total number of data points in the feed. Note, this value may be up to 5 minutes behind real time.

Start (X-Pagination-Start) is the timestamp on the oldest value; End ( X-Pagination-End) is the timestamp on the newest value; and Count ( X-Pagination-Count) is the number of data points in the current request.

Whenever Limit and Count are both 1000 and Total is more than 1000, that’s evidence that more data is available.

You can get the next 1000 data points by using the X-Pagination-Start value OR the created_at value of the oldest data point in the API response as the end_time parameter in your next request to the data API.

When visualized on a timeline, the concept of pagination looks like this: api-pagination.png

Note that long running, frequently updated feeds could have more than a hundred “pages” of data. If you make requests without a delay in between, you could hit a rate limit. To avoid this, watch for 429 HTTP error responses and handle them in code by adding a 30 second timeout between requests.

Regarding the data storage and feed history, storage size in this instance is the per-data point value size limit.

With history on, meaning we preserve every data point, each data point value can be at most 1KB. With history off, meaning we only preserve the most recent data point, each value can be at most 100KB.

Post originally written by AdamB on the Adafruit IO Forum, read the entire thread here…