Adafruit IO Internals: Pagination
Attempting to access the data points on your Adafruit IO feed and only receiving 1000 data points back? kevinljxljx on the Adafruit IO Forum was running into the same issue:
Adafruit IO’s API only returns 1000 data points at a time. To get the rest of the data (if your feed is over 1000 data points), you’ll need to “paginate” the data.
What does this mean?
When you perform a data query, the results are always sorted newest-to-oldest and include *x-pagination-** headers. For example:
X-Pagination-Limit: 1000 X-Pagination-Total: 84548 X-Pagination-Start: 2019-02-11T22:52:18.103+0000 X-Pagination-End: 2019-02-12T16:03:00.694+0000 X-Pagination-Count: 1000
X-Pagination-Limit) is either the requested limit or 1000, whichever is less; Total (
X-Pagination-Total) is the total number of data points in the feed. Note, this value may be up to 5 minutes behind real time.
X-Pagination-Start) is the timestamp on the oldest value; End (
X-Pagination-End) is the timestamp on the newest value; and Count (
X-Pagination-Count) is the number of data points in the current request.
Whenever Limit and Count are both 1000 and Total is more than 1000, that’s evidence that more data is available.
You can get the next 1000 data points by using the
X-Pagination-Start value OR the
created_at value of the oldest data point in the API response as the end_time parameter in your next request to the data API.
When visualized on a timeline, the concept of pagination looks like this:
Note that long running, frequently updated feeds could have more than a hundred “pages” of data. If you make requests without a delay in between, you could hit a rate limit. To avoid this, watch for 429 HTTP error responses and handle them in code by adding a 30 second timeout between requests.
Regarding the data storage and feed history, storage size in this instance is the per-data point value size limit.
With history on, meaning we preserve every data point, each data point value can be at most 1KB. With history off, meaning we only preserve the most recent data point, each value can be at most 100KB.
Post originally written by AdamB on the Adafruit IO Forum, read the entire thread here…