[Tumblr] Tumblr official API bug would cause posts being skipped in pagination

[fireattack]

Tumblr official API has a serious bug (tumblr/docs#76) where if some posts have same "user-set publish time" (according to their employee), using the next link returned by the API for pagination would actually skip some posts. See the original issue for a still-valid example.

And they refused to fix it.

I see gdl uses this API endpoint and data["_links"]["next"]["href"], so I think I should share this as a PSA.

To workaround this issue, you can use offset param instead of using the href provided by the API itself. (Actually, the API used to just return new href with changed offset, before they switched to the current page_number hash fashion.)

Or use before calculated based on posts fetched from the previous call, but make sure to include some overlapping. Because as said above, some posts would have the exactly same timestamp due to being scheduled.

[mikf]

#5810 might be related. At the very least it is a case where using data["_links"]["next"]["href"] doesn't work and stops after the first page of results, so it would also need a different pagination strategy.

[mikf]

I made an attempt: 540eaa5

[Hrxn]

@fireattack Can you still reproduce this issue with 540eaa5?

[fireattack]

I didn't try; but by the look of it it should have solved it. Closing.