Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Specify the maximum number of elements per page each endpoint can return. #359

Open
wants to merge 3 commits into
base: master
Choose a base branch
from

Conversation

graille
Copy link

@graille graille commented Nov 13, 2022

General warning

Note: This PR is the first step in my desire to provide a way for the user to request a specific number of result on endpoint allowing pagination, as discussed with several developers on Discord and completing the work previosuly started in #276 (and thus use the internal pagination of the request function, which is currently not used by high-level functions!). This feature is not implemented in this PR.

However, the features implemented in this PR can be considered independent and already offer new cool stuff, so I propose it for review.

In this PR, I make a difference between the first parameter, which is the parameter sent to Twitch API when performing the HTTP request, and the first argument exposed by some methods as User.fetch_pools()

Changes

In the current version of TwitchIO, the number of elements requested to endpoints when pagination is enabled is 100 by default, this is set by the get_limit() subfunction:

def get_limit():
    if limit is None:
        return "100"
    to_get = limit - len(data)
    return str(to_get) if to_get < 100 else "100"

This leads to two main issues:

  • For endpoints where the maximum value allowed for firstis less than 100, we have to do a little trick by adding a "first" parameter when calling the fetch_ method, this is currently the case for 4 methods: get_reward_redemptions, get_polls, get_predictions, get_channel_schedule. These functions are the ONLY functions exposing a first parameter. All the other ones suppose by default that first=100 due to the get_limit() function.
  • For endpoints allowing higher values than 100, you will receive 100 data and no more. Endpoints like entitlements/drops allows 1000 results per page.

This PR fix all of this by specifying the maximum value allowed by first when requesting the endpoint. This is done by adding a new parameter, max_elements_per_page to the request() method, containing this information.

Below, I listed all endpoint using the "first" parameter, with their min, default and max value. For each endpoint, the value of the max_elements_per_page argument is the maximum value allowed to first by the documentation.

Method Endpoint min default max Implemented in TwitchIO
GET analytics/extensions 1 20 100 X
GET analytics/games 1 20 100 X
GET extensions/transactions 1 20 100 get_extension_transactions
GET channel_points/custom_rewards/redemptions 1 20 50 get_reward_redemptions
GET chat/chatters 1 100 1000 X, BETA
GET clips 1 20 100 get_clips
GET entitlements/drops 1 20 1000 get_entitlements
GET extensions/live 1 20 100 X
GET games/top 1 20 100 get_top_games
GET hypetrain/events 1 1 100 get_hype_train
GET moderation/banned 1 20 100 get_channel_bans
GET moderation/blocked_terms 1 20 100 X
GET moderation/moderators 1 20 100 get_channel_moderators
GET channels/vips 1 20 100 get_channel_vips
GET polls 1 20 20 get_polls
GET predictions 1 20 25 get_predictions
GET schedule 1 20 25 get_channel_schedule
GET search/categories 1 20 100 get_search_categories
GET search/channels 1 20 100 get_search_channels
GET soundtrack/playlist 1 20 50 X
GET soundtrack/playlists 1 20 50 X
GET streams 1 20 100 get_streams
GET streams/followed 1 100 100 X
GET streams/markers 1 20 100 get_stream_markers
GET subscriptions 1 20 100 get_channel_subscriptions
GET tags/streams 1 20 100 get_stream_tags
GET users/follows 1 20 100 get_user_follows
GET users/blocks 1 20 100 X
GET videos 1 20 100 get_videos

(Note that for /videos, we can specify first parameter only if we specify the game_id or user_id query parameter. This condition has been implemented in this PR)

In fetch_ methods exposing a first argument, it has been rerouted to the limit attribute of request() to avoid any breaking change.

Moreover, the following functions:

  • get_predictions
  • get_channel_schedule
  • get_polls

Can implement pagination, but were delivered with pagination=False.
Activate the pagination in these functions doesn't change or break anything and allow them to benefit from this PR. So I did it.

Finally, I disabled the assertion breaking the request when full_body was True as the same time as paginate, since the body is returned before the evaluation of paginate. This will prevent any futher bug in the future.

I tested all helix endpoints with both app token and user token to validate this PR.

Checklist

  • If code changes were made then they have been tested.
    • I have updated the documentation to reflect the changes.
    • I have updated the changelog with a quick recap of my changes.
  • This PR fixes an issue.
  • This PR adds something new (e.g. new method or parameters).
  • This PR is a breaking change (e.g. methods or parameters removed/renamed)
  • This PR is not a code change (e.g. documentation, README, ...)

- Add max_elements_per_page attribute in function signature and function documentation.
- Add type hints in function signature.
- Break some comments to respect the 120 characters limit.
- Add possibility to modify the maximum elements per page a function can return. Was 100 before that.
- The limit argument can now be None, if so, it takes the value of max_elements_per_page.
- Update requesting system to support that.

Note that limit argument is NOT used anywhere for now. So this commit doesn't break anything.
- Add the value of max_elements_per_page attribute for each endpoint supporting irst parameter (according to Twitch documentation)
- Put the default value of max_elements_per_page to None in request function
- Put paginate to True in get_channel_schedule method in http.py
- Put paginate to True get_predictions method in http.py
- Put paginate to True get_polls method in http.py

Before this commit, the maximum number of elements returned by a request was always 100.
Now, each endpoint specifies the maximum number of items it expect to receive according to Twitch Documentation.
For example, entitlements/drops will now ask a maximum of 1000 elements per request instead of 100.

Some issues with some functions:
- get_reward_redemptions
- get_channel_schedule
- get_polls
- get_channel_vips

Were the only one to implement a rist kwargs attribute mapping the irst HTTP parameter.
This attribute has been rerouted to the limit attribute of request() to avoid any breaking change.

Moreover, the following functions:
- get_predictions
- get_channel_schedule
- get_polls

Can implement pagination, but were delivered with pagination=False.
Activate the pagination in these functions doesn't change or break anything and allows to receive the maximum number of elements according to the max_elements_per_page attribute
@IAmTomahawkx IAmTomahawkx added new feature HTTP HTTP Endpoint related items. 2.0 TwitchIO 2.0 labels Jan 13, 2023
@IAmTomahawkx IAmTomahawkx self-assigned this Jan 13, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
2.0 TwitchIO 2.0 HTTP HTTP Endpoint related items. new feature
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants