some error handling cases to cover/be aware of

[coletdjnz]

webpage:

• HTTP ERROR 429
• this is generally separate from the API rate limit (i.e. if webpage gets rate limited, API usually doesn't)

API:

Network

• HTTP ERROR 429
  • this is prob where we need to sleep
• Alerts
  • On network error YouTube provides info under data['error']['message']

Server Side issues

• 'Unknown Error' Alert (type: ERROR)
  • Comes under HTTP 200 OK
  • see alerts under general errors below
  • retry the request should work
• Missing onResponseReceivedEndpoints or reloadContinuationItemsCommand
  • retrying the request usually solves (may require many, exponential back-off might be good idea)
  • in some cases it will keep getting stuck on retrying.
• Retry on any 5XX error

General errors

• YouTube provides alerts (type info or error) under under data['alerts'] of normal responses
• need to handle:
  • channel doesn't exist/deleted

Misc

No comments received on page we had a continuation for (**updated)

• There may be no continuationItems in appendContinuationItemsAction or reloadContinuationItemsCommand
• yt-dlp solution:
  • check how many comments were extracted
  • if no comments extracted, and no extract continuation, treat as end of replies / comments
  • if another continuation detected (note: this shouldn't happen in newest sorting mode):
    • if replies: treat as end of replies
    • if root: warn and treat as end of replies
• test cases:
  • replies, midway through: UC5nc_ZtjKW1htCVZVRxlQAQ
  • root comments, at start: UC9nWWJFKULXtjTfEDEPjX6A

API returns a response that is not a comment section

• needs investigation, currently this clashes with the handling under server side issues.
• maybe check for continuationContents and contents since that key is never present for comments?
• test case: UC-9-kyTW8ZkZNDHQJ6FgpwQ

Recently deleted channels

Recently deleted channels may return one page of comments, but any continuation given for e.g. replies appear to fail.

In this these cases, treat of end of comments when YouTube gives "Requested entity was not found." or "The caller does not have permission" alert.

Missing comment data

Most the time appears to be due to deleted users' comments.
Missing data may include:

• contentText
• authorEndpoint
• authorText

[tech234a]

Wow, thanks for the detailed list!

[tech234a]

Also one other thing to add to this list is that when retrieving the autogenerated Music channel it returned what appeared to be the homepage data (a list of playlist shelves)

[tech234a]

Added backoff retry on 3XX, 429, 5XX

[tech234a]

Also disallowed redirects (for the sorry page)

[tech234a]

Rewrote the retry code for more flexibility

[tech234a]

What keys can we expect data["alerts"] to contain?

[tech234a]

UC-9-kyTW8ZkZNDHQJ6FgpwQ appears to be handled by the check for onResponseReceivedEndpoints

[tech234a]

Now handles missing continuationItems key.

[tech234a]

I think most cases are handled now, though some trigger a sleep when they probably shouldn't, such as invalid UCID, non-comment response (music channel), and 400 errors. Deleted channels/valid, non-existent UCIDs are treated as 0 comments currently.

[coletdjnz]

What keys can we expect data["alerts"] to contain?

Check it out when you pass a dead channel, e.g. UC__aJC8gjH9ZGi-Jpd7_6Kw. I believe this gives an alert with type 'error'

this might be of use: https://github.com/yt-dlp/yt-dlp/blob/2333ea102986f5ae792d3f297aac04cf8065d9f3/yt_dlp/extractor/youtube.py#L725-L735

[tech234a]

No alert on that example, still returns the comment header

[tech234a]

With an incorrect-length UCID I get an error

{
  "error": {
    "code": 400,
    "message": "Request contains an invalid argument.",
    "errors": [
      {
        "message": "Request contains an invalid argument.",
        "domain": "global",
        "reason": "badRequest"
      }
    ],
    "status": "INVALID_ARGUMENT"
  }
}

[tech234a]

TODO: improve handling for non-comment responses (currently sleeps based on existing detections) and handle data["alerts"] (need an example). Also the script currently always treats a lack of continuationItems as end of comments/replies and warns.