Map Explorer unavailable

[shaunanoordin]

API Outage

Affects: Map

We are currently investigating an issue where the Classroom Maps API is returning a 504 Gateway Timeout error.

• Impact: Teachers & Students can't use the WildCam Gorongosa/Darien/Kenya Lab map explorer in their classroom environment.
  • The geographic maps display, but all camera trap data (orange circles) is unavailable.
• Affected pages:
  • WildCam Gorongosa Map Explorer: https://classroom.zooniverse.org/#/wildcam-gorongosa-lab/explorers/map/
    • Expected: When I load the page (in an incognito window) I should see orange circles corresponding to camera trap data
    • Actual: webpage loads, geographical map (Leaflet) loads, but no orange circles are visible. "Loading" message on map controls shows for a long time, before turning into a generic "ERROR".
  • URL: http://classroom-maps-api.zooniverse.org/
    • Expected: datasette-powere homepage should be visible
    • Actual: "504 Gateway Timeout, nginx"
• Tested with macOS + Chrome/FF (WiFi), Android + Chrome (4G). Doesn't seem to be a client-side issue.
• I can confirm the API has been out since 11:48 GMT on Mon 6 Nov 2023 (at least), and continues until 21:09 GMT (so far).
• I can confirm the API was working normally on Tue 2 Nov 2023 at 19:17 (whatever timezone Freshdesk is in).
• There have been reports of intermittent outages previously, but I can't confirm those, sorry.

Related Freshdesk tickets (requires login):

• Troubleshoot Gorongosa Wildcam, 1 Nov 2023

Status

Investigation ongoing. Temporary "service unavailable" message will be set up.

[shaunanoordin]

Update

This is a bit of a rollercoaster, so investigation is ongoing.

• At 6 Nov 2023 21:59 GMT, the Classroom Maps API appeared to be working again. Woo!
  • Zach noted that the problem was likely due to deployment issues. See Update docker registry to ghcr.io classroom-maps-api#16 and Deploy needs build/push classroom-maps-api#17
  • Slack mesage
• A little later, Michelle noted that "classroom-maps-api does hit its limit pretty often", referring to the server's usage stats.
  • In the shared screenshot, there were 2 pods - one at 100% capacity, the other unavailable with a warning status. Shaun proposed a possible chain of events: 1 of 2 pods down => 1 pod carrying double weight => that pod reaches capacity => 504 gateway error? This is unconfirmed, however
  • Slack message
• However, as of 22:47 GMT, the Classroom Maps API appear to have issues again. Boo!
  • Zach noted that the 504s are appearing again, and a follow up will need to be done.
  • Slack message

As noted with my conversation with Michelle:

If the Classroom Maps API is actually hitting its limits, that may correlate with the number of students trying to use the map being... higher than anticipated? I have no idea what the intended capacity of the server is... but I do know that the Map Explorer can support some pretty complex SQL queries, which may be taxing on even the Datasette system.