Notes for point dev investigations 2021-05-26

[AAkindele]

Using this to keep notes while gathering information about various issues. Will put issues on the board after initial investigations.

• Email about unsubscribing from Azure Monitor action group.
  • Does anyone have more information about this?
  • Is there any action item to take here?
  • Is it possible that someone unsubscribed from one of the alerts, which unsubscribed the email group?

  • You’ve unsubscribed from an Azure Monitor action group
    You have unsubscribed from the ngsa-pre-log action group and will no longer receive notifications sent to the group.

• Dashboard is missing some data https://aka.ms/ngsadashboard
  • Some fluentbit pods in all 3 clusters are not sending logs. They don't show recent log entries.
  • ngsa-memory is not showing data in the dashboard https://aka.ms/ngsadashboard
  • Central cluster is not reporting data in dashboard. https://aka.ms/ngsadashboard
  • Fluenbit pods were manually restarted at 2021-05-26 around 3pm central.
  • Action Item: Alerts for missing data
• Noisy alerts
  • Alerts are firing A LOT. Initial investigation of the data suggests that the thresholds are set too low
  • Example: HighClientResponseTimeWestUS2-cosmos has a threshold of 178ms. The data for this month show that 98th percentile is 178.28ms. This mean this specific alert will fire on any small increase in network latency.
  • Action Item: Discuss as a team. a criteria for identifying what is the acceptable threshold for alerts.
  • Action Item: Use this criteria to periodically evaluate if alerts thresholds are still valid.
  • Action Item: Check thresholds for all "HighServerResponseTime" and "HighClientResponseTime" requests and make sure they are not too low when taking minor network blips into account.
• ngsa and loderunner logs in ngsa-pre-log seem to only go back 30 days.
  • Is that intentional or is more investigation needed here?
• NGSA dependabot alert for https://github.com/retaildevcrews/ngsa/security/dependabot/spikes/ngsa-typescript/package-lock.json/mixme/open
• Dashboard is showing 204 status code as Loderunner failures. https://aka.ms/ngsadashboard

cc: @gortegaMS

[AAkindele]

Notes have been converted to items in the ngsa devx board https://github.com/orgs/retaildevcrews/projects/4