[CheckMK] Temp & Humidity Alert levels should be set separately per sensor

[acozine]

User story

As operations folks, we want to know when there's an HVAC problem in a data center, but we don't want continual alerts for normal readings.

We have been monitoring these sensors for a few months now, so we have some idea of what the typical readings for temperature and humidity look like. Now we can determine what appropriate alerts look like for each sensor, and implement those levels as checks in CheckMK.

Acceptance criteria

• Each temperature and humidity alert is set appropriately for its data center, rack, and position
• We no longer see frequent alerts under normal operating conditions
• If temperature rises or humidity falls out of normal bounds, we get an alert
• We have documentation for how to set different alert levels for specific sensors in CheckMK

Implementation notes, if any

Here are the levels we agreed on for the five racks in the "A" Data Center:

• set E22 Front Temp to warn at 94 and critical at 97
• set E22 Front Humidity to warn at 18% and critical at 13%
• set E23 Rear Temp for E23 to warn at 94 and critical at 97
• set E23 Rear Humidity to warn at 18% and critical at 13%
• set E24 Rear Humidity to warn at 18% and critical at 13%
• set E25 Rear Humidity to warn at 18% and critical at 13%
• set E26 Front Temp to warn at 94 and critical at 97
• set E26 Front Humidity to warn at 18% and critical at 13%
• all other sensor settings for this data center can retain the default warning and critical settings

Once those are done, we can review and discuss appropriate settings for the "B" data center. - once we know how to set sensor-specific levels, we can implement those more easily/