Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Handle scenario where creating an alert policy w/ a condition causes error #71

Open
sanderblue opened this issue May 26, 2020 · 0 comments
Labels
bug Something isn't working kubernetes

Comments

@sanderblue
Copy link
Contributor

sanderblue commented May 26, 2020

Description

If you have an alert policy configured with an "invalid" alert condition, the policy will be created, but an invisible, sneaky, silent error will occur under the hood for the invalid alert condition (such as missing a required condition config attribute).

Objective

At a minimum we probably want to surface an error/warning message. Welcome to suggestions for a better UX as well.

Steps To Reproduce

Steps to reproduce the behavior:

  1. run kubectl apply -f ./policy.yaml

    # policy.yaml
    
    apiVersion: nr.k8s.newrelic.com/v1
    kind: Policy
    metadata:
      name: my-policy
    spec:
      api_key: API_KEY
      name: "Alert Policy Created With k8s"
      region: "us"
    
    # Invalid NRQL Alert Condition spec (hypothetical scenario)
    # `nrql.since_value` is missing
    # `enabled` is missing 
    conditions:
      - spec:
          name: "NRQL Alert Condition Created With k8s"
          nrql:
            query: "SELECT average(duration) FROM Transaction WHERE appName = 'Dummy App'"
          terms:
            - threshold: "1"
              time_function: "all"
              duration: "4"
              priority: "critical"
              operator: "above"
  2. Check New Relic. You should see a new alert policy, but no alert conditions have been added due to the error occurring on the NRQL alert condition request.

Note: You can tail the logs to see the requests and errors using the following commands:

k describe nodes -n newrelic-kubernetes-operator-system | grep newrelic-kubernetes

# Then using the hash from the other command, tail the logs
kubectl logs -f -n newrelic-kubernetes-operator-system -c manager newrelic-kubernetes-operator-controller-manager-<HASH GOES HERE>
@sanderblue sanderblue added the bug Something isn't working label May 26, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working kubernetes
Projects
None yet
Development

No branches or pull requests

2 participants