You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
It's possible to calculate the perplexity of a given response. This is basically a metric for how confused a model is. This would be an interesting metric when comparing one prompt against another. It also makes me wonder if there are existing OTEL integrations that would calculate this and potentially other metrics for us easily
The text was updated successfully, but these errors were encountered:
Since we're calling OpenAI in multiple locations, it would be good if we could centralise the helper functions: https://github.com/overmindtech/api-server/blob/main/server/risks/shared.go#L81-L127
This would mean it would be standardised across all usage
Extra Credit: Perplexity
As per this article: https://medium.com/@furqanshaikh/measuring-llm-confusion-9529a4b5e907
It's possible to calculate the perplexity of a given response. This is basically a metric for how confused a model is. This would be an interesting metric when comparing one prompt against another. It also makes me wonder if there are existing OTEL integrations that would calculate this and potentially other metrics for us easily
The text was updated successfully, but these errors were encountered: