diff --git a/doc/cookbook/cabal.project b/doc/cookbook/cabal.project index f82acd464..49a0e80b8 100644 --- a/doc/cookbook/cabal.project +++ b/doc/cookbook/cabal.project @@ -4,6 +4,7 @@ packages: db-mysql-basics/ db-sqlite-simple/ db-postgres-pool/ + expose-prometheus/ using-custom-monad/ jwt-and-basic-auth/ hoist-server-with-context/ diff --git a/doc/cookbook/expose-prometheus/ExposePrometheus.lhs b/doc/cookbook/expose-prometheus/ExposePrometheus.lhs new file mode 100644 index 000000000..88463798d --- /dev/null +++ b/doc/cookbook/expose-prometheus/ExposePrometheus.lhs @@ -0,0 +1,236 @@ +# Expose Prometheus metrics + +Production services require monitoring to operate reliably and efficiently. In +a production setup, you may want to record a variety of things like the number +of access to a feature when doing some A-B tests, the duration of database queries to +optimize performance when needed, the number of third-party API calls to avoid +hitting rate-limits, the number of failed logins to report suspicious +activity, etc. Observability is the umbrella term for techniques and +technologies concerned with exposing such _metrics_ and _traces_ about +internals of services. +A prevalent tool and format to expose metrics is +[Prometheus](https://prometheus.io/). + +Prometheus proposes a simple mechanism: services who want to expose metrics add +a web-API route returning a series of metrics in a well-known text-format. +Prometheus collectors then periodically (often at a short interval) request +this route for one or many services. + +This cookbook shows how to expose Prometheus counters using Servant so that a +Prometheus collector can then collect metrics about your application. We +leverage the [`prometheus-client`](https://hackage.haskell.org/package/prometheus-client) package to provide most of the instrumentation +primitives. While packages exist to direcly expose Prometheus counters, this +cookbook guides you to write your own exposition handler. Writing your own +handler allows you to tailor the metrics endpoint to your needs. Indeed, you +may want to re-use Servant combinators to expose different subsets of metrics +onto different endpoints. Another usage of Servant combinators would be to +protect the endpoint so that only trusted clients can read the metrics. Here we +propose to augment the endpoint with a +[CORS](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing) header so +that a browser clients (such as +[prometheus-monitor](https://dicioccio.fr/prometheus-monitor.html)) can query the Prometheus +metrics endpoint. + +First, the imports. + +``` haskell +{-# LANGUAGE DataKinds #-} +{-# LANGUAGE OverloadedStrings #-} +{-# LANGUAGE TypeOperators #-} +{-# LANGUAGE MultiParamTypeClasses #-} +{-# LANGUAGE GeneralizedNewtypeDeriving #-} + +import Control.Monad (forever) +import Control.Concurrent (ThreadId, forkIO, threadDelay) +import Control.Monad.IO.Class (liftIO) +import Data.ByteString.Lazy (ByteString) +import Data.Text (Text) +import Network.Wai.Handler.Warp (run) +import qualified Prometheus as Prometheus +import Prometheus.Metric.GHC (ghcMetrics) +import Servant +``` + +In this cookbook we will write a dummy "hello-world" API route. A counter will +count accesses to this API route. For the purpose of this cookbook the +"hello-world" API route will count how many times whom got a hello. For +instance "Hello, Bob" and "Hello, Alice" means we got "1 for Bob" and "1 for +Alice", in short, we record a counter _breakdown_. Another counter will report +values counted in a background-thread, here, a counter in a sleep-loop. Such +counters can serve as watchdog for other applications: if the counter stops +increasing, then something is amiss. + +In a real-application you may want to avoid +exposing counters broken-down by a value chosen by an untrusted-user (i.e., if +our hello-world API is public, you open the door to unbounded +memory-requirements as counter breakdowns persist in memory). However, for the +purpose of this cookbook, we assume the risk is mitigated. + +The Prometheus library we use requires us to register the counters ahead of +time. Let's define a datatype to refer to all the counters needed +in this cookbook. An `initCounters` function performs all the needed +registration. + +``` haskell +data Counters + = Counters + { countHellos :: Prometheus.Vector (Text) Prometheus.Counter + , countBackground :: Prometheus.Counter + } + +initCounters :: IO Counters +initCounters = + Counters + <$> Prometheus.register + (Prometheus.vector "who" + (Prometheus.counter + (Prometheus.Info "cookbook_hello" "breakdown of hello worlds"))) + <*> Prometheus.register + (Prometheus.counter + (Prometheus.Info "cookbook_background" "number of background thread steps")) +``` + +We next implement the dummy "hello-world" API route. We add a Servant type and +implement a Servant Handler. We want the API route to have a `who` query-param +so that one can call `hello?who=Alice` and get a greeting like "hello, Alice" +in return. We use our first Prometheus counter to record how many times Alice, +Bob, or anyone got a greeting. + +The handler will defaults to `n/a` as a magic value to represent the absence of +`who` query-parameter. + +``` haskell +type Greeting = Text + +newtype HelloWho = HelloWho { getWho :: Text } + deriving FromHttpApiData + +type HelloAPI = + Summary "a dummy hello-world route" + :> "api" + :> "hello" + :> QueryParam "who" HelloWho + :> Get '[JSON] Greeting + +-- | A function to turn an input object into a key that we use as breakdown for +-- the `countHellos` counter. In a real-world setting you want to ponder +-- security and privacy risks of recording user-controlled values as breakdown values. +helloWhoToCounterBreakdown :: HelloWho -> Text +helloWhoToCounterBreakdown (HelloWho txt) = txt + +handleHello :: Counters -> Maybe HelloWho -> Handler Greeting +handleHello counters Nothing = do + let breakdown = "n/a" + liftIO $ Prometheus.withLabel (countHellos counters) breakdown Prometheus.incCounter + pure "hello, world" +handleHello counters (Just who) = do + let breakdown = helloWhoToCounterBreakdown who + liftIO $ Prometheus.withLabel (countHellos counters) breakdown Prometheus.incCounter + pure $ "hello, " <> getWho who +``` + +The second metrics we use to instrument our program is a background thread +incrementing a counter every second. + +``` haskell +startBackgroundThread :: Counters -> IO ThreadId +startBackgroundThread counters = forkIO $ forever go + where + go :: IO () + go = do + Prometheus.incCounter (countBackground counters) + threadDelay 1000000 +``` + +Now we need to implement the part where we expose the Prometheus metrics on the +web API. +Let's define an API route: it's a simple `HTTP GET` returning some _Metrics_. +In this example we also add the CORS header we discussed in intro. + +``` haskell +type PrometheusAPI = + Summary "Prometheus metrics" + :> "metrics" + :> Get '[PlainText] + (Headers '[Header "Access-Control-Allow-Origin" CORSAllowOrigin] Metrics) +``` + +With this API type, we now need to fill-in the blanks. We define a `Metrics` +object that serializes as the Prometheus text format. We want to keep it +simple and use `Prometheus.exportMetricsAsText` to collect the metrics as a +text-formatted payload. This function is an IO object returning the whole text +payload, thus, our `Metrics` object contains the raw payload in a +"pre-rendered" format for the MimeRender instance. + +``` haskell +newtype Metrics = Metrics {toLBS :: ByteString} + +instance MimeRender PlainText Metrics where + mimeRender _ = toLBS +``` + +We define the CORS header helper. + +``` haskell +newtype CORSAllowOrigin = CORSAllowOrigin Text + deriving ToHttpApiData +``` + +Finally, we define the Prometheus collection endpoint proper. The magic +function `Prometheus.exportMetricsAsText` provided by our Prometheus library +ensures that we collect all registered metrics in a process-global variable. +The implementation uses a where-clause to recall the Handler data-type so that everything is front of our eyes. + +``` haskell +handlePrometheus :: CORSAllowOrigin -> Server ServePrometheusAPI +handlePrometheus corsAllow = handleMetrics + where + handleMetrics :: Handler (Headers '[Header "Access-Control-Allow-Origin" CORSAllowOrigin] Metrics) + handleMetrics = do + metrics <- liftIO $ Prometheus.exportMetricsAsText + pure $ addHeader corsAllow $ Metrics metrics +``` + +Finally, we bundle everything in an application. + +The complete API consists in the "hello-world" api aside the Prometheus metrics +endpoint. As a bonus we register the `ghcMetrics` (from the +`prometheus-metrics-ghc` package) which allows to collects a lot of runtime +information (such as the memory usage of the program), provided that your +binary is compiled with `rtsopts` and run your progam with `+RTS -T` (cf. the [GHC docs about the RTS](https://downloads.haskell.org/ghc/latest/docs/users_guide/runtime_control.html)). + +``` haskell +type API + = HelloAPI + :<|> ServePrometheusAPI + +api :: Proxy API +api = Proxy + +server :: CORSAllowOrigin -> Counters -> Server API +server policy counters = handleHello counters :<|> handlePrometheus policy + +runApp :: Counters -> IO () +runApp counters = do + let allowEveryone = CORSAllowOrigin "*" + run 8080 (serve api (server allowEveryone counters)) + +main :: IO () +main = do + _ <- Prometheus.register ghcMetrics + counters <- initCounters + _ <- startBackgroundThread counters + runApp counters +``` + +Now you can navigate to various pages: +- http://localhost:8080/metrics +- http://localhost:8080/api/hello +- http://localhost:8080/api/hello?who=prometheus +- http://localhost:8080/metrics +- http://localhost:8080/api/hello?who=prometheus again +- http://localhost:8080/api/hello?who=world +- http://localhost:8080/metrics + +You can also see counter increases live by installing a Prometheus collector. +For development and troubleshooting purpose, you can use [prometheus-monitor](https://dicioccio.fr/prometheus-monitor.html) also available as [Firefox extension](https://addons.mozilla.org/en-GB/firefox/addon/prometheus-monitor/). diff --git a/doc/cookbook/expose-prometheus/expose-prometheus.cabal b/doc/cookbook/expose-prometheus/expose-prometheus.cabal new file mode 100644 index 000000000..0eb813e23 --- /dev/null +++ b/doc/cookbook/expose-prometheus/expose-prometheus.cabal @@ -0,0 +1,30 @@ +cabal-version: 2.2 +name: expose-prometheus +version: 0.1 +synopsis: Expose Prometheus cookbook example +homepage: http://docs.servant.dev/ +license: BSD-3-Clause +license-file: ../../../servant/LICENSE +author: Servant Contributors +maintainer: haskell-servant-maintainers@googlegroups.com +build-type: Simple +tested-with: GHC==9.4.2 + +executable cookbook-expose-prometheus + main-is: ExposePrometheus.lhs + build-depends: base == 4.* + , text >= 1.2 + , bytestring >= 0.11 + , containers >= 0.5 + , servant + , servant-server + , prometheus-client + , prometheus-metrics-ghc + , warp >= 3.2 + , wai >= 3.2 + , http-types >= 0.12 + , markdown-unlit >= 0.4 + , http-client >= 0.5 + default-language: Haskell2010 + ghc-options: -rtsopts -Wall -pgmL markdown-unlit + build-tool-depends: markdown-unlit:markdown-unlit diff --git a/doc/cookbook/index.rst b/doc/cookbook/index.rst index cb24c2165..565a815ed 100644 --- a/doc/cookbook/index.rst +++ b/doc/cookbook/index.rst @@ -23,6 +23,7 @@ you name it! db-mysql-basics/MysqlBasics.lhs db-sqlite-simple/DBConnection.lhs db-postgres-pool/PostgresPool.lhs + expose-prometheus/ExposePrometheus.lhs using-custom-monad/UsingCustomMonad.lhs using-free-client/UsingFreeClient.lhs custom-errors/CustomErrors.lhs