Add cookbook to expose Prometheus counters.

doc/cookbook/cabal.project

@@ -4,6 +4,7 @@ packages:
   db-mysql-basics/
   db-sqlite-simple/
   db-postgres-pool/
+  expose-prometheus/
   using-custom-monad/
   jwt-and-basic-auth/
   hoist-server-with-context/

doc/cookbook/expose-prometheus/ExposePrometheus.lhs

@@ -0,0 +1,233 @@
+# Expose Prometheus metrics
+
+Production services require monitoring to operate reliably and efficiently.  In
+a production setup, you may want to record a variety of things like the number
+of access to a feature when doing some A-B tests, the duration of database queriest to
+optimize performance when needed, the number of third-party API calls to avoid
+hitting rate-limits, the count the number of failed logins to report suspicious
+activity etc.  Observability is the umbrella term for techniques and
+technologies to expose such _metrics_ and _traces_ about internals of services.
+A prevalent tool and format to expose metrics is
+[Prometheus](https://prometheus.io/).
+
+Prometheus proposes a simple mechanism: services who want to expose metrics add
+a web-API route returning a series of metrics in a well-known text-format.
+Prometheus collectors then periodically (often at a short interval) request
+this route for one or many services.
+
+This cookbook shows how to expose Prometheus counters using Servant so that a
+Prometheus collector can then collect metrics about your application. We
+leverage the `prometheus-client` package to provide most of the instrumentation
+primitives.  While packages exist to direcly expose Prometheus counters, this
+cookbook guides you to write your own exposition handler. Writing your own
+handler allows you to tailor the metrics endpoint to your needs. Indeed, you
+may want to re-use Servant combinators to expose different subsets of metrics
+onto different endpoints. Another usage of Servant combinators would be to
+protect the endpoint so that only trusted clients can read the metrics. Here we
+propose to augment the endpoint with a
+[CORS](https://en.wikipedia.org/wiki/Cross-origin_resource_sharing) header so
+that a browser clients (such as
+[mine](https://dicioccio.fr/prometheus-monitor.html)) can query the Prometheus
+metrics endpoint.
+
+First, the imports.
+
+``` haskell
+{-# LANGUAGE DataKinds #-}
+{-# LANGUAGE OverloadedStrings #-}
+{-# LANGUAGE TypeOperators #-}
+{-# LANGUAGE MultiParamTypeClasses #-}
+{-# LANGUAGE GeneralizedNewtypeDeriving #-}
+
+import Control.Monad (forever)
+import Control.Concurrent (ThreadId, forkIO, threadDelay)
+import Control.Monad.IO.Class (liftIO)
+import Data.ByteString.Lazy (ByteString)
+import Data.Text (Text)
+import Network.Wai.Handler.Warp (run)
+import qualified Prometheus as Prometheus
+import Prometheus.Metric.GHC (ghcMetrics)
+import Servant
+```
+
+In this cookbook we will write a dummy "hello-world" API route.  A counter will
+count accesses to this API route.  For the purpose of this cookbook the
+"hello-world" API route will count how many times whom got a hello. For
+instance "Hello, Bob" and "Hello, Alice" means we got "1 for Bob" and "1 for
+Alice", in short, we record a counter _breakdown_.  Another counter will report
+values counted in background-thread, here, a counter in a sleep-loop.  Such
+counters can serve as watchdog for other applications: if the counter stops
+increasing, then something is aloof.
+
+In a real-application you may want to avoid
+exposing counters broken-down by a value chosen by an untrusted-user (i.e., if
+our hello-world API is public, you open the door to unbounded
+memory-requirements as counter breakdowns persist in memory). However, for the
+purpose of this cookbook, we assume the risk is mitigated.
+
+The Prometheus library we use requires us to register the counters ahead of
+time. I recommend that we define a datatype to refer to all the counters needed
+in this cookbook. A `initCounters` function performs all the needed
+registration.
+
+``` haskell
+data Counters
+  = Counters
+  { countHellos :: Prometheus.Vector (Text) Prometheus.Counter
+  , countBackground :: Prometheus.Counter
+  }
+
+initCounters :: IO Counters
+initCounters =
+  Counters
+    <$> Prometheus.register
+          (Prometheus.vector "who"
+            (Prometheus.counter
+              (Prometheus.Info "cookbook_hello" "breakdown of hello worlds")))
+    <*> Prometheus.register
+          (Prometheus.counter
+            (Prometheus.Info "cookbook_background" "number of background thread steps"))
+```
+
+We next implement the dummy "hello-world" API route.  We add a Servant type and
+implement a Servant Handler.  We want the API route to have a `who` query-param
+so that one can call `hello?who=Alice` and  get a greeting like "hello, Alice"
+in return. We use our first Prometheus counter to record how many times Alice,
+Bob, or anyone got a greeting.
+
+The handler will defaults to `n/a` as a magic value to represent the absence of
+`who` query-parameter.
+
+``` haskell
+type Greeting = Text
+
+newtype HelloWhom = HelloWhom { getWhom :: Text }
+  deriving FromHttpApiData
+
+type HelloAPI =
+  Summary "a dummy hello-world route"
+    :> "api"
+    :> "hello"
+    :> QueryParam "who" HelloWhom
+    :> Get '[JSON] Greeting
+
+-- | A function to turn an input object into a key that we use as breakdown for
+-- the `countHellos` counter. In a real-world setting you want to ponder
+-- security and privacy risks of recording user-controlled values as breakdown values.
+helloWhomToCounterBreakdown :: HelloWhom -> Text
+helloWhomToCounterBreakdown (HelloWhom txt) = txt
+
+handleHello :: Counters -> Maybe HelloWhom -> Handler Greeting
+handleHello counters Nothing = do
+  let breakdown = "n/a"
+  liftIO $ Prometheus.withLabel (countHellos counters) breakdown Prometheus.incCounter
+  pure "hello, world"
+handleHello counters (Just who) = do
+  let breakdown = helloWhomToCounterBreakdown who
+  liftIO $ Prometheus.withLabel (countHellos counters) breakdown Prometheus.incCounter
+  pure $ "hello, " <> getWhom who
+```
+
+The second metrics we use to instrument our program is a background thread
+incrementing a counter every second.
+
+``` haskell
+startBackgroundThread :: Counters -> IO ThreadId
+startBackgroundThread counters = forkIO $ forever go
+  where
+    go :: IO ()
+    go = do
+      Prometheus.incCounter (countBackground counters)
+      threadDelay 1000000
+```
+
+Now we need to implement the part where we expose the Prometheus metrics on the
+web API.
+Let's define an API route: it's a simple `HTTP GET` returning some _Metrics_.
+In this example we also add the CORS header we discussed in intro.
+
+``` haskell
+type ServePrometheusAPI =
+  Summary "Prometheus metrics"
+    :> "metrics"
+    :> Get '[PlainText]
+         (Headers '[Header "Access-Control-Allow-Origin" CORSAllowOrigin] Metrics)
+```
+
+With this API type, we now need to fill-in the blanks.  We define a `Metrics`
+object that serializes as the Prometheus text format.  We want to keep it
+simple and use `Prometheus.exportMetricsAsText` to collect the metrics as a
+text-formatted payload. Thus, our `Metrics` object contains the raw payload.
+
+``` haskell
+newtype Metrics = Metrics {toLBS :: ByteString}
+
+instance MimeRender PlainText Metrics where
+  mimeRender _ = toLBS
+```
+
+We define the CORS header helper.
+
+``` haskell
+newtype CORSAllowOrigin = CORSAllowOrigin Text
+  deriving ToHttpApiData
+```
+
+Finally, we define the Prometheus collection endpoint proper. The magic
+function `Prometheus.exportMetricsAsText` provided by our Prometheus library
+ensures that we collect all registered metrics in a process-global variable.
+The implementation uses a where-clause to recall the Handler data-type so that everything is front of our eyes.
+
+``` haskell
+handlePrometheus :: CORSAllowOrigin -> Server ServePrometheusAPI
+handlePrometheus corsAllow = handleMetrics
+  where
+    handleMetrics :: Handler (Headers '[Header "Access-Control-Allow-Origin" CORSAllowOrigin] Metrics)
+    handleMetrics = do
+      metrics <- liftIO $ Prometheus.exportMetricsAsText
+      pure $ addHeader corsAllow $ Metrics metrics
+```
+
+Finally, we bundle everything in an application.
+
+The complete API consists in the "hello-world" api aside the Prometheus metrics
+endpoint.  As a bonus we register the `ghcMetrics` (from the
+`prometheus-metrics-ghc` package) which allows to collects a lot of runtime
+information (such as the memory usage of the program), provided that your
+binary is compiled with `rtsopts` and run your progam with `+RTS -T` (cf. the [GHC docs about the RTS](https://downloads.haskell.org/ghc/latest/docs/users_guide/runtime_control.html)).
+
+``` haskell
+type API
+  = HelloAPI
+  :<|> ServePrometheusAPI
+
+api :: Proxy API
+api = Proxy
+
+server :: CORSAllowOrigin -> Counters -> Server API
+server policy counters = handleHello counters :<|> handlePrometheus policy
+
+runApp :: Counters -> IO ()
+runApp counters = do
+  let allowEveryone = CORSAllowOrigin "*"
+  run 8080 (serve api (server allowEveryone counters))
+
+main :: IO ()
+main = do
+  _ <- Prometheus.register ghcMetrics
+  counters <- initCounters
+  _ <- startBackgroundThread counters
+  runApp counters
+```
+
+Now you can navigate to various pages:
+- http://localhost:8080/metrics
+- http://localhost:8080/api/hello
+- http://localhost:8080/api/hello?who=prometheus
+- http://localhost:8080/metrics
+- http://localhost:8080/api/hello?who=prometheus again
+- http://localhost:8080/api/hello?who=world
+- http://localhost:8080/metrics
+
+You can also see counter increases live by installing a Prometheus collector.
+For development and troubleshooting purpose, I only use my [own monitor tool](https://dicioccio.fr/prometheus-monitor.html) also available as [Firefox extension](https://addons.mozilla.org/en-GB/firefox/addon/prometheus-monitor/).

doc/cookbook/expose-prometheus/expose-prometheus.cabal

@@ -0,0 +1,30 @@
+cabal-version:       2.2
+name:                expose-prometheus
+version:             0.1
+synopsis:            Expose Prometheus cookbook example
+homepage:            http://docs.servant.dev/
+license:             BSD-3-Clause
+license-file:        ../../../servant/LICENSE
+author:              Servant Contributors
+maintainer:          [email protected]
+build-type:          Simple
+tested-with:         GHC==9.4.2
+
+executable cookbook-expose-prometheus
+  main-is:             ExposePrometheus.lhs
+  build-depends:       base == 4.*
+                     , text >= 1.2
+                     , bytestring >= 0.11
+                     , containers >= 0.5
+                     , servant
+                     , servant-server
+                     , prometheus-client
+                     , prometheus-metrics-ghc
+                     , warp >= 3.2
+                     , wai >= 3.2
+                     , http-types >= 0.12
+                     , markdown-unlit >= 0.4
+                     , http-client >= 0.5
+  default-language:    Haskell2010
+  ghc-options:         -rtsopts -Wall -pgmL markdown-unlit
+  build-tool-depends: markdown-unlit:markdown-unlit

doc/cookbook/index.rst

@@ -23,6 +23,7 @@ you name it!
   db-mysql-basics/MysqlBasics.lhs
   db-sqlite-simple/DBConnection.lhs
   db-postgres-pool/PostgresPool.lhs
+  expose-prometheus/ExposePrometheus.lhs
   using-custom-monad/UsingCustomMonad.lhs
   using-free-client/UsingFreeClient.lhs
   custom-errors/CustomErrors.lhs