Skip to content

Latest commit

 

History

History
64 lines (43 loc) · 2.5 KB

5-model-monitoring.md

File metadata and controls

64 lines (43 loc) · 2.5 KB
Raw

Exercise 5: Model Monitoring and Observability

Goals

  • Learn about EFK Stack (Elasticsearch, FluentD, and Kibana)
  • Configure and deploy our application to log prediction events to Elastic Search
  • Visualize events on Kibana dashboard
  • Learn how to close the data feedback loop

Introduction

To close the data feedback loop, we can log events in production to collect data about how our model is performing against real data. This data can later be curated and labeled to improve the dataset used during training. This allows us to continuously improve our models in production!

In this workshop, we use the EFK stack for our monitoring and observability infrastructure. It is composed of three main components:

  • Elasticsearch: an open source search engine.
  • FluentD: an open source data collector for unified logging layer.
  • Kibana: an open source web UI that makes it easy to explore and visualize the data indexed by Elasticsearch.

Step by Step instructions

  1. In GoCD, click on the little gear symbol (gear) next to ci-workshop-app-X to edit your deployment pipeline configuration.

  2. Open the "Environment Variables" tab and configure the FluentD host and port:

  • FLUENTD_HOST = elastic-stack-fluentd-elasticsearch.elk.svc.cluster.local
  • FLUENTD_PORT = 24224

Configure deployment pipeline to log to FluentD

  1. Save and return to the Dashboard page.

  2. Trigger a new application deployment pipeline and wait for it to succeed.

  3. Visit your application in production to make a few predictions.

  4. Visit the Kibana URL http://kibana.cd4ml.net and click on the "Discover" menu.

  5. In the search field, find the entries tagged with your user, with a query string tag:"userX.prediction" (substitute X with your user ID).

  6. Click "Refresh" and you should see your predictions logged!

Prediction events in Kibana

  1. Done!

NOTE: after the end of the workshop, we delete all the infrastructure and GoCD pipelines for security and cost reasons.

You don’t need to use the same tools we chose to implement CD4ML. Get in touch with us if you want to learn how to run this workshop with your teams in your company!