New Machine Learning Monitoring & Interactive Drill-Down Features – Seldon Deploy 1.3 Released!

The lifecycle of a machine learning only begins once it’s deployed to production. This is why this release of our enterprise product Seldon Deploy focuses heavily on extending our existing monitoring capabilities to enable data science practitioners and domain experts to dynamically visualise & drill down into the inference data of their production machine learning models.

The 1.3 release of Seldon Deploy introduces a new feature distributions drill-down screen, together with the underlying scalable architecture and cloud native data infrastructure that enables it at massive scale. It also introduces various improvements to the model catalogue such as an advanced search with a new DSL query language, as well as the ability to deploy models from the model catalogue enabling for deeper integrations with frameworks in the model experimentation space.

Monitoring Drilldown

A new feature added to our machine learning monitoring within our enterprise stack is the ability to drill down into the input and output payloads from deployed models to provide advanced visualizations of feature distributions. Users can dynamically filter by specific feature values and ranges, as well as specifying time-ranges for the data to take into consideration.

This is extremely powerful when combined with the existing advanced machine learning explainability, outlier detection, drift detection and statistical performance features of Seldon. To build an intuition of the powerful nature of these advanced data science monitoring workflows we cover a brief example case study below. 

Let’s take as an example the well-known income-classifier machine learning challenge where we predict whether an individual earns more or less than 50k. With Seldon’s  enterprise monitoring features, a user can identify a sudden divergence raised from a configured drift detector or threshold alert from statistical performance metrics which can then lead to a domain expert investigating further. The domain expert can leverage the new drill down features for Seldon Deploy 1.3 which allows them to answer questions such as “what is the ratio of individuals classified with more than 50k across a certain age group or gender”. This example is illustrated in the image below.

Model Prediction Metadata Schema 

To allow for the most advanced visualizations to be provided automatically in the feature distribution drilldown data science practitioners can provide core metadata describing the input and output features for their model with a new open standard model prediction schema

The open model prediction schema extends the Seldon Deploy metadata management to encompass input and output payloads of a model. This provides the data infrastructure that allows for the scalable interactive visualizations shown in the last section.

An example for an image model is shown below.

{
  "requests": [
    {
      "name": "Input Image",
      "type": "TENSOR",
      "data_type": "FLOAT",
      "shape": [32, 32, 3]
    }
  ],
  "responses": [
    {
      "name": "Image Class",
      "type": "CATEGORICAL",
      "data_type": "INT",
      "n_categories": 10,
      "category_map": {
        "0": "Airplane",
        "1": "Automobile",
        "2": "Bird",
        "3": "Cat",
        "4": "Deer",
        "5": "Dog",
        "6": "Frog",
        "7": "Horse",
        "8": "Ship",
        "9": "Truck"
      }
    }
  ]
}

With this release of our enterprise product we also extend the existing metadata interface to allow via the UI or API the ability to load a defined prediction schema for a model. An illustrated example is shown in the animated image below.

Advanced Metadata Query Language

We have also introduced a new query language to enable users to perform discover on their machine learning models. More specifically users are now able to search across their deployed models and model catalogue artifacts by using advanced queries such as:

  • All models with name iris and version 2.0
    • QUERY: name=iris AND version=2.0
  • Either the results above OR models with version 3
    • (name=iris AND version=2.0) OR version=3.0
  • All models with metrics with specific name and NOT a specific tag
    • QUERY: metrics[metricName]>1.0 AND tags[tagKey]!=someValue

From these examples it is possible to see the power and flexibility that this could provide for practitioners that want to explore and discover models with specific characteristics, in some cases to deploy, and for deployed models to consume by requesting inference.

Extending Enterprise APIs

All of the functionality that is exposed through the user interface is also exposed through our Seldon Deploy Enterprise APIs, which  can then be programmatically integrated to enterprise systems using our Seldon Deploy SDK. This means that data scientists are also able to consume the features available in a programmatic manner to perform further advanced drilldowns in any relevant data science environment. 

Below is a simple example that shows how easy it can be to consume some of the data available in the Seldon Deploy store, which allows data scientists and other advanced users to extract deeper insights from their production data.

from seldon_deploy_sdk import MonitorApi, DeploymentFeatureData

deployment_name="income-classifier"

deployment_namespace="seldon"

feature_data = DeploymentFeatureData(

feature="Education",

interaction="request",

aggregate_over_time=True

)

api_response = api_instance.seldon_deployment_feature_distributions(deployment_name, deployment_namespace, feature_data)

groups = []

counts = []

for bucket in api_response.distribution.buckets:

groups.append(bucket.key)

counts.append(bucket.count)

import matplotlib.pyplot as plt

fig = plt.figure()

ax = fig.add_axes([0,0,1,1])

fig = plt.figure()

 

The output of that plot can be visualised below. Similarly the other features including the metadata query language, as well as the filtering can be accessed through the SDK.

If you are interested to try out the new Seldon Deploy metadata capabilities simply fill out the form below:

Blog Form

  • This field is for validation purposes and should be left unchanged.