As organisations implement mature, scalable MLOps workflows, the requirement for robust observability and auditability becomes increasingly critical. In this latest release of Seldon Deploy Enterprise we have focused on increasing the advanced monitoring capabilities of the platform as well as the thousands of models it manages at scale.  

Seldon Deploy 1.4 extends the monitoring and drilldown capabilities of feature distributions visualisations by allowing both inference and training data to be compared side by side. Also introduced are automated alerts, together with a new usage audit logs engine, and significant improvements on the advanced ML Drift Monitoring. 

Training Data Distributions Comparison 

A feature that was added in Seldon Deploy v1.3 enabled practitioners to drill down into advanced visualisations with filtering of the feature distributions of input and output payloads from deployed models, enabling deeper exploration of live prediction data.  

Seldon Deploy 1.4 lets you import training data from external sources, leveraging prediction metadata features to process this data into machine readable format, enabling rich and meaningful comparisons across live production models with powerful filtering. 

Toggle training data on and off to compare training data distributions appear with inference data distributions across all relevant features. Identify more target labels for comparison  by leveraging the offline ETL integration introduced via Argo Workflows integrations. In the animation below you can see how the training data can be uploaded by providing the respective data location, and similar to all our data download components, this one supports over 100 different data storage sources. 

Real Time ML Monitoring Alerts 

Operational monitoring, also known as observability, in machine learning operations is key. In this version of Seldon Deploy we extend the advanced machine learning monitoring features that are being used to manage thousands of models in production through automated alerts. 

We have added support for alerts to be captured through heuristics. These can be leveraged through battle-tested integrations with backend systems like alert-manager, and support interoperability with proven operational alerting tools like PagerDuty and OpsGenie. 

In the animation above you can see how both operational alerts and system alerts are now displayed in the user interface. However the more important update is the backend integrations that allow DevOps and MLE users to set and receive alerts that can be configured on any relevant service level objectives and metrics thresholds. 

Usage Audit Logs Engine 

Organisations that adopt machine learning at scale often face complex and abstract compliance and regulatory requirements that highlight the need for mature enterprise features. As an extension of the existing deployment audits and prediction request audits, Seldon Deploy 1.4 adds a core usage audit logs engine that enables administrators of Seldon Deploy to ensure all interactions are stored with relevant metadata.  

We have put in place this new architecture such that the audit log engine enables for a robust and reliable process that ensures the usage metrics can not only be backed up but that they are secure, and available in the relevant storage location. 

Drift Detection Workflows 

The drift detection screen has now been extended to show feature level drift exposed by the detectors that are configured to provide much richer insights into the respective windows of data used to compute drift, allowing for more in-depth workflows as users can now click on the respective drift instance to see the requests in the specified timeframe that were used to calculate the particular drift instance. 

In the dashboard below you can see how the view can be configured across different timelines, displaying the main metrics, p-values, thresholds and distance scores at every feature level. Other drift detection techniques show these metrics at batch level as well. 

Try Out Seldon Deploy! 

Set up a free trial today to test the new Seldon Deploy metadata capabilities for yourself.