whylogs

The open standard for data logging

Documentation • Slack Community • Python Quickstart • WhyLabs Quickstart

What is whylogs

whylogs is an open source library for logging any kind of data. With whylogs, users are able to generate summaries of their datasets (called whylogs profiles) which they can use to:

1. Track changes in their dataset
2. Create data constraints to know whether their data looks the way it should
3. Quickly visualize key summary statistics about their datasets

These three functionalities enable a variety of use cases for data scientists, machine learning engineers, and data engineers:

• Detect data drift in model input features
• Detect training-serving skew, concept drift, and model performance degradation
• Validate data quality in model inputs or in a data pipeline
• Perform exploratory data analysis of massive datasets
• Track data distributions & data quality for ML experiments
• Enable data auditing and governance across the organization
• Standardize data documentation practices across the organization
• And more

If you have any questions, comments, or just want to hang out with us, please join our Slack Community. In addition to joining the Slack Community, you can also help this project by giving us a ⭐ in the upper right corner of this page.

Python Quickstart

import whylogs as why
import pandas as pd

#dataframe
df = pd.read_csv("path/to/file.csv")
results = why.log(df)

• whylogs Profiles
• Data Constraints
• Profile Visualization
• Integrations
• Supported Data Types
• Examples
• Usage Statistics
• Community
• Contribute

whylogs Profiles

What are profiles

whylogs profiles are the core of the whylogs library. They capture key statistical properties of data, such as the distribution (far beyond simple mean, median, and standard deviation measures), the number of missing values, and a wide range of configurable custom metrics. By capturing these summary statistics, we are able to accurately represent the data and enable all of the use cases described in the introduction.

whylogs profiles have three properties that make them ideal for data logging: they are efficient, customizable, and mergeable.

Efficient: whylogs profiles efficiently describe the dataset that they represent. This high fidelity representation of datasets is what enables whylogs profiles to be effective snapshots of the data. They are better at capturing the characteristics of a dataset than a sample would be—as discussed in our Data Logging: Sampling versus Profiling blog post—and are very compact.

Customizable: The statistics that whylogs profiles collect are easily configured and customizable. This is useful because different data types and use cases require different metrics, and whylogs users need to be able to easily define custom trackers for those metrics. It’s the customizability of whylogs that enables our text, image, and other complex data trackers.

Mergeable: One of the most powerful features of whylogs profiles is their mergeability. Mergeability means that whylogs profiles can be combined together to form new profiles which represent the aggregate of their constituent profiles. This enables logging for distributed and streaming systems, and allows users to view aggregated data across any time granularity.

How do you generate profiles

Once whylogs is installed, it’s easy to generate profiles in both Python and Java environments.

To generate a profile from a Pandas dataframe in Python, simply run:

import whylogs as why
import pandas as pd

#dataframe
df = pd.read_csv("path/to/file.csv")
results = why.log(df)

What can you do with profiles

Once you’ve generated whylogs profiles, a few things can be done with them:

In your local Python environment, you can set data constraints or visualize your profiles. Setting data constraints on your profiles allows you to get notified when your data don’t match your expectations, allowing you to do data unit testing and some baseline data monitoring. With the Profile Visualizer, you can visually explore your data, allowing you to understand it and ensure that your ML models are ready for production.

In addition, you can send whylogs profiles to the SaaS ML monitoring and AI observability platform WhyLabs. With WhyLabs, you can automatically set up monitoring for your machine learning models, getting notified on both data quality and data change issues (such as data drift). If you’re interested in trying out WhyLabs, check out the always free Starter edition, which allows you to experience the entire platform’s capabilities with no credit card required.

WhyLabs

import whylogs as why
import os

os.environ["WHYLABS_API_KEY"] = "YOUR-API-KEY"
os.environ["WHYLABS_DEFAULT_DATASET_ID"] = "model-0" # The selected model project "MODEL-NAME" is "model-0"

results = why.log(df)

results.writer("whylabs").write()

If you’re interested in trying out WhyLabs, check out the always free Starter edition, which allows you to experience the entire platform’s capabilities with no credit card required.

Data Constraints

import whylogs as why
from whylogs.core.constraints import Constraints, ConstraintsBuilder
from whylogs.core.constraints.factories import greater_than_number

profile_view = why.log(df).view()
builder = ConstraintsBuilder(profile_view)
builder.add_constraint(greater_than_number(column_name="col_name", number=0.15))

constraints = builder.build()
constraints.report()

To learn more about constraints, check out: the Constraints Example.

Profile Visualization

import whylogs as why

from whylogs.viz import NotebookProfileVisualizer

result = why.log(pandas=df_target)
prof_view = result.view()

result_ref = why.log(pandas=df_reference)
prof_view_ref = result_ref.view()

visualization = NotebookProfileVisualizer()
visualization.set_profiles(target_profile_view=prof_view, reference_profile_view=prof_view_ref)

visualization.summary_drift_report()

To learn more about visualizing your profiles, check out: the Visualizer Example

Data Types

If you want to check out our complete list, please refer to our integrations examples page.

Examples

For a full set of our examples, please check out the examples folder.

Benchmarks of whylogs

By design, whylogs run directly in the data pipeline or in a sidecar container and use highly scalable streaming algorithms to compute statistics. Since data logging with whylogs happens in the same infrastructure where the raw data is being processed, it’s important to think about the compute overhead. For the majority of use cases, the overhead is minimal, usually under 1%. For very large data volumes with thousands of features and 10M+ QPS it can add ~5% overhead. However, for large data volumes, customers are typically in a distributed environment, such as Ray or Apache Spark. This means they benefit from whylogs parallelization—and the map-reducible property of the whylogs profiles keeping the compute overhead to a minimum. Below are benchmarks to demonstrate how efficient whylogs is at processing tabular data with default configurations (tracking distributions, missing values, counts, cardinality, and schema). Two important advantages of this approach are that parallelization speeds up the calculation and whylogs scales with the number of features, rather than the number of rows. Learn more about how whylogs scales here.

Usage Statistics

To read more about what usage statistics whylogs collects, check out the relevant documentation.

To turn off Usage Statistics, simply set the WHYLOGS_NO_ANALYTICS environment variable to True, like so:

import os
os.environ['WHYLOGS_NO_ANALYTICS']='True'

Community

If you have any questions, comments, or just want to hang out with us, please join our Slack channel.

Contribute

How to Contribute

We welcome contributions to whylogs. Please see our contribution guide and our development guide for details.

Contributors

Made with contrib.rocks.