Quality

Quality metrics capture the statistical similarity between real data and synthetic data. If the synthetic and real data are statistically similar, we refer to the synthetic data as being high quality. We intend the quality metrics to be aspirational, as it may not always be possible to achieve 100% quality on all metrics.

Synthetic data can be measured in two ways. Much of the focus has been on measuring statistical data differences between the real and synthetic data, such as quality measures. But this is not enough. Synthetic data needs to provide a return-on-investment (ROI) for the task it is ultimately meant to accomplish — whether it's software testing, machine learning development, or more. When possible, it's important to include metrics that measure ROI in your evaluation.

SDMetrics includes metrics for statistical data differences as well as for the ultimate ROI for different tasks. The two may or may not correlate.

Quality Report

Measure the statistical quality of your entire dataset. The Quality Report is designed to capture statistical quality measurements across multiple tables and columns. It determines which metrics to apply based on the type of columns, providing a consolidated score.

from sdmetrics.reports import QualityReport

report = QualityReport()
report.generate(real_data, synthetic_data, metadata)

Generating report ...

(1/4) Evaluating Column Shapes: |██████████| 15/15 [00:00<00:00, 564.15it/s]|
Column Shapes Score: 85.61%

(2/4) Evaluating Column Pair Trends: |██████████| 55/55 [00:00<00:00, 110.40it/s]|
Column Pair Trends Score: 71.97%

(3/4) Evaluating Cardinality: |██████████| 1/1 [00:00<00:00, 53.27it/s]|
Cardinality Score: 70.0%

(4/4) Evaluating Intertable Trends: |██████████| 50/50 [00:00<00:00, 86.54it/s]|
Intertable Trends Score: 68.49%

Overall Score (Average): 74.02%

How does it work?

The quality report captures the Column Shapes, Column Pair Trends and Cardinality. This guide contains some technical details about each property.

Column Shapes

Does the synthetic data capture the shape of each column?

The shape of a column describes its overall distribution. The higher the score, the more similar the distributions of real and synthetic data.

Methodology

This property applies metrics based on the column types.

Column Type	Metric
numerical	KSComplement
datetime	KSComplement
boolean	TVComplement
categorical	TVComplement

This yields a separate score for every column. The final Column Shapes score is the average of all columns.

You may notice that column shape quality is better for discrete columns (categorical, boolean) as opposed to continuous columns (numerical, datetime). Generally, we've found that it's much easier to create synthetic data for a small number of known categories than large ranges of numerical values.

Column Pair Trends

Does the synthetic data capture trends between pairs of columns?

The trend between two columns describes how they vary in relation to each other, for example the correlation. The higher the score, the more the trends are alike.

Methodology

This property applies a different metric metric based on the type of data

Column Types	Metric
numerical (or datetime) with another numerical (or datetime)	CorrelationSimilarity
categorical (or boolean) with another categorical (or boolean)	ContingencySimilarity
numerical (or datetime) with a categorical (or boolean)	Discretize the numerical columns into bins, then apply ContingencySimilarity

This yields a score between every pair of columns.* The Column Pair Trends score is the average of all the scores.

*Starting from SDMetrics version 0.27.0, the Quality Report discards pairs that do not exhibit a strong pattern in the real data to begin with. A strong correlation is defined as a Pearson correlation of >0.5 or <-0.5, or a Cramer's association of >0.3.

The CorrelationSimilarity metric works by computing a separate value for the real vs. the synthetic data. The Quality Report shows a side-by-side visualization for real vs. synthetic data when applicable.

Cardinality

This property is only available for multi table datasets. (In older versions of SDMetrics, it was known as "Table Relationships".)

Does the synthetic data capture the number of connections between parent and child tables? This is also known as the cardinality of the tables.

Methodology

This property applies the CardinalityShapeSimilarity metric for every set of connected tables: parent table and child table.

Intertable Trends

This property is only available for multi table datasets.

Does the synthetic data capture trends between columns across different tables?

This is similar to the Column Pair Trends property, but it is applied across parent/child tables. For example, a column in a parent table might be correlated with a column in the child.

Methodology

This property denormalizes the parent and child table into a single, flat table. Then, it applies the same metrics as the Column Pair Trends property.

Column Types	Metric
numerical (or datetime) with another numerical (or datetime)	CorrelationSimilarity
categorical (or boolean) with another categorical (or boolean)	ContingencySimilarity
numerical (or datetime) with a categorical (or boolean)	Discretize the numerical columns into bins, then apply ContingencySimilarity

This yields a score between every pair of columns*. The Intertable Trends score is the average of all the scores.

*Starting from SDMetrics version 0.27.0, the Quality Report discards pairs that do not exhibit a strong pattern in the real data to begin with. A strong correlation is defined as a Pearson correlation of >0.5 or <-0.5, or a Cramer's association of >0.3.

Browse Metrics

Alternatively, you can apply quality metrics to individual columns and tables in your data:

• KSComplement, TVComplement: compare column shapes (aka marginal distributions, histograms)

• ContingencySimilarity, CorrelationSimilarity: compare 2D distributions & pairwise correlations

• CardinalityShapeSimilarity: compare the frequency of parent/child connections (aka cardinality)

• CategoryCoverage, RangeCoverage: measure whether the overall synthetic data spans all the possibilities

• InterRowMSAS, SequenceLengthSimilarity, StatisticMSAS: compares the quality of real and synthetic data that represents sequential information

• MissingValueSimilarity, StatisticSimilarity: compare individual statistics of the data