Results Summary

Benchmark results are available for every synthesizer and dataset pair. The summary is returned in Python as a pandas DataFrame object, and also written in a CSV file if you've provided a destination folder.

Synthesizer                Dataset   Dataset_Size_MB   Train_Time   Peak_Memory_MB   Synthesizer_Size_MB    Sample_Time    Evaluate_Time   Diagnostic_Score  Quality_Score   Privacy_Score    Adjusted_Total_Time    Adjusted_Quality_Score
GaussianCopulaSynthesizer  alarm     34.5              123.56       300101           0.981                  2012.1         1001.2          1.00000           0.9991991       0.998191         2136.76                0.9991991
GaussianCopulaSynthesizer  census    130.2             2356.12      201011           1.232                  2012.2         1012.1          1.00000           0.689101        1.0              4383.82                0.689101
CTGANSynthesizer           alarm     34.5              NaN          99999999         NaN                    NaN            NaN             1.00000           NaN             NaN              3605.3                 0.5829
CTGANSynthesizer           census    130.2             3140.4       9929188110       12.10                  NaN            NaN             1.00000           NaN             NaN              3166.6                 0.49102  
UniformSynthesizer         alarm     34.5              1.1          10               0.010                  5.2            1000            1.00000           0.5829          1.0              6.3                    0.5829
UniformSynthesizer         census    130.2             15.5         2012.2           0.031                  10.7           0.321           1.00000           0.49102         1.0              26.2                   0.49102

Description

The summary provides information about the setup, execution performance, and the overall evaluation. Browse through the tabs below to learn more about what each result means.

Setup

These results summarize the setup of your benchmarking run.

• Synthesizer: The name of the synthesizer used to model and create the synthetic data

• Dataset: The name of the dataset that the synthesizer learned to create

• Dataset_Size_MB: The overall size of the dataset when loaded into Python, in MB

Execution

Evaluation

Adjusted Times

Graceful Handling of Errors

In the case that some synthesizers crash or time out, the results will show a NaN value from that point of the execution process. For example, if your synthesizer ran out of memory during the sampling phase, you'll see NaN values for the sample time, evaluation time and other metrics.

However in an enterprise setting, it is best practice to gracefully degrade by reverting to a backup synthesizer in the case of an error. To simulate this, SDGym always trains a UniformSynthesizer as backup. The UniformSynthesizer creates random values within the correct ranges. It represents a fast, lightweight backup.

The final columns you see (for time and quality score) are adjusted to simulate having the backup synthesizer. These columns will not contain any missing values, ensuring that you can continue to compare synthesizers fairly even if there are errors for certain datasets.