Metadata

Metadata is a description of your dataset. It helps the library understand the type and format of your data, so that it can apply the correct metrics.

If the metadata is incorrect, this library may apply the wrong metrics to your data which leads to inaccurate scores.

What does metadata include?

Your metadata includes:

• The type of data that each column represents

• The primary keys and other identifiers of the table

• The relationships between the tables, if you have multiple tables

Write your metadata once and use it anywhere in the SDMetrics library.

If you used the SDV library to create your metadata, you can reuse the same file for SDMetrics.

For example, your data may be present in multiple tables with rows and columns. The tables may be connected to each other through primary and foreign key references.

This example of a Multi Table dataset has a table for users and a table for their sessions. Each user can have multiple sessions recorded.

Your data description is called metadata. SDMetrics expects metadata as a Python dictionary object.

Click to see the metadata

This is the metadata dictionary for the illustrated table

{
    "tables": {
        "users": {
            "primary_key": "user_id",
            "columns": {
                "user_id": {
                    "sdtype": "id",
                    "regex_format": "U_[0-9]{3}"
                },
                "age": {
                    "sdtype": "numerical"
                },
                "address": {
                    "sdtype": "address",
                    "pii": True
                }
            }
        },
        "sessions": {
            "primary_key": "session_id",
            "columns": {
                "session_id": {
                    "sdtype": "id"
                },
                "user": {
                    "sdtype": "id",
                    "regex_format": "U_[0-9]{3}"
                },
                "date": {
                    "sdtype": "datetime",
                    "datetime_format": "%Y-%m-%d"
                },
                "browser": {
                    "sdtype": "categorical"
                },
                "bounced": {
                    "sdtype": "boolean"
                }
            }
        }
    },
    "relationships": [{
        "parent_table_name": "users",
        "parent_primary_key": "user_id",
        "child_table_name": "sessions",
        "child_foreign_key": "user_id"
    ]}
}

Metadata Specification

The file is an object that includes a dictionary named "tables".

{
    "tables": {
        <tables information>
    },
}

Tables

The "tables" dictionary contains the information about each individual table of your application. Its keys are the table names and the values are dictionaries that describe each single table. This includes:

• "primary_key": the column name used to identify a row in your table. For a composite key, provide a list of column names.

• (required) "columns": a dictionary description of each column

{
    "tables": {
        "users": {
            "primary_key": "user_id",
            "columns": { <column information> }
         },
        "sessions": {
            "primary_key": "session_id",
            "columns": { <column information> }
        }
    },
    ...
}

Column Information

Inside "columns", you will describe each column. You'll start with the name of the column. Then you'll specify the type of data and any other information about it.

There are specific data types to choose from. Expand the options below to learn about the data types.

boolean

Boolean columns represent True or False values.

"active": { 
    "sdtype": "boolean"
}

Properties (None)

categorical

Categorical columns describe discrete data.

"tier": {
    "sdtype": "categorical",
}

Properties (None)

datetime

Date columns represent a point in time

"renew_date": {
    "sdtype": "datetime",
    "format": "%Y-%m-%d"
}

Properties

• datime_format: A string describing the format as defined by Python's strftime module. This is required if your datetime columns are represented as strings.

The format string has special values to describe the components. For example, Jan 06, 2022 is represented as "%b %d, %Y". Common values are:

• Year: "%Y" for a 4-digit year like 2022, or "%y" for a 2-digit year like 22

• Month: "%m" for a 2-digit month like 01, "%b" for an abbreviated month like Jan

• Day: "%d" for a 2-digit day like 06

numerical

Numerical columns represents discrete or continuous numerical values.

"age": {
    "sdtype": "numerical"
},
"paid_amt": {
    "sdtype": "numerical",
    "compute_representation": "Float"
}

Properties

• computer_representation: A string that represents how you'll ultimately store the data. This determines the min and max values allowed Available options are: 'Float', 'Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64'

id

ID columns represent identifiers that do not have any special mathematical or semantic meaning

"user_id": { 
    "sdtype": "id",
    "regex_format": "U_[0-9]{3}"
}

Properties

• regex_format: A string describing the format of the ID as a regular expression

Do you have embedded context within the Regex? For example maybe the first two letters in th ID must refer to a known country code such as "US_49102" or "CA_19341". In this case, use a named capture groups within the Regex to note the context. Denoted by a parenthesis and the group name, named capture groups let the SDMetrics know about the context. In the example above, the Regex format would be:

(?P<country>[A-Z]{2})_[0-9A-Z]{5}

other

You can input any other data type such as 'phone_number', 'ssn' or 'email'. See the Sdtypes Reference for a full list.

"address": {
    "sdtype": "address",
    "pii": True
}

Properties

• pii: A boolean denoting whether the data is sensitive

  • (default) True: The column is sensitive, meaning the synthetic data is anonymized

  • False: The column is not sensitive, meaning the synthetic data may not be anonymized

Relationships

Inside the "relationships" section of the metadata, provide a list of relationships that exist between tables. Each relationship is represented as a dictionary with the following keys:

• "parent_table_name": The name of the parent table

• "parent_primary_key": The primary key column in the parent table. This column uniquely identifies each row in the parent table. For a composite key, provide a list of column names.

• "child_table_name": The name of the child table that refers to the parent

• "child_foreign_key": The foreign key column in the child table. The values in this column contain a reference to a row in the parent table. For a composite key, provide a list of column names.

Use multiple dictionaries to represent multiple tables.

"relationships": [{
        "parent_table_name": "users",
        "parent_primary_key": "user_id",
        "child_table_name": "sessions",
        "child_foreign_key": "user_id"
 ]}

Saving & Loading Metadata

After creating your dictionary, you can save it as a JSON file. For example, my_metadata_file.json.

import json

with open('my_metadata_file.json', 'w') as f:
    json.dump(my_metadata_dict, f)

In the future, you can load the Python dictionary by reading from the file.

import json 

with open('my_metadata_file.json') as f:
    my_metadata_dict = json.load(f)

# use my_metadata_dict in the SDMetrics library

Adding Multi-Sequence Information

In some cases, your data table may contain multiple, independent sequences belonging to different entities. See the diagram below for an illustration of sequential data.

This example shows sequential data related to vital signs. The table contains multiple sequences, each corresponding to a different patient. For each sequences, health measurements change over time.

In this case, you can add some information about the sequential nature of this data to your metadata specification. In the dictionary for the table, add:

• "sequence_key": the name of a column that identifies each unique sequence in your data

• "sequence_index": the column name used to order the rows in the table

An example for the table is provided below.

Click to see the sequential table's metadata

This is the metadata dictionary for the illustrated sequential table

{
    "tables": {
        "patients": {
            "sequence_key": "Patient ID",
            "sequence_index": "Time",
            "columns": {
                "Patient ID": {
                    "sdtype": "id",
                    "regex_format": "ID_[0-9]{3}"
                },
                "Address": {
                    "sdtype": "address",
                    "pii": True
                },
                "Smoker": {
                    "sdtype": "boolean"
                },
                "Time": {
                    "sdtype": "datetime",
                    "datetime_format": "%m/%d/%Y"
                },
                "Heart Rate": {
                    "sdtype": "categorical"
                },
                "Systolic BP": {
                    "sdtype": "numerical"
                }
            }
        }
    }
}

At this time, SDMetrics offers limited evaluation support for data involving sequences. You may run the Diagnostic and Quality reports with your data, but these reports do not factor in any order or sequence-related measures.

In addition to the report, you can run individual metrics that are suited for sequential data. See:

• SequenceLengthSimilarity

• StatisticMSAS

• Metrics in Beta: Sequential Detection, Sequential ML Efficacy