Single Table Metadata API

This guide will walk you through creating the metadata using the Python API.
Creation API
Get started by creating a blank SingleTableMetadata 
from sdv.metadata import SingleTableMetadata
​
metadata = SingleTableMetadata()
Auto detect metadata
Automatically detect the metadata based on your actual data. Different methods are available based on the format of your data.
DataFrame
CSV
detect_from_dataframe: Use this function to automatically detect metadata from your data that is available in a pandas.DataFrame object
metadata.detect_from_dataframe(data=my_pandas_dataframe)
Parameters
(required) data: A pandas.DataFrame containing your real data
Output (None)
detect_from_csv: Use this function to automatically detect metadata from your data that is available in a CSV file
metadata.detect_from_csv(filepath='data/guests.csv')
Parameters
(required) filepath: The location of the CSV file that contains your data
Output (None)
The detected metadata is not guaranteed to be accurate or complete. Be sure to carefully inspect the metadata and update information.
Primary keys and other identifiers are auto-detected, but may be incorrect or incomplete. See set_primary_key and add_alternate_keys method to add them.
Sensitive information may not be auto-detected. Check for columns with an 'unknown' sdtype and use the update_column method to update them.
Inspection API
At any point, you can inspect the current state of the metadata.
to_dict
Use this to get a copy of the Python dictionary that corresponds to the metadata.
Parameters (None)
Output A Python dictionary that corresponds to the metadata
python_dict = metadata.to_dict()
Note that the returned object is a representation of the metadata. Changing it will not modify the original metadata object in any way.
visualize
Use this to this to see a visual representation of the metadata. Use the parameters to control the level of details in the visualization and for saving the image.
Parameters 
show_table_details: Toggle the display of column details
(default) 'full'
Show all the different column names, primary keys and foreign keys
'summarized'
Summarize the columns based on the data type
output_filepath: If provided, save the image at the given location in the given format
The output_filepath must end with the filetype that you want to save as. Popular examples are png, jpg or pdf.
Output A graphviz.graphs.Digraph​
metadata.visualize(
    show_table_details='summarized',
    output_filepath='my_metadata.png'
)
Validation API
validate
Use this to validate that the metadata is written according to the specification. This function will throw descriptive errors if there is anything wrong with the metadata.
Parameters (None)
Output  (None) 
metadata.validate()
InvalidMetadataError: The metadata is not valid
​
Error: Invalid values ("pii") for datetime column "start_date".
Error: Invalid regex format string "[A-{6}" for id column "user_id"
validate_data
Use this method to validate that the metadata accurately describes a particular dataset. This function will throw descriptive errors if there is any mismatch between the metadata and data.
Parameters:
(required) data: A pandas.DataFrame containing data. The data should have the same columns as described in the metadata.
Output (None)
metadata.validate_data(data=my_dataset)
Update API
It is important to verify and update any inaccuracies in the metadata
update_column
Use this method to modify the information about a column in your metadata
Parameters
(required) column_name:  The name of the column to update
(required) sdtype: A string describing the statistical data type.
Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available (see SDTypes).
<other properties>: Based on the sdtype, provide other parameters. See below for some options.
boolean
categorical
datetime
numerical
id
other
Boolean columns represent True or False values.
metadata.update_column(
    column_name='has_rewards',
    sdtype='boolean')
Properties (None)
Categorical columns represent discrete data
metadata.update_column(
    column_name='room_type',
    sdtype='categorical')
Properties (None)
Date columns represent a point in time
metadata.update_column(
    column_name='checkin_date',
    sdtype='datetime',
    datetime_format='%d %b %Y')
Properties
datetime_format: A string describing the format as defined by Python's strftime module.
Numerical columns represents discrete or continuous numerical values. 
metadata.update_column(
    column_name='amenities_fee',
    sdtype='numerical',
    computer_representation='Float')
Properties
computer_representation: A string that represents how you'll ultimately store the data. This determines the min and max values allowed
Available options are: 'Float', 'Int8', 'Int16', 'Int32', 'Int64', 'UInt8', 'UInt16', 'UInt32', 'UInt64'
ID columns represent identifiers that do not have any special mathematical or semantic meaning
metadata.update_column(
    column_name='user_id',
    sdtype='id',
    regex_format='U_[0-9]{3}')
Properties
regex_format: A string describing the format of the ID as a regular expression​
You can input other data types such as 'phone_number', 'ssn' or 'email'. See the Sdtypes Reference for a full list.
metadata.update_column(
    column_name='billing_address',
    sdtype='address',
    pii=True
)
Properties
pii: A boolean denoting whether the data is sensitive
(default) True: The column is sensitive, meaning the values should be anonymized 
False: The column is not sensitive
Output (None)
set_primary_key
Use this function to set the primary key of the table. Any existing primary keys will be removed.
The primary key uniquely identifies every row in the table. When you set a primary key, the SDV will guarantee that every value in the table is unique.
At this time, the SDV does not support composite keys.
Parameters
(required) column_name: The column name of the primary key. The column name must already be defined in the metadata and it must be an ID or another PII sdtype.
Output (None)
metadata.set_primary_key(column_name='guest_email')
add_alternate_keys
Use this function to set alternate keys of the table. This method will add to any existing alternate keys you may have.
Similar to primary keys, alternate keys are also unique in your table. However, other tables do not reference alternate keys.
Parameters
(required) column_names: A list of column names that represent the alternate keys in the table. All column names must already be defined in the metadata and they must be IDs or another PII sdtype.
Output (None)
metadata.add_alternate_keys(column_names=['credit_card_number'])
Saving, Loading & Sharing Metadata
You can save the metadata object as a JSON file and load it again for future use.
save_to_json
Use this to save the metadata object to a new JSON file that will be compatible with SDV 1.0 and beyond. We recommend you write the metadata to a new file every time you update it.
Parameters
(required) filepath: The location of the file that will be created with the JSON metadata
Output (None) 
metadata.save_to_json(filepath='my_metadata_v1.json')
Load previously saved metadata
If you already have a metadata JSON file, you can load it in as a SingleTableMetadata object. Use the method based on the version of your JSON file.
JSON file for SDV 1.0+
Older JSON files
load_from_json: If you recently wrote your JSON file for SDV, use this class method to load it as a SingleTableMetadata object.
from sdv.metadata import SingleTableMetadata
​
metadata = SingleTableMetadata.load_from_json(
    filepath='my_metadata_v1.json')
Parameters
(required) filepath: The name of the file containing the JSON metadata
Output A SingleTableMetadata object
upgrade_metadata: If you wrote a JSON file for any SDV version before 1.0, use this class method to upgrade the metadata.
You have older metadata if you see a key named "fields" instead of "columns".
from sdv.metadata import SingleTableMetadata
​
metadata = SingleTableMetadata.upgrade_metadata(
    filepath='my_old_metadata.json'
)
Parameters
(required) old_filepath: The filepath to your older metadata JSON file
Output A SingleTableMetadata object with the new metadata
Tip! After upgrading, save your metadata so you can use it again
metadata.save_to_json('my_new_metadata.json')
You can also load the metadata from a Python dictionary with the information.
load_from_dict
Use this class method to load a Python dictionary as a SingleTableMetadata object.
Parameters
(required) metadata_dict: A Python dictionary representation of the metadata. See Metadata Spec for more details.
Output A SingleTableMetadata object
from sdv.metadata import SingleTableMetadata
​
metadata_obj = SingleTableMetadata.load_from_dict(metadata_dict)
＊ anonymize
Use this method to anonymize the column names of your metadata. This makes it easier to share your metadata, eg. for debugging purposes.
Parameters (None)
Output A new SingleTableMetadata object that represents the anonymized metadata
anonymized_metadata = original_metadata.anonymize()
＊This feature is only available for licensed, enterprise users. To learn more about the SDV Enterprise features and purchasing a license, get in touch with us.
The anonymized metadata contains new column names. The original names are obfuscated, but the sdtypes and other formatting information remains the same.
>>> anonymized_metadata.to_dict()
{
    'primary_key': 'id_0',
    'columns': {
        'id_0': { 'sdtype': 'id', 'regex_format': 'ID_[0-9]{10}' },
        'num_0': { 'sdtype': 'numerical' },
        'num_1': { 'sdtype': 'numerical' },
        'cat_0': { 'sdtype': 'categorical' },
        'dt_0': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
        'pii_0': { 'sdtype': 'ssn' },
        ...
    }
}
​
Last modified 10d ago
(default) `'full'`	Show all the different column names, primary keys and foreign keys
`'summarized'`	Summarize the columns based on the data type