Metadata API

Creation API

There are various ways to create your metadata. For example, if you are an SDV Enterprise user, you can directly connect to a database to load data and create metadata all at once.

Otherwise, if you are in possession of your data, you can auto-detect the metadata.

detect_from_dataframes

Use this function to automatically detect metadata from your data that you've loaded as a pandas.DataFrame objects.

Parameters:

• (required) data: Your data, represented as a dictionary. The keys are your table names and values are the pandas.DataFrame objects containing your data.

• infer_sdtypes: A boolean describing whether to infer the sdtypes of each column

  • (default) True: Infer the sdtypes of each column based on the data.

  • False: Do not infer the sdtypes. All columns will be marked as unknown, ready for you to manually update.

• infer_keys: A string describing whether to infer the primary and/or foreign keys.

  • (default) 'primary_and_foreign': Infer the primary keys in each table, and the foreign keys in other tables that refer to them

  • 'primary_only': Infer the primary keys in each table. You can manually add the foreign key relationships later.

  • None: Do not infer any primary or foreign keys. You can manually add these later.

Output A Metadata object that describes the data

from sdv.metadata import Metadata

metadata = Metadata.detect_from_dataframes(
    data={
        'hotels': hotels_dataframe,
        'guests': guests_dataframe
    })

The detected metadata is not guaranteed to be accurate or complete. Be sure to carefully inspect the metadata and update information

Inspection API

At any point, during the metadata creation or updates, you can inspect the current state of the metadata.

to_dict

Use this to get a copy of the Python dictionary that corresponds to the metadata.

Parameters (None)

Output A Python dictionary that corresponds to the metadata

python_dict = metadata.to_dict()

Note that the returned object is a representation of the metadata. Changing it will not modify the original metadata object in any way.

visualize

Use this to this to see a visual representation of the metadata. Use the parameters to control the level of details in the visualization and for saving the image.

Parameters

• show_table_details: Toggle the display of column details

(default) 'full'	Show all the different column names, primary keys and foreign keys
'summarized'	Summarize the columns based on the data type
None	Hide the details. Only show the table name.

• show_relationship_labels: Toggle the display of the table relationships

(default) True	Label each relationship between 2 tables with the column names
False	Do not label the relationships. Only show an arrow between tables.

• output_filepath: If provided, save the image at the given location in the given format

The output_filepath must end with the filetype that you want to save as. Popular examples are png, jpg or pdf.

Output A graphviz.graphs.Digraph

metadata.visualize(
    show_table_details='full',
    show_relationship_labels=True,
    output_filepath='my_metadata.png'
)

get_column_names

Use this function to look up column names based on the metadata properties that they have.

This is particularly useful if you want to list all columns assigned to a specific sdtype, such as unknown in order to update it.

Parameters

• (required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available (see SDTypes).
• table_name: The name of the table. This is required if you have multiple tables.

• <other properties>: Based on the sdtype, provide other parameters. For more information, see the Metadata Spec.

Output A list of strings, with the column names that match the criteria. If no columns match the criteria, then an empty string will be returned.

metadata.get_column_names(sdtype='unknown', table_name='products')

[ 'product_id', 'product_code_name', 'code_type']

Validation API

validate

Use this to validate that the metadata is written according to the specification. This function will throw descriptive errors if there is anything wrong with the metadata.

Parameters (None)

Output (None)

metadata.validate()

InvalidMetadataError: The metadata is not valid

Error: Invalid values ("pii") for datetime column "start_date".
Error: Invalid regex format string "[A-{6}" for id column "hotel_id"

validate_data

Use this method to validate that the metadata accurately describes a particular dataset. This function will throw descriptive errors if there is any mismatch between the metadata and data.

Parameters:

• (required) data: A dictionary containing your multi-table data. Each key should be the name of a table and the value should be a pandas.DataFrame containing its data. The data should have the same tables and columns as described in the metadata.

Output (None)

metadata.validate_data(data={
    'hotels': hotels_dataframe,
    'guests': guests_dataframe
})

validate_table

Use this method to validate that the metadata accurately describes a single data table. This function will throw descriptive errors if there is any mismatch between the metadata and data.

Parameters:

• (required) data: A pandas.DataFrame containing data. The data should have the same columns as described in the metadata.

• table_name: The name of the table. This is required if you have multiple tables.

Output (None)

metadata.validate_table(data=my_dataframe)

Output (None)

Update API

It is important to verify and update any inaccuracies in the metadata

update_column

Use this method to modify the information about a column in your metadata

Parameters

• (required) column_name: The name of the column to update

• (required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available. For more information, see SDTypes docs.
• table_name: The name of the table. This is required if you have multiple tables.

• <other properties>: Based on the sdtype, provide other parameters. For more information, see SDTypes docs.

Output (None)

metadata.update_column(
    column_name='start_date',
    sdtype='datetime',
    table_name='guests',
    datetime_format='%Y-%m-%d')
    
metadata.update_column(
    column_name='user_cell',
    sdtype='phone_number',
    table_name='guests',
    pii=True)

update_columns

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the same parameters for a group of columns.

Parameters

• (required) column_names: A list of strings representing the column names to update. All columns must be in the table.

• (required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. But other types such as 'phone_number' are also available (see SDTypes).
• table_name: The name of the table. This is required if you have multiple tables.

• <other properties>: Based on the sdtype, provide other parameters

Output (None)

metadata.update_columns(
    column_names=['age', 'transactions', 'session_length'],
    sdtype='numerical',
    table_name='users',
    computer_representation='Float'
)

update_columns_metadata

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the different parameters for each column

Parameters

• (required) column_metadata: A dictionary mapping each column name you want to update to the metadata information for that column. All columns must be in the table. For the exact format, see the Metadata Spec.
• table_name: The name of the table. This is required if you have multiple tables.

Output (None)

metadata.update_columns_metadata(
    column_metadata={
        'age': { 'sdtype': 'numerical' },
        'ssn': { 'sdtype': 'ssn', 'pii': True },
        'gender': { 'sdtype': 'categorical' },
        'dob': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
        ...
    },
    table_name='users',
)

add_column

Use this function to add a column to your metadata object.

Parameters

• (required) column_name : Name of the column to be added

• (required) sdtype: A string describing the statistical data type. Common types are 'boolean', 'categorical', 'datetime', 'numerical' and 'id'. Other types such as 'phone_number' are also available (see SDTypes).
• table_name: The name of the table. This is required if you have multiple tables.

• **kwargs: Any other parameters you need that describe metadata for a column.

Output (None)

metadata.add_column(
  column_name='cell_phone_numbers',
  sdtype='phone_number',
  table_name='users',
  pii=True
)

remove_column

Use this function to remove a column from your metadata object.

Parameters

• (required) column_name: The name of the column to bedeleted

• table_name: The name of the table. This is required if you have multiple tables.

Output (None). This removes the column anywhere it appears in the metadata — in a relationship, as a primary key, etc.

metadata.remove_column(
    column_name='cell_phone_numbers',
    table_name='users'
)

add_column_relationship

Use this function to specify when a group of columns within the same table represent the same concept.

While anyone can add column relationships to their data, SDV Enterprise users will see the highest quality data for the relationships. To learn more about the SDV Enterprise and its extra features, visit our website.

Parameters

• (required) relationship_type: A string with the type of relationship. This represents a higher level concept. See the tabs below for options.

• (required) column_names: A list of column names that are part of that relationship. Make sure that these columns are compatible with the relationship type. See the tabs below for more information.

• table_name: The name of the table. This is required if you have multiple tables.

address

An address is defined by 2 or more columns that have the following sdtypes: country_code, administrative_unit, state, state_abbr, city, postcode, street_address and secondary_address.

metadata.add_column_relationship(
    relationship_type='address',
    column_names=['addr_line1', 'addr_line2', 'city', 'zipcode', 'state']
)

gps

A GPS coordinate pair is defined by 2 columns:

• sdtype latitude &

• sdtype longitude

metadata.add_column_relationship(
    relationship_type='gps',
    column_names=['location_lat', 'location_lon']
)

More coming soon!

Additional column relationships coming soon!

Do you have a request for a type of column relationship? Please file a feature request describing your use case.

Output (None)

set_primary_key

Use this function to set the primary key of the table. Any existing primary keys will be removed.

The primary key uniquely identifies every row in the table. When you set a primary key, the SDV will guarantee that every value in the table is unique. At this time, the SDV does not support composite keys.

Parameters

• (required) column_name: The column name of the primary key. The column name must already be defined in the metadata and it must be an ID or another PII sdtype.

• table_name: The name of the table. This is required if you have multiple tables.

Output (None)

metadata.set_primary_key(
    column_name='hotel_id',
    table_name='hotels',
)

remove_primary_key

Use this function to remove any existing primary keys in a table.

Parameters

• table_name: The name of the table. This is required if you have multiple tables.

Output (None) The primary key will be removed. Any existing relationships that use the primary key will be removed too.

metadata.remove_primary_key(table_name='guests')

add_alternate_keys

Use this function to set alternate keys of the table. This method will add to any existing alternate keys you may have.

Similar to primary keys, alternate keys are also unique in your table. However, other tables do not reference alternate keys.

Parameters

• (required) column_names: A list of column names that represent the alternate keys in the table. All column names must already be defined in the metadata and they must be IDs or other PII sdtypes.

• table_name: The name of the table. This is required if you have multiple tables.

Output (None)

metadata.add_alternate_keys(
    column_names=['credit_card_number'],
    table_name='guests',
)

add_relationship

Use this method to add a relationship between 2 connected tables: A parent and child table. The parent table contains the primary key references while the child table has rows that refer to its parent. Multiple child rows can refer to the same parent row.

Parameters:

• (required) parent_table_name: The name of the parent table

• (required) child_table_name: The name of the child table that refers to the parent

• (required) parent_primary_key: The primary key column in the parent table. This column uniquely identifies each row in the parent table .

• (required) child_foreign_key: The foreign key column in the child table. The values in this column contain a reference to a row in the parent table

Output (None)

metadata.add_relationship(
    parent_table_name='hotels',
    child_table_name='guests',
    parent_primary_key='hotel_id',
    child_foreign_key='hotel_id'
)

remove_relationship

Use this method to remove the connection between a parent and child table. In the case where there are multiple connections, this method will remove all the connections. Use this if the metadata has incorrectly detected relationships.

Parameters:

• (required) parent_table_name: The name of the parent table

• (required) child_table_name: The name of the child table that refers to the parent

Output (None)

metadata.remove_relationship(
    parent_table_name='hotels',
    child_table_name='guests'
)

add_table

Use this method to add a new table to your metadata.

Parameters:

• (required) table_name: The name of the table to add.

Output (None) After the table added, the new table will be empty, so be sure to add columns to it using the add_column function.

metadata.add_table(
    table_name='travel_details'
)

remove_table

Use this method to remove an existing table from your metadata.

Parameters:

• (required) table_name: The name of the table to remove

Output (None) The table will be removed, as well as any relationships that included the table.

metadata.remove_table(
    table_name='travel_details'
)

Adding multi-sequence information

If your data includes multiple sequences, use these methods to add information about them.

set_sequence_key

Use this function to set the sequence key of your table. Any existing sequence keys will be removed.

The sequence key is a column that identify which row(s) belong to which sequences. This is usually an ID column but it may also be a PII sdtype (such as "phone_number"). At this time, SDV does not support composite keys.

This is important for tables that contain multiple sequences. In our example, the sequence key is 'Patient ID' because this column is used to break up the sequences.

If you don't supply a sequence key, the SDV assumes that your table only contains a single sequence. Note: The SDV sequential models do not fully support single sequence data.

Parameters

• (required) column_name: The column name of the sequence key. The column name must already be defined in the metadata and it must be an ID or another PII sdtype.

Output (None)

metadata.set_sequence_key(column_name='Patient ID')

set_sequence_index

Use this function to set the sequence index of your table. Any existing sequence indices will be removed.

The sequence index determines the spacing between the rows in a sequence. Use this if you have an explicit index such as a timestamp. If you don't supply a sequence index, the SDV assumes there is equal spacing of an unknown unit.

Parameters

• (required) column_name: The column name of the sequence index. The column name must already be defined in the metadata. It must be either a numerical or datetime column.

Output (None)

metadata.set_sequence_index(column_name='Time')

Saving, Loading & Sharing Metadata

You can save the metadata object as a JSON file and load it again for future use.

save_to_json

Use this to save the metadata object to a new JSON file that will be compatible with SDV 1.0 and beyond. We recommend you write the metadata to a new file every time you update it.

Parameters

• (required) filepath: The location of the file that will be created with the JSON metadata

• mode: A string describing the mode to use when creating the JSON file

  • (default) 'write': Write the metadata to the file, raising an error if the file already exists

  • 'overwrite': Write the metadata to the file, replacing the contents if the file already exists

Output (None)

metadata.save_to_json(filepath='metadata.json')

load_from_json

Use this method to load your file as a Metadata object.

Parameters

• (required) filepath: The name of the file containing the JSON metadata

Output: A Metadata object.

load_from_dict

Use this class method to load a Python dictionary as a Metadata object.

Parameters

• (required) metadata_dict: A Python dictionary representation of the metadata.

Output A Metadata object

from sdv.metadata import Metadata

metadata_obj = Metadata.load_from_dict(metadata_dict)

anonymize

Use this method to anonymize the column names of your metadata. This makes it easier to share your metadata, eg. for debugging purposes.

Parameters (None)

Output A new Metadata object that represents the anonymized metadata

anonymized_metadata = original_metadata.anonymize()

The anonymized metadata contains new column names. The original names are obfuscated, but the sdtypes and other formatting information remains the same.

>>> anonymized_metadata.to_dict()
{
    'tables': {
        '3oc2d': {
            'primary_key': 'id_0',
            'columns': {
                'id_0': { 'sdtype': 'id', 'regex_format': 'ID_[0-9]{10}' },
                'num_0': { 'sdtype': 'numerical' },
                'num_1': { 'sdtype': 'numerical' },
                'cat_0': { 'sdtype': 'categorical' },
                'dt_0': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
                'pii_0': { 'sdtype': 'ssn' },
                ...
            }
        }
    }
}

copy

Use this method to create a copy of your metadata in its current state.

Parameters (None)

Output A new Metadata object that is a copy of the current metadata. Note that the originsl metadata and the copy are separate objects, so modifying one will not modify the other.

metadata_copy = metadata.copy()