# Multi Table Metadata API

This guide will walk you through creating the metadata using the Python API.

## Creation API

Get started by creating a blank `MultiTableMetadata` object.

```python
from sdv.metadata import MultiTableMetadata

metadata = MultiTableMetadata()
```

Automatically detect the metadata based on your actual data. Different methods are available based on the format of your data.

{% tabs %}
{% tab title="DataFrame" %}
**`detect_from_dataframes`**: Use this function to automatically detect metadata from *multiple* tables at a time. 

```python
metadata.detect_from_dataframes(
    data={
        'hotels': hotels_table,
        'guests': guests_table
    }
)
```

**Parameters**

* (required) `data`: A dictionary mapping the name of the table to the pandas.DataFrame that contains its data

**Output** (None)

{% hint style="info" %}
*Deprecation Notice: In previous versions of the SDV, you could detect one table at a time using `detect_table_from_dataframe`. We do not recommend using this older function.*
{% endhint %}
{% endtab %}

{% tab title="CSV" %}
**`detect_from_csvs`**: Use this function to automatically detect metadata from muliple CSV files

```python
metadata.detect_from_csvs(
    folder_name='my_data_folder/'
)
```

**Parameters**

* (required) `folder_name`: The name of the folder that contains all your CSV files. This method assumes each CSV file is a separate table of your metadata.

**Output** (None)

{% hint style="info" %}
*Deprecation Notice: In previous versions of the SDV, you could detect one table at a time using `detect_table_from_csv`. We do not recommend using this older function.*
{% endhint %}
{% endtab %}

{% tab title="＊DDL" %}
＊**`detect_from_ddl`**: Use this function to automatically detect metadata from your data that is available in a DDL file from an SQL schema.

*This functionality is currently in Beta testing. Please let us know if you encounter any issues.*

```python
metadata.detect_from_ddl(
    filepath='my_schema.ddl',
    db_type='ibm_db2',
    database_name='users'
)
```

**Parameters**

* (required) `filepath`: The location of the DDL file that contains the schema
* (required) `db_type`: A string with the type of SQL variant. Currently only `'ibm_db2'` is supported
* `database_name`: A string with the name of the database. Only the tables from the database will be included in the metadata. *In the DDL file, the database name is usually prepended to the table name, such as `users.transaction_log`.*
  * (default) `None`: Detect data from all database names found in the DDL file.

**Output** (None)

{% hint style="info" %}
**＊SDV Enterprise Feature.** This feature is only available for licensed, enterprise users. To learn more about the SDV Enterprise features and purchasing a license, [visit our website](https://datacebo.com/pricing/).
{% endhint %}
{% endtab %}
{% endtabs %}

{% hint style="warning" %}
**The detected metadata is not guaranteed to be accurate or complete.** Be sure to carefully inspect the metadata and update information.

* Primary keys and other identifiers are auto-detected, but may be incorrect or incomplete. See [`set_primary_key`](#set_primary_key) and [`add_alternate_keys`](#add_alternate_keys) method to add them.
* Sensitive information may not be auto-detected. Check for columns with an `'unknown'` sdtype and use the [`update_column`](#update_column) method to update them.
* Simple relationships between tables are auto-detected, but may be incomplete. Use [`add_relationship`](#add_relationship) to add them.
  {% endhint %}

## Inspection API

At any point, during the metadata creation or updates, you can inspect the current state of the metadata.

### to\_dict

Use this to get a copy of the Python dictionary that corresponds to the metadata.

**Parameters** (None)

**Output** A Python dictionary that corresponds to the metadata

```python
python_dict = metadata.to_dict()
```

{% hint style="info" %}
Note that the returned object is a representation of the metadata. Changing it will not modify the original metadata object in any way.
{% endhint %}

### visualize

Use this to this to see a visual representation of the metadata. Use the parameters to control the level of details in the visualization and for saving the image.

**Parameters** 

* `show_table_details`: Toggle the display of column details

<table data-header-hidden><thead><tr><th width="212"></th><th></th></tr></thead><tbody><tr><td>(default) <code>'full'</code></td><td>Show all the different column names, primary keys and foreign keys</td></tr><tr><td><code>'summarized'</code></td><td>Summarize the columns based on the data type</td></tr><tr><td><code>None</code></td><td><em>Hide the details. Only show the table name.</em></td></tr></tbody></table>

* `show_relationship_labels`: Toggle the display of the table relationships

<table data-header-hidden><thead><tr><th width="170"></th><th></th></tr></thead><tbody><tr><td>(default) <code>True</code></td><td>Label each relationship between 2 tables with the column names</td></tr><tr><td><code>False</code></td><td>Do not label the relationships. Only show an arrow between tables.</td></tr></tbody></table>

* `output_filepath`: If provided, save the image at the given location in the given format

{% hint style="warning" %}
The `output_filepath` must end with the filetype that you want to save as. Popular examples are `png`, `jpg` or `pdf`.
{% endhint %}

**Output** A [graphviz.graphs.Digraph](https://graphviz.readthedocs.io/en/stable/manual.html)

```python
metadata.visualize(
    show_table_details='full',
    show_relationship_labels=True,
    output_filepath='my_metadata.png'
)
```

<figure><img src="https://1967107441-files.gitbook.io/~/files/v0/b/gitbook-x-prod.appspot.com/o/spaces%2FfNxEeZzl9uFiJ4Zf4BRZ%2Fuploads%2FP08Tdn7GbwbDH4Jo6dD2%2FMultiTable%20Metadata%20Schema.png?alt=media&#x26;token=5a8a308c-3ee6-45c8-b4b4-e4058ecdd45b" alt=""><figcaption></figcaption></figure>

### get\_column\_names

Use this function to look up column names based on the metadata properties that they have.

{% hint style="info" %}
This is particularly useful if you want to list all columns assigned to a specific sdtype, such as `unknown` in order to update it.
{% endhint %}

**Parameters**

* (required) `table_name`: A string describing the name of the table to look into
* (required) `sdtype`: A string describing the statistical data type.\
  Common types are `'boolean'`, `'categorical'`, `'datetime'`, `'numerical'` and `'id'`. But other types such as `'phone_number'` are also available (see [SDTypes](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec/sdtypes)).
* `<other properties>`: Based on the sdtype, provide other parameters. For more information, see the [Metadata Spec](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec).

**Output** A list of strings, with the column names that match the criteria. If no columns match the criteria, then an empty string will be returned.

```python
metadata.get_column_names(table_name='products', sdtype='unknown')
```

```python
[ 'product_id', 'product_code_name', 'code_type']
```

## Validation API

### validate

Use this to validate that the metadata is written according to the specification. This function will throw descriptive errors if there is anything wrong with the metadata.

**Parameters** (None)

**Output** (None)&#x20;

```python
metadata.validate()
```

```
InvalidMetadataError: The metadata is not valid

Error: Invalid values ("pii") for datetime column "start_date".
Error: Invalid regex format string "[A-{6}" for id column "hotel_id"
```

### validate\_data

Use this method to validate that the metadata accurately describes a particular dataset. This function will throw descriptive errors if there is any mismatch between the metadata and data.

**Parameters:**

* (required) `data`: A dictionary containing your multi-table data. Each key should be the name of a table and the value should be a pandas.DataFrame containing its data. The data should have the same tables and columns as described in the metadata.

**Output** (None)

```python
metadata.validate_data(data={
    'hotels': hotels_dataframe,
    'guests': guests_dataframe
})
```

## Update API

It is important to verify and update any inaccuracies in the metadata

### update\_column

Use this method to modify the information about a column in your metadata

**Parameters**

* (required) `table_name`: The name of the table to update
* (required) `column_name`:  The name of the column to update
* (required) `sdtype`: A string describing the statistical data type.\
  Common types are `'boolean'`, `'categorical'`, `'datetime'`, `'numerical'` and `'id'`. But other types such as `'phone_number'` are also available. For more information, see [SDTypes docs](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec/sdtypes).
* `<other properties>`: Based on the sdtype, provide other parameters. For more information, see [SDTypes docs](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec/sdtypes).

**Output** (None)

```python
metadata.update_column(
    table_name='guests',
    column_name='start_date',
    sdtype='datetime',
    datetime_format='%Y-%m-%d')
    
metadata.update_column(
    table_name='guests'
    column_name='user_cell',
    sdtype='phone_number',
    pii=True)
```

### update\_columns

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the same parameters for a group of columns.

**Parameters**

* (required) `table_name`: A string with the name of the table
* (required) `column_names`: A list of strings representing the column names to update. All columns must be in the table.
* (required) `sdtype`: A string describing the statistical data type.\
  Common types are `'boolean'`, `'categorical'`, `'datetime'`, `'numerical'` and `'id'`. But other types such as `'phone_number'` are also available (see [SDTypes](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec/sdtypes)).
* `<other properties>`: Based on the sdtype, provide other parameters

**Output** (None)

```python
metadata.update_columns(
    table_name='users',
    column_names=['age', 'transactions', 'session_length'],
    sdtype='numerical',
    computer_representation='Float'
)
```

### update\_columns\_metadata

Use this function to make a bulk update to multiple columns at once. This function will allow you to set the different parameters for each column

**Parameters**

* (required) `table_name`: A string with the name of the table
* (required) `column_metadata`: A dictionary mapping each column name you want to update to the metadata information for that column. All columns must be in the table. For the exact format, see the [Metadata Spec](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec).

**Output** (None)

```python
metadata.update_columns_metadata(
    table_name='users',
    column_metadata={
        'age': { 'sdtype': 'numerical' },
        'ssn': { 'sdtype': 'ssn', 'pii': True },
        'gender': { 'sdtype': 'categorical' },
        'dob': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
        ...
    }
)
```

### add\_column

Use this function to add a column to your MultiTableMetadata object.

**Parameters**

* (required) `table_name`: Name of the table that contains the column
* (required) `column_name` : Name of the column to be added
* (required) `sdtype`: A string describing the statistical data type. Common types are `'boolean'`, `'categorical'`, `'datetime'`, `'numerical'` and `'id'`. Other types such as `'phone_number'` are also available (see [SDTypes](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec/sdtypes)).
* `**kwargs`: Any other parameters you need that describe metadata for a column.

```python
metadata.add_column(
  table_name='users',
  column_name='cell_phone_numbers',
  sdtype='phone_number',
  pii=True
)
```

### add\_column\_relationship

Use this function to specify when a group of columns within the same table represent the same concept.

{% hint style="info" %}
While anyone can add column relationships to their data, SDV Enterprise users will see the highest quality data for the relationships. To learn more about the SDV Enterprise and its extra features, [visit our website](https://datacebo.com/pricing/).
{% endhint %}

**Parameters**

* (required) `table_name`: The name of the table involved
* (required)  `relationship_type`: A string with the type of relationship. This represents a higher level concept. See the tabs below for options.
* (required) `column_names`: A list of column names that are part of that relationship. Make sure that these columns are compatible with the relationship type. See the tabs below for more information.

{% tabs %}
{% tab title="address" %}
An address is defined by 2 or more columns that have the following sdtypes: `country_code`, `administrative_unit`, `state`, `state_abbr`, `city`, `postcode`, `street_address` and `secondary_address`.

```python
metadata.add_column_relationship(
    table_name='users',
    relationship_type='address',
    column_names=['addr_line1', 'addr_line2', 'city', 'zipcode', 'state']
)
```

{% endtab %}

{% tab title="gps" %}
A GPS coordinate pair is defined by 2 columns:&#x20;

* sdtype `latitude` &
* sdtype `longitude`

```python
metadata.add_column_relationship(
    table_name='users',
    relationship_type='gps',
    column_names=['location_lat', 'location_lon']
)
```

{% endtab %}

{% tab title="More coming soon!" %}
Additional column relationships coming soon!

*Do you have a request for a type of column relationship? Please* [*file a feature request*](https://github.com/sdv-dev/SDV/issues/new/choose) *describing your use case.*
{% endtab %}
{% endtabs %}

**Output** (None)

### set\_primary\_key

Use this function to set the primary key of the table. Any existing primary keys will be removed.

{% hint style="info" %}
The primary key uniquely identifies every row in the table. When you set a primary key, the SDV will guarantee that every value in the table is unique. At this time, the SDV does not support composite keys.
{% endhint %}

**Parameters**

* (required) `table_name`: The name of the table
* (required) `column_name`: The column name of the primary key. The column name must already be defined in the metadata and it must be an ID or another PII sdtype.

**Output** (None)

```python
metadata.set_primary_key(
    table_name='hotels',
    column_name='hotel_id'
)

metadata.set_primary_key(
    table_name='guests',
    column_name='guest_email'
)
```

### remove\_primary\_key

Use this function to remove any existing primary keys in a table.

**Parameters**

* (required) `table_name`: The string name of the table that you'd like to remove the primary key for

**Output** (None) The primary key will be removed. Any existing relationships that use the primary key will be removed too.

```python
metadata.remove_primary_key(table_name='guests')
```

### add\_alternate\_keys

Use this function to set alternate keys of the table. This method will add to any existing alternate keys you may have.

{% hint style="info" %}
Similar to primary keys, alternate keys are also unique in your table. However, other tables do not reference alternate keys.
{% endhint %}

**Parameters**

* (required) `table_name`: The name of the table
* (required) `column_names`: A list of column names that represent the alternate keys in the table. All column names must already be defined in the metadata and they must be IDs or other PII sdtypes.

**Output** (None)

```python
metadata.add_alternate_keys(
    table_name='guests',
    column_names=['credit_card_number']
)
```

### add\_relationship

Use this method to add a relationship between 2 connected tables: A parent and child table. The parent table contains the primary key references while the child table has rows that refer to its parent. Multiple child rows can refer to the same parent row.

**Parameters:**

* (required) `parent_table_name`: The name of the parent table
* (required) `child_table_name`: The name of the child table that refers to the parent
* (required) `parent_primary_key`: The primary key column in the parent table. This column uniquely identifies each row in the parent table .
* (required) `child_foreign_key`: The foreign key column in the child table. The values in this column contain a reference to a row in the parent table

**Output** (None)

```python
metadata.add_relationship(
    parent_table_name='hotels',
    child_table_name='guests',
    parent_primary_key='hotel_id',
    child_foreign_key='hotel_id'
)
```

### remove\_relationship

Use this method to remove the connection between a parent and child table. In the case where there are multiple connections, this method will remove all the connections. Use this if the metadata has incorrectly detected relationships.

**Parameters:**

* (required) `parent_table_name`: The name of the parent table
* (required) `child_table_name`: The name of the child table that refers to the parent

**Output** (None)

```python
metadata.remove_relationship(
    parent_table_name='hotels',
    child_table_name='guests'
)
```

## Saving, Loading & Sharing Metadata

You can save the metadata object as a JSON file and load it again for future use.

### save\_to\_json

Use this to save the metadata object to a new JSON file that will be compatible with SDV 1.0 and beyond. We recommend you write the metadata to a new file every time you update it.

**Parameters**

* (required) `filepath`: The location of the file that will be created with the JSON metadata

**Output** (None)&#x20;

```python
metadata.save_to_json(filepath='metadata.json')
```

### Load previously saved metadata

If you already have a metadata JSON file, you can load it in as a `MultiTableMetadata` object. Use the method based on the version of your JSON file.

{% tabs %}
{% tab title="JSON file for SDV 1.0+" %}
**`load_from_json`**: If you recently wrote your JSON file for SDV, use this class method to load it as a `MultiTableMetadata` object.

```python
from sdv.metadata import MultiTableMetadata

metadata = MultiTableMetadata.load_from_json(
    filepath='metadata.json')
```

**Parameters**

* (required) `filepath`: The name of the file containing the JSON metadata

**Output** A `MultiTableMetadata` object
{% endtab %}

{% tab title="Older JSON files" %}
**`upgrade_metadata`**: If you wrote a JSON file for any SDV version before 1.0, use this class method to upgrade the metadata.

{% hint style="warning" %}
You have older metadata if you see a key named `"fields"` instead of `"columns"`.
{% endhint %}

```python
from sdv.metadata import MultiTableMetadata

metadata = MultiTableMetadata.upgrade_metadata(
    filepath='my_old_metadata.json'
)
```

**Parameters**

* (required) `old_filepath`: The filepath to your older metadata JSON file

**Output** A `MultiTableMetadata` object with the new metadata

{% hint style="info" %}
**Tip!** After upgrading, save your metadata so you can use it again

```python
metadata.save_to_json('my_new_metadata.json')
```

{% endhint %}
{% endtab %}
{% endtabs %}

### load\_from\_dict

Use this class method to load a Python dictionary as a `MultiTableMetadata` object.

#### Parameters

* (required) `metadata_dict`: A Python dictionary representation of the metadata. See [Metadata Spec](https://docs.sdv.dev/sdv/~/changes/T3ZD1DOoRUEqkmrAGBZp/reference/metadata-spec/multi-table-metadata-json) for more details.

**Output** A MultiTableMetadata object

```python
from sdv.metadata import MultiTableMetadata

metadata_obj = MultiTableMetadata.load_from_dict(metadata_dict)
```

### ＊ anonymize

Use this method to anonymize the column names of your metadata. This makes it easier to share your metadata, eg. for debugging purposes.

**Parameters** (None)

**Output** A new MultiTableMetadata object that represents the anonymized metadata

```python
anonymized_metadata = original_metadata.anonymize()
```

*＊This feature is only available for licensed, enterprise users. To learn more about the SDV Enterprise features and purchasing a license,* [*get in touch with us*](https://datacebo.com/contact/)*.*

{% hint style="info" %}
**The anonymized metadata contains new table and column names.** The original names are obfuscated, but the sdtypes and other formatting information remains the same.

```python
>>> anonymized_metadata.to_dict()
{
    'tables': {
        '3oc2d': {
            'primary_key': 'id_0',
            'columns': {
                'id_0': { 'sdtype': 'id', 'regex_format': 'ID_[0-9]{10}' },
                'num_0': { 'sdtype': 'numerical' },
                'cat_0': { 'sdtype': 'categorical' },
                'dt_0': { 'sdtype': 'datetime', 'datetime_format': '%Y-%m-%d' },
                'pii_0': { 'sdtype': 'ssn' },
                ...
        },
        'dpco1': {
            ...
        }
    },
    'relationships': [{
        'parent_table_name': '3oc2d',
        'child_table_name': 'dpco1',
        'parent_primary_key': 'id_0',
        'child_foreign_key': 'id_2'
    }]   
}
```

{% endhint %}