> For the complete documentation index, see [llms.txt](https://docs.sdv.dev/sdmetrics/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.sdv.dev/sdmetrics/getting-started/metadata.md).
# Metadata
Metadata is a description of your dataset. It helps the library understand the type and format of your data, so that it can apply the correct metrics.
{% hint style="danger" %}
If the metadata is incorrect, this library may apply the wrong metrics to your data which leads to inaccurate scores.
{% endhint %}
## What does metadata include?
Your metadata includes:
* The type of data that each column represents
* The primary keys and other identifiers of the table
* The relationships between the tables, if you have multiple tables
{% hint style="success" %}
Write your metadata once and use it anywhere in the SDMetrics library.
If you used the [SDV library](https://docs.sdv.dev/sdv/integration/metadata) to create your metadata, you can reuse the same file for SDMetrics.
{% endhint %}
For example, your data may be present in multiple tables with rows and columns. The tables may be connected to each other through primary and foreign key references.
Your data description is called metadata. SDMetrics expects metadata as a **Python dictionary** object.
Click to see the metadata
This is the metadata dictionary for the illustrated table
```python
{
"tables": {
"users": {
"primary_key": "user_id",
"columns": {
"user_id": {
"sdtype": "id",
"regex_format": "U_[0-9]{3}"
},
"age": {
"sdtype": "numerical"
},
"address": {
"sdtype": "address",
"pii": True
}
}
},
"sessions": {
"primary_key": "session_id",
"columns": {
"session_id": {
"sdtype": "id"
},
"user": {
"sdtype": "id",
"regex_format": "U_[0-9]{3}"
},
"date": {
"sdtype": "datetime",
"datetime_format": "%Y-%m-%d"
},
"browser": {
"sdtype": "categorical"
},
"bounced": {
"sdtype": "boolean"
}
}
}
},
"relationships": [{
"parent_table_name": "users",
"parent_primary_key": "user_id",
"child_table_name": "sessions",
"child_foreign_key": "user_id"
]}
}
```
## Metadata Specification
The file is an object that includes a dictionary named `"tables"`.
```python
{
"tables": {
},
}
```
### Tables
The `"tables"` dictionary contains the information about each individual table of your application. Its keys are the table names and the values are dictionaries that describe each single table. This includes:
* `"primary_key"`: the column name used to identify a row in your table. *For a composite key, provide a list of column names.*
* (required) `"columns"`: a dictionary description of each column
```python
{
"tables": {
"users": {
"primary_key": "user_id",
"columns": { }
},
"sessions": {
"primary_key": "session_id",
"columns": { }
}
},
...
}
```
#### Column Information
Inside `"columns"`, you will describe each column. You'll start with the name of the column. Then you'll specify the type of data and any other information about it.
There are specific data types to choose from. Expand the options below to learn about the data types.
{% tabs %}
{% tab title="boolean" %}
Boolean columns represent True or False values.
```python
"active": {
"sdtype": "boolean"
}
```
**Properties** (None)
{% endtab %}
{% tab title="categorical" %}
Categorical columns describe discrete data.
```python
"tier": {
"sdtype": "categorical",
}
```
**Properties** (None)
{% endtab %}
{% tab title="datetime" %}
Date columns represent a point in time
```python
"renew_date": {
"sdtype": "datetime",
"format": "%Y-%m-%d"
}
```
**Properties**
* (required) `datime_format`: A string describing the format as defined by [Python's strftime module](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes).
{% hint style="info" %}
The format string has special values to describe the components. For example, `Jan 06, 2022` is represented as `"%b %d, %Y".` Common values are:
* **Year**: `"%Y"` for a 4-digit year like 2022, or `"%y"` for a 2-digit year like 22
* **Month**: `"%m"` for a 2-digit month like 01, `"%b"` for an abbreviated month like Jan
* **Day**: `"%d"` for a 2-digit day like 06
{% endhint %}
{% endtab %}
{% tab title="numerical" %}
Numerical columns represents discrete or continuous numerical values.
```python
"age": {
"sdtype": "numerical"
},
"paid_amt": {
"sdtype": "numerical",
"compute_representation": "Float"
}
```
**Properties**
* `computer_representation`: A string that represents how you'll ultimately store the data. This determines the min and max values allowed\
Available options are: `'Float'`, `'Int8'`, `'Int16'`, `'Int32'`, `'Int64'`, `'UInt8'`, `'UInt16'`, `'UInt32'`, `'UInt64'`
{% endtab %}
{% tab title="id" %}
ID columns represent identifiers that do not have any special mathematical or semantic meaning
```python
"user_id": {
"sdtype": "id",
"regex_format": "U_[0-9]{3}"
}
```
**Properties**
* `regex_format`: A string describing the format of the ID as a [regular expression](https://docs.python.org/3/library/re.html)
{% endtab %}
{% tab title="other" %}
You can input any other data type such as `'phone_number'`, `'ssn'` or `'email'`. See the [Sdtypes Reference](https://docs.sdv.dev/sdv/reference/metadata-spec/sdtypes) for a full list.
```python
"address": {
"sdtype": "address",
"pii": True
}
```
**Properties**
* `pii`: A boolean denoting whether the data is sensitive
* (default) `True`: The column is sensitive, meaning the synthetic data is anonymized
* `False`: The column is not sensitive, meaning the synthetic data may not be anonymized
{% endtab %}
{% endtabs %}
### Relationships
Inside the `"relationships"` section of the metadata, provide a list of relationships that exist between tables. Each relationship is represented as a dictionary with the following keys:
* `"parent_table_name"`: The name of the parent table
* `"parent_primary_key"`: The primary key column in the parent table. This column uniquely identifies each row in the parent table. *For a composite key, provide a list of column names.*
* `"child_table_name"`: The name of the child table that refers to the parent
* `"child_foreign_key"`: The foreign key column in the child table. The values in this column contain a reference to a row in the parent table. *For a composite key, provide a list of column names.*
Use multiple dictionaries to represent multiple tables.
```python
"relationships": [{
"parent_table_name": "users",
"parent_primary_key": "user_id",
"child_table_name": "sessions",
"child_foreign_key": "user_id"
]}
```
## Saving & Loading Metadata
After creating your dictionary, you can save it as a JSON file. For example, `my_metadata_file.json`.
```python
import json
with open('my_metadata_file.json', 'w') as f:
json.dump(my_metadata_dict, f)
```
In the future, you can load the Python dictionary by reading from the file.
```python
import json
with open('my_metadata_file.json') as f:
my_metadata_dict = json.load(f)
# use my_metadata_dict in the SDMetrics library
```
## Adding Multi-Sequence Information
In some cases, your data table may contain multiple, independent sequences belonging to different entities. See the diagram below for an illustration of sequential data.
In this case, you can add some information about the sequential nature of this data to your metadata specification. In the dictionary for the table, add:
* `"sequence_key"`: the name of a column that identifies each unique sequence in your data
* `"sequence_index"`: the column name used to order the rows in the table
An example for the table is provided below.
Click to see the sequential table's metadata
This is the metadata dictionary for the illustrated sequential table
```python
{
"tables": {
"patients": {
"sequence_key": "Patient ID",
"sequence_index": "Time",
"columns": {
"Patient ID": {
"sdtype": "id",
"regex_format": "ID_[0-9]{3}"
},
"Address": {
"sdtype": "address",
"pii": True
},
"Smoker": {
"sdtype": "boolean"
},
"Time": {
"sdtype": "datetime",
"datetime_format": "%m/%d/%Y"
},
"Heart Rate": {
"sdtype": "categorical"
},
"Systolic BP": {
"sdtype": "numerical"
}
}
}
}
}
```
{% hint style="warning" %}
At this time, SDMetrics offers limited evaluation support for data involving sequences. You may run the Diagnostic and Quality reports with your data, but these reports do not factor in any order or sequence-related measures.
In addition to the report, you can run individual metrics that are suited for sequential data. See:
* [SequenceLengthSimilarity](/sdmetrics/data-metrics/quality/sequencelengthsimilarity.md)
* [StatisticMSAS](/sdmetrics/data-metrics/quality/statisticmsas.md)
* Metrics in Beta: [Sequential Detection](/sdmetrics/data-metrics/metrics-in-beta/detection-sequential.md), [Sequential ML Efficacy](/sdmetrics/data-metrics/metrics-in-beta/ml-efficacy-sequential.md)
{% endhint %}
---
# Agent Instructions
This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com.
## Querying This Documentation
If you need additional information that is not directly available in this page, you can query the documentation dynamically by asking a question.
Perform an HTTP GET request on the current page URL with the `ask` query parameter, and the optional `goal` query parameter:
```
GET https://docs.sdv.dev/sdmetrics/getting-started/metadata.md?ask=&goal=
```
`ask` is the immediate question: it should be specific, self-contained, and written in natural language.
`goal` is optional and describes the broader end goal you are ultimately trying to accomplish on behalf of the user. GitBook uses it to tailor the answer towards what is most useful for that goal.
The response will contain a direct answer to the question and relevant excerpts and sources from the documentation.
Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.