> For the complete documentation index, see [llms.txt](https://docs.sdv.dev/sdv/llms.txt). Markdown versions of documentation pages are available by appending `.md` to page URLs; this page is available as [Markdown](https://docs.sdv.dev/sdv/integration/metadata/sdtypes.md). # Sdtypes Let SDV know what type of data you have in each column of your table. In the SDV, the data types are specified by **sdtype**, denoting a semantic or statistical meaning. SDV is designed to create synthetic data differently based on each sdtype. {% hint style="info" %} An sdtype is a high level concept that tells SDV what your data *means*. An sdtype does not depend on how a computer stores the data. A single sdtype (such as `"categorical"`) can be stored by a computer in several ways (string, integer, etc). {% endhint %} ## Common Sdtypes Below are common sdtypes that describe columns in a dataset. ### Categorical Sdtype `categorical` describes columns that contain discrete, unordered categories. {% hint style="info" %} An example of categorical data is tax payer status such as `Single`, `Married filing jointly`, `Head of household`, etc. These discrete categories are unordered. {% endhint %} ```json { "tax_payer_status": { "sdtype": "categorical", "range_values": ["single", "married_filing_jointly", "married_filing_separately", "head_of_household", "surviving_spouse"], "range_is_nullable": false }, "tax_location_code": { "sdtype": "categorical", "high_cardinality": true } } ``` **Properties**: * `range_values`: A list of all possible non-null values that can appear in the categorical column. Because categorical data is unordered, this list can appear in any order. *This list may include category values that are not present in your training data.* * (default) If this is not provided, then SDV assumes that the full list of possible values is not known. It may look at your training data to determine the comprehensive list. * `high_cardinality`: A true/false value describing whether the categorical column has a large number of possible category values that cannot be fully and meaningfully listed in `range_values`. We recommend setting this when there are >500 unique values. If this is set to true, then the `range_values` should not be present. * `range_is_nullable`: Whether the missing values are allowed in the column. * (default) If this is not provided, then SDV looks at your training data as the main example that determines whether nulls should be allowed. ### Ordinal Sdtype `ordinal` describes columns that contain discrete elements with an inherent order. {% hint style="info" %} An example of ordinal data is survey responses such as `strongly_disagree`, `disagree`, etc., as there is an order between these values. Ordinal data might also be present as integers. For example, the responses could be coded instead as `1`, `2`, `3`, `4`, or `5`. {% endhint %} ```json { "likert_responses": { "sdtype": "ordinal", "range_values": ["strongly_disagree", "disagree", "neither_agree_nor_disagree", "agree", "strongly_agree"] } } ``` **Properties:** * `range_values`: A list of all possible non-null values that can appear in the ordinal column. Because ordinal data is ordered, this list should appear in order from lowest to highest. *This list may include ordinal values that are not present in your training data.* * (default) If this is not provided, then SDV assumes that the full list of possible values is not known. It may look at your training data to determine the comprehensive list. * `high_cardinality`: A true/false value describing whether the ordinal column has a large number of possible values that cannot be fully and meaningfully listed in `range_values`. We recommend setting this when there are >500 unique values. If this is set to true, then the `range_values` should not be present. * `range_is_nullable`: Whether the missing values are allowed in the column. * (default) If this is not provided, then SDV looks at your training data as the main example that determines whether nulls should be allowed. ### Numerical Sdtype `numerical` describes continuous data represented as numbers. You can apply a variety of mathematical computations to the values (average, sum, etc.) to numerical data. The actual values may follow a specific format, such as being rounded to 2 decimal digits and remaining between min/max bounds. {% hint style="info" %} Some data may appear numerical but actually represents distinct categories. For example, HTTP response codes such as `200`, `404`, etc. are categorical data. These numbers don't have a specific order, and they cannot be combined or averaged. {% endhint %} **Properties** * `range_min`: The absolute lowest value that is possible for this column. *This might be lower than what your training data contains.* * (default) If this is not provided, then SDV assumes that the lowest value is not known. It may look at your training data to determine the lowest value. * `range_max`: The absolute highest value that is possible for this column. *This might be higher than what your training data contains.* * (default) If this is not provided, then SDV assumes that the highest value is not known. It may look at your training data to determine the highest value. * `range_is_nullable`: Whether the missing values are allowed in the column. * (default) If this is not provided, then SDV looks at your training data as the main example that determines whether nulls should be allowed. * `decimal_places`: An integer representing the number of decimal places that the numerical values should be rounded to, for example `2` for 2 decimal digits. Note that a designation of `0` could mean either that there are whole numbers (integers) or that all values contain `.0` at the end such as `1.0`, `2.0`, etc. ```json { "age": { "sdtype": "numerical", "range_min": 18, "range_max": 120, "decimal_places": 0 }, "transaction_amt": { "sdtype": "numerical", "range_is_nullable": false, "decimal_places": 2 } } ``` ### Datetime Sdtype `datetime` describes columns that indicate a point of time. This can be at any granularity: to the nearest day, minute, second or even nanosecond. Typically, the datetime will be represented as a string. ```json { "start_date": { "sdtype": "datetime", "datetime_format": "%Y-%m-%d", "range_min": "1970-01-01", "range_max": "2050-12-31" } } ``` **Properties** * (required) `datetime_format`: A string describing the format as defined by [Python's strftime module](https://docs.python.org/3/library/datetime.html#strftime-and-strptime-format-codes). * `range_min`: The absolute earliest datetime that is possible for this column. *This might be earlier than what your training data contains.* * (default) If this is not provided, then SDV assumes that the earliest value is not known. It may look at your training data to determine the earliest value. * `range_max`: The absolute latest value that is possible for this column. *This might be later than what your training data contains.* * (default) If this is not provided, then SDV assumes that the latest value is not known. It may look at your training data to determine the latest value. * `range_is_nullable`: Whether the missing values are allowed in the column. * (default) If this is not provided, then SDV looks at your training data as the main example that determines whether nulls should be allowed. ### Boolean A `boolean` columns contains `True` or `False` values and may contain some missing data. ```json { "is_active": { "sdtype": "boolean", "range_is_nullable": false } } ``` **Properties**: * `range_is_nullable`: Whether the missing values are allowed in the column. * (default) If this is not provided, then SDV looks at your training data as the main example that determines whether nulls should be allowed. ### ID Sdtype `id` describes columns that are used to identify rows (eg. as a primary or foreign key). ID columns may not have a statistical or mathematical meaning behind them. Typically, an ID column follows a particular structure, for example being exactly 8 digits long with a `-` in the middle. **Properties** * `regex_format`: A string describing the format of the ID as a [regular expression](https://docs.python.org/3/library/re.html) ```json { "product_code": { "sdtype": "id", "regex_format": "[0-9]{4}-[0-9]{4}" } } ``` {% hint style="success" %} **What will the synthetic data contain?** Your synthetic data will contain brand new, randomly generated IDs based on the regex. If you have multiple tables, the primary and foreign key IDs will match up. {% endhint %} #### ＊ ID Regex Formats in SDV Enterprise SDV Enterprise is designed to automatically detect Regexes based on your data. You can view or update these Regexes. SDV Enterprise can also detect and accept Regexes that contain **embedded context**. {% hint style="info" %} **What is embedded context?** Many ID columns have completely random values that do not contain any statistical meaning. But other times, a portion of your ID column may have a meaning — it is not completely random. We call this portion of the ID column *embedded context.* For example consider an ID that starts with a 2-letter country code followed by a 5-character random value such as `US-3RO9P` or `CA-99QC4`. In this case, the first 2 letters form the *context* because there is a meaning behind them. SDV should not invent random countries. However, the final 5 characters are random without any statistical meaning. {% endhint %} SDV Enterprise detects embedded context using [*named capture groups*](https://docs.python.org/3/howto/regex.html#non-capturing-and-named-groups) within the Regex. Denoted by a parenthesis and the group name, named capture groups let the model know to learn the context exactly as-is and not invent new random values for it. In the example above, the Regex format would be: ``` (?P[A-Z]{2})_[0-9A-Z]{5} ``` The name of group (in this case `country`) is important. * If it represents and entirely new concept that is not found anywhere else in the data, then it should contain a brand new name that is not the same as any other column. * If the concept already exists as another column in the data, then it should be exactly the same as the column name. *SDV Enterprise is designed to auto-detect Regexes, including Regexes with context.* ## Additional Sdtypes: Domain-Specific Concepts & PII You may find that some of the columns in your dataset represent high-level, concepts in your domain. Such data might also contain sensitive, Personal Identifiable Information (PII). **Properties** * `pii`: A boolean denoting whether the data is sensitive * (default) `true`: The column is sensitive, meaning the values should be anonymized. If not provided, we assume that the column is PII. * `false`: The column is not sensitive, meaning the exact set of values can be reused in the synthetic data * `range_is_nullable`: Whether the missing values are allowed in the column. * (default) If this is not provided, then SDV looks at your training data as the main example that determines whether nulls should be allowed. ```json { "user_ssn": { "sdtype": "ssn", "range_is_nullable": true }, "user_city": { "sdtype": "city", "pii": false, "range_is_nullable": false } } ``` {% hint style="success" %} **Synthetic data goals:** Your synthetic data will contain entirely new values that do not necessarily appear in the original data. If you are using SDV Enterprise, the synthetic data may conform to some generic properties that are non-identifiable (eg. an area code of a phone number). {% endhint %} Browse below for some common sdtypes related to different concepts. {% tabs %} {% tab title="Personal Info" %} These sdtypes describe the information about a person.
＊phone_numberA local or international phone number such as '+1(555)123-4567'. Different countries have different formats.
＊emailA person's email such as 'first_last@gmail.com'
ssnA social security number such as 000-00-0000
first_nameA person's first name
last_nameA person's last name
*＊ Licensed, enterprise users will see higher quality data. The SDV will extract the deeper meaning and replicate it in the synthetic data. To learn more,* [*visit our website*](https://datacebo.com/pricing/)*.* {% endtab %} {% tab title="Location" %} These sdtypes describe a location around the world.
＊country_codeA 2-character country code such as 'US'
＊administrative_unitThe name of a region inside the country such as 'Massachusetts'. Countries call this concept different names such as state or province.
＊state_abbrFor countries that call their regions states, this refers to the 2-character code such as 'MA'
＊cityThe full name of the city such as 'Boston'
＊postcodeThe internationally-recognized, 5-digit postcode such as 02116
＊street_addressThe street and building number such as '123 Main St'. The exact format of this may vary by country
＊secondary_addressAdditional information about units in the building, such as 'Apartment #3'.
latitudeThe latitude of a location, expressed as a decimal
longitudeThe longitude of a location, expressed as a decimal
*＊ Licensed, enterprise users will see higher quality data. The SDV will extract the deeper meaning and replicate it in the synthetic data. To learn more,* [*visit our website*](https://datacebo.com/pricing/)*.* {% endtab %} {% tab title="Networking" %} These sdtypes describe information about computer networks and the internet.
ipv4_addressAn IP address, using the v4 protocol
ipv6_addressAn IP address, using the v6 protocol
mac_addressA media access control address
user_agent_stringA user agent string sent by the HTTP protocol
{% endtab %} {% tab title="Banking" %} These sdtypes describe information needed for banking functions such as payment transfers.
ibanAn international bank account number
swift11A SWIFT bank code that uses 11 digits
swift8A SWIFT bank code that uses 8 digits
credit_card_numberA credit card number, expressed using digits
{% endtab %} {% tab title="Automotive" %} These sdtypes describe concepts from the automotive industry.
vinA vehicle identification number
license_plateA license plate number, expressed using digits, letters or other characters. The format varies by country.
{% endtab %} {% tab title="Other" %} Many other sdtypes are possible. The SDV models can use the [Python Faker library](https://faker.readthedocs.io/en/master/providers.html) for new data types. You can input any of the function names as sdtypes. For example, inputting the sdtype `passport_number` will use [this function](https://faker.readthedocs.io/en/master/providers/faker.providers.passport.html#faker.providers.passport.Provider.passport_number) to generate meaningful numbers. For full SDV support, [file a request](https://github.com/sdv-dev/SDV/issues/new/choose) to help us prioritize other data types. {% endtab %} {% endtabs %} ### FAQs
How does the SDV factor in different countries or languages Many concepts vary based on the country and language. For example, phone numbers are represented differently in different countries. The SDV aims to provide worldwide support. You can specify the locales when you create a synthesizer. This lets the SDV know that any higher-level concepts should conform to the right formatting rules for that country. For example, assume you provide the following info: ```python from sdv.single_table import GaussianCopulaSynthesizer synthesizer = GaussianCopulaSynthesizer( metadata, locales=['en_US', 'nl_BE']) ``` Then for every concept described in your metadata, the SDV will generate values only from the US or Belgium in the appropriate language (English or Dutch).
Does the SDV understand the context of PII data? The public SDV randomly creates values corresponding to the concept, without taking additional context into account. Sometimes this may not be enough. For example, you may want to extract geographical areas from `phone_number` to ensure that it follows the same patterns. These features are available to licensed users. To learn more, [contact us](https://datacebo.com/contact/).
--- # Agent Instructions This documentation is published with GitBook. GitBook is the documentation platform designed so that both humans and AI agents can read, navigate, and reason over technical content effectively. Learn more at gitbook.com. ## Querying This Documentation If you need additional information that is not directly available in this page, you can query the documentation by asking a question. Perform an HTTP GET request on the following URL with the `ask` and `goal` query parameters: ``` GET https://docs.sdv.dev/sdv/integration/metadata/sdtypes.md?ask=&goal= ``` `ask` is the immediate question: it should be specific, self-contained, and written in natural language. `goal` is what the user is ultimately trying to achieve, the reason they need the answer. Sharing it helps GitBook give you a better, more relevant answer. A goal is most helpful when it describes the outcome the user wants rather than restating the question. For example, with `ask=how do I create an API token`, a goal like `automate deployments from our CI pipeline` lets GitBook tailor the answer to that use case. The response will contain a direct answer to the question and relevant excerpts and sources from the documentation. Use this mechanism when the answer is not explicitly present in the current page, you need clarification or additional context, or you want to retrieve related documentation sections.