-
Notifications
You must be signed in to change notification settings - Fork 42
Adding New Datasets
Data USA is built around four major types of data: geographies, occupations, industries and educational courses. In order to integrate new data with existing data in the platform, it is important that the data are linkable. Below are details on how each one of the four major data types are structured and how they should be structured for any new data source.
Geographic identifiers are special strings that denote different levels of the US geographic hierarchy. Here are the currently supported identifiers for geographies in Data USA:
| Description | Code Format | Notes |
|---|---|---|
| Nation (United States) | 01000US | |
| State | 04000USXX | Where XX is a 2-digit FIPS code |
| County | 05000USXXAAA | Where XX is a 2-digit FIPS state code and AAA is a 3-digit county code |
| Place | 1600USXXBBBBB | Where XX is a 2-digit FIPS state code and BBBBB is a 5-digit place code |
| Metropolitan Statistical Area | 3100USCCCCC | Where CCCCC is a 5-digit MSA code |
| Tract | 14000USXXAAADDDDDD | Where XX is a 2-digit FIPS state code and AAA is a 3-digit county code and DDDDD is a 5-digit tract code |
By convention, all geography columns should be text fields named geo.
Data USA is primarily built around the PUMS NAICS. For a full listing of all PUMS NAICS codes visit the attribute list at https://api.datausa.io/attrs/naics/. As a secondary option, data sets may also use BLS NAICS codes provided as an attribute list at https://api.datausa.io/attrs/bls_naics/.
For a dataset to work appropriately, it should either be completely contained by the list of PUMS NAICS or completely contained by the list of BLS NAICS codes (mixing the two lists is considered invalid). Every row of data in a new source should correspond to a valid NAICS code (either PUMS or BLS) to be considered valid data.
By convention, all NAICS industry columns should be text fields named naics.
Like the industry (NAICS) codes, in Data USA the occupational data is built primarily around the SOC PUMS standard. For a complete listing of valid PUMS codes visit the SOC attribute list https://api.datausa.io/attrs/soc/. Also available as a secondary option are the BLS style SOC codes from https://api.datausa.io/attrs/bls_soc/. Again, for a dataset to be considered valid, it should either be completely contained by the list of PUMS SOC or completely contained by the list of BLS SOC codes (mixing the two lists is not considered valid). Every row of data in a new source should contain a SOC code that is present in the attribute data.
By convention, all SOC occupation columns should be text fields named soc.
For data on educational majors, Data USA uses the 2010 CIP classification standard. A complete list of valid CIP codes and their descriptions may be found at https://api.datausa.io/attrs/cip/.
When including CIP data, ensure that each CIP code is found in the CIP attribute list.
All CIP fields should be text fields named cip.
Occasionally, when working with a new dataset data may be provided in a classification system that does not match the Geographic, Occupation, Industry or Educational Course classification systems used in Data USA. In these instances, the way to proceed is to provide an additional table that maps the new codes to into the existing code space. An important element of any new crosswalk table is that every existing attribute should map to at least one new attribute if possible as this will allow the new data to be pulled in on every existing profile page.
The crosswalk table should be of the format new_attr_name to existing_attr_name. For instance, if we were to crosswalk industrial codes to the ISIC classiciation we would create a crosswalk table that had two text columns: isic & naics. Note that attribute columns should always be represented as strings.