Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CDC_Environmental_Health_Toxicology_Refresh #1097

Open
wants to merge 13 commits into
base: master
Choose a base branch
from
Open

CDC_Environmental_Health_Toxicology_Refresh #1097

Show file tree
Hide file tree
Changes from 12 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
ee25ba6
fixed process script
SudhishaK Oct 25, 2024
537b71b
fixed process script
SudhishaK Oct 25, 2024
c1ec163
fixed process script
SudhishaK Oct 25, 2024
1fe3dd1
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Nov 12, 2024
fc1e5fc
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Dec 2, 2024
4a477ea
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Dec 30, 2024
aab107f
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Dec 30, 2024
39aa35c
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Dec 30, 2024
27721f0
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Jan 2, 2025
4fdb106
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Jan 2, 2025
c8318b7
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Jan 2, 2025
e4f26ad
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Jan 2, 2025
a77f179
CDC_Environmental_Health_Toxicology_Refresh
SudhishaK Jan 4, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion scripts/us_cdc/environmental_health_toxicology/OzoneCensusTractPollution.tmcf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ typeOf: dcs:StatVarObservation
observationDate: C:OzoneCTPollution->date
variableMeasured: dcs:Mean_Concentration_AirPollutant_Ozone
observationPeriod: "P8H"
unit: parts per billion (ppb)
unit: PartsPerBillion
value: C:OzoneCTPollution->Value

Node: E:OzoneCTPollution->E2
SudhishaK marked this conversation as resolved.
Show resolved Hide resolved
Expand Down
16 changes: 8 additions & 8 deletions scripts/us_cdc/environmental_health_toxicology/OzoneCountyPollution.tmcf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,34 +6,34 @@ Node: E:OzoneCountyPollution->E1
observationAbout: C:OzoneCountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:OzoneCountyPollution->date
value: C:OzoneCountyPollution->O3_mean_pred
value: C:OzoneCountyPollution->o3_mean_pred
observationPeriod: "P8H"
unit: parts per billion (ppb)
unit: PartsPerBillion
variableMeasured: dcs:Mean_Concentration_AirPollutant_Ozone

Node: E:OzoneCountyPollution->E2
observationAbout: C:OzoneCountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:OzoneCountyPollution->date
value: C:OzoneCountyPollution->O3_med_pred
value: C:OzoneCountyPollution->o3_med_pred
observationPeriod: "P8H"
unit: parts per billion (ppb)
unit: PartsPerBillion
variableMeasured: dcs:Median_Concentration_AirPollutant_Ozone

Node: E:OzoneCountyPollution->E3
observationAbout: C:OzoneCountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:OzoneCountyPollution->date
value: C:OzoneCountyPollution->O3_max_pred
value: C:OzoneCountyPollution->o3_max_pred
observationPeriod: "P8H"
unit: parts per billion (ppb)
unit: PartsPerBillion
variableMeasured: dcs:Max_Concentration_AirPollutant_Ozone

Node: E:OzoneCountyPollution->E4
observationAbout: C:OzoneCountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:OzoneCountyPollution->date
value: C:OzoneCountyPollution->O3_pop_pred
value: C:OzoneCountyPollution->o3_pop_pred
observationPeriod: "P8H"
unit: parts per billion (ppb)
unit: PartsPerBillion
variableMeasured: dcs:PopulationWeighted_Concentration_AirPollutant_Ozone
2 changes: 1 addition & 1 deletion scripts/us_cdc/environmental_health_toxicology/PM25CensusTractPollution.tmcf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ typeOf: dcs:StatVarObservation
observationDate: C:PM25CTPollution->date
variableMeasured: dcs:Mean_Concentration_AirPollutant_PM2.5
observationPeriod: "P24H"
unit: μg/m3
unit: dcs:MicrogramsPerCubicMeter
value: C:PM25CTPollution->Value

Node: E:PM25CTPollution->E2
Expand Down
16 changes: 8 additions & 8 deletions scripts/us_cdc/environmental_health_toxicology/PM25CountyPollution.tmcf
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,34 +2,34 @@ Node: E:PM25CountyPollution->E1
observationAbout: C:PM25CountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:PM25CountyPollution->date
value: C:PM25CountyPollution->PM25_mean_pred
value: C:PM25CountyPollution->pm25_mean_pred
observationPeriod: "P24H"
unit: μg/m3
unit: MicrogramsPerCubicMeter
variableMeasured: dcs:Mean_Concentration_AirPollutant_PM2.5

Node: E:PM25CountyPollution->E2
observationAbout: C:PM25CountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:PPM25CountyPollution->date
value: C:PM25CountyPollution->PM25_med_pred
value: C:PM25CountyPollution->pm25_med_pred
observationPeriod: "P24H"
unit: μg/m3
unit: MicrogramsPerCubicMeter
variableMeasured: dcs:Median_Concentration_AirPollutant_PM2.5

Node: E:PM25CountyPollution->E3
observationAbout: C:PM25CountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:PM25CountyPollution->date
value: C:PM25CountyPollution->PM25_max_pred
value: C:PM25CountyPollution->pm25_max_pred
observationPeriod: "P24H"
unit: μg/m3
unit: MicrogramsPerCubicMeter
variableMeasured: dcs:Max_Concentration_AirPollutant_PM2.5

Node: E:PM25CountyPollution->E4
observationAbout: C:PM25CountyPollution->dcid
typeOf: dcs:StatVarObservation
observationDate: C:PM25CountyPollution->date
value: C:PM25CountyPollution->PM25_pop_pred
value: C:PM25CountyPollution->pm25_pop_pred
observationPeriod: "P24H"
unit: μg/m3
unit: MicrogramsPerCubicMeter
variableMeasured: dcs:PopulationWeighted_Concentration_AirPollutant_PM2.5
58 changes: 55 additions & 3 deletions scripts/us_cdc/environmental_health_toxicology/README.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -109,7 +109,7 @@ The expected output of this test can be found in [`small_Palmer_expected.csv`](h

#### Processing Steps

`@input_file_name` - path to the input csv file to be cleaned
<!-- `@input_file_name` - path to the input csv file to be cleaned

`@output_file_name` - path to write the cleaned csv file

Expand All @@ -122,5 +122,57 @@ $ python3 parse_air_quality.py input_file_name output_file_name
To clean the precipitation index data files, run:

```bash
$ python3 parse_precipitation_index.py input_file_name output_file_name
```
$ python3 parse_precipitation_index.py input_file_name output_file_name -->

1. Import name: CDC_PM25CensusTract

command to download the file
============================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_PM25CensusTract --mode=download


Command to process the file
===========================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_PM25CensusTract --mode=process


2. Import name: CDC_OzoneCensusTract

command to download the file
============================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_OzoneCensusTract --mode=download


Command to process the file
===========================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_OzoneCensusTract --mode=process

3. Import name: CDC_PM25County

command to download the file
============================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_PM25County --mode=download


Command to process the file
===========================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_PM25County --mode=process

4. Import name: CDC_OzoneCounty

command to download the file
============================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_OzoneCounty --mode=download


Command to process the file
===========================

$ python3 scripts/us_cdc/environmental_health_toxicology/parse_air_quality.py CDC_OzoneCounty --mode=process
104 changes: 104 additions & 0 deletions scripts/us_cdc/environmental_health_toxicology/manifest.json
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
{
"import_specifications": [
{
"import_name": "CDC_PM25CensusTract",
"curator_emails": [
"[email protected]"
],
"provenance_url": "https://data.cdc.gov/browse?category=Environmental+Health+%26+Toxicology",
"provenance_description": "Environmental Health Toxicology",
"scripts": [
"parse_air_quality.py CDC_PM25CensusTract"
],
"import_inputs": [
{
"template_mcf": "PM25CensusTractPollution.tmcf",
"cleaned_csv": "output/PM2.5CensusTract_0.csv"
},
{
"template_mcf": "PM25CensusTractPollution.tmcf",
"cleaned_csv": "output/PM2.5CensusTract_1.csv"
},
{
"template_mcf": "PM25CensusTractPollution.tmcf",
"cleaned_csv": "output/PM2.5CensusTract_2.csv"
},
{
"template_mcf": "PM25CensusTractPollution.tmcf",
"cleaned_csv": "output/PM2.5CensusTract_3.csv"
}

],
"cron_schedule": "0 1 2 * *"
},
{
"import_name": "CDC_OzoneCensusTract",
"curator_emails": [
"[email protected]"
],
"provenance_url": "https://data.cdc.gov/browse?category=Environmental+Health+%26+Toxicology",
"provenance_description": "Environmental Health Toxicology",
"scripts": [
"parse_air_quality.py CDC_OzoneCensusTract"
],
"import_inputs": [
{
"template_mcf": "OzoneCensusTractPollution.tmcf",
"cleaned_csv": "output/Census_Tract_Level_Ozone_Concentrations_0.csv"
},
{
"template_mcf": "OzoneCensusTractPollution.tmcf",
"cleaned_csv": "output/Census_Tract_Level_Ozone_Concentrations_1.csv"
},
{
"template_mcf": "OzoneCensusTractPollution.tmcf",
"cleaned_csv": "output/Census_Tract_Level_Ozone_Concentrations_2.csv"
},
{
"template_mcf": "OzoneCensusTractPollution.tmcf",
"cleaned_csv": "output/Census_Tract_Level_Ozone_Concentrations_3.csv"
}

],
"cron_schedule": "0 1 3 * *"
},
{
"import_name": "CDC_PM25County",
"curator_emails": [
"[email protected]"
],
"provenance_url": "https://data.cdc.gov/browse?category=Environmental+Health+%26+Toxicology",
"provenance_description": "Environmental Health Toxicology",
"scripts": [
"parse_air_quality.py CDC_PM25County"
],
"import_inputs": [
{
"template_mcf": "PM25CountyPollution.tmcf",
"cleaned_csv": "output/PM25county.csv"
}

],
"cron_schedule": "0 1 4 * *"
},
{
"import_name": "CDC_OzoneCounty",
"curator_emails": [
"[email protected]"
],
"provenance_url": "https://data.cdc.gov/browse?category=Environmental+Health+%26+Toxicology",
"provenance_description": "Environmental Health Toxicology",
"scripts": [
"parse_air_quality.py CDC_OzoneCounty"
],
"import_inputs": [
{
"template_mcf": "OzoneCountyPollution.tmcf",
"cleaned_csv": "output/OzoneCounty.csv"
}

],
"cron_schedule": "0 1 5 * *"
}
]
}
Loading
Loading