Project Meeting 2023.06.29

Agenda

Caitlin has set up an ActivitySim Google Drive. Joe to set up permissions and such to make it easy to access development and administrative stuff.

Pandera implementation
- Components/steps include input checker file; specify class for each input table; inherit the class; and then you can define checks, field, etc.
- It can integrate with the enums file, where you can specify enumerated variables that makes checking easy.
- You can specify custom checks.
- It is very simple to call.
- The down side is that it’s not a true “data model." It does not explicitly embed the relationships (you have to join files).
Pydantic
- Explicitly embedded relationships
- Operates on “serialize” data
- Similar to pandera but a little more overhead because of this serialize step
- Not the most computationally efficient but it is useful for understanding and building a tool based on relationships.
Runtime for MTC is ~30 minutes with pydantic approach, versus pandera which is ~30 seconds. You could implement a pydantic approach that doesn’t serialize. There may be something incorrect with the implementation, exceptations for runtime are about 2 to 5 minutes. RSG can look into optimizations and/or Pydantic 2.0.
Anticipating validators on zone files and other inputs, they just weren't presented today.
Path forward
- Retain and improve on Pandera approach
- Leave Pydantic example available for those that want to use it or if we move to that in the future