Skip to content

Latest commit

 

History

History
178 lines (117 loc) · 5.35 KB

README.md

File metadata and controls

178 lines (117 loc) · 5.35 KB
Raw

Introduction

Perseus combines intuitive and easy to use Web-based UI for design and implement ETL (extract, transform, and load) configuration and service for conversion the native/raw data to the OMOP Common Data Model (CDM).

Additionally, Perseus has embedded tools for search in the standardized vocabularies, generates documentation for the ETL process, create the code mappings and data quality check.

Wiki

Getting started

Contact Us: [email protected]

Features

  • Map source data to tables and columns to CDM
  • Combine source tables
  • Use pre-built sql functions (replace, concat…)
  • Use pre-built source to source and source to standard vocabulary lookups (icd9, icd10, ndc…)
  • Create custom lookups for the concept_id fields
  • Set constant values to the CDM fields
  • Use System/Auto generated values for the CDM fields
  • Auto mapping for similar fields
  • OMOP Vocabulary search
  • Data Quality check
  • Search mapping between new codes and OMOP standard concepts
  • Convert data from native format to CDM
  • Logic for creating eras (DRUG_ERAs, CONDITION_ERAs…)
  • Logic for grouping visit occurrence/observation_period records
  • Auto domain switching
  • Create ETL specification

Screenshots

Start page

Link tables

Link fields

Concept configuration

Lookup configuration

Code mappings - Import

Code mappings

Technology

  • Angular 12
  • Python 3.7
  • Java 17
  • R 4.1.3
  • PostgreSQL 13.2
  • .NET Core 3.1

Deployment server requirements

  • Unix / Windows OS, Docker,
  • 4GB RAM,
  • ~10 GB HDD (Depend on Vocabulary size),
  • Open ports: 443, 80.

Getting Started

Docker Compose and Podman

Install docker compose as a plugin

apt install docker-compose-plugin

You can use Podman instead of Docker

apt install python3-pip
pip install podman-compose

If you have an issue with downloading from Docker Hub check configuration file:

/etc/containers/registries.conf

It need to contain the following string: unqualified-search-registries = ["docker.io"]

podman-compose pull
podman-compose up -d

Vocabulary

Get the link to the vocabulary from Athena.

cd vocabulary-db

Install vocabulary archive and extract to vocabulary directory. Full path vocabulary-db/vocabulary. For example:

unzip <downloaded archive> -d vocabulary

Database deployment can take a long time if the dictionary size is large enough.

To Docker Compose.

SMTP server

Multi-user

(Optional)

cd user
  • To get user registration links by e-mail you should configure SMTP server settings first. Edit file named user-envs.txt in the user directory with the following content (without spaces):

SMTP_SERVER=<your SMTP server host address>
SMTP_PORT=<your SMTP port>
SMTP_EMAIL=<email from which registration links will be sent to users>
SMTP_USER=<SMTP login>
SMTP_PWD=<SMPT password>
TOKEN_SECRET_KEY=<token encoding key>

to Docker Compose

Test user

Single-user

If you want to skip multi-user mode use user with these credential:

Email:

[email protected]

Password:

perseus

Starting with Docker Compose

To start all containers at once using docker-compose please

  • make sure docker-compose is installed
  • set vocabulary link, see Vocabulary section
  • configure SMTP server as it described in SMTP section (Optional)

Unix:

./startup.sh

Windows:

startup.cmd

Open localhost:80 in your browser, preferably Google Chrome

Starting each container separately

CONTAINERS

Perseus uses auxiliary services to scan, convert and validate data.

Below are links to these services, which should be included in the app build.

White-rabbit service

https://github.com/SoftwareCountry/WhiteRabbit

Cdm-builder service

https://github.com/SoftwareCountry/ETL-CDMBuilder

Data-quality-check service

https://github.com/SoftwareCountry/DataQualityDashboard

Finally

Open localhost:80 in your browser, preferably Google Chrome.

Getting Involved

License

Perseus is licensed under Apache License 2.0