Skip to content

Commit

Permalink
syntax fix for pandoc with latex
Browse files Browse the repository at this point in the history
  • Loading branch information
k-wolski committed Oct 7, 2024
1 parent ad35e75 commit ccea421
Showing 1 changed file with 9 additions and 5 deletions.
14 changes: 9 additions & 5 deletions docs/data_platform/lifecycle.md
Original file line number Diff line number Diff line change
Expand Up @@ -9,25 +9,28 @@ The data engineering lifecycle encompasses the various stages of designing, buil

The data engineering lifecycle is supported by several critical undercurrents, including security, data governance, DataOps, data architecture, orchestration, and software engineering practices. While these foundational concepts are integral to the success and scalability of data engineering workflows, they will not be covered in detail in this document. However, they remain crucial to our overall approach and should be considered throughout every stage of the lifecycle.

**Diagram Placeholder: Data Engineering Lifecycle to be replaced**

![data-engineering-lifecycle](../assets/images/data-platform_data-engineering-lifecycle.png)
![Data Engineering Lifecycle](../assets/images/data-platform_data-engineering-lifecycle.png)

---

### 1. **Data Generation**

Data generation is the first step in the lifecycle, where raw data is created or collected from various sources. These sources can include user activity on applications, sensor data from IoT devices, logs from systems, and external APIs. The goal of this phase is to collect raw, unprocessed data that will be used in downstream processes.

```{=latex}
\newpage
```
#### Key Activities:
- **Event Logging**: Capturing user or system events.
- **IoT and Sensor Data**: Collecting data from physical devices.
- **API Requests**: Pulling data from external sources.
- **Manual Input**: Data entered by users or operators.

**Diagram Placeholder: Data Generation Process**
![data generation - application database](../assets/images/data-platform_data-generation-application-database.png)
![data generation - IoT swarm and message queue](../assets/images/data-platform_data-generation-iot-swarm-and-message-queue.png)

![Data Generation - Application Database](../assets/images/data-platform_data-generation-application-database.png)

![Data Generation - IoT Swarm and Message Queue](../assets/images/data-platform_data-generation-iot-swarm-and-message-queue.png)

---

Expand Down Expand Up @@ -112,6 +115,7 @@ Reverse ETL is the process of moving data from your centralized data warehouse o
- **Data Enrichment**: Enhancing third-party tools with enriched or aggregated data from internal systems.

**Diagram Placeholder: Reverse ETL Process**

![Reverse ETL](../assets/images/data-platform_reverse-etl.png)

---
Expand Down

0 comments on commit ccea421

Please sign in to comment.