Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add in remove persisted data command #43

Merged
merged 2 commits into from
Jun 20, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 30 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
# insta-infra

Spin up any tool or service straight away on your local laptop. Tells you how to connect to it.
Spin up any service straight away on your local laptop. Tells you how to connect to it.

- Simple commands
- Add custom data (i.e startup SQL scripts)
Expand Down Expand Up @@ -49,6 +49,14 @@ mysql mysql:3306 localhost:3306 host.docker.internal:3306
./run.sh list
```

### Remove persisted data

```shell
./run.sh [remove|-r] <services>
./run.sh -r #remove all service persisted data
./run.sh remove postgres
```

### Run from anywhere

In your `.bashrc, .zshrc, ...`, add:
Expand All @@ -64,18 +72,39 @@ insta -l
insta postgres
insta -c postgres
insta -d
insta -r postgres
```

### Custom data

Alter data in [`data`](data) folder.

You may notice that for some services (such as Cassandra, Postgres, MySQL), they follow the same pattern for custom
data. They have a `sql` directory which contains data files with DDL statements and an `init.sh` script that will help
execute them at startup. This allows you to dump all your `.sql` files into the directory, and it will be automatically
run at startup.


### Persisted data

If any data is persisted from the services to carry across sessions, it gets pushed to folder:

`./data/<service>/persist`

### Authentication

By default, users and passwords follow what is default in the service. For those services where the user and password
can be altered at startup, it can be altered using environment variable pattern:
```shell
<service>_USER=...
<service>_PASSWORD=...
```

For example:
```shell
POSTGRES_USER=my-user POSTGRES_PASSWORD=my-password ./run.sh postgres
```

## Services

| Service Type | Service | Supported |
Expand Down
File renamed without changes.
16 changes: 16 additions & 0 deletions data/cassandra/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash

count=0
total=0

for f in $(ls /tmp/data/*.cql);
do
total=$((total + 1))
cqlsh -u "${CASSANDRA_USER:-cassandra}" -p "${CASSANDRA_PASSWORD:-cassandra}" -f "${f}" cassandra 9042
if [[ $? -eq 0 ]];
then
count=$((count + 1))
fi;
done;

echo "Executed ${count} out of ${total} files"

This file was deleted.

15 changes: 15 additions & 0 deletions data/clickhouse/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
#!/usr/bin/env bash

count=0
total=0

for f in $(ls /tmp/data/*.sql);
do
clickhouse client --host clickhouse-server --multiquery < "$f"
if [[ $? -eq 0 ]];
then
count=$((count + 1))
fi;
done;

echo "Executed ${count} out of ${total} files"
5 changes: 5 additions & 0 deletions data/clickhouse/sql/create_postgres_connection.sql
Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
SET allow_experimental_database_materialized_postgresql=1;

SET allow_experimental_materialized_postgresql_table=1;

CREATE DATABASE customer_postgres_db ENGINE = PostgreSQL('postgres:5432', 'customer', 'postgres', 'postgres', 'account', 0);
2 changes: 0 additions & 2 deletions data/druid/environment
Original file line number Diff line number Diff line change
Expand Up @@ -34,8 +34,6 @@ druid_zk_service_host=zookeeper
druid_metadata_storage_host=
druid_metadata_storage_type=postgresql
druid_metadata_storage_connector_connectURI=jdbc:postgresql://postgres:5432/druid
druid_metadata_storage_connector_user=postgres
druid_metadata_storage_connector_password=postgres

druid_coordinator_balancer_strategy=cachingCost

Expand Down
18 changes: 18 additions & 0 deletions data/kafka/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,18 @@
#!/usr/bin/env bash

host="kafka"
# blocks until kafka is reachable
kafka-topics --bootstrap-server ${host}:29092 --list

if [[ -z "$KAFKA_TOPICS" ]]; then
echo "No Kafka topics provided via KAFKA_TOPICS environment variable"
exit 0
fi

echo "Creating kafka topics"
for t in $(echo "$KAFKA_TOPICS" | sed s/,/\\n/g); do
kafka-topics --create --if-not-exists --topic "${t}" --bootstrap-server ${host}:9092 --replication-factor 1 --partitions 1
done

echo "Successfully created the following topics:"
kafka-topics --bootstrap-server ${host}:29092 --list
11 changes: 0 additions & 11 deletions data/kafka/my_data.sh

This file was deleted.

16 changes: 16 additions & 0 deletions data/mysql/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash

count=0
total=0

for f in $(ls /tmp/data/*.sql);
do
total=$((total + 1))
mysql -u root -p"${MYSQL_PASSWORD:-root}" -h "mysql" < "${f}"
if [[ $? -eq 0 ]];
then
count=$((count + 1))
fi;
done;

echo "Executed ${count} out of ${total} files"
2 changes: 1 addition & 1 deletion data/mysql/my_data.sql → data/mysql/sql/my_data.sql
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
CREATE DATABASE customer;
CREATE DATABASE IF NOT EXISTS customer;

CREATE TABLE IF NOT EXISTS customer.accounts
(
Expand Down
16 changes: 16 additions & 0 deletions data/postgres/init.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
#!/usr/bin/env bash

count=0
total=0

for f in $(ls /tmp/data/*.sql);
do
total=$((total + 1))
psql -U${POSTGRES_USER:-postgres} -hpostgres -f "${f}"
if [[ $? -eq 0 ]];
then
count=$((count + 1))
fi;
done;

echo "Executed ${count} out of ${total} files"
File renamed without changes.
Loading