Skip to content

Commit

Permalink
Updates config string, snapshot, & adds intro/overivew polish (#34)
Browse files Browse the repository at this point in the history
  • Loading branch information
goodroot authored Aug 1, 2024
1 parent c18a287 commit c2956f2
Show file tree
Hide file tree
Showing 13 changed files with 489 additions and 344 deletions.
83 changes: 43 additions & 40 deletions clients/ingest-c-and-cpp.md
Original file line number Diff line number Diff line change
Expand Up @@ -23,7 +23,6 @@ Key features of the QuestDB C & C++ client include:
health monitoring
- **Automatic write retries**: Reuse connections and retry after interruptions


### Requirements

- Requires a C/C++ compiler and standard libraries.
Expand All @@ -32,16 +31,16 @@ Key features of the QuestDB C & C++ client include:

### Client Installation

You need to add the client as a dependency to your project. Depending on your environment,
you can do this in different ways. Please check the documentation at the
You need to add the client as a dependency to your project. Depending on your
environment, you can do this in different ways. Please check the documentation
at the
[client's repository](https://github.com/questdb/c-questdb-client/blob/main/doc/DEPENDENCY.md).


## C++

:::note

This section is for the QuestDB C++ client.
This section is for the QuestDB C++ client.

For the QuestDB C Client, see the below seciton.

Expand All @@ -52,7 +51,6 @@ For the QuestDB C Client, see the below seciton.
Explore the full capabilities of the C++ client via the
[C++ README](https://github.com/questdb/c-questdb-client/blob/main/doc/CPP.md).


## Authentication

The QuestDB C++ client supports basic connection and authentication
Expand All @@ -70,7 +68,8 @@ auto sender = questdb::ingress::line_sender::from_conf(

```

You can also pass the connection configuration via the `QDB_CLIENT_CONF` environment variable:
You can also pass the connection configuration via the `QDB_CLIENT_CONF`
environment variable:

```bash
export QDB_CLIENT_CONF="http::addr=localhost:9000;username=admin;password=quest;"
Expand All @@ -83,8 +82,8 @@ auto sender = questdb::ingress::line_sender::from_env();
```

When using QuestDB Enterprise, authentication can also be done via REST token.
Please check the [RBAC docs](/docs/operations/rbac/#authentication) for more info.

Please check the [RBAC docs](/docs/operations/rbac/#authentication) for more
info.

### Basic data insertion

Expand Down Expand Up @@ -123,7 +122,8 @@ These are the main steps it takes:

In this case, the designated timestamp will be the one at execution time.

Let's see now an example with timestamps, custom timeout, basic auth, and error control.
Let's see now an example with timestamps, custom timeout, basic auth, and error
control.

```cpp
#include <questdb/ingress/line_sender.hpp>
Expand Down Expand Up @@ -183,26 +183,25 @@ int main()
}
```

As you can see, both events now are using the same timestamp. We recommended using the original event timestamps when
ingesting data into QuestDB. Using the current timestamp will hinder the ability to deduplicate rows which is
As you can see, both events now are using the same timestamp. We recommended
using the original event timestamps when ingesting data into QuestDB. Using the
current timestamp will hinder the ability to deduplicate rows which is
[important for exactly-once processing](/docs/reference/api/ilp/overview/#exactly-once-delivery-vs-at-least-once-delivery).


## C

:::note

This sectioni s for the QuestDB C client.
This sectioni s for the QuestDB C client.

Skip to the bottom of this page for information relating to both the C and C++ clients.
Skip to the bottom of this page for information relating to both the C and C++
clients.

:::
<ILPClientsTable language="C" />
::: <ILPClientsTable language="C" />

Explore the full capabilities of the C client via the
[C README](https://github.com/questdb/c-questdb-client/blob/main/doc/C.md).


### Connection

The QuestDB C client supports basic connection and authentication
Expand All @@ -225,13 +224,15 @@ if (!sender) {
}
```

You can also pass the connection configuration via the `QDB_CLIENT_CONF` environment variable:
You can also pass the connection configuration via the `QDB_CLIENT_CONF`
environment variable:

```bash
export QDB_CLIENT_CONF="http::addr=localhost:9000;username=admin;password=quest;"
```

Then you use it like this:

```c
#include <questdb/ingress/line_sender.h>
...
Expand Down Expand Up @@ -317,9 +318,8 @@ error:

In this case, the designated timestamp will be the one at execution time.

Let's see now an example with timestamps, custom timeout, basic auth, error control, and transactional
awareness.

Let's see now an example with timestamps, custom timeout, basic auth, error
control, and transactional awareness.

```c
// line_sender_trades_example.c
Expand Down Expand Up @@ -417,11 +417,11 @@ error:

```

As you can see, both events use the same timestamp. We recommended using the original event timestamps when
ingesting data into QuestDB. Using the current timestamp hinder the ability to deduplicate rows which is
As you can see, both events use the same timestamp. We recommended using the
original event timestamps when ingesting data into QuestDB. Using the current
timestamp hinder the ability to deduplicate rows which is
[important for exactly-once processing](#/docs/clients/java_ilp/#exactly-once-delivery-vs-at-least-once-delivery).


## Other Considerations for both C and C++

### Configuration options
Expand All @@ -433,21 +433,23 @@ general structure is:
<transport>::addr=host:port;param1=val1;param2=val2;...
```

`transport` can be `http`, `https`, `tcp`, or `tcps`. The C/C++ and Rust clients share
the same codebase. Please refer to the
[Rust client's documentation](https://docs.rs/questdb-rs/latest/questdb/ingress) for the
full details on configuration.
`transport` can be `http`, `https`, `tcp`, or `tcps`. The C/C++ and Rust clients
share the same codebase. Please refer to the
[Rust client's documentation](https://docs.rs/questdb-rs/latest/questdb/ingress)
for the full details on configuration.

Alternatively, for a breakdown of Configuration string options available across
all clients, see the [Configuration string](/docs/configuration-string/) page.

### Don't forget to flush

The sender and buffer objects are entirely decoupled. This means that the sender
won't get access to the data in the buffer until you explicitly call
`sender.flush` or `line_sender_flush`.
This may lead to a pitfall where you drop a buffer that still has some data in it,
resulting in permanent data loss.
`sender.flush` or `line_sender_flush`. This may lead to a pitfall where you drop
a buffer that still has some data in it, resulting in permanent data loss.

Unlike other official QuestDB clients, the Rust client does not supports auto-flushing
via configuration.
Unlike other official QuestDB clients, the Rust client does not supports
auto-flushing via configuration.

A common technique is to flush periodically on a timer and/or once the buffer
exceeds a certain size. You can check the buffer's size by calling
Expand All @@ -459,20 +461,21 @@ QuestDB instances), call `sender.flush_and_keep(&mut buffer)` instead.

### Transactional flush

As described in the [ILP overview](/docs/reference/api/ilp/overview#http-transaction-semantics),
the HTTP transport has some support for transactions.
As described in the
[ILP overview](/docs/reference/api/ilp/overview#http-transaction-semantics), the
HTTP transport has some support for transactions.

To ensure in advance that a flush will not affect more than one table, call
`buffer.transactional()` or `line_sender_buffer_transactional(buffer)` as we demonstrated on
the examples in this document.
`buffer.transactional()` or `line_sender_buffer_transactional(buffer)` as we
demonstrated on the examples in this document.

This call will return false if the flush wouldn't be data-transactional.

## Next Steps

Please refer to the [ILP overview](/docs/reference/api/ilp/overview) for details
about transactions, error control, delivery guarantees, health check, or table and
column auto-creation.
about transactions, error control, delivery guarantees, health check, or table
and column auto-creation.

With data flowing into QuestDB, now it's time to for analysis.

Expand Down
24 changes: 14 additions & 10 deletions clients/ingest-dotnet.md
Original file line number Diff line number Diff line change
Expand Up @@ -91,7 +91,8 @@ TCP authentication can be configured using JWK tokens:
using var sender = Sender.New("tcp::addr=localhost:9000;username=admin;token=<token>");
```

The connection string can also be built programatically. See [Configuration](#configuration) for details.
The connection string can also be built programatically. See
[Configuration](#configuration) for details.

## Basic insert

Expand All @@ -117,7 +118,9 @@ await sender.Table("trades")
await sender.SendAsync();
```

In this case, the designated timestamp will be the one at execution time. Let's see now an example with timestamps, custom auto-flushing, basic auth, and error reporting.
In this case, the designated timestamp will be the one at execution time. Let's
see now an example with timestamps, custom auto-flushing, basic auth, and error
reporting.

```csharp
using QuestDB;
Expand Down Expand Up @@ -159,11 +162,11 @@ class Program
}
```

As you can see, both events use the same timestamp. We recommended using the original event timestamps when
ingesting data into QuestDB. Using the current timestamp hinder the ability to deduplicate rows which is
As you can see, both events use the same timestamp. We recommended using the
original event timestamps when ingesting data into QuestDB. Using the current
timestamp hinder the ability to deduplicate rows which is
[important for exactly-once processing](/docs/reference/api/ilp/overview/#exactly-once-delivery-vs-at-least-once-delivery).


## Configuration

Construct new Senders via the `Sender` factory.
Expand All @@ -177,7 +180,7 @@ Optionally, TCP uses `9009`.
### With a configuration string

It is recommended, where possible, to initialise the sender using a
[configuration string](https://questdb.io/docs/reference/api/ilp/overview/#client-side-configuration).
[configuration string](/docs/configuration-string/).

Configuration strings provide a convenient shorthand for defining client
properties, and are validated during construction of the `Sender`.
Expand Down Expand Up @@ -310,7 +313,6 @@ QuestDB's deduplication feature, and should be avoided where possible.

:::


## Flushing

Once the buffer is filled with data ready to be sent, it can be flushed to the
Expand Down Expand Up @@ -415,7 +417,9 @@ batch only for a single table.

:::caution

As described in the [ILP overview](/docs/reference/api/ilp/overview#http-transaction-semantics), the HTTP transport has some limitations for transactions when adding new columns.
As described in the
[ILP overview](/docs/reference/api/ilp/overview#http-transaction-semantics), the
HTTP transport has some limitations for transactions when adding new columns.

:::

Expand Down Expand Up @@ -574,8 +578,8 @@ using var sender =
## Next Steps

Please refer to the [ILP overview](/docs/reference/api/ilp/overview) for details
about transactions, error control, delivery guarantees, health check, or table and
column auto-creation.
about transactions, error control, delivery guarantees, health check, or table
and column auto-creation.

Dive deeper into the .NET client capabilities by exploring more examples
provided in the
Expand Down
43 changes: 26 additions & 17 deletions clients/ingest-go.md
Original file line number Diff line number Diff line change
Expand Up @@ -78,26 +78,27 @@ Or, set the QDB_CLIENT_CONF environment variable and call
client, err := questdb.LineSenderFromEnv(context.TODO())
```

Alternatively, you can use the built-in Go API to specify the connection options.
Alternatively, you can use the built-in Go API to specify the connection
options.

```go
package main
```go
package main

import (
"context"
qdb "github.com/questdb/go-questdb-client/v3"
"context"
qdb "github.com/questdb/go-questdb-client/v3"
)


func main() {
ctx := context.TODO()
ctx := context.TODO()

client, err := qdb.NewLineSender(context.TODO(), qdb.WithHttp(), qdb.WithAddress("localhost:9000"), qdb.WithBasicAuth("admin", "quest"))
client, err := qdb.NewLineSender(context.TODO(), qdb.WithHttp(), qdb.WithAddress("localhost:9000"), qdb.WithBasicAuth("admin", "quest"))
```
When using QuestDB Enterprise, authentication can also be done via REST token.
Please check the [RBAC docs](/docs/operations/rbac/#authentication) for more info.
Please check the [RBAC docs](/docs/operations/rbac/#authentication) for more
info.
## Basic Insert
Expand Down Expand Up @@ -139,7 +140,9 @@ func main() {
}
```
In this case, the designated timestamp will be the one at execution time. Let's see now an example with an explicit timestamp, custom auto-flushing, and basic auth.
In this case, the designated timestamp will be the one at execution time. Let's
see now an example with an explicit timestamp, custom auto-flushing, and basic
auth.
```Go
package main
Expand Down Expand Up @@ -183,29 +186,35 @@ func main() {
}
}
```
We recommended to use User-assigned timestamps when ingesting data into QuestDB.
Using the current timestamp hinder the ability to deduplicate rows which is
Using the current timestamp hinder the ability to deduplicate rows which is
[important for exactly-once processing](/docs/reference/api/ilp/overview/#exactly-once-delivery-vs-at-least-once-delivery).
## Configuration options
The minimal configuration string needs to have the protocol, host, and port, as in:
The minimal configuration string needs to have the protocol, host, and port, as
in:
```
http::addr=localhost:9000;
```
In the Go client, you can set the configuration options via the standard config string,
which is the same across all clients, or using [the built-in API](https://pkg.go.dev/github.com/questdb/go-questdb-client/v3#LineSenderOption).
In the Go client, you can set the configuration options via the standard config
string, which is the same across all clients, or using
[the built-in API](https://pkg.go.dev/github.com/questdb/go-questdb-client/v3#LineSenderOption).
For all the extra options you can use, please check [the client docs](https://pkg.go.dev/github.com/questdb/go-questdb-client/v3#LineSenderFromConf)
For all the extra options you can use, please check
[the client docs](https://pkg.go.dev/github.com/questdb/go-questdb-client/v3#LineSenderFromConf)
Alternatively, for a breakdown of Configuration string options available across
all clients, see the [Configuration string](/docs/configuration-string/) page.
## Next Steps
Please refer to the [ILP overview](/docs/reference/api/ilp/overview) for details
about transactions, error control, delivery guarantees, health check, or table and
column auto-creation.
about transactions, error control, delivery guarantees, health check, or table
and column auto-creation.
Explore the full capabilities of the Go client via
[Go.dev](https://pkg.go.dev/github.com/questdb/go-questdb-client/v3).
Expand Down
Loading

0 comments on commit c2956f2

Please sign in to comment.