From aaa6ba8ff3a7b954f8db850aa6ec9ad022a8c76e Mon Sep 17 00:00:00 2001 From: goodroot <9484709+goodroot@users.noreply.github.com> Date: Thu, 1 Aug 2024 12:54:39 -0700 Subject: [PATCH 1/7] minor intro polish --- introduction.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/introduction.md b/introduction.md index 5e712a67..080511a3 100644 --- a/introduction.md +++ b/introduction.md @@ -46,10 +46,6 @@ Developers are most enthusiastic about the following key features: If you are running into throughput bottlenecks using an existing storage engine or time series database, QuestDB can help. -#### Familiar SQL analytics - -No obscure domain-specific languages required. Use extended SQL. - #### High performance deduplication & out-of-order indexing [High data cardinality](/glossary/high-cardinality/) will not lead to @@ -60,10 +56,12 @@ performance degradation. Strong, cost-saving performance on very mninimal hardware, including sensors and Raspberry Pi. -#### Time series SQL extensions +#### SQL with time series extensions Fast, SIMD-optimized SQL extensions to cruise through querying and analysis. +No obscure domain-specific languages required. + Greatest hits include: - [`SAMPLE BY`](/docs/reference/sql/sample-by/) summarizes data into chunks From 075011576e6f3375460d7da2ebf9a251ebfc6b8a Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Tue, 6 Aug 2024 18:34:25 +0100 Subject: [PATCH 2/7] Fix nesting for BETWEEN operator (#35) The nesting was incorrect. --- reference/operators/date-time.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/reference/operators/date-time.md b/reference/operators/date-time.md index 326738cb..5f7aba9d 100644 --- a/reference/operators/date-time.md +++ b/reference/operators/date-time.md @@ -129,7 +129,7 @@ results on Jan 1-2 in 2018 and in 2019: | ... | ... | | 2019-01-02T23:59:59.999999Z | 103.8 | -### `BETWEEN` value1 `AND` value2 +## `BETWEEN` value1 `AND` value2 The `BETWEEN` operator allows you to specify a non-standard range. It includes both upper and lower bounds, similar to standard SQL. The order of these bounds From 2851d3c61035037abaa3081a024723683817fe89 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?javier=20ram=C3=ADrez?= Date: Thu, 8 Aug 2024 18:10:36 +0200 Subject: [PATCH 3/7] Adding reference for Date to Timestamp conversion (#36) Added new page /docs/clients/date-to-timestamp-conversion and an intro in /docs/reference/function/date-time/#timestamp-to-date-conversion. This should improve developer experience and should help position the page better in search engines --------- Co-authored-by: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> --- clients/date-to-timestamp-conversion.md | 394 ++++++++++++++++++++++++ reference/function/date-time.md | 148 +++++---- sidebars.js | 5 + 3 files changed, 487 insertions(+), 60 deletions(-) create mode 100644 clients/date-to-timestamp-conversion.md diff --git a/clients/date-to-timestamp-conversion.md b/clients/date-to-timestamp-conversion.md new file mode 100644 index 00000000..18898178 --- /dev/null +++ b/clients/date-to-timestamp-conversion.md @@ -0,0 +1,394 @@ +--- +title: Date to Timestamp Conversion +sidebar_label: Date to Timestamp +description: Python, Go, JAVA, JavaScript, C/C++, Rust, .Net, PHP, or Ruby. +--- + + +# Date to Timestamp Conversion in Different Programming Languages + +Most languages have a dedicated type for dates or timestamps, with the notable exception of C. In this guide, we show how to convert from a literal string representing a date into the native `Date` type, and then +into a `Timestamp` type using Python, Go, Java, C, C++, Rust, C#/.NET, JavaScript/Node.js, Ruby, and PHP. + +QuestDB offers clients for Python, Go, Java, C, C++, Rust, C#/.NET, and JavaScript/Node.js. Some of the clients +can directly use a `Timestamp` type when using the client, while others need to convert the timestamp into a +long representing the epoch time in microseconds. We add such required conversions into the snippets. + +Please refer to the [ingestion overview](https://questdb.io/docs/ingestion-overview/) to learn more about the details of the client library for your language. + +## Date to Timestamp in Python + +The `datetime.date` type stores only date information, while `datetime.datetime` stores both date and time information. +The QuestDB Python client accepts either a `datetime.datetime` object, or a `pandas.timestamp`. + +```python +from datetime import datetime, date +import pandas as pd +from questdb.ingress import Sender + +date_str = '2024-08-05' + +# We could parse the string directly into a datetime object, +# but for this example, we will show how to convert it via a date. +date_obj = datetime.strptime(date_str, '%Y-%m-%d').date() +print(f"Date object: {date_obj}") + +# Convert to datetime object. This object can be passed as a column value in the QuestDB Python client's row() API. +datetime_obj = datetime.combine(date_obj, datetime.min.time()) +print(f"DateTime object: {datetime_obj}") + +# Now you can pass the datetime_obj to the QuestDB sender, as in +# ... +# columns={'NonDesignatedTimestampColumnName': datetime_obj, +# ... + + +# Optionally, you can convert to a pandas Timestamp. The QuestDB Python client offers the dataframe() API, which accepts pd.Timestamp columns +pd_timestamp = pd.Timestamp(datetime_obj) +print(f"Pandas Timestamp: {pd_timestamp}") + +``` +Learn more about the [QuestDB Python Client](/docs/clients/ingest-python/) + +## Date to Timestamp in Go + +The `time.Time` type stores both date and time information. It is used for most time-related tasks in Go, such as +parsing dates, formatting dates, and time arithmetic. + +The QuestDB Go client expects the timestamp as an `int64`, in unix microseconds, so we will need to convert it. + +```go +package main + +import ( + "fmt" + "time" + "github.com/questdb/go-questdb-client/v3" +) + +func main() { + dateStr := "2024-08-05" + + // Layout string to match the format of dateStr + layout := "2006-01-02" + + // Parse the date string into a time.Time object + dateObj, err := time.Parse(layout, dateStr) + if err != nil { + fmt.Println("Error parsing date:", err) + return + } + + // Convert the dateObj to a timestamp in microseconds + timestamp := dateObj.UnixNano() / int64(time.Microsecond) + + fmt.Println("Date:", dateObj) + fmt.Println("Timestamp (microseconds):", timestamp) + + // Now you can call add the column to the QuestDB client, as in + // ... + // TimestampColumn("NonDesignatedTimestampColumnName", timestamp). + // ... +} + +``` +Learn more about the [QuestDB Go Client](/docs/clients/ingest-go/) + +## Date to Timestamp in Java + +The `java.time.LocalDate` type stores only date information, while `java.time.LocalDateTime` stores both date and time +information. + +The QuestDB Java Client expects either a `java.time.Instant` object or a `long` value representing an epoch number. When using a `long`, you must provide the time unit using `java.time.temporal.ChronoUnit`. + +Example using `Instant` + +```java +import java.time.LocalDate; +import java.time.ZoneId; +import java.time.Instant; +import java.time.format.DateTimeFormatter; +import io.questdb.client.Sender; + +public class Main { + public static void main(String[] args) { + String dateStr = "2024-08-05"; + DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd"); + // Note we are only converting to Date for educational purposes, but we really need the timestamp for QuestDB + LocalDate date = LocalDate.parse(dateStr, formatter); + + // Convert LocalDate to Instant at the start of the day in the default time zone + Instant instant = date.atStartOfDay(ZoneId.systemDefault()).toInstant(); + + System.out.println("Date: " + date); + System.out.println("Instant: " + instant); + + // Example method call using QuestDB API + try (Sender sender = Sender.fromConfig("http::addr=localhost:9000;")) { + sender.table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "sell") + .timestampColumn("NonDesignatedTimestampColumnName", instant) + .doubleColumn("price", 2615.54) + .doubleColumn("amount", 0.00044) + .atNow(); + } +} +``` + +Example using a `long` with epoch microseconds and `java.time.temporal.ChronoUnit` + +```java +import java.time.LocalDate; +import java.time.LocalDateTime; +import java.time.ZoneOffset; +import java.time.format.DateTimeFormatter; +import java.time.temporal.ChronoUnit; +import io.questdb.client.Sender; + +public class Main { + public static void main(String[] args) { + String dateStr = "2024-08-05"; + DateTimeFormatter formatter = DateTimeFormatter.ofPattern("yyyy-MM-dd"); + // Note we are only converting to Date for educational purposes, but we really need the timestamp for QuestDB + LocalDate date = LocalDate.parse(dateStr, formatter); + + // Convert LocalDate to LocalDateTime at the start of the day + LocalDateTime dateTime = date.atStartOfDay(); + + // Convert LocalDateTime to epoch microseconds + long epochMicro = dateTime.toInstant(ZoneOffset.UTC).toEpochMilli() * 1000; + + System.out.println("Epoch in Microseconds: " + epochMicro); + + // Example method call using QuestDB API + try (Sender sender = Sender.fromConfig("http::addr=localhost:9000;")) { + sender.table("trades") + .symbol("symbol", "ETH-USD") + .symbol("side", "sell") + .timestampColumn("NonDesignatedTimestampColumnName", epochMicro, ChronoUnit.MICROS) + .doubleColumn("price", 2615.54) + .doubleColumn("amount", 0.00044) + .atNow(); + + } + +} +``` + +Learn more about the [QuestDB Java Client](/docs/clients/java-ilp/) + +## Date to Timestamp in C + +Standard C does not have a native `Date type`, but `struct tm` in `` can store both date and time information. + +The QuestDB C client expects timestamp to be an `int64_t` in microseconds, so we will need to convert it. + +```c +#include +#include +#include +#include + + +int main() { + char dateStr[] = "2024-08-05"; + struct tm tm = {0}; + strptime(dateStr, "%Y-%m-%d", &tm); + + // Convert to time_t (seconds since Epoch) + time_t seconds = mktime(&tm); + + // Convert to microseconds + int64_t microseconds = (int64_t)seconds * 1000000; + printf("Date: %s, Timestamp (microseconds): %ld +", dateStr, microseconds); + + // you can now pass the microseconds variable to the QuestDB client line_sender_buffer_column_ts_micros function + + return 0; +} +``` + +Learn more about the [QuestDB C Client](/docs/clients/ingest-c-and-cpp/#c-1) + +## Date to Timestamp in C++ + +The `std::chrono::year_month_da`y type from C++20 stores only date information, while `std::chrono::time_point` stores +both date and time information. + +The QuestDB C++ client accepts an int64_t as microseconds, but also a +`std::chrono::time_point` or a `std::chrono::time_point`. + +```cpp +#include +#include +#include +#include +#include + + +int main() { + std::string dateStr = "2024-08-05"; + std::istringstream iss(dateStr); + std::tm tm = {}; + iss >> std::get_time(&tm, "%Y-%m-%d"); + + // Convert to time_point (timestamp) + auto tp = std::chrono::system_clock::from_time_t(std::mktime(&tm)); + auto microseconds = std::chrono::duration_cast(tp.time_since_epoch()).count(); + + std::cout << "Date: " << std::put_time(&tm, "%Y-%m-%d") << ", Timestamp (microseconds): " << microseconds << std::endl; + + // You can now pass the microseconds variable to the QuestDB column function, converting to int64_t using the + // timestamp_micros function + // .column("NotDesignatedTimestampColumnName",questdb::ingress::timestamp_micros(microseconds) + + return 0; +} +``` +Learn more about the [QuestDB C++ Client](/docs/clients/ingest-c-and-cpp/) + +## Date to Timestamp in Rust + +The `chrono::NaiveDate` type stores only date information, while `chrono::NaiveDateTime` stores both date and time +information. + +The QuestDB Rust client accepts either a `i64` Epoch in microseconds, or a `chrono::Datetime`. + +```rust +extern crate chrono; +use questdb::ingress::{Sender, Buffer, TimestampMicros} +use chrono::{NaiveDate, NaiveDateTime}; + +fn main() { + let date_str = "2024-08-05"; + let date_obj = NaiveDate::parse_from_str(date_str, "%Y-%m-%d").expect("Failed to parse date"); + + // Convert to NaiveDateTime for timestamp + let datetime = date_obj.and_hms(0, 0, 0); + let timestamp = datetime.timestamp_micros(); + + println!("Date: {}", date_obj); + println!("Timestamp (microseconds): {}", timestamp); + + // You can now use this timestamp to call the .column_ts QuestDB API + // .column_ts("NonDesignatedTimestampColumnName", TimestampMicros::new(timestamp))? + + + +} +``` +Learn more about the [QuestDB Rust Client](/docs/clients/ingest-rust/) + +## Date to Timestamp in C#/.NET + +The `System.DateTime` type stores both date and time information. There is also `System.DateOnly` for only date information +in .NET 6 and later. + +The QuestDB Dotnet client accepts `DateTime` or `DateTimeOffset` objects. Showing both options below. + +```csharp +using System; +using QuestDB; + + +class Program +{ + static void Main() + { + string dateStr = "2024-08-05"; + + // Parse the date string into a DateTime object + DateTime date = DateTime.ParseExact(dateStr, "yyyy-MM-dd", null); + Console.WriteLine("DateTime: " + date); + + // You can now call the QuestDB API adding a column, as in + // sender.Column("NonDesignatedTimestampColumnName", date); + + + // Parse the date string into a DateTimeOffset object + DateTimeOffset dateOffset = DateTimeOffset.ParseExact(dateStr, "yyyy-MM-dd", null); + Console.WriteLine("DateTimeOffset: " + dateOffset); + + // You can now call the QuestDB API adding a column, as in + // sender.Column("NonDesignatedTimestampColumnName", dateOffset); + } +} + +``` +Learn more about the [QuestDB .NET Client](/docs/clients/ingest-dotnet/) + +## Date to Timestamp in JavasScript/Node.js + +The Date type stores both date and time information. + +The QuestDB Node.js client accepts an epoch in microseconds, which can be a `number` or `bigint`. + +```javascript +const { Sender } = require("@questdb/nodejs-client") + +const dateStr = '2024-08-05'; +const dateObj = new Date(dateStr + 'T00:00:00Z'); + +// Convert to timestamp (milliseconds since Epoch) then convert to microseconds +const timestamp = BigInt(dateObj.getTime()) * 1000n; +console.log("Date:", dateObj.toISOString().split('T')[0]); +console.log("Timestamp (microseconds):", timestamp.toString()); + +// You can now add the column using QuestDB client, as in +// .timestampColumn("NonDesignatedTimestampColumnName", timestamp) +``` + +Learn more about the [QuestDB Node.js Client](/docs/clients/ingest-node/) + +## Date to Timestamp in Ruby + +The `Date` class stores only date information, while the `DateTime` class stores both date and time information. + +QuestDB does not have an official Ruby client, but you can send a POST request comprising the ILP messages. Within this messages, you can pass an epoch timestamp in nanoseconds as the designated timestamp, and pass epoch timestamps in microseconds for other timestamp columns. + +Alternatively, you can use the [InfluxDB Ruby Client](https://github.com/influxdata/influxdb-client-ruby), which is compatible with QuestDB. + +```ruby +require 'date' + +date_str = '2024-08-05' +date_obj = Date.parse(date_str) + +# Convert to DateTime for timestamp in microseconds +datetime_obj = DateTime.parse(date_str) +timestamp = (datetime_obj.to_time.to_i * 1_000_000) + (datetime_obj.to_time.usec) + +puts "Date: #{date_obj}" +puts "Timestamp (microseconds): #{timestamp}" +``` + +Learn more about the [ILP text format](/docs/reference/api/ilp/advanced-settings/). + + +## Date to Timestamp in PHP + +Both of the `DateTime` and `DateTimeImmutable` classes store date and time information + +QuestDB does not have an official PHP client, but you can send a POST request comprising the ILP messages. Within this messages, you can pass an epoch timestamp in nanoseconds as the designated timestamp, and pass epoch timestamps in microseconds for other timestamp columns. + +Alternatively, you can use the [InfluxDB PHP Client](https://github.com/influxdata/influxdb-client-php), which is compatible with QuestDB. + +```php +getTimestamp() * 1000000; + +echo "Date: " . $date_obj->format('Y-m-d') . " +"; +echo "Timestamp (microseconds): " . $timestamp . " +"; +?> +``` + +Learn more about the [ILP text format](/docs/reference/api/ilp/advanced-settings/). + diff --git a/reference/function/date-time.md b/reference/function/date-time.md index e37c02f6..949f3723 100644 --- a/reference/function/date-time.md +++ b/reference/function/date-time.md @@ -65,9 +65,37 @@ The interpretation of `y` depends on the input digit number: | `005-03` | `0005-03-01T00:00:00.000000Z` | Greedily parsing the number as it is | | `0005-03` | `0005-03-01T00:00:00.000000Z` | Greedily parsing the number as it is | +## Timestamp to Date conversion + +As described at the [data types section](/docs/reference/sql/datatypes), the only difference between `Timestamp` and `Date` +in QuestDB type system is the resolution. Whilst `Timestamp` stores resolution as an offset from Unix epoch in microseconds, +`Date` stores the offset in milliseconds. + +Since both types are backed by a signed long, this means the `DATE` type has a +wider range. A `DATE` column can store about ±2.9 million years from the Unix epoch, whereas a `TIMESTAMP` has an approximate +range of ±290,000 years. + +For most purposes a `TIMESTAMP` is preferred, as it offers a wider range of functions whilst still being 8 bytes in size. + +Be aware that, when using a `TIMESTAMP` as the designated timestamp, you cannot set it to any value before the Unix epoch +(`1970-01-01T00:00:00.000000Z`). + +To explicitely convert from `TIMESTAMP` to `DATE` you can use `CAST(ts_column AS DATE)`. To convert +from `DATE` to `TIMESTAMP` you can `CAST(date_column AS TIMESTAMP)`. + +### Programmatically convert from language-specific datetimes into QuestDB timestamps + +Different programming languages use different types of objects to represent the `Date` type. To learn how to convert +from the `Date` type into a `Timestamp` object in Python, Go, Java, JavaScript, C/C++, Rust, or C#/.NET, please visit +our [Date to Timestamp conversion](/docs/clients/date-to-timestamp-conversion) reference. + +--- + +# Function Reference + ## date_trunc -`date_trunc(unit, timestamp)` - returns a timestamps truncated to the selected +`date_trunc(unit, timestamp)` - returns a timestamp truncated to the specified precision **Arguments:** @@ -299,6 +327,33 @@ SELECT to_str(ts,'EE'),day_of_week_sunday_first(ts) FROM myTable; | Saturday | 7 | | Sunday | 1 | +## days_in_month + +`days_in_month(value)` - returns the number of days in a month from a given +timestamp or date. + +**Arguments:** + +- `value` is any `timestamp` or `date` + +**Return value:** + +Return value type is `int` + +**Examples:** + +```questdb-sql +SELECT month(ts), days_in_month(ts) FROM myTable; +``` + +| month | days_in_month | +| :---- | :------------ | +| 4 | 30 | +| 5 | 31 | +| 6 | 30 | +| 7 | 31 | +| 8 | 31 | + ## extract `extract (unit, timestamp)` - returns the selected time unit from the input @@ -413,33 +468,6 @@ SELECT year(ts), is_leap_year(ts) FROM myTable; | 2024 | true | | 2025 | false | -## days_in_month - -`days_in_month(value)` - returns the number of days in a month from a provided -timestamp or date. - -**Arguments:** - -- `value` is any `timestamp` or `date` - -**Return value:** - -Return value type is `int` - -**Examples:** - -```questdb-sql -SELECT month(ts), days_in_month(ts) FROM myTable; -``` - -| month | days_in_month | -| :---- | :------------ | -| 4 | 30 | -| 5 | 31 | -| 6 | 30 | -| 7 | 31 | -| 8 | 31 | - ## micros `micros(value)` - returns the `micros` of the millisecond for a given date or @@ -708,83 +736,83 @@ SELECT second(ts), count() FROM transactions; | 58 | 9876 | | 59 | 2567 | -## systimestamp +## sysdate -`systimestamp()` - offset from UTC Epoch in microseconds. Calculates -`UTC timestamp` using system's real time clock. The value is affected by -discontinuous jumps in the system time (e.g., if the system administrator -manually changes the system time). +`sysdate()` - returns the timestamp of the host system as a `date` with +`millisecond` precision. -`systimestamp()` value can change within the query execution timeframe and -should **NOT** be used in WHERE clause to filter designated timestamp column. +Calculates `UTC date` with millisecond precision using system's real time clock. +The value is affected by discontinuous jumps in the system time (e.g., if the +system administrator manually changes the system time). + +`sysdate()` value can change within the query execution timeframe and should +**NOT** be used in WHERE clause to filter designated timestamp column. :::tip -Use now() with WHERE clause filter. +Use `now()` with WHERE clause filter. ::: **Arguments:** -- `systimestamp()` does not accept arguments. +- `sysdate()` does not accept arguments. **Return value:** -Return value type is `timestamp`. +Return value type is `date`. **Examples:** -```questdb-sql title="Insert current system timestamp" +```questdb-sql title="Insert current system date along with a value" INSERT INTO readings -VALUES(systimestamp(), 123.5); +VALUES(sysdate(), 123.5); ``` -| ts | reading | +| sysdate | reading | | :-------------------------- | :------ | | 2020-01-02T19:28:48.727516Z | 123.5 | -## sysdate +```questdb-sql title="Query based on last minute" +SELECT * FROM readings +WHERE date_time > sysdate() - 60000000L; +``` -`sysdate()` - returns the timestamp of the host system as a `date` with -`millisecond` precision. +## systimestamp -Calculates `UTC date` with millisecond precision using system's real time clock. -The value is affected by discontinuous jumps in the system time (e.g., if the -system administrator manually changes the system time). +`systimestamp()` - offset from UTC Epoch in microseconds. Calculates +`UTC timestamp` using system's real time clock. The value is affected by +discontinuous jumps in the system time (e.g., if the system administrator +manually changes the system time). -`sysdate()` value can change within the query execution timeframe and should -**NOT** be used in WHERE clause to filter designated timestamp column. +`systimestamp()` value can change within the query execution timeframe and +should **NOT** be used in WHERE clause to filter designated timestamp column. :::tip -Use `now()` with WHERE clause filter. +Use now() with WHERE clause filter. ::: **Arguments:** -- `sysdate()` does not accept arguments. +- `systimestamp()` does not accept arguments. **Return value:** -Return value type is `date`. +Return value type is `timestamp`. **Examples:** -```questdb-sql title="Insert current system date along with a value" +```questdb-sql title="Insert current system timestamp" INSERT INTO readings -VALUES(sysdate(), 123.5); +VALUES(systimestamp(), 123.5); ``` -| sysdate | reading | +| ts | reading | | :-------------------------- | :------ | | 2020-01-02T19:28:48.727516Z | 123.5 | -```questdb-sql title="Query based on last minute" -SELECT * FROM readings -WHERE date_time > sysdate() - 60000000L; -``` - ## timestamp_ceil `timestamp_ceil(unit, timestamp)` - performs a ceiling calculation on a diff --git a/sidebars.js b/sidebars.js index e5536b2c..3c6de3ea 100644 --- a/sidebars.js +++ b/sidebars.js @@ -68,6 +68,11 @@ module.exports = { type: "doc", label: ".NET", }, + { + id: "clients/date-to-timestamp-conversion", + type: "doc", + label: "Date to Timestamp Conversion", + }, ], }, { From 32326969525ef02775679c4422e55d5a7aa3f732 Mon Sep 17 00:00:00 2001 From: Nick Woolmer <29717167+nwoolmer@users.noreply.github.com> Date: Mon, 12 Aug 2024 17:18:53 +0100 Subject: [PATCH 4/7] Update Kafka connect ports from 9009 -> 9000 (#39) The article uses port 9009 but with http addresses. Later, it uses http addresses with port 9000. I assume this should be consistently 9000. --- third-party-tools/kafka/questdb-kafka.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/third-party-tools/kafka/questdb-kafka.md b/third-party-tools/kafka/questdb-kafka.md index 944bbf3a..fa8aa52e 100644 --- a/third-party-tools/kafka/questdb-kafka.md +++ b/third-party-tools/kafka/questdb-kafka.md @@ -74,13 +74,13 @@ information on how to configure properties, see the This example file: -1. Assumes a running InfluxDB Line Protocol default port of `9009` +1. Assumes a running InfluxDB Line Protocol default port of `9000` 2. Creates a reader from a Kafka topic: `example-topic` 3. Creates & writes into a QuestDB table: `example_table` ```shell title="Create a configuration file" name=questdb-sink -client.conf.string=http::addr=localhost:9009; +client.conf.string=http::addr=localhost:9000; topics=example-topic table=example_table @@ -481,7 +481,7 @@ Example: ```shell title="Configuration file an explicit table name" name=questdb-sink connector.class=io.questdb.kafka.QuestDBSinkConnector -client.conf.string=http::addr=localhost:9009; +client.conf.string=http::addr=localhost:9000; topics=example-topic table=example_table @@ -496,7 +496,7 @@ Example: ```shell title="Configuration file with a templated table name" name=questdb-sink connector.class=io.questdb.kafka.QuestDBSinkConnector -client.conf.string=http::addr=localhost:9009; +client.conf.string=http::addr=localhost:9000; topics=example-topic table=from_kafka_${topic}_${key}_suffix @@ -590,7 +590,7 @@ you want to remove the `address` field, you can use the following configuration: "name": "questdb-sink", "config": { "connector.class": "io.questdb.kafka.QuestDBSinkConnector", - "host": "localhost:9009", + "host": "localhost:9000", "topics": "Orders", "table": "orders_table", "key.converter": "org.apache.kafka.connect.storage.StringConverter", From fffcd4d709b980bd431ec7c85fa3d949e5229d2f Mon Sep 17 00:00:00 2001 From: goodroot <9484709+goodroot@users.noreply.github.com> Date: Mon, 12 Aug 2024 15:07:52 -0700 Subject: [PATCH 5/7] Add Pulumi GCP config (#40) Courtesy of our friend Nelson Griffiths. --- deployment/google-cloud-platform.md | 61 +++++++++++++++++++++++++++-- 1 file changed, 57 insertions(+), 4 deletions(-) diff --git a/deployment/google-cloud-platform.md b/deployment/google-cloud-platform.md index 09cbfd55..a73480ec 100644 --- a/deployment/google-cloud-platform.md +++ b/deployment/google-cloud-platform.md @@ -41,8 +41,7 @@ import Screenshot from "@theme/Screenshot" general-purpose instance with 4GB memory and should be enough to run this example. - {" "} - :9000/exec ``` + +## Set up GCP with Pulumi + +If you're using [Pulumi](https://www.pulumi.com/gcp/) to manage your +infrastructure, you can create a QuestDB instance with the following: + +```python +import pulumi +import pulumi_gcp as gcp + +# Create a Google Cloud Network +firewall = gcp.compute.Firewall( + "questdb-firewall", + network="default", + allows=[ + gcp.compute.FirewallAllowArgs( + protocol="tcp", + ports=["9000", "8812"], + ), + ], + target_tags=["questdb"], + source_ranges=["0.0.0.0/0"], +) + +# Create a Compute Engine Instance +instance = gcp.compute.Instance( + "questdb-instance", + machine_type="e2-medium", + zone="us-central1-a", + boot_disk={ + "initialize_params": { + "image": "ubuntu-os-cloud/ubuntu-2004-lts", + }, + }, + network_interfaces=[ + gcp.compute.InstanceNetworkInterfaceArgs( + network="default", + access_configs=[{}], # Ephemeral public IP + ) + ], + metadata_startup_script="""#!/bin/bash + sudo apt-get update + sudo apt-get install -y docker.io + sudo docker run -d -p 9000:9000 -p 8812:8812 \ + --env QDB_HTTP_USER="admin" \ + --env QDB_HTTP_PASSWORD="quest" \ + questdb/questdb + """, + tags=["questdb"], +) + +# Export the instance's name and public IP +pulumi.export("instanceName", instance.name) +pulumi.export("instance_ip", instance.network_interfaces[0].access_configs[0].nat_ip) +``` From 8f7ef325a8b3f01459f181d8baac233c5cbce748 Mon Sep 17 00:00:00 2001 From: =?UTF-8?q?javier=20ram=C3=ADrez?= Date: Wed, 14 Aug 2024 10:42:33 +0200 Subject: [PATCH 6/7] bump version of questdb connect at superset docs (#41) Version was pointing to 1.0.11 instead of 1.0.12 --- third-party-tools/superset.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/third-party-tools/superset.md b/third-party-tools/superset.md index 2b5bc9d7..01b93289 100644 --- a/third-party-tools/superset.md +++ b/third-party-tools/superset.md @@ -39,7 +39,7 @@ git clone https://github.com/apache/superset.git ```bash touch ./docker/requirements-local.txt -echo "questdb-connect==1.0.11" > docker/requirements-local.txt +echo "questdb-connect==1.0.12" > docker/requirements-local.txt ``` 4. Run Apache Superset: @@ -67,7 +67,7 @@ Superset without Docker, you need to install the following requirements : Install QuestDB Connect using `pip`: ```bash -pip install 'questdb-connect==1.0.11' +pip install 'questdb-connect==1.0.12' ``` ## Connecting QuestDB to Superset From 6e6139c88957e9163bbdc9a7f40f8e56669b8e36 Mon Sep 17 00:00:00 2001 From: Li Yuanheng <520dhh@gmail.com> Date: Thu, 15 Aug 2024 05:36:16 +0800 Subject: [PATCH 7/7] fix port and wording mismatch in docker deployment (#38) Co-authored-by: goodroot <9484709+goodroot@users.noreply.github.com> --- deployment/docker.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/deployment/docker.md b/deployment/docker.md index c1774e7c..5dbc607b 100644 --- a/deployment/docker.md +++ b/deployment/docker.md @@ -62,7 +62,7 @@ This parameter will expose a port to the host. You can specify: All ports are optional, you can pick only the ones you need. For example, it is enough to expose `8812` if you only plan to use -[InfluxDB line protocol](/docs/reference/api/ilp/overview/). +[Postgres wire protocol](/docs/reference/api/postgres/). ### `-v` parameter to mount storage