Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Us 108392 add static data producer #6

Merged
merged 8 commits into from
Sep 15, 2021
+451 −32
Merged

Us 108392 add static data producer #6

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
8 commits
Select commit Hold shift + click to select a range
2f02cab
Add description of variants
ingeknudsen Sep 14, 2021
3395191
Added static data box
ingeknudsen Sep 14, 2021
7341700
Refactored to be aligned
ingeknudsen Sep 14, 2021
42ff2b7
Merge branch 'main' into US-108392-AddStaticDataProducer
ingeknudsen Sep 14, 2021
1361f1d
Corrected typing
ingeknudsen Sep 14, 2021
58a9082
Merge branch 'main' into US-108392-AddStaticDataProducer
espentj Sep 15, 2021
37225b9
Merge in from main
espentj Sep 15, 2021
cdd8d14
Update documentation
espentj Sep 15, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions das-producer/src/main/resources/application.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -44,6 +44,14 @@ das:
numberOfLoci: ${NUMBER_OF_LOCI}
numberOfPrePopulatedValues: ${NUMBER_OF_PREPOPULATED_VALUES:100}
amplitudesPrPackage: ${PACKAGE_SIZE:8192}
static:
numberOfShots: ${NUMBER_OF_SHOTS:-1}
secondsToRun: ${SECONDS_TO_RUN:120}
boxUUID: ${BOX_UUID}
opticalPathUUID: ${OPTICAL_PATH_UUID}
disableThrottling: ${DISABLE_THROTTLING:0}
numberOfLoci: ${NUMBER_OF_LOCI}
amplitudesPrPackage: ${PACKAGE_SIZE:8192}
kafka:
config:
client.id: ${DAS_PRODUCER_KAFKA_CONFIG_CLIENT_ID:Simulator}
Expand Down
16 changes: 13 additions & 3 deletions pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -24,6 +24,7 @@
<modules>
<module>simulator-common</module>
<module>simulator-box-unit</module>
<module>static-data-unit</module>
<module>das-producer</module>
</modules>

Expand Down Expand Up @@ -52,6 +53,7 @@
<spring-cloud.version>2020.0.3</spring-cloud.version>
<confluent.version>6.2.0</confluent.version>
<fiberoptics-schemas-version>2.0.2</fiberoptics-schemas-version>
<lombok.version>1.18.20</lombok.version>
<java.version>13</java.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<project.reporting.outputEncoding>UTF-8</project.reporting.outputEncoding>
Expand All @@ -75,6 +77,11 @@
<artifactId>simulator-box-unit</artifactId>
<version>2.0.3-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>com.equinor.fiberoptics</groupId>
<artifactId>static-data-unit</artifactId>
<version>2.0.3-SNAPSHOT</version>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-dependencies</artifactId>
Expand All @@ -100,6 +107,12 @@
<artifactId>kafka-avro-serializer</artifactId>
<version>${confluent.version}</version>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>${lombok.version}</version>
<scope>provided</scope>
</dependency>
</dependencies>
</dependencyManagement>

Expand All @@ -123,7 +136,4 @@
</repository>
</repositories>



</project>

64 changes: 44 additions & 20 deletions readme.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,7 +20,6 @@ found int the same directory as this file.

== Basic data flow


A successful DAS acquisition is made by initially contacting the Stream initiator (INITIATOR_URL variable).
[source,json]
----
Expand Down Expand Up @@ -71,36 +70,20 @@ To notable value here is the partition assignments map. It tells the kafka produ
to a specific partition. It is very important that this is being followed as strict ordering of loci-data can only be guaranteed in
Kafka within the same partition.


== Environment values
Control of the parameters of the process in this example is done via environment values.


.DAS simulator config
|===
|Setting |Default value |Description

|VARIANT_PLUGIN|SimulatorBoxUnit|The implementation producing data for simulation. Possible values are SimulatorBoxUnit and StaticDataUnit. The bean need to be available on classpath as it is loaded at runtime
|VENDOR_CODE| Simulator| The vendor code identifying the vendor
|ACQUISITION_START_VERSION| V1| The schema version used. Can be one of V1,V2
|VARIANT_PLUGIN|SimulatorBoxUnit|The implementation producing data for simulation. The bean need to be available on classpath as it is loaded at runtime
|PACKAGE_SIZE|8192|The number of amplitude delivered pr Kafka message.
|INITIATOR_URL| http://localhost:8080| The API endpoint base URL for the stream initiation service.
|INITIATOR_API_KEY| No value| An API key that needs to be set in the service behind the initiator url e.g. 1aa111a11aa11a0a1a1aa1111a1a1a1a
|KAFKA_SERVER_OVERRIDE| No value| If the Kafka server is on a network that has a specific naming that has no meaning on the calling end (e.g. inside a docker container where you would use localhost:9092) you might get a service name. Here you can override the name with an IP address.
|SCHEMA_REGISTRY_URL_OVERRIDE| No value| If the Schema registry server is on a network that has a specific naming that has no meaning on the calling end (e.g. inside a docker container where you would use http://localhost:8081) you might get a service name. Here you can override the name with an IP address.
|NUMBER_OF_SHOTS| No value| Will override value set on SECONDS_TO_RUN to have a predefined number of shots sent
|SECONDS_TO_RUN| 120| Use this to limit the time the simulator should run. Can be overridden by NUMBER_OF_SHOTS
|PULSE_RATE| 10000| as pr ProdML - laser pulse rate.
|MAX_NYQ_FREQ| 5000| as pr ProdML - max frequency available in signal (Nyquist).
|MIN_NYQ_FREQ| 0| as pr ProdML - min frequency available in signal (Nyquist).
|BOX_UUID| No value| the id (UUID) assigned by the stream initiator server.
|OPTICAL_PATH_UUID| No value| the id (UUID) assigned to the optical fiber where the data is aquisitioned.
|SPATIAL_SAMPLING_INTERVAL| 1.1| as pr ProdML - distance between data channel(loci) on the fiber.
|PULSE_WIDTH| 100.50| as pr ProdML - laser pulse width
|START_LOCUS_INDEX| 0| the point (index as in integer) where locus (channel) 0 is on the fiber.
|DISABLE_THROTTLING| 0| Ignore the time interval between data as pr. sampling frequency and package size. Instead deliver data as fast as possible.
|GAUGE_LENGTH|10.209524| as pr ProdML
|NUMBER_OF_LOCI| No value| The total number of data channels that will be delivered
|DAS_PRODUCER_KAFKA_CONFIG_CLIENT_ID|Simulator| http://kafka.apache.org/documentation.html#producerconfigs
|DAS_PRODUCER_KAFKA_CONFIG_ACKS|1| http://kafka.apache.org/documentation.html#producerconfigs
|DAS_PRODUCER_KAFKA_CONFIG_BATCH_SIZE|2500| http://kafka.apache.org/documentation.html#producerconfigs
Expand All @@ -117,7 +100,6 @@ Note that any env value that starts with DAS_PRODUCER_KAFKA_CONFIG_ will be appl

Details for ProdML can be found here: http://w3.energistics.org/energyML/data/prodml/v2.0/doc/prodml_schema_overview.html


== Building and running

Note:
Expand All @@ -133,7 +115,49 @@ source ./das-producer/defaults.env
java -jar ./das-producer/target/fiberoptics-das-simulator-<version>.jar
----

== Data that can be used in the configuration of simulator box
== Variants

=== Simulation Box Unit

Emulation of an interrogator with random data.

Config name: SimulatorBoxUnit
Config Options:
|===
|Setting |Default value |Description
|BOX_UUID| No value| the id (UUID) assigned by the stream initiator server.
|OPTICAL_PATH_UUID| No value| the id (UUID) assigned to the optical fiber where the data is requisitioned.
|GAUGE_LENGTH|10.209524| as pr ProdML
|SPATIAL_SAMPLING_INTERVAL| 1.1| as pr ProdML - distance between data channel(loci) on the fiber.
|PULSE_WIDTH| 100.50| as pr ProdML - laser pulse width
|START_LOCUS_INDEX| 0| the point (index as in integer) where locus (channel) 0 is on the fiber.
|PULSE_RATE| 10000| as pr ProdML - laser pulse rate.
|MAX_FREQ| 5000| as pr ProdML - max frequency available in signal (Nyquist).
|MIN_FREQ| 0| as pr ProdML - min frequency available in signal (Nyquist).
|NUMBER_OF_LOCI| No value| The total number of data channels that will be delivered
|DISABLE_THROTTLING| 0| Ignore the time interval between data as pr. sampling frequency and package size. Instead deliver data as fast as possible.
|AMPLITUDES_PR_PACKAGE|8192|The number of amplitude delivered pr Kafka message.
|NUMBER_OF_SHOTS| No value| Will override value set on SECONDS_TO_RUN to have a predefined number of shots sent
|SECONDS_TO_RUN| 120| Use this to limit the time the simulator should run. Can be overridden by NUMBER_OF_SHOTS
|===

=== Static data unit

An emulated interrogator that produces static data. Useful for testing that known data ends up as expected.
Config name: StaticDataUnit
Config Options:
|===
|Setting |Default value |Description
|BOX_UUID| No value| the id (UUID) assigned by the stream initiator server.
|OPTICAL_PATH_UUID| No value| the id (UUID) assigned to the optical fiber where the data is requisitioned.
|NUMBER_OF_LOCI| No value| The total number of data channels that will be delivered
|AMPLITUDES_PR_PACKAGE|8192|The number of amplitude delivered pr Kafka message.
|DISABLE_THROTTLING| 0| Ignore the time interval between data as pr. sampling frequency and package size. Instead deliver data as fast as possible.
|NUMBER_OF_SHOTS| No value| Will override value set on SECONDS_TO_RUN to have a predefined number of shots sent
|SECONDS_TO_RUN| 120| Use this to limit the time the simulator should run. Can be overridden by NUMBER_OF_SHOTS
|===

==== Data that can be used in the configuration of simulator box

Fiber optic path UUIDs:
[source,text]
Expand Down
3 changes: 0 additions & 3 deletions simulator-box-unit/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -46,12 +46,9 @@
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
</dependency>

<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
<version>1.18.20</version>
<scope>provided</scope>
</dependency>

<!-- Test dependencies -->
Expand Down
11 changes: 5 additions & 6 deletions ...java/com/equinor/fiberoptics/das/producer/variants/simulatorboxunit/SimulatorBoxUnit.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,8 @@
import java.time.Instant;
import java.util.ArrayList;
import java.util.List;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

/**
* This is an example DAS box unit implementation.
Expand Down Expand Up @@ -76,12 +78,9 @@ public Flux<List<PartitionKeyValueEntry<DASMeasurementKey, DASMeasurement>>> pro
.interval(Duration.ofMillis(delay))
.take(take)
.map(tick -> {
List<PartitionKeyValueEntry<DASMeasurementKey, DASMeasurement>> data = new ArrayList<>();

for (int currentLocus = 0; currentLocus < _configuration.getNumberOfLoci(); currentLocus++) {
var message = constructAvroObjects(currentLocus, dataCache.getFloat());
data.add(message);
}
List<PartitionKeyValueEntry<DASMeasurementKey, DASMeasurement>> data = IntStream.range(0, _configuration.getNumberOfLoci())
.mapToObj(currentLocus -> constructAvroObjects(currentLocus, dataCache.getFloat()))
.collect(Collectors.toList());

_stepCalculator.increment(1);
return data;
Expand Down
132 changes: 132 additions & 0 deletions static-data-unit/pom.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,132 @@
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>

<parent>
<groupId>com.equinor.fiberoptics</groupId>
<artifactId>parent</artifactId>
<version>2.0.3-SNAPSHOT</version>
</parent>

<artifactId>static-data-unit</artifactId>
<description>Plugin for testing streaming functionality</description>
<packaging>jar</packaging>


<developers>
<developer>
<id>iknu</id>
<name>Inge Knudsen</name>
<email>[email protected]</email>
</developer>
</developers>

<dependencies>
<dependency>
<groupId>com.equinor.fiberoptics</groupId>
<artifactId>simulator-common</artifactId>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-context</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.integration</groupId>
<artifactId>spring-integration-feed</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-starter-stream-kafka</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-configuration-processor</artifactId>
</dependency>
<dependency>
<groupId>io.confluent</groupId>
<artifactId>kafka-avro-serializer</artifactId>
</dependency>
<dependency>
<groupId>org.projectlombok</groupId>
<artifactId>lombok</artifactId>
</dependency>

<!-- Test dependencies -->
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>org.springframework</groupId>
<artifactId>spring-test</artifactId>
<scope>test</scope>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit</artifactId>
<scope>test</scope>
</dependency>
</dependencies>

<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>3.8.1</version>
<configuration>
<source>${java.version}</source>
<target>${java.version}</target>
<encoding>${project.build.sourceEncoding}</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-resources-plugin</artifactId>
<version>3.1.0</version>
<configuration>
<encoding>${project.build.sourceEncoding}</encoding>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.22.2</version>
</plugin>
<plugin>
<groupId>org.codehaus.mojo</groupId>
<artifactId>license-maven-plugin</artifactId>
<version>1.9</version>
<configuration>
<verbose>false</verbose>
<includes>
<includes>**/*.java</includes>
</includes>
</configuration>
<executions>
<execution>
<id>generate-license-headers</id>
<goals>
<goal>update-file-header</goal>
</goals>
<phase>process-sources</phase>
<configuration>
<licenseName>apache_v2</licenseName>
<roots>
<root>src/main/java</root>
<root>src/test/java</root>
</roots>
<addJavaLicenseAfterPackage>false</addJavaLicenseAfterPackage>
<processStartTag>========================LICENSE_START=================================</processStartTag>
<processEndTag>=========================LICENSE_END==================================</processEndTag>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>

</project>
Loading