This repository has been archived by the owner on Sep 9, 2022. It is now read-only.
-
Notifications
You must be signed in to change notification settings - Fork 9
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Merge pull request #51 from jpatton-USGS/hazdevbroker-update
Bug fix and Hazdevbroker update
- Loading branch information
Showing
51 changed files
with
741 additions
and
182 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -179,7 +179,7 @@ if (BUILD_GLASS-BROKER-APP) | |
| if (GIT_CLONE_PUBLIC) | ||
| ExternalProject_Add(HazdevBroker | ||
| GIT_REPOSITORY https://github.com/usgs/hazdev-broker.git | ||
| #GIT_TAG working | ||
| GIT_TAG v0.2.1 | ||
| TIMEOUT 10 | ||
| CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${INSTALL_LOCATION} | ||
| -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} | ||
|
|
@@ -191,7 +191,7 @@ if (BUILD_GLASS-BROKER-APP) | |
| else() | ||
| ExternalProject_Add(HazdevBroker | ||
| GIT_REPOSITORY [email protected]:usgs/hazdev-broker.git | ||
| #GIT_TAG working | ||
| GIT_TAG v0.2.1 | ||
| TIMEOUT 10 | ||
| CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${INSTALL_LOCATION} | ||
| -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} | ||
|
|
@@ -352,6 +352,7 @@ if (BUILD_GLASS-BROKER-APP) | |
| CMAKE_ARGS -DCMAKE_INSTALL_PREFIX=${INSTALL_LOCATION} | ||
| -DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE} | ||
| -DCMAKE_MODULE_PATH=${CMAKE_MODULE_PATH} | ||
| -DRAPIDJSON_PATH=${RAPIDJSON_PATH} | ||
| -DLIBRDKAFKA_PATH=${LIBRDKAFKA_PATH} | ||
| -DLIBRDKAFKA_C_LIB=${LIBRDKAFKA_C_LIB} | ||
| -DLIBRDKAFKA_CPP_LIB=${LIBRDKAFKA_CPP_LIB} | ||
|
|
||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,7 @@ | ||
| # include_rapidjson.cmake - a CMake script that finds the rapidjson | ||
| # include files | ||
| # | ||
| # rapidjson | ||
| set(RAPIDJSON_PATH "${CMAKE_CURRENT_LIST_DIR}/lib/rapidjson" CACHE PATH "Path to rapidjson") | ||
|
|
||
| include_directories(${RAPIDJSON_PATH}) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,4 +1,4 @@ | ||
| # version.cmake - a CMake script that defines the overall project version | ||
| set (PROJECT_VERSION_MAJOR 1) | ||
| set (PROJECT_VERSION_MINOR 0) | ||
| set (PROJECT_VERSION_PATCH 1) | ||
| set (PROJECT_VERSION_PATCH 2) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,76 @@ | ||
| # gen-travel-times-app | ||
|
|
||
| gen-travel-times-app is an application that uses the glasscore traveltime libraries to generate the traveltime lookup files (.trv) used by neic-glass3 from a model file. Please note that this application is currently not optimized, and is **extremely** slow. | ||
|
|
||
| ## Building | ||
|
|
||
| To build gen-travel-times-app, set the `BUILD_GEN-TRAVELTMES-APP` option equal to true (1) in the cmake command or GUI. | ||
|
|
||
| ## Configuration | ||
|
|
||
| An example configuration for gen-travel-times-app is available in the [gen-travel-times-app params directory](https://github.com/usgs/neic-glass3/tree/master/gen-travel-times-app/params) | ||
|
|
||
| ### gen-travel-times.d | ||
|
|
||
| ```json | ||
| { | ||
| "Configuration": "gen-travel-times-app", | ||
| "FileExtension": ".trv", | ||
| "OutputPath": "./", | ||
| "Model": "./params/ak135_mod.d", | ||
| "Branches": [ | ||
| { | ||
| "Cmd": "GenerateTraveltime", | ||
| "Branch": "P", | ||
| "Rays": [ | ||
| "Pup", | ||
| "P", | ||
| "Pdiff" | ||
| ], | ||
| "DeltaTimeWarp": { | ||
| "MinimumDistance": 0.0, | ||
| "MaximumDistance": 360.0, | ||
| "SlopeDecayConstant": 0.10, | ||
| "SlopeZero": 0.05, | ||
| "SlopeInfinite": 1.0 | ||
| }, | ||
| "DepthTimeWarp": { | ||
| "MinimumDepth": -10.0, | ||
| "MaximumDepth": 800.0, | ||
| "SlopeDecayConstant": 0.10, | ||
| "SlopeZero": 1.0, | ||
| "SlopeInfinite": 10.0 | ||
| } | ||
| } | ||
| ] | ||
| } | ||
| ``` | ||
|
|
||
| * **FileExtension** - The file extension to use for travel time files | ||
| * **OutputPath** - The output directory to write travel time files to | ||
| * **Model** - The earth model file to use | ||
| * **Branches** - The list of travel time files to generate | ||
| * **Branch** - The branch name for the travel time file | ||
| * **Rays** - The rays (phases) to use in generating the file | ||
| * **DeltaTimeWarp** - The distance warp for this travel time file | ||
| * **MinimumDistance** - Start of warp in degrees | ||
| * **MaximumDistance** - End of warp in degrees | ||
| * **SlopeDecayConstant** - The decay exponent for the warp | ||
| * **SlopeZero** - The warp slope value at minimum | ||
| * **SlopeInfinite** - The warp slope value at maximum | ||
| * **DepthTimeWarp** - The depth warp for this travel time file | ||
| * **MinimumDepth** - Start of warp in kilometers | ||
| * **MaximumDepth** - End of warp in kilometers | ||
| * **SlopeDecayConstant** - The decay exponent for the warp | ||
| * **SlopeZero** - The warp slope value at minimum | ||
| * **SlopeInfinite** - The warp slope value at maximum | ||
|
|
||
| ## Documentation | ||
|
|
||
| Further documentation of the gen-traveltimes-app software is available [here](https://usgs.github.io/neic-glass3/gen-travel-times-app/html/) | ||
|
|
||
| ## Running | ||
|
|
||
| To run gen-travel-times-app, use the following command: `gen-travel-times-app <configfile> [logname]` where `<configfile>` is the required path the the gen-travel-times.d configuration fileand `[logname]` is an optional command that when present specifies the log file name and enables logging. | ||
|
|
||
| gen-travel-times-app uses the environment variable GLASS_LOG to define the location to write log files |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,100 @@ | ||
| # glass-app | ||
|
|
||
| glass-app is an implementation of the neic-glass3 libraries that reads input from a file system directory, and writes output to a file system directory, with a static stationlist and configuration. | ||
|
|
||
| ## Building | ||
|
|
||
| To build glass-app, set the `BUILD_GLASS-APP` option equal to true (1) in the cmake command or GUI. | ||
|
|
||
| ## Configuration | ||
|
|
||
| An example configuration for glass-app is available in the [glass-app params directory](https://github.com/usgs/neic-glass3/tree/master/glass-app/params) | ||
|
|
||
| ### Glass-App | ||
|
|
||
| ```json | ||
| { | ||
| "Configuration":"glass-app", | ||
| "LogLevel":"debug", | ||
| "ConfigDirectory":"./params", | ||
| "InitializeFile":"initialize.d", | ||
| "StationList":"stationlist.d", | ||
| "GridFiles":[ | ||
| "ak_grid.d", | ||
| ], | ||
| "InputConfig":"input.d", | ||
| "OutputConfig":"output.d" | ||
| } | ||
| ``` | ||
|
|
||
| * **LogLevel** - Sets the minimum logging level, trace, debug, info, warning, error, or criticalerror | ||
| * **ConfigDirectory** - The path to directory containing the other glass subcomponent configuration files | ||
| * **InitializeFile** - Configuration file containing the neic-glass3 Algorithm configuration | ||
| * **StationList** - File containing the initial neic-glass3 station list | ||
| * **GridFiles** - One or more files defining detection grids used by neic-glass3 | ||
| * **InputConfig** - Configuration file containing the file input configuration | ||
| * **OutputConfig** - Configuration file containing the file output configuration | ||
|
|
||
| ### input.d | ||
|
|
||
| ```json | ||
| { | ||
| "Configuration":"GlassInput", | ||
| "InputDirectory":"./input", | ||
| "ErrorDirectory":"./error", | ||
| "ArchiveDirectory":"./archive", | ||
| "Format":"gpick", | ||
| "QueueMaxSize":1000, | ||
| "ShutdownWhenNoData":true, | ||
| "ShutdownWait":300, | ||
| "DefaultAgencyID":"US", | ||
| "DefaultAuthor":"glassConverter" | ||
| } | ||
| ``` | ||
|
|
||
| * **InputDirectory** - The directory to read input files from | ||
| * **ErrorDirectory** - The optional directory to archive erroneous input files to. | ||
| * **ArchiveDirectory** - The optional directory to archive input files to. | ||
| * **Format** - The format to accept. glass-app currently understands the gpick, jsonpick, jsonhypo, and ccdata (dat) formats. Note that the only way to use multiple inputs (Picks, Correlations, and Detections at the same time) | ||
| * **QueueMaxSize** - The maximum size of the input queue | ||
| * **ShutdownWhenNoData** - Optional Flag indicating whether to shut down when there is no more input data | ||
| * **ShutdownWait** - The time in seconds to wait before shutting down due to there being no input data | ||
| * **DefaultAgencyID** - The default agency identifier to use when converting data to json | ||
| * **DefaultAuthor** - The default author to use when converting data to json | ||
|
|
||
| ### Output | ||
|
|
||
| ```json | ||
| { | ||
| "Configuration":"GlassOutput", | ||
| "PublicationTimes":[20,180], | ||
| "PublishOnExpiration":true, | ||
| "OutputDirectory":"./output", | ||
| "OutputFormat":"json", | ||
| "TimeStampFileName":true, | ||
| "OutputAgencyID":"US", | ||
| "OutputAuthor":"glass" | ||
| } | ||
| ``` | ||
|
|
||
| * **PublicationTimes** - The time(s), in seconds since the detections was first found, to publish | ||
| * **PublishOnExpiration** - Flag indicating whether to always publish a final version of a detection when it expires out of glass | ||
| * **OutputDirectory** - The directory to write output to | ||
| * **OutputFormat** - The format to write output in for now, the only format is json | ||
| * **TimeStampFileName** - Optional flag to define whether to timestamp output file names, defaults to true | ||
| * **OutputAgencyID** - The agency identifier to use when generating output data | ||
| * **OutputAuthor** - The author to use when generating output data | ||
|
|
||
| ### neic-glass3 Algorithm | ||
|
|
||
| For neic-glass3 algorithmic configuration, see [neic-glass3 Configuration](https://github.com/usgs/neic-glass3/blob/master/doc/GlassConfiguration.md). | ||
|
|
||
| ## Documentation | ||
|
|
||
| Further documentation of the glass-app software is available [here](https://usgs.github.io/neic-glass3/glass-app/html/) | ||
|
|
||
| ## Running | ||
|
|
||
| To run glass-app, use the following command: `glass-app <configfile> [logname] [noconsole]` where `<configfile>` is the required path the the glass.d configuration file, `[logname]` is an optional command that defining an alternate name for the glass-app log file, and `[noconsole]` is an optional command specifying that glass-app should not write messages to the console. | ||
|
|
||
| glass-app uses the environment variable GLASS_LOG to define the location to write log files |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,142 @@ | ||
| # glass-broker-app | ||
|
|
||
| glass-broker-app is an implementation of the neic-glass3 libraries that reads | ||
| input, station, and configuration updates from | ||
| [HazDev Broker](https://github.com/usgs/hazdev-broker) topics, and writes output | ||
| and station information requests to HazDev Broker topics. | ||
|
|
||
| ## Building | ||
|
|
||
| To build glass-broker-app, set the `BUILD_GLASS-BROKER-APP` option equal to | ||
| true (1) in the cmake command or GUI. Building glass-broker-app requires | ||
| dependencies, such as [hazdev-broker](https://github.com/usgs/hazdev-broker) | ||
| and [librdkafka](https://github.com/edenhill/librdkafka). | ||
|
|
||
| ## Configuration | ||
|
|
||
| An example configuration for glass-broker-app is available in the | ||
| [glass-broker-app params directory](https://github.com/usgs/neic-glass3/tree/master/glass-broker-app/params) | ||
|
|
||
| ### glass.d | ||
|
|
||
| ```json | ||
| { | ||
| "Configuration":"glass-broker-app", | ||
| "LogLevel":"debug", | ||
| "ConfigDirectory":"./params", | ||
| "StationList":"stationlist.d", | ||
| "InitializeFile":"initialize.d", | ||
| "GridFiles":[ | ||
| "ak_grid.d", | ||
| ], | ||
| "InputConfig":"input.d", | ||
| "OutputConfig":"output.d" | ||
| } | ||
| ``` | ||
|
|
||
| * **LogLevel** - Sets the minimum logging level, trace, debug, info, warning, error, or criticalerror | ||
| * **ConfigDirectory** - The path to directory containing the other glass subcomponent configuration files | ||
| * **InitializeFile** - Configuration file containing the neic-glass3 Algorithm configuration | ||
| * **StationList** - File containing the initial neic-glass3 station list | ||
| * **GridFiles** - One or more files defining detection grids used by neic-glass3 | ||
| * **InputConfig** - Configuration file containing the broker input configuration | ||
| * **OutputConfig** - Configuration file containing the broker output configuration | ||
|
|
||
| ### input.d | ||
|
|
||
| ```json | ||
| { | ||
| "Configuration":"GlassInput", | ||
| "HazdevBrokerConfig": { | ||
| "Type":"ConsumerConfig", | ||
| "Properties":{ | ||
| "client.id":"glass3Default", | ||
| "group.id":"1", | ||
| "metadata.broker.list":"<HazDev Brokers>", | ||
| "enable.auto.commit":"false" | ||
| } | ||
| }, | ||
| "HazdevBrokerTopicConfig": { | ||
| "Type":"TopicConfig", | ||
| "Properties":{ | ||
| "auto.commit.enable":"false", | ||
| "auto.offset.reset":"latest" | ||
| } | ||
| }, | ||
| "Topics":["Dev-RayPicker-1", "Station-Data"], | ||
| "HeartbeatDirectory":"./", | ||
| "BrokerHeartbeatInterval":300, | ||
| "QueueMaxSize":1000, | ||
| "DefaultAgencyID":"US", | ||
| "DefaultAuthor":"glassConverter" | ||
| } | ||
| ``` | ||
|
|
||
| * **HazdevBrokerConfig** - The HazDev Broker configuration to use for input, see [HazDev-Broker](https://github.com/usgs/hazdev-broker) | ||
| * **HazdevBrokerTopicConfig** - The HazDev Broker topic configuration to use, see [HazDev-Broker](https://github.com/usgs/hazdev-broker) | ||
| * **Topics** - The HazDev Broker topic(s) to receive input data from | ||
| * **HeartbeatDirectory** - An optional key defining where HazDev Broker heartbeat files should be written, if not defined, heartbeat files will not be written. | ||
| * **BrokerHeartbeatInterval** - An optional key defining the interval in seconds to expect HazDev Broker heartbeats, if not defined, heatbeats are not expected. | ||
| * **QueueMaxSize** - The maximum size of the input queue | ||
| * **DefaultAgencyID** - The default agency identifier to use when converting data to json | ||
| * **DefaultAuthor** - The default author to use when converting data to json | ||
|
|
||
| ### output.d | ||
|
|
||
| ```json | ||
| { | ||
| "Configuration":"GlassOutput", | ||
| "PublishOnExpiration":false, | ||
| "PublicationTimes":[20,180], | ||
| "HazdevBrokerConfig": { | ||
| "Type":"ProducerConfig", | ||
| "Properties":{ | ||
| "client.id":"glass3Default", | ||
| "group.id":"0", | ||
| "metadata.broker.list":"<HazDev Brokers>", | ||
| "retries":"0" | ||
| } | ||
| }, | ||
| "OutputTopics":[ | ||
| { | ||
| "TopicName":"OK", | ||
| "TopLatitude":38.0, | ||
| "LeftLongitude":-101.0, | ||
| "BottomLatitude":33.0, | ||
| "RightLongitude":-94.0 | ||
| }, | ||
| { "TopicName":"DefaultWorld" | ||
| } | ||
| ], | ||
| "StationRequestTopic":"Station-Lookup", | ||
| "SiteListDelay":7200, | ||
| "StationFile":"./params/stationlist.d", | ||
| "OutputAgencyID":"US", | ||
| "OutputAuthor":"glass" | ||
| } | ||
| ``` | ||
|
|
||
| * **PublishOnExpiration** - Flag indicating whether to always publish a final version of a detection when it expires out of glass | ||
| * **PublicationTimes** - The time(s), in seconds since the detections was first found, to publish | ||
| * **HazdevBrokerConfig** - The HazDev Broker configuration to use for output, see [HazDev-Broker](https://github.com/usgs/hazdev-broker) | ||
| * **OutputTopics** - The list of HazDev Broker topics to write output to. | ||
| * **TopicName** - | ||
| * **TopLatitude** - | ||
| * **LeftLongitude** - | ||
| * **BottomLatitude** - | ||
| * **RightLongitude** - | ||
| * **StationRequestTopic** - Optional HazDev Broker topic to request station information. | ||
| * **SiteListDelay** - Optional delay between writing updated station files to disk | ||
| * **StationFile** - Optional file name of updated station file | ||
| * **OutputAgencyID** - The agency identifier to use when generating output data | ||
| * **OutputAuthor** - The author to use when generating output data | ||
|
|
||
| ### neic-glass3 Algorithm | ||
|
|
||
| For neic-glass3 algorithmic configuration, see [GLASS 3 Configuration](https://github.com/usgs/neic-glass3/blob/master/doc/GlassConfiguration.md). | ||
|
|
||
| ## Running | ||
|
|
||
| To run glass-broker-app, use the following command: `glass-broker-app <configfile> [logname] [noconsole]` where `<configfile>` is the required path to the glass.d configuration file, `[logname]` is an optional command that defining an alternate name for the glass-broker-app log file, and `[noconsole]` is an optional command specifying that glass-broker-app should not write messages to the console. | ||
|
|
||
| glass-broker-app uses the environment variable GLASS_LOG to define the location to write log files |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.