From 0b5eea371c120254866e7744fd4592c7d792d9a7 Mon Sep 17 00:00:00 2001 From: Andrew O'Neill Date: Mon, 27 Feb 2017 15:49:36 -0800 Subject: [PATCH 1/7] docs/elwin-usage: added initial docs for elwin usage --- docs/elwin-usage.md | 74 +++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/elwin-usage.md diff --git a/docs/elwin-usage.md b/docs/elwin-usage.md new file mode 100644 index 0000000..df8a58d --- /dev/null +++ b/docs/elwin-usage.md @@ -0,0 +1,74 @@ +# Elwin Usage + +* [Elwin endpoints](#elwin-endpoints) +* [Elwin request](#elwin-request) +* [Elwin response](#elwin-response) + +## Elwin endpoints + +Elwin uses two endpoints for experiments: _dev_ and _prod_. The _prod_ +endpoint will serve tests that are considered live to customers. The +_dev_ endpoint can be used to preview experiments before enabling it +for your customers. + +`http://elwin.ttoapps.aws.cloud.nordstorm.net` is our current _prod_ +endpoint. + +`http://dev.elwin.aws.cloud.nordstrom.net` is our current _dev_ +endpoint. + +We are in the process of migrating to a supported Kubernetes cluster. +When that migration occurs we will update our _prod_ endpoint to +`http://prod.elwin.aws.cloud.nordstrom.net`. + +## Elwin request + +Elwin's request structure is very simple. In it's current state, +clients will make a GET request to one of the endpoints. The request +requires two params. One is specifies the team making the request. The +second specifies the user's id. + +The team name can currently be specified in query params `label`, +`teamid`, or `group-id`. If any of those are params are not blank, +they will be used to filter the experiments. If more than one is used, +it will prefer them in the order specified before. + +The user id is specified in the param `userid`. In most cases this +should be the `ExperimentID` from the `experiments` cookie on web +requests. + +## Elwin response + +All the experiments a user qualifies for will be returned in the +experiment response. The response is JSON. + +```javascript +{ + "experiments": { + "some-experiment": { + "namespace": "aaaaaa", + "params": { + "my-param-name": "value-for-my-user" + } + }, + "another-experiment": { + "namespace": "bbbbbb", + "params": { + "first-param": "some-value", + "second-param": "another-value" + } + } + } +} +``` + +The top level of the object will contain only a single key, +`experiments`. The `experiments` key contains a map of experiment +names to experiment values, represented as an object. Experiment names +will be keys in the object. Experiment values will be objects with two +keys. `namespace`, the first key, contains a string value. It is not +essential to the experiment but is required for the data collection. +`params`, the second key, contains an object of param names and param +values. If you are running an A/B/N test there will only be one key. +If you are running a multivariate test then there be keys for each +arm. The values for params will always be returned as strings. \ No newline at end of file From 69ab440f1dafb4546060a050dc68e67fb2c87192 Mon Sep 17 00:00:00 2001 From: Andrew O'Neill Date: Mon, 27 Feb 2017 15:55:30 -0800 Subject: [PATCH 2/7] docs: add request example --- docs/elwin-usage.md | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/docs/elwin-usage.md b/docs/elwin-usage.md index df8a58d..d83dac7 100644 --- a/docs/elwin-usage.md +++ b/docs/elwin-usage.md @@ -15,7 +15,7 @@ for your customers. endpoint. `http://dev.elwin.aws.cloud.nordstrom.net` is our current _dev_ -endpoint. +endpoint. This is in the We are in the process of migrating to a supported Kubernetes cluster. When that migration occurs we will update our _prod_ endpoint to @@ -37,6 +37,13 @@ The user id is specified in the param `userid`. In most cases this should be the `ExperimentID` from the `experiments` cookie on web requests. +A request for experiments that are labeled with `ato` for the userid +`andrew` would look like the following. + +``` +http://dev.elwin.aws.cloud.nordstrom.net/?label=ato&userid=andrew +``` + ## Elwin response All the experiments a user qualifies for will be returned in the From 305b29110f07b9da8375f9731627fdf0f94ef5f6 Mon Sep 17 00:00:00 2001 From: Andrew O'Neill Date: Mon, 27 Feb 2017 18:46:06 -0800 Subject: [PATCH 3/7] docs: better examples and notes --- docs/elwin-usage.md | 50 ++++++++++++++++++++++++++++++--------------- 1 file changed, 33 insertions(+), 17 deletions(-) diff --git a/docs/elwin-usage.md b/docs/elwin-usage.md index d83dac7..059c3c8 100644 --- a/docs/elwin-usage.md +++ b/docs/elwin-usage.md @@ -14,12 +14,17 @@ for your customers. `http://elwin.ttoapps.aws.cloud.nordstorm.net` is our current _prod_ endpoint. +> We are in the process of migrating to a supported Kubernetes +> cluster. When that migration occurs we will update our _prod_ +> endpoint to `http://prod.elwin.aws.cloud.nordstrom.net`. + `http://dev.elwin.aws.cloud.nordstrom.net` is our current _dev_ -endpoint. This is in the +endpoint. -We are in the process of migrating to a supported Kubernetes cluster. -When that migration occurs we will update our _prod_ endpoint to -`http://prod.elwin.aws.cloud.nordstrom.net`. +> There are other dev endpoints but they should pointed to the same +> service as the above. They are +> `http://elwin.k8s-a.ttoapps.aws.cloud.nordstrom.net` and +> `http://elwin-test.ttoapps.aws.cloud.nordstrom.net`. ## Elwin request @@ -52,30 +57,41 @@ experiment response. The response is JSON. ```javascript { "experiments": { - "some-experiment": { + "personalized-header-experiment": { "namespace": "aaaaaa", "params": { - "my-param-name": "value-for-my-user" + "personalized": "default" } }, - "another-experiment": { + "button-experiment": { "namespace": "bbbbbb", "params": { - "first-param": "some-value", - "second-param": "another-value" + "button-color": "blue", + "button-size": "large" } } } } ``` +> In this example there are two experiments: +> `personalized-header-experiment` and `button-experiment`. In +> `personalized-header-experiment` the user was hashed into the +> `default` experience. In `button-experiment` they were hashed into +> `blue` + `large` MVT experience. + The top level of the object will contain only a single key, `experiments`. The `experiments` key contains a map of experiment -names to experiment values, represented as an object. Experiment names -will be keys in the object. Experiment values will be objects with two -keys. `namespace`, the first key, contains a string value. It is not -essential to the experiment but is required for the data collection. -`params`, the second key, contains an object of param names and param -values. If you are running an A/B/N test there will only be one key. -If you are running a multivariate test then there be keys for each -arm. The values for params will always be returned as strings. \ No newline at end of file +names to experiment values, represented as an object. + +Experiment names will be keys in the `experiments` object. The values +for the experiment names will be objects with two keys. `namespace`, +the first key, contains a string value. It is not essential to the +experiment but is required for the data collection. `params`, the +second key, contains an object of param names and param values. + +The `params` object has keys that correspond to param names. The +values of these keys will be the experience the user has been hashed +into. If you are running an A/B/N test there will only be one key. If +you are running a multivariate test then there be keys for each arm. +The values for params will always be returned as strings. \ No newline at end of file From fac1a4633d6adfb5703a666691d9969c24d53eb2 Mon Sep 17 00:00:00 2001 From: Andrew O'Neill Date: Mon, 6 Mar 2017 21:11:58 -0800 Subject: [PATCH 4/7] update docs for new label support --- docs/elwin-usage.md | 27 ++++++++++++++++++++------- 1 file changed, 20 insertions(+), 7 deletions(-) diff --git a/docs/elwin-usage.md b/docs/elwin-usage.md index 059c3c8..24a379d 100644 --- a/docs/elwin-usage.md +++ b/docs/elwin-usage.md @@ -33,20 +33,33 @@ clients will make a GET request to one of the endpoints. The request requires two params. One is specifies the team making the request. The second specifies the user's id. -The team name can currently be specified in query params `label`, -`teamid`, or `group-id`. If any of those are params are not blank, -they will be used to filter the experiments. If more than one is used, -it will prefer them in the order specified before. +The team name can currently be specified in query params `team`, +`label`, `teamid`, or `group-id`. If any of those params are not +blank, they will be used to filter the experiments. The user id is specified in the param `userid`. In most cases this should be the `ExperimentID` from the `experiments` cookie on web requests. -A request for experiments that are labeled with `ato` for the userid -`andrew` would look like the following. +You can also supply other query params to match labels on your +experiments. For example if you are running a test that should only be +shown to desktop users, you could set the label `platform` with the +value `desktop` on your experiment. When you query for desktop +experiments you would then include the `platform=desktop` query param. +Another example is if you wanted to run a test internal only before +deploying it to customers you could add a label +`traffic-source=internal` and query for it the same way. You can query +for multiple values for the same label key by repeating the query in +the request. For example, `?env=prod&env=dev`. The query params create +an *and* selection on the labels of your experiments. The results +returned will be the union of all experiments whose labels match. + +The full request for experiments that are for the ato team in the dev +and prod _env_ironment for customers browsing on desktop platform for +the userid `andrew` would look like the following. ``` -http://dev.elwin.aws.cloud.nordstrom.net/?label=ato&userid=andrew +http://dev.elwin.aws.cloud.nordstrom.net/?team=ato&env=prod&env=dev&platform=desktop&userid=andrew ``` ## Elwin response From 595d223b3c366b84cf5c997e4a024e883910b50b Mon Sep 17 00:00:00 2001 From: Andrew O'Neill Date: Mon, 6 Mar 2017 21:15:19 -0800 Subject: [PATCH 5/7] fix unformatted env --- docs/elwin-usage.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/elwin-usage.md b/docs/elwin-usage.md index 24a379d..5ff84b5 100644 --- a/docs/elwin-usage.md +++ b/docs/elwin-usage.md @@ -55,7 +55,7 @@ an *and* selection on the labels of your experiments. The results returned will be the union of all experiments whose labels match. The full request for experiments that are for the ato team in the dev -and prod _env_ironment for customers browsing on desktop platform for +and prod environment for customers browsing on desktop platform for the userid `andrew` would look like the following. ``` From bdbf627e5e9452ba57e2685a931ff1314a6ff0fc Mon Sep 17 00:00:00 2001 From: Andrew O'Neill Date: Wed, 8 Mar 2017 13:09:02 -0800 Subject: [PATCH 6/7] initial building elwin docs --- docs/building-elwin.md | 109 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 109 insertions(+) create mode 100644 docs/building-elwin.md diff --git a/docs/building-elwin.md b/docs/building-elwin.md new file mode 100644 index 0000000..df3b85b --- /dev/null +++ b/docs/building-elwin.md @@ -0,0 +1,109 @@ +# Compiling and Running Elwin from Source + +# 1. Overview + +Elwin is an executable that can perform a/b/c and multivariate +testing. The main library that performs the majority of the work is +called `choices`. It is written in go. Elwin contains the server logic +and evaluates experiments based on the requests it receives. There are +two binaries that make up a basic elwin deployment. + +# 2. Compiling + +## 2.1 Go compiler + +In order to compile an Elwin executable you will need a go compiler. I +use the most up to date compiler version, currently go1.8. You can +download go at [https://golang.org/dl/](https://golang.org/dl/). + +## 2.2 Getting source code + +The first step will be to clone the elwin repo locally. Go requires +files to be placed in certain directories based on your `GOPATH` +environment variable. If you are using the default `GOPATH` in go1.8 +It will be `$HOME/go`. There are two options for getting the elwin +code. One option is to use Go's built in `get` command. The second +option is to manually download and create the necessary directory +structure. + +### 2.2.1 Using `go get` + +In your terminal run the following command. + +```bash +go get github.com/Nordstrom/choices +``` + +> This requires `go` and `git` to be installed + +You should find the elwin files at +`$GOPATH/src/github.com/Nordstrom/choices` if you have `GOPATH` set or +`$HOME/go/src/github.com/Nordtsrom/choices` if you are using the +default `GOPATH` in go1.8. + +### 2.2.2 Manual `git clone` + +You should check if your `GOPATH` is set. + +```bash +echo $GOPATH +# If nothing is displayed run the following commands. +mkdir $HOME/go +export GOPATH=$HOME/go +``` + +> If you set the `GOPATH` it will only be set for this terminal. If +> you close the terminal or start a new session it will need to be set +> again. + +Next you need to create the directory structure that go expects to +hold the files. Then you can clone the repo. + +```bash +mkdir -p $GOPATH/src/github.com/Nordstrom/ +cd $GOPATH/src/github.com/Nordstrom/ +git clone https://github.com/Nordstrom/choices +``` + +## 2.3 Compiling source components + +### 2.3.1 Compiling Elwin binary + +Assuming you have the downloaded the code and set your `GOPATH`, now +you can compile the elwin executable. To compile an executable that +can run on your local machine you could run the following. + +```bash +cd $GOPATH/src/github.com/Nordstrom/choices/cmd/elwin +go build +``` + +### 2.3.2 Compiling Storage binary + +The most up to date storage binary is the bolt-store implementation. +You compile this in a similar way to elwin. + +```bash +cd $GOPATH/src/github.com/Nordstrom/choices/cmd/bolt-store +go build +``` + +# 3. Running Elwin + +> TODO: use `spf13/viper` for configuration + +# 3.1 Running Elwin Locally + +To run elwin locally you will need to supply some configuration values so the ports don't collide. + +```bash +cd $GOPATH/src/github.com/Nordstrom/choices/cmd/bolt-store +./bolt-store +``` + +In a separate terminal, run the following. + +```bash +cd $GOPATH/src/github.com/Nordstrom/choices/cmd/elwin +JSON_ADDRESS=:8082 GRPC_ADDRESS=:8083 MONGO_ADDRESS=:8080 ./elwin +``` From 59a1cc0d2bf0af0fd2d9abff31c8ffec11297d54 Mon Sep 17 00:00:00 2001 From: Andrew O'Neill Date: Mon, 13 Mar 2017 17:17:43 -0700 Subject: [PATCH 7/7] doc/something-broke: added some info for when stuff is broken --- docs/something-broke.md | 74 +++++++++++++++++++++++++++++++++++++++++ 1 file changed, 74 insertions(+) create mode 100644 docs/something-broke.md diff --git a/docs/something-broke.md b/docs/something-broke.md new file mode 100644 index 0000000..1d0473f --- /dev/null +++ b/docs/something-broke.md @@ -0,0 +1,74 @@ +# Something Broke + +Steps for checking what's broken. + +## A test I created is not showing up when I query elwin. + +1. Check if your labels match the query params + +```bash +curl prod.elwin.aws.cloud.nordstrom.net/?userid=andrew&team=blah&env=dev +``` + +Your experiment should have labels that match all the query params +except `userid`. In this example, your experiment would need labels +`team` and `env` with the values `blah` and `dev`. + +2. Check the test was actually created. + +```bash +# dev environment +curl -X POST -d '{"environment": 0}' json-gateway.ato.platform.prod.aws.cloud.nordstrom.net/api/v1/all +# prod environment +curl -X POST -d '{"environment": 1}' json-gateway.ato.platform.prod.aws.cloud.nordstrom.net/api/v1/all +``` + +If you do not see the experiment in either output then it needs to be +recreated. Use the [form][neo-form] to create an experiment. + +If your test is in the wrong environment, you should use +[houston][houston] to launch or delete tests from dev and prod. + +3. Check if elwin is failing to update + +If you verfied the test is in storage and the query is correct then +elwin may not be able to contact the storage server. You can first +check the logs. + +```bash +$ kubectl get po -l run=elwin +NAME READY STATUS RESTARTS AGE +elwin-3750452436-4vshr 1/1 Running 0 10d +elwin-3750452436-6m78g 1/1 Running 0 11d +elwin-3750452436-7rnmx 1/1 Running 0 10d +elwin-3750452436-ffnq0 1/1 Running 0 11d +elwin-3750452436-gh607 1/1 Running 0 10d +elwin-3750452436-jlkcd 1/1 Running 0 11d +elwin-3750452436-p298j 1/1 Running 0 11d +elwin-3750452436-rqh9l 1/1 Running 0 11d +elwin-3750452436-sp9c0 1/1 Running 0 11d +elwin-3750452436-znccr 1/1 Running 0 11d +elwin-dev-3169672050-0sq6s 1/1 Running 0 10d +elwin-dev-3169672050-3r0w2 1/1 Running 0 14d +elwin-dev-3169672050-f3kcd 1/1 Running 0 11d +elwin-dev-3169672050-gxlqj 1/1 Running 0 10d +elwin-dev-3169672050-lzf50 1/1 Running 0 14d +``` + +Select a pod and check it's logs. + +```bash +kubectl logs --tail=50 elwin-3750452436-4vshr +``` + +Check for lines like the following. + +```bash +2017/03/13 17:14:17 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: lookup elwin-storage: no such host"; Reconnecting to {elwin-storage:80 } +2017/03/13 17:14:19 grpc: addrConn.resetTransport failed to create client transport: connection error: desc = "transport: dial tcp: lookup elwin-storage: no such host"; Reconnecting to {elwin-storage:80 } +``` + +This means grpc has lost it's connection to storage. You can try restarting the pod, but it should restart automatically after `UPDATE_FAIL_TIMEOUT` passes (default is 15m). + +[neo-form]: http://127.0.0.1 "neo form" +[houston]: http://houston.ato.platform.prod.aws.cloud.nordstrom.net "Launch Control"