Skip to content

Commit

Permalink
[CI] Update docs for load-image and register-apps.sh for separation o…
Browse files Browse the repository at this point in the history 
…f Stream Apps Release Train and Version. [skip ci]
  • Loading branch information
@corneil
corneil committed Jul 9, 2024
1 parent 5c4ad1f commit 59d0790
Show file tree
Hide file tree
Showing 4 changed files with 106 additions and 48 deletions.
28 changes: 19 additions & 9 deletions spring-cloud-dataflow-docs/src/main/asciidoc/appendix-local-k8s-development.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -8,7 +8,7 @@ You will need to install kubectl and then kind or minikube for a local cluster.

All the examples assume you have cloned the `spring-cloud-dataflow` repository and are executing the scripts from `deploy/k8s`.

On macOS you may need to install `realpath` from link:https://ports.macports.org/port/realpath/[Macports] or `brew install realpath`
On macOS, you may need to install `realpath` from link:https://ports.macports.org/port/realpath/[Macports] or `brew install realpath`

NOTE: The scripts require a shell like `bash` or `zsh` and should work on Linux, WSL 2 or macOS.

Expand Down Expand Up @@ -83,7 +83,7 @@ source ./k8s/use-mk-docker.sh postgresql rabbitmq --namespace test-ns

===== TMC or GKE Cluster in Cloud

The cluster must exist before use and you should use the relevant cli to login before executing `source ./k8s/use-gke.sh`
The cluster must exist before use, and you should use the relevant cli to login before executing `source ./k8s/use-gke.sh`

===== Create Local Cluster.

Expand Down Expand Up @@ -139,13 +139,19 @@ This example shows the versions of the current development snapshot.

[source,shell]
....
export DATAFLOW_VERSION=2.11.3-SNAPSHOT
export SKIPPER_VERSION=2.11.3-SNAPSHOT
export DATAFLOW_VERSION=2.11.4-SNAPSHOT
export SKIPPER_VERSION=2.11.4-SNAPSHOT
....

Before you can install SCDF you will need to pull the following images to ensure they are present for uploading to the k8s cluster.

You can configure the before `pull-app-images` and `install-scdf`:

* `STREAM_APPS_RT_VERSION` Stream Apps Release Train Version. _Default is 2022.0.0_.
* `STREAM_APPS_VERSION` Stream Apps Version. _Default is 4.0.0_.

Use:

[source,shell]
....
./images/pull-app-images.sh
Expand Down Expand Up @@ -197,13 +203,17 @@ The following list of utilities may prove useful.


* Using kail to log activity related to a specific stream.
```shell

[source,shell]
----
kail --label=spring-group-id=<stream-name>
```
----
* Using kail to log all pods in specific namespace.
```shell

[source,shell]
----
kail --ns=<namespace>
```
----

==== Scripts

Expand All @@ -225,7 +235,7 @@ Some of the scripts apply to local containers as well and can be found in `src/l
| ./k8s/delete-scdf.sh | Delete all Kubernetes resources create by the deployment.
| ./k8s/destroy-k8s.sh | Delete cluster, kind or minikube.
| ./k8s/export-dataflow-ip.sh | Export the url of the data flow server to `DATAFLOW_IP`
| ./k8s/export-http-url.sh | Export the url of an http source of a specific flow by name to `HTTP_APP_URL`
| ./k8s/export-http-url.sh | Export the url of the http source of a specific flow by name to `HTTP_APP_URL`
| ./k8s/install-scdf.sh | Configure and deploy all the containers for Spring Cloud Dataflow
| ./k8s/load-images.sh | Load all container images required by tests into kind or minikube to ensure you have control over what is used.
| ./k8s/load-image.sh | Load a specific container image into local kind or minikube.
Expand Down
11 changes: 8 additions & 3 deletions spring-cloud-dataflow-docs/src/main/asciidoc/configuration-carvel.adoc
View file
Original file line number Diff line number Diff line change
Expand Up @@ -52,9 +52,8 @@ These scripts assume you are connected to a Kubernetes cluster and `kubectl` is
| Creates `scdf-values.yml` in current directory based on `scdf-pro-values.yml` or `scdf-oss-values.yml`

| carvel-import-secret.sh
| <secret-name> <namespace> [secret-namespace] [--import|--placeholder]
| [--placeholder] Creates place holder secret in namespace. [--import] Creates `SecretImport` for secretgen-controller.

| <secret-name> <namespace> [secret-namespace] [--import\|--placeholder]
| Creates an import secret, placeholder or import using secretgen-controller.
| setup-scdf-repo.sh
| [scdf-type] (oss, pro)
| Creates the namespace and installs the relevant Carvel package and credentials. If the optional _scdf-type_ is not provided the environmental variable `SCDF_TYPE` will be used.
Expand Down Expand Up @@ -92,6 +91,7 @@ The default _app-name_ is `scdf-${SCDF_TYPE}`.
| <broker> [stream-application-version]
| _broker_ must be one of rabbit or kafka.
_stream-application-version_ is optional and will install the latest version. The latest version is 2021.1.2

|===

NOTE: Take note that the registration of application in the _pro_ version can take a few minutes since it retrieves all version information and metadata upfront.
Expand Down Expand Up @@ -248,6 +248,11 @@ In the case where and existing prometheus and prometheus proxy is deployed the p

=== Deploy Spring Cloud Data Flow

You can configure the before `register-apps.sh`:

* `STREAM_APPS_RT_VERSION` Stream Apps Release Train Version. _Default is 2022.0.0_.
* `STREAM_APPS_VERSION` Stream Apps Version. _Default is 4.0.0_.

[source,shell]
....
./carvel/deploy-scdf.sh
Expand Down
115 changes: 79 additions & 36 deletions src/deploy/README.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@
<meta charset="UTF-8">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<meta name="generator" content="Asciidoctor 2.0.22">
<meta name="generator" content="Asciidoctor 2.0.23">
<title>Deployment to Kubernetes</title>
<link rel="stylesheet" href="https://fonts.googleapis.com/css?family=Open+Sans:300,300italic,400,400italic,600,600italic%7CNoto+Serif:400,400italic,700,700italic%7CDroid+Sans+Mono:400,700">
<style>
Expand Down Expand Up @@ -669,61 +669,78 @@ <h3 id="_scripts">Scripts</h3>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>carvel-import-secret.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;secret-name&gt; &lt;namespace&gt; [secret-namespace] [--import</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;secret-name&gt; &lt;namespace&gt; [secret-namespace] [--import|--placeholder]</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>--placeholder]</p>
<p>Creates an import secret, placeholder or import using secretgen-controller.</p>
</div></div></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>[--placeholder] Creates place holder secret in namespace. [--import] Creates <code>SecretImport</code> for secretgen-controller.</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">setup-scdf-repo.sh</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>setup-scdf-repo.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">[scdf-type] (oss, pro)</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>[scdf-type] (oss, pro)</p>
<p>Creates the namespace and installs the relevant Carvel package and credentials. If the optional <em>scdf-type</em> is not provided the environmental variable <code>SCDF_TYPE</code> will be used.</p>
</div></div></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>Creates the namespace and installs the relevant Carvel package and credentials. If the optional <em>scdf-type</em> is not provided the environmental variable <code>SCDF_TYPE</code> will be used.</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">configure-prometheus-proxy.sh</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>configure-prometheus-proxy.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;host&gt; &lt;port&gt; [step]</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>&lt;host&gt; &lt;port&gt; [step]</p>
<p>Configures Spring Boot Actuator properties for Data Flow, Skipper, Streams and Tasks. Default <code>step</code> is 10s</p>
</div></div></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>Configures Spring Boot Actuator properties for Data Flow, Skipper, Streams and Tasks. Default <code>step</code> is 10s</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">configure-database.sh</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>configure-database.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;app&gt; &lt;database&gt; &lt;url&gt; &lt;username/secret-name&gt; [password/secret-username-key] [secret-password-key]</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>&lt;app&gt; &lt;database&gt; &lt;url&gt; &lt;username/secret-name&gt; [password/secret-username-key] [secret-password-key]</p>
<p>If only <em>secret-name</em> is provided then <em>secret-username-key</em> defaults to <code>username</code> and <em>secret-password-key</em> defaults to <code>password</code>.</p>
</div>
<div class="paragraph">
<p>The following 3 combinations are allowed after the <em>url</em>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p>&lt;secret-name&gt;</p>
</li>
<li>
<p>&lt;secret-name&gt; &lt;username-key&gt;
&lt;password-key&gt;</p>
</li>
<li>
<p>&lt;username&gt; &lt;password&gt;</p>
</li>
</ul>
</div></div></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>If only <em>secret-name</em> is provided then <em>secret-username-key</em> defaults to <code>username</code> and <em>secret-password-key</em> defaults to <code>password</code>.</code></p>
<p class="tableblock"><code>The following 3 combinations are allowed after the <em>url</em>:</code></p>
<p class="tableblock"><code>* &lt;secret-name&gt;
* &lt;secret-name&gt; &lt;username-key&gt;
&lt;password-key&gt;
* &lt;username&gt; &lt;password&gt;</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">deploy-scdf.sh</p></td>
<td class="tableblock halign-left valign-top"><div class="content"></div></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>deploy-scdf.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">[app-name]</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>Deploys the application using the package and <code>scdf-values.yml</code> in the current directory.
The default <em>app-name</em> is <code>scdf-${SCDF_TYPE}</code>.</p>
</div></div></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>Deploys the application using the package and <code>scdf-values.yml</code> in the current directory.
The default <em>app-name</em> is <code>scdf-${SCDF_TYPE}</code>.</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">update-scdf.sh</p></td>
<td class="tableblock halign-left valign-top"><div class="content"></div></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>update-scdf.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">[app-name]</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>Updated the deployed application using a modified values file.
The default <em>app-name</em> is <code>scdf-${SCDF_TYPE}</code>.</p>
</div></div></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>Updated the deployed application using a modified values file.
The default <em>app-name</em> is <code>scdf-${SCDF_TYPE}</code>.</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">export-dataflow-ip.sh</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>export-dataflow-ip.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">N/A</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>N/A</p>
<p>Will print the URL to access dataflow. If you use <code>source ./export-dataflow-ip.sh</code> it will export <code>DATAFLOW_URL</code> to be used by <code>register-apps.sh</code></p>
</div></div></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>Will print the URL to access dataflow. If you use <code>source ./export-dataflow-ip.sh</code> it will export <code>DATAFLOW_URL</code> to be used by <code>register-apps.sh</code></code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">register-apps.sh</p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>register-apps.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">&lt;broker&gt; [stream-application-version]</p></td>
<td class="tableblock halign-left valign-top"><div class="content"><div class="paragraph">
<p>&lt;broker&gt; [stream-application-version]</p>
<p><em>broker</em> must be one of rabbit or kafka.
<em>stream-application-version</em> is optional and will install the latest version. The latest version is 2021.1.2</p>
</div></div></td>
</tr>
</tbody>
Expand Down Expand Up @@ -977,6 +994,19 @@ <h3 id="_configure_prometheus_proxy">Configure Prometheus proxy</h3>
</div>
<div class="sect2">
<h3 id="_deploy_spring_cloud_data_flow">Deploy Spring Cloud Data Flow</h3>
<div class="paragraph">
<p>You can configure the before <code>register-apps.sh</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>STREAM_APPS_RT_VERSION</code> Stream Apps Release Train Version. <em>Default is 2022.0.0</em>.</p>
</li>
<li>
<p><code>STREAM_APPS_VERSION</code> Stream Apps Version. <em>Default is 4.0.0</em>.</p>
</li>
</ul>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="shell">./carvel/deploy-scdf.sh
Expand All @@ -1003,7 +1033,7 @@ <h4 id="_prerequisites">Prerequisites</h4>
<p>All the examples assume you have cloned the <code>spring-cloud-dataflow</code> repository and are executing the scripts from <code>deploy/k8s</code>.</p>
</div>
<div class="paragraph">
<p>On macOS you may need to install <code>realpath</code> from <a href="https://ports.macports.org/port/realpath/">Macports</a> or <code>brew install realpath</code></p>
<p>On macOS, you may need to install <code>realpath</code> from <a href="https://ports.macports.org/port/realpath/">Macports</a> or <code>brew install realpath</code></p>
</div>
<div class="admonitionblock note">
<table>
Expand Down Expand Up @@ -1165,7 +1195,7 @@ <h4 id="_configure_k8s_environment">Configure k8s environment</h4>
<div class="sect4">
<h5 id="_tmc_or_gke_cluster_in_cloud">TMC or GKE Cluster in Cloud</h5>
<div class="paragraph">
<p>The cluster must exist before use and you should use the relevant cli to login before executing <code>source ./k8s/use-gke.sh</code></p>
<p>The cluster must exist before use, and you should use the relevant cli to login before executing <code>source ./k8s/use-gke.sh</code></p>
</div>
</div>
<div class="sect4">
Expand Down Expand Up @@ -1250,14 +1280,27 @@ <h6 id="_configure_database">Configure Database</h6>
</div>
<div class="listingblock">
<div class="content">
<pre class="CodeRay highlight"><code data-lang="shell">export DATAFLOW_VERSION=2.11.3-SNAPSHOT
export SKIPPER_VERSION=2.11.3-SNAPSHOT</code></pre>
<pre class="CodeRay highlight"><code data-lang="shell">export DATAFLOW_VERSION=2.11.4-SNAPSHOT
export SKIPPER_VERSION=2.11.4-SNAPSHOT</code></pre>
</div>
</div>
<div class="paragraph">
<p>Before you can install SCDF you will need to pull the following images to ensure they are present for uploading to the k8s cluster.</p>
</div>
<div class="paragraph">
<p>You can configure the before <code>pull-app-images</code> and <code>install-scdf</code>:</p>
</div>
<div class="ulist">
<ul>
<li>
<p><code>STREAM_APPS_RT_VERSION</code> Stream Apps Release Train Version. <em>Default is 2022.0.0</em>.</p>
</li>
<li>
<p><code>STREAM_APPS_VERSION</code> Stream Apps Version. <em>Default is 4.0.0</em>.</p>
</li>
</ul>
</div>
<div class="paragraph">
<p>Use:</p>
</div>
<div class="listingblock">
Expand Down Expand Up @@ -1433,7 +1476,7 @@ <h4 id="_scripts_2">Scripts</h4>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./k8s/export-http-url.sh</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Export the url of an http source of a specific flow by name to <code>HTTP_APP_URL</code></p></td>
<td class="tableblock halign-left valign-top"><p class="tableblock">Export the url of the http source of a specific flow by name to <code>HTTP_APP_URL</code></p></td>
</tr>
<tr>
<td class="tableblock halign-left valign-top"><p class="tableblock"><code>./k8s/install-scdf.sh</code></p></td>
Expand Down
Binary file modified src/deploy/README.pdf
View file
Binary file not shown.

0 comments on commit 59d0790

Please sign in to comment.