-
Notifications
You must be signed in to change notification settings - Fork 23
/
development.html.md.erb
164 lines (114 loc) · 8.14 KB
/
development.html.md.erb
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
---
title: Development Steps at Each Level
owner: Services
---
<strong><%= modified_date %></strong>
This topic describes the typical steps that tile developers follow when developing a tile at a given [integration level](./stages.html),
such as brokered service tile, a managed service tile, or an on-demand service tile.
## <a id="overview"></a> Overview
The ultimate deliverable of a Pivotal Cloud Foundry (PCF) integration is almost always a **tile**, a
PCF installation package that is delivered through our
[Marketplace](http://network.pivotal.io) and installed through Pivotal's
Ops Manager. But when developing an integration, it is advisable to start
with smaller components of that tile, as it allows you to iterate on those
components much faster. Pivotal recommends that you approach development in the following phases:
1. [Develop](#components) and [test](#testing) the Tile Generator inputs individually
2. [Describe](#tile-generator) your tile in `tile.yml` and use [Tile Generator](./tile-generator.html) to create the tile components
3. Test the generated deploy errands individually
4. [Deploy](#deploy-tile) and test the generated BOSH release and the complete tile
5. [Implement](#concourse) Continuous Integration (CI) for the complete tile
If you follow this approach, you may not have a dependency on a complete
PCF installation until step 3, and your iterations on the components will
be much faster than if you attempt to test them through actual deployment
to PCF.
Each of the development phases is described in more detail below.
## <a id="components"></a> Develop the Tile Generator Inputs
Tiles are a packaging format to deliver ISV software to PCF customers. Most
tiles contain one or more of the following types of components:
- Service Brokers
- Managed Services
- Buildpacks
- Apps
It is much more efficient to develop and test these components individually
than it is to test them through tile deployment. So before you start generating
and deploying tiles, *always* make sure that the components you are deploying
already work, individually and in whatever combination you intend to deploy
them as a tile.
In most cases, you do not need a full PCF installation to complete these early phases.
You can set up a [light-weight PCF development environment](./environments.html#pcfdev) on your laptop or desktop,
possibly including BOSH-Lite if your are developing managed services.
## <a id="tile-generator"></a> Describe and Generate Your Tile
After your components are in working order, download and install the
[Tile Generator](tile-generator.html) and follow the instructions to describe
and generate your actual tile. This is where you list all the components that
are to be included, add an icon and a description, and have the option to add
forms for values that are to be configured by the PCF operator at installation
time.
## <a id="deploy-tile"></a> Deploy Your Generated BOSH Release and Full Tile
After you have verified that all individual [components](#components) and
[errands](#test-errands) work, you are ready to deploy your tile's generated BOSH release from the command line.
Finally, test deploy your tile as your customers would, by uploading it to Ops Manager, installing and configuring the tile,
and having Ops Manager apply your changes to the PCF deployment.
The only things you should be testing in this phase are the things that could not be tested in earlier ones:
- The appearance of the tile forms in Ops Manager
- The upgrade/migration steps from one version of a tile to another (if applicable)
Everything else should work exactly as it did in prior steps.
## <a id="testing"></a> Tile Testing
Good testing assures tile developers that their product installs and runs properly on diverse platforms,
and assures PCF platform operators that the tile they install can provide its service successfully on their platform.
Pivotal recommends a pyramid structure for testing, starting with unit tests and stepping up to successively broader and more automated levels of integration.
Pivotal uses and recommends [Concourse](./concourse.html) for creating build pipelines that follow this test structure.
Other continuous integration tools should also support a pyramid testing approach.
### <a id="pyramid"></a> Tile Test Pyramid
For PCF tiles, a typical test pyramid progresses as follows:
1. Unit tests for each tile **component** (e.g. service components, broker, adapter, and metrics emitter), manual by developer and in automated pipeline.
1. System tests of the tile's **BOSH release**, including:
- **Functional tests** covering the main features of the service.
The main features typically interact with almost all important external integration points, so these tests confirm product functionality.
- **Smoke tests** (lifecycle tests) for service instances that create and bind a service instance, call it from a test app,
check the logs it generates, and delete it.
For a typical end-to-end test sequence, see [Smoke Tests](#smoke) below.
1. System tests of **tile** operation within Ops Manager.
- These include:
* **Configuration checks** that test every external configurable integration point and connection to remote servers using configured credentials
* **Default checks** that confirm "happy path" functionality.
- Use the Ops Manager API to verify that property blueprints in the tile metadata are correct
and that they translate correctly to the BOSH manifest that Ops Manager generates.
- Use the [Om](https://github.com/pivotal-cf/om) tool to call the Ops Manager API programmatically from Go.
Avoid the unsupported opsmgr gem that called the Ops Manager API from Ruby.
- Confirm manually that the tile wires property blueprints to the expected pane and form controls in the UI.
<p class="note"><strong>Note</strong>: System tests might incur costs from using third party services, IaaS resources, etc.</p>
### <a id="smoke"></a> Smoke Tests
Smoke tests are end-to-end lifecycle tests for service instances that you can include as [post-deploy errands](./tile-errands.html#post-deploy)
within a tile and also automate in [Concourse](./concourse.html) or other integration platforms.
A typical smoke test runs as follows:
1. Create an org and space for the test to run in.
1. Register the tile's service broker.
1. Enable service access for the created org.
1. Iterate through all service plans (or a subset of them) to do the following:
1. Create a service instance for the plan.
1. Push a test app.
1. Bind the service instance to the app.
1. Use the app in a way that exercises the service instance. For a data service, for example, write and read from the service instance.
1. Unbind the service instance.
1. Delete the service instance.
1. Delete the test app.
1. Delete the service broker.
1. Delete the test org and space.
### <a id="general-recs"></a> General Recommendations
The following are general recommendations for designing and running tests on PCF tiles:
* Clean up after yourself. Leave the environment exactly as it was before the test was run.
* Generate verbose logging with lots of contextual data to make troubleshooting easier.
* Design test suites for re-usability by making them highly parameterizable. Important parameters include:
- External settings such as domains, creds, and certs
- Plans to test against.
For example, the [Redis for PCF](https://github.com/pivotal-cf/cf-redis-smoke-tests) smoke tests use identical code
for two different service plans, pre-provisioned and on-demand.
- Timeouts, numbers of retries, and other things that you need to adjust for different environments
- Switches to include or exclude portions of the tests such as generating metrics or backups
* Re-use tests that exist already, for example in Concourse.
* Use an example CF app that uses your service. This app can serve for testing, demoing your tile capabilities, and as a code code example.
See the [MySQL Test App](https://github.com/cloudfoundry-incubator/cf-mysql-acceptance-tests/tree/master/assets/sinatra_app) an example.
* When testing manually, using the UI is better than calling the underlying API directly.
Use UIs and APIs the way a customer would.