-
Notifications
You must be signed in to change notification settings - Fork 15
/
09-ci-testing.html
339 lines (292 loc) · 16.4 KB
/
09-ci-testing.html
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
330
331
332
333
334
335
336
337
338
339
<!doctype html>
<html lang="en">
<head>
<meta charset="utf-8">
<title>CS 4970: 09-ci-testing slide set</title>
<meta name="description" content="A set of slides for UVa's Service Learning Practicum course">
<meta name="author" content="Aaron Bloomfield">
<meta name="apple-mobile-web-app-capable" content="yes" />
<meta name="apple-mobile-web-app-status-bar-style" content="black-translucent" />
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no, minimal-ui">
<link rel="stylesheet" href="../slides/reveal.js/css/reveal.css">
<link rel="stylesheet" href="../slides/reveal.js/css/theme/black.css" id="theme">
<link rel="stylesheet" href="../slides/css/slp.css">
<!-- Code syntax highlighting -->
<link rel="stylesheet" href="../slides/reveal.js/lib/css/zenburn.css">
<!-- Printing and PDF exports -->
<script>
var link = document.createElement( 'link' );
link.rel = 'stylesheet';
link.type = 'text/css';
link.href = window.location.search.match( /print-pdf/gi ) ? '../slides/reveal.js/css/print/pdf.css' : '../slides/reveal.js/css/print/paper.css';
document.getElementsByTagName( 'head' )[0].appendChild( link );
</script>
<!--[if lt IE 9]>
<script src="../slides/reveal.js/lib/js/html5shiv.js"></script>
<![endif]-->
<script type="text/javascript" src="../slides/js/dhtmlwindow.js"></script>
<script type="text/javascript" src="../slides/js/canvas.js"></script>
<link rel="stylesheet" href="../slides/css/dhtmlwindow.css" type="text/css">
</head>
<body>
<div id="dhtmlwindowholder"><span style="display:none"></span></div>
<div class="reveal">
<!-- Any section element inside of this container is displayed as a slide -->
<div class="slides">
<section data-markdown><script type="text/template">
# CS 4970
### Capstone Practicum I
<center><small>[Aaron Bloomfield](http://www.cs.virginia.edu/~asb) / [[email protected]](mailto:[email protected]) / [@bloomfieldaaron](http://twitter.com/bloomfieldaaron)</small></center>
<center><small>Repository: [github.com/aaronbloomfield/slp](http://github.com/aaronbloomfield/slp) / [↑](index.html) / <a href="09-ci-testing.html?print-pdf"><img class="print" width="20" src="images/print-icon.png"></a></small></center>
## Continuous Integration Testing
</script></section>
<section data-markdown><script type="text/template">
# Contents
[Introduction](#/intro)
[Writing Unit Tests](#/unittests)
[CI Testing Services](#/ciservices)
[Conclusions](#/conclusions)
</script></section>
<section>
<section data-markdown id="intro"><script type="text/template">
# Introduction
</script></section>
<section data-markdown><script type="text/template">
## Testing lectures
- This is one of two lectures on testing
- It focuses only on unit tests and CI testing services
- The other is the [full testing lecture](12-testing.html#/)
- Given in the spring
- Contents: test plan, other types of tests, etc.
</script></section>
<section data-markdown><script type="text/template">
## What is Continuous Integration?
- As you add code to a code base, you need to know that this code will *integrate* into the existing code base without errors
- Given a large number of unit tests, run those tests on the system after each code change
- This is done *continuously* (as opposed to once a week, before release, etc.)
- This can mean upon every git push, or multiple times a day at fixed times, etc.
- The hitch: running all those tests can take hours!
</script></section>
<section data-markdown><script type="text/template">
## Background: Fixtures
- A given test will likely need some information in the database
- But not much!
- That information is called a *fixture*
- It is a few database records necessary to complete the test
- Specifying fixtures
- CakePHP: using PHP arrays (see [here](http://book.cakephp.org/3.0/en/development/testing.html#creating-fixtures))
- Django: using [JSON](http://en.wikipedia.org/wiki/JSON) or [YAML](http://en.wikipedia.org/wiki/YAML) (see [here](https://docs.djangoproject.com/en/dev/howto/initial-data/))
- Rails: using [YAML](http://en.wikipedia.org/wiki/YAML) (see [here](http://guides.rubyonrails.org/testing.html#the-low-down-on-fixtures))
</script></section>
</section>
<section>
<section data-markdown id="unittests"><script type="text/template">
# Writing Unit Tests
</script></section>
<section data-markdown><script type="text/template">
## Step 1: Configure the Database
- You need to have a test database
- Either MySQL or [SQLite](http://en.wikipedia.org/wiki/SQLite)
- Note that unit tests wipe the ENTIRE database!
- That's why we provided a `slp_tag_test` database on the course server
- Your DB passwords should ***NOT*** be kept in the github repo!
</script></section>
<section data-markdown><script type="text/template">
## Step 2: Write some Unit Tests
- See the appropriate tutorial
- CakePHP: [CakePHP: Testing](http://book.cakephp.org/3.0/en/development/testing.html)
- Django: [Testing in Django](https://docs.djangoproject.com/en/1.11/topics/testing/)
- But start with [page 5 of the intro tutorial](https://docs.djangoproject.com/en/1.11/intro/tutorial05/)
- Rails: [A Guide to Testing Rails Applications](http://guides.rubyonrails.org/testing.html)
</script></section>
<section data-markdown><script type="text/template">
## Step 3: Call the Testing Command
- CakePHP: `phpunit`
- This requires installing phpunit (`sudo apt-get install phpunit`), which has already been done on the VirtualBox image
- Django: `python manage.py test`
- Rails: `bundle exec rake`
- In all cases, this should run the unit tests, and they should all succeed
</script></section>
<section data-markdown><script type="text/template">
## What to Test
- Your system is an MVC (Model-View-Controller) based system
- So you should write tests for each of the components
- Model tests: does data validate properly? what if required data is missing? what about invalid associations?
- Controller tests: for each function in the controller, test out *each* flow of control through that method
- Views: given various data in the fixture, does the view render properly?
</script></section>
<section data-markdown><script type="text/template">
## What to Test, take 2
- This adapted from [here](https://developer.salesforce.com/page/How_to_Write_Good_Unit_Tests)
- Unit tests should test:
- Normal conditions
- Unexpected conditions (user performs functions in the wrong order, etc.)
- Bad input values
- Boundary conditions
- That bugs were fixed (and thus that the bug is not later re-introduced)
</script></section>
<section>
<h2>What to Test, take 3</h2>
<div align="center">
<blockquote class="twitter-tweet" lang="en"><p lang="nl" dir="ltr">QA Engineer walks into a bar. Orders a beer. Orders 0 beers. Orders 999999999 beers. Orders a lizard. Orders -1 beers. Orders a sfdeljknesv.</p>— Bill Sempf (@sempf) <a href="https://twitter.com/sempf/status/514473420277694465">September 23, 2014</a></blockquote>
<script async src="http://platform.twitter.com/widgets.js" charset="utf-8"></script>
</div>
</section>
<section data-markdown><script type="text/template">
## Code Coverage
- How much of the code base was "covered" (i.e., executed) by a given run?
- Or, in our case, by a series of tests
- There are many code coverage tools, here are some:
- CakePHP: [phpunit](https://phpunit.de/); see [here](http://book.cakephp.org/3.0/en/development/testing.html#generating-code-coverage)
- Django: [coverage](http://nedbatchelder.com/code/coverage/), see [here](https://docs.djangoproject.com/en/1.6/topics/testing/advanced/#integration-with-coverage-py)
- Rails: the [simplecov](https://github.com/colszowka/simplecov) gem; see [here](http://www.rubydoc.info/gems/simplecov/frames)
- Later in the year we will need to ensure that our code is fully covered by our unit tests
</script></section>
<section data-markdown><script type="text/template">
## Unit Tests are not to find bugs!
- (well, not *only*...)
- Finding bugs is *integration testing*
- As bugs tend to appear more when all the parts of the system are interacting
- Unit tests are just for a single *unit*
- Unit tests are for:
- Ensuring that some functionality works properly
- A way to ensure that this functionaity works in the future when system changes are made
</script></section>
<section data-markdown><script type="text/template">
## Unit Testing Tips
- These are from [here](http://blog.stevensanderson.com/2009/08/24/writing-great-unit-tests-best-and-worst-practises/)
- Make each test orthogonal
- Don't unit-test configuration settings
- Name your unit tests clearly and consistently
</script></section>
</section>
<section>
<section data-markdown id="ciservices"><script type="text/template">
# CI Testing Services
</script></section>
<section data-markdown><script type="text/template">
## Background: Virtualization
- Types
- System-level: emulates the entire machine; what you are used to with VirtualBox
- Process-level: runs a single process inside an execution environment; an example is a Java program running on the JRE; also .NET
- Operating system-level: the OS partitions user space to run different "containers", each of which appears as a separate machine
</script></section>
<section data-markdown><script type="text/template">
## Background: Containers
- Boot-up time is only a few seconds
- But the kernel version is the same as the running host
- Really the kernel "splits" itself to run the container
- This means that processes can be run *very* fast inside a container
- Your CS 2150 submissions were all run within a Linux Container for security reasons
</script></section>
<section data-markdown><script type="text/template">
## Continuous Integration Methods
- This is the software that automates the running of your tests
- Some are software you download or buy, and run on your server; an example is [Jenkins](http://jenkins-ci.org/)
- Others run your code (and tests) on their servers, or in the "cloud"
- We'll see one: [Travis CI](https://magnum.travis-ci.com/)
- We have used [Circle CI](https://circleci.com) in the past, but we are not using it this year
- Wikipedia has a [long list](http://en.wikipedia.org/wiki/Comparison_of_continuous_integration_software) of such software
</script></section>
<section data-markdown><script type="text/template">
## How a CI service works
- A *webhook* is set up in the github repo to notify the CI service on each push
- Commits are ignored, only pushes are monitored (as only pushes modify the github repo)
- When notified of a push, the service creates a container, clones the repo, then configures that container as per the configuration file in the repo
- The tests are run, and the result is (somehow) reported back to the user
</script></section>
<section data-markdown><script type="text/template">
## Circle CI versus Travis CI
- They are remarkably the same:
- Both integrate nicely with github, use Linux Containers and [YAML](http://en.wikipedia.org/wiki/YAML) configuration files, and run the major platforms
- [Travis CI](https://magnum.travis-ci.com/) is a bit easier to configure, as it can infer a bit more about the system
- It also provides free testing for public github repos
- [Circle CI](https://circleci.com) has much better debugging facilities (you can SSH into the container, for example)
- In the end, these two are a wash, and the services are largely identical for our purposes
- For this year, all projects will use TravisCI
</script></section>
<section data-markdown><script type="text/template">
## Configuring your CI service
- See the [CI setup page](../docs/ci-setup.html) ([md](../docs/ci-setup.md))
- You will have to do this in iteration 3
</script></section>
</section>
<section>
<section data-markdown id="conclusions"><script type="text/template">
# Conclusions
</script></section>
<!--
<section data-markdown id="assignments"><script type="text/template">
## Who is assigned to which group
- Travis CI: aim (6, django), meals (8, django), pacem (7, django), readykids (6, cake), twice (7, django)
- Circle CI: gcadc (8, django), helpline (8, django), pb&j (8, cake), rtf (8, django),
- Methodology
- 2 groups requested a CI service (one for Travis, one for Circle), and were granted that one
- The groups were balanced by the number of group members, not the number of groups
- There are 34 students on Travis and 32 on Circle
</script></section>
-->
<section data-markdown><script type="text/template">
## Your tasks
1. Remove all DB passwords from the repo
2. Write 2 (or so) unit tests, as per the tutorials
3. Configure your project to run unit tests on the assigned CI service
4. Have the status badge for your project's build shown at the top of your project's readme.md file
5. Write a number of unit tests, beyond what was done in (2); see the next slide
6. Ensure that all the unit tests written by the group (at least 30, 35, or 40, depending on the group size) run successfully on the CI service
</script></section>
<section data-markdown><script type="text/template">
## Writing the unit tests
- Each *person* in the group must write at least 5 (five!) unit tests; you must commit your own unit tests to the repo
- Each person in the group must write at least one fixture, which is used in some/all of their unit tests; you must also commit your own fixture (need not be a separate commit)
- These unit tests must be actually test something, and it must be a valid unit test
- They must all succeed (if they don't, then fix the code to make them succeed)
</script></section>
<section data-markdown><script type="text/template">
## Your build should never fail!
- Under no circumstances should you push code to the repo that you know breaks the build (i.e., causes the tests to fail)
- If you do so accidentally, then fix it right away
- If you have tests, then they should all pass!
- You can run the test specific to the code you are writing; the rest are run by the CI service
- If they don't pass, then either fix the code or fix the test
</script></section>
<section data-markdown><script type="text/template">
## Your build should never fail!
- You are explicitly emailed by the CI service if your build fails, so there is no excuse why it should stay failing
</script></section>
<section data-markdown id="howtobegin"><script type="text/template">
## How to begin, part 1
- The [CI setup page](../docs/ci-setup.html) ([md](../docs/ci-setup.md)) page in the repo that goes over how to configure your project
- There are three repos that are now available to you: ~~[rails-test](https://github.com/uva-slp/rails-test)~~, ~~[cake3-test](https://github.com/uva-slp/cake3-test)~~, and [django-test](https://github.com/uva-slp/django-test)
- Each is configured with *both* Travis and Circle (even though we aren't using Circle)
- Each of those repos start with the default site, run through the tutorials from the [Frameworks homework](hw-frameworks.html) ([md](framework-hw.md)), and then proceed to configure the CI tests
- You can view the commit diffs, if desired
</script></section>
<section data-markdown id="howtobegin"><script type="text/template">
## How to begin, part 2
- About the three testing repos (continued)
- Note that the repos are slightly older versions of the frameworks, but the setup is still the same
- The .travis.yml and circle.yml files in those repos are the sample files shown in the [CI setup page](../docs/ci-setup.html) ([md](../docs/ci-setup.md)) page
- And theys how how to display the status badges (although you have to get your own API key)
</script></section>
<section data-markdown id="howtobegin"><script type="text/template">
## How to begin, part 3
- The CI servers will be set up, by me, to run the tests on your repo
- You have to create the appropriate configuration file
- The [CI setup page](../docs/ci-setup.html) ([md](../docs/ci-setup.md)) lists, at the very bottom, a series of steps to follow
- And that page describes the configuration files
- You have to *each* write 5 unit tests
</script></section>
</section>
</div>
</div>
<script src="../slides/reveal.js/lib/js/head.min.js"></script>
<script src="../slides/reveal.js/js/reveal.js"></script>
<script src="../slides/js/settings.js"></script>
</body>
</html>