Run Karma tests with BrowserStack and reconfigure Travis #452

funkyboy · 2018-01-23T12:33:19Z

Adding BrowserStack for karma tests.
Fix #449
Fix #460
Fix #465

add script for CI
add more combinations of os/browsers
inspect with IE hits timeouts and takes 4x than Firefox tests
remove local Firefox
add all browsers to CI script (in batches)
add Safari for Mac
add one Android device
Set up Travis to run 2 jobs (see Pick on CI to run unit and browser tests on #460 (comment))
Re-add local Firefox

funkyboy · 2018-01-26T08:31:38Z

IE10 issues have been fixed here: #453

SimonWoolf · 2018-01-26T09:11:17Z

IE10 issues have been fixed here

To be clear, those appear to have been regressions introduced since a couple years ago (genuine bugs affecting IE10), rather than the IE10 problems we had with saucelabs/testingbot that couldn't be reproduced in local vms (and which, as #449 notes, caused every test after some initial number to time out with connectivity issues, rather than only the crypto tests). Looking forward to seeing how we fare now on saucelabs or browserstack.

funkyboy · 2018-02-08T12:24:39Z

test/bin/ci-travis-all.sh

@@ -1,3 +1,3 @@
 grunt $B0
-grunt test:karma --browsers bs_firefox_sierra,bs_chrome_sierra,bs_ie11_win81,bs_ie10_win81
+grunt test:karma --browsers $B1
 grunt test:karma --browsers bs_ie9_win7,bs_ie16_win10,bs_safari_10_3,bs_safari_11


This would be nice but as soon as $B0 (nodeunit) tests end, Travis starts running the script on line 3. But given that line 2 has not finished yet, BrowserStack will hang, because we are allowed at most 5 tests in parallel. :(

When running tests in a serial fashion, we hit travis (.org) limits of 50m per build. Sequence tested:

nodeunit

bs_firefox_sierra,bs_chrome_sierra,bs_ie11_win81,bs_ie10_win81

bs_ie9_win7,bs_ie16_win10,bs_safari_10_3,bs_safari_11

Example here: https://api.travis-ci.org/v3/job/338955785/log.txt

funkyboy · 2018-02-08T14:32:04Z

We hit Travis 50m timeout also when removing iOS10 tests, thus running tests on 8 browsers (plus nodeunit).

paddybyers

I'll defer to @SimonWoolf

SimonWoolf

lgtm with comments 👍
Now the bigger piece of work: fixing the bugs and intermittent tests that this exposes

SimonWoolf · 2018-02-20T08:42:11Z

karma.conf.js

-      browserName: 'internet explorer',
-      platform: 'Windows 7',
-      version: '10'
+    bs_safari_10_3: {


consistent naming: OS not just version, eg bs_safari_iOS_10_3

@SimonWoolf My bad. This is not referenced in the the travis file. Will substitute it with IE8.

SimonWoolf · 2018-02-20T08:52:41Z

karma.conf.js

@@ -166,24 +169,14 @@ module.exports = function(config) {

    // level of logging
    // possible values: config.LOG_DISABLE || config.LOG_ERROR || config.LOG_WARN || config.LOG_INFO || config.LOG_DEBUG
-    logLevel: config.LOG_INFO,
+    logLevel: process.env.ABLY_LOG_LEVEL,


Not sure why you made this change? It won't work -- karma log level is an enum whose values are uppercase strings; Ably log level is an integer

@SimonWoolf That's why it doesn't work :)
Goal was to keep the Travis log shorter. Will set to WARN to see what happens.

karma INFO-level logs add around 10 lines to the beginning of the test; the remaining ~40k lines are mostly from ably-js Logger. I'd focus on the 40k rather than cutting those 10 lines down to 5 ☺. Check out the envPreprocessor config line for exposing external env vars to karma tests

@SimonWoolf Oh so this doesn't work because ABLY_LOG_LEVEL doesn't appear in the envPreprocessor "enumeration"?

Answering myself: yes :)

SimonWoolf · 2018-02-20T09:02:57Z

test/bin/ci-travis-all.sh

@@ -0,0 +1,3 @@
+grunt test:nodeunit


No point duplicating the nodeunit tests in the browserstack run. (If you're only doing that get the concat task, you can just do grunt concat directly. have a look in the gruntfile to see what tasks do what)

@SimonWoolf True, but browser tests failed if not preceded by nodeunit tests because of #462

No, grunt concat doesn't run the minifier, so is unaffected by #462.

SimonWoolf · 2018-02-20T09:06:56Z

karma.conf.js

+      os: 'android',
+      os_version: '6.0',
+      device: 'Google Nexus 6',
+      real_mobile: true
    }
  };


No IE8? (I'm not against dropping IE8 if usage numbers are finally low enough, but that should be a deliberate decision, as long as we claim we support it we should be testing on it)

@SimonWoolf Fixing. See above.

SimonWoolf · 2018-02-20T09:13:42Z

karma.conf.js

    }
  };

  config.set({

+    browserStack: {
+      username: process.env.BROWSERSTACK_USERNAME,
+      accessKey: process.env.BROWSERSTACK_ACCESSKEY,


Would be convenient if these were in the private repo in a script in sourcable

@SimonWoolf Added

funkyboy · 2018-02-20T11:50:44Z

@SimonWoolf one last check before merging?

SimonWoolf · 2018-02-20T12:27:34Z

👍 . Git history's a bit messed up though, why is there a commit merging master into this branch? Get rid of that and rebase this on master, and regen package-lock.json. Let me know when done and I'll check it and merge

funkyboy · 2018-02-20T13:34:49Z

@SimonWoolf that was the result of of using GH web UI to resolve a conflict :(

funkyboy · 2018-02-20T14:17:05Z

@SimonWoolf here you go

SimonWoolf · 2018-02-20T14:23:11Z

Thanks @funkyboy

This was referenced Feb 1, 2018

Document supported platforms in the README file #457

Closed

Pick on CI to run unit and browser tests on #460

Closed

funkyboy force-pushed the add-browserstack-testing branch from fc7ec88 to a21795d Compare February 7, 2018 18:04

funkyboy mentioned this pull request Feb 7, 2018

Update to official closure compiler plugin #462

Closed

funkyboy commented Feb 8, 2018

View reviewed changes

funkyboy force-pushed the add-browserstack-testing branch from e463217 to 492e6d1 Compare February 9, 2018 14:49

funkyboy changed the title ~~WIP: Add BrowserStack config for karma~~ WIP: Run Karma tests with BrowserStack and reconfigure Travis Feb 14, 2018

funkyboy changed the title ~~WIP: Run Karma tests with BrowserStack and reconfigure Travis~~ Run Karma tests with BrowserStack and reconfigure Travis Feb 14, 2018

funkyboy requested review from paddybyers and SimonWoolf February 14, 2018 08:40

paddybyers reviewed Feb 14, 2018

View reviewed changes

funkyboy mentioned this pull request Feb 20, 2018

WIP: Update outdated dependencies #469

Closed

SimonWoolf requested changes Feb 20, 2018

View reviewed changes

funkyboy force-pushed the add-browserstack-testing branch 2 times, most recently from 3dc079c to 2fb08e8 Compare February 20, 2018 14:04

funkyboy added 9 commits February 20, 2018 15:07

Add BrowserStack config for karma

1f75026

Add more test configurations for windows

604aa7f

Update Firefox version

048d33a

Update win version

762991e

Add IE9

c40c5b6

Remove redundant combination

8c8f740

Add Edge16 on Win 10

fcd5195

Add mobile Safari 10.3

c6d5b12

Remove old node version

c918257

funkyboy added 25 commits February 20, 2018 15:10

Try parallel builds

cd705eb

Add nodeunit to Travis matrix

dcb6d2c

Group BrowserStack tests in batches of 4

b213cd5

Run nodeunit in parallel and BS tests in a series

6f80a83

Run tests in a serial queue

e697a5a

Try removing iOS10 tests

5e2726a

Remove SauceLab custom launchers

b05f814

Remove SauceLab bins and scripts

1d229e8

Add Safari High Sierra

1ecb6a8

Rename Safari iOS

1c6d356

Add BS tests on Android

0f36cbc

Use global log level in Karma

7c5efed

Split short and long test sessions

ff6a6f6

Fix script name in Travis

f576ae7

Add +x permissions to script file

701e129

Remove SauceLab settings

e6ec8f5

Add Firefox local tests to short session

f6e1a78

Set up virtual display for Firefox

6771cbf

Swap order of jobs

c6616c4

Remove unneeded default

9daaf1d

Remove iOS10 profile and add IE8

0768faa

Set log level to WARN

953ee33

Run concat before browser tests

aa475c5

Revert ably log level

8a4495e

Add global log level to karma preprocessor

ae67028

funkyboy force-pushed the add-browserstack-testing branch from 2fb08e8 to ae67028 Compare February 20, 2018 14:14

SimonWoolf merged commit 9ea1585 into master Feb 20, 2018

SimonWoolf deleted the add-browserstack-testing branch February 20, 2018 14:23

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Run Karma tests with BrowserStack and reconfigure Travis #452

Run Karma tests with BrowserStack and reconfigure Travis #452

funkyboy commented Jan 23, 2018 •

edited

Loading

funkyboy commented Jan 26, 2018

SimonWoolf commented Jan 26, 2018 •

edited

Loading

funkyboy Feb 8, 2018

funkyboy Feb 8, 2018 •

edited

Loading

funkyboy commented Feb 8, 2018

paddybyers left a comment

SimonWoolf left a comment

SimonWoolf Feb 20, 2018

funkyboy Feb 20, 2018

SimonWoolf Feb 20, 2018

funkyboy Feb 20, 2018

SimonWoolf Feb 20, 2018 •

edited

Loading

funkyboy Feb 20, 2018 •

edited

Loading

funkyboy Feb 20, 2018

SimonWoolf Feb 20, 2018

funkyboy Feb 20, 2018

SimonWoolf Feb 20, 2018

SimonWoolf Feb 20, 2018

funkyboy Feb 20, 2018

SimonWoolf Feb 20, 2018

funkyboy Feb 20, 2018

funkyboy commented Feb 20, 2018

SimonWoolf commented Feb 20, 2018

funkyboy commented Feb 20, 2018

funkyboy commented Feb 20, 2018

SimonWoolf commented Feb 20, 2018

Run Karma tests with BrowserStack and reconfigure Travis #452

Run Karma tests with BrowserStack and reconfigure Travis #452

Conversation

funkyboy commented Jan 23, 2018 • edited Loading

funkyboy commented Jan 26, 2018

SimonWoolf commented Jan 26, 2018 • edited Loading

Choose a reason for hiding this comment

funkyboy Feb 8, 2018 • edited Loading

Choose a reason for hiding this comment

funkyboy commented Feb 8, 2018

paddybyers left a comment

Choose a reason for hiding this comment

SimonWoolf left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SimonWoolf Feb 20, 2018 • edited Loading

Choose a reason for hiding this comment

funkyboy Feb 20, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

funkyboy commented Feb 20, 2018

SimonWoolf commented Feb 20, 2018

funkyboy commented Feb 20, 2018

funkyboy commented Feb 20, 2018

SimonWoolf commented Feb 20, 2018

funkyboy commented Jan 23, 2018 •

edited

Loading

SimonWoolf commented Jan 26, 2018 •

edited

Loading

funkyboy Feb 8, 2018 •

edited

Loading

SimonWoolf Feb 20, 2018 •

edited

Loading

funkyboy Feb 20, 2018 •

edited

Loading