Merge pull request #491 from exadel-inc/feature/EAK-490

[EAK-490] Implemented a regression testing profile

core/pom.xml

@@ -136,7 +136,7 @@
         </dependency>
         <dependency>
             <groupId>javax.servlet</groupId>
-            <artifactId>servlet-api</artifactId>
+            <artifactId>javax.servlet-api</artifactId>
         </dependency>
         <dependency>
             <groupId>javax.servlet.jsp</groupId>

docs/content/dev-tools/component-management/customizing-toolkit.md

@@ -291,11 +291,11 @@ Read more on debugging a Maven plugin e.g. [here](https://spin.atomicobject.com/
 
 ### Running integration tests
 
-Starting from version _2.3.0_, the ToolKit supports integration tests powered by [Selenide](https://selenide.org). They are mainly for checking the browser-bound functions as well as checking the connection to a live AEM server and/or 3rd-party services on the Internet.
+The ToolKit supports integration tests powered by [Selenide](https://selenide.org). They are mainly for checking the browser-bound functions and rely on the connection to a live AEM server and/or 3rd-party services on the Internet.
 
 The integration tests are localed in the `it.tests` module. Important: unlike unit tests, integration tests are not run in frames of a "regular" build. To run them, you need to specify the dedicated Maven profile like the following:
 ```
-mvn clean install -Pintegration
+mvn clean test -Pintegration
 ```
 
 As the integration tests start, a synthetic content package containing test data will be created and deployed to a live AEM instance (the one specified by the _aem.host_ and/or _aem.port_ properties). You can find the data in AEM under _/apps/etoolbox-authoring-kit-test_ and also in the same folders under _/conf_ and _/content_.
@@ -308,6 +308,47 @@ After the integration tests are run, the synthetic package and its content are r
 
 A complete command line for running integration tests with different properties specified may look like the following:
 ```
-mvn clean install -PautoInstallPackage -Pintegration -D"aem.host"=192.168.0.81 -D"aem.port"=8080 -D"aem.login"=siteadmin -D"aem.password"=MyPa$$w0rd -Dnouninstall=true
+mvn clean test -Pintegration -D"aem.host"=192.168.0.81 -D"aem.port"=8080 -D"aem.login"=siteadmin -D"aem.password"=MyPa$$w0rd -Dnouninstall=true
+```
+
+### Doing regression testing
+
+The ToolKit supports regression testing as well. Regression testing is done using an external AEM project. The aim is to make sure that a <newer> version of ToolKit (namely, the plugin) produces essentially the same XML markup as an <older> one. The <older> version is considered a "reference" and is used as a baseline for comparison.
 
+To run regression tests, you need to specify the dedicated Maven profile like the following:
 ```
+mvn test -Pregression "-Dproject=e:\projects\aem\my_project -Deak.version=${version}"
+```
+The focus here is the _-Dproject=..._ option. It is the command line to build the target project.
+
+The ToolKit will navigate to the folder e:\projects\aem\my_project_ (and expectedly will find project's POM file in it). It will build the project (without doing unit tests and deployment) with the version of ToolKit that is specified in the project. It will collect the content packages created  upon the build.
+
+Then it will switch the version of ToolKit to the latest one and run the build for the second time. The content packages will be collected again. Then the "former" and the "newer" packages will be compared content-wise. If there are no significant disparities, the test is passed. Otherwise, the test is failed.
+
+We assume that your project has a _Maven property_ that stores the version of ToolKit, and the dependencies within your project use that property (like `<dependency><groupId>com.exadel.etoolbox</groupId><artifactId>...</artifactId><version>${eak.version}</dependency>`). Name of the property does not matter. But you are expected
+ to mention it in the _-Dproject=..._ as shown above. The _${version}_ part is the placeholder that will be used by the regression code.
+
+Within _-Dproject=..._, you can specify more property keys that are specific for your project if you need to. You can also specify the _-pl_ option to select particular modules for the build, or also some _-P_ keys to specify build profiles, etc. Do not forget to enclose _-Dproject=..._ in double quotes if there are spaces inside.
+
+You can process more than one project at a time. Separate projects in the _-Dproject=..._ option with a semicolon.
+
+We've said that the second part of the regression is building a target project with the current ToolKit version. If you however wish to test another version, specify it with the optional switch like `-Deak.version=1.2.3-SNAPSHOT`.`
+
+Besides, you can override the _Maven_ executable used to build the target AEM project by specifying `-Dmaven.cmd=/path/to/mvn/or/a/cmd/file`. Optionally you can specify a local Maven repository address if it differs from _$HOME$/.m2_, with `-Dmaven.dir=/path/to/maven/repository`.
+
+#### Filtering disparities
+
+Certain disparities that may be found in regression testing are expected and, therefore, can be omitted. This is done with the _filters_ option. A filter is a JavaScript file composed per the EToolbox-Coconut documentation.
+
+By default, the filters are looked for at the current project's path in the `eak.regression/filters` subdirectory. E.g., `/home/projects/my-aem-project/eak.regression/filters`.
+
+This subdirectory can contain another subdir that refers to the "older" and "newer" ToolKit's versions delimited with _-to-_ like in the following example: `/home/projects/my-aem-project/eak.regression/filters/2.4.0-to-2.4.1-SNAPSHOT`. If such a subdirectory exists, the filters are borrowed from it. But if the there isn't a subdir matching the "older" and "newer" versions, the filters are borrowed from just `eak.regression/filters`.
+
+The comparison is done inside a temp folder. By default, the folder is erased after the regression is done. If you want to keep it, specify `-Dnocleanup=true`.
+
+#### Regression tips
+
+Be sure to _build_ the ToolKit completely before doing the regression, or else include the _install_ phase into the regression Maven run.
+
+Also, make sure that your target project(-s) are fully built with the newest changes, especially if you do regression with a selection of modules (via _-pl_). Otherwise there may occur a dependency resolution issue, or else some compilation errors.
+

it.tests/pom.xml

@@ -71,7 +71,7 @@
                                 <aem.host>${aem.host}</aem.host>
                                 <aem.port>${aem.port}</aem.port>
                                 <aem.login>${aem.login}</aem.login>
-                                <aem.password>${aem.login}</aem.password>
+                                <aem.password>${aem.password}</aem.password>
                             </systemPropertyVariables>
                             <includes>
                                 <include>**/AllTests.class</include>
@@ -81,6 +81,28 @@
                 </plugins>
             </build>
         </profile>
+        <profile>
+            <id>regression</id>
+            <build>
+                <plugins>
+                    <plugin>
+                        <groupId>org.apache.maven.plugins</groupId>
+                        <artifactId>maven-surefire-plugin</artifactId>
+                        <configuration>
+                            <skipTests>false</skipTests>
+                            <systemPropertyVariables>
+                                <eak.group>${project.parent.groupId}</eak.group>
+                                <eak.id>${project.parent.artifactId}</eak.id>
+                                <eak.version>${project.parent.version}</eak.version>
+                            </systemPropertyVariables>
+                            <includes>
+                                <include>**/RegressionTest.class</include>
+                            </includes>
+                        </configuration>
+                    </plugin>
+                </plugins>
+            </build>
+        </profile>
     </profiles>
 
     <dependencies>
@@ -104,6 +126,14 @@
             <groupId>com.codeborne</groupId>
             <artifactId>selenide</artifactId>
         </dependency>
+        <dependency>
+            <groupId>org.codehaus.plexus</groupId>
+            <artifactId>plexus-utils</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>com.exadel.etoolbox</groupId>
+            <artifactId>etoolbox-anydiff-core</artifactId>
+        </dependency>
         <dependency>
             <groupId>com.fasterxml.jackson.core</groupId>
             <artifactId>jackson-annotations</artifactId>
@@ -112,6 +142,10 @@
             <groupId>com.fasterxml.jackson.core</groupId>
             <artifactId>jackson-databind</artifactId>
         </dependency>
+        <dependency>
+            <groupId>com.google.code.findbugs</groupId>
+            <artifactId>jsr305</artifactId>
+        </dependency>
         <dependency>
             <groupId>com.google.jimfs</groupId>
             <artifactId>jimfs</artifactId>
@@ -140,6 +174,14 @@
             <groupId>org.apache.httpcomponents</groupId>
             <artifactId>httpmime</artifactId>
         </dependency>
+        <dependency>
+            <groupId>org.apache.maven</groupId>
+            <artifactId>maven-core</artifactId>
+        </dependency>
+        <dependency>
+            <groupId>org.apache.maven</groupId>
+            <artifactId>maven-model</artifactId>
+        </dependency>
         <dependency>
             <groupId>org.slf4j</groupId>
             <artifactId>slf4j-api</artifactId>

it.tests/src/test/java/com/exadel/aem/toolkit/it/AllowedChildrenTest.java

@@ -18,6 +18,7 @@
 import org.openqa.selenium.Keys;
 import com.codeborne.selenide.CollectionCondition;
 import com.codeborne.selenide.Condition;
+import com.codeborne.selenide.Configuration;
 import com.codeborne.selenide.ElementsCollection;
 import com.codeborne.selenide.Selenide;
 
@@ -50,6 +51,8 @@ public class AllowedChildrenTest {
 
     @BeforeClass
     public static void login() {
+        Configuration.timeout = AemConnection.TIMEOUT;
+        Configuration.pollingInterval = AemConnection.POLLING_INTERVAL;
         AemConnection.login();
     }
 

it.tests/src/test/java/com/exadel/aem/toolkit/it/IgnoreFreshnessTest.java

@@ -16,6 +16,7 @@
 import org.junit.BeforeClass;
 import org.junit.Test;
 import com.codeborne.selenide.Condition;
+import com.codeborne.selenide.Configuration;
 import com.codeborne.selenide.SelenideElement;
 
 import com.exadel.aem.toolkit.core.CoreConstants;
@@ -37,6 +38,8 @@ public class IgnoreFreshnessTest {
 
     @BeforeClass
     public static void login() {
+        Configuration.timeout = AemConnection.TIMEOUT;
+        Configuration.pollingInterval = AemConnection.POLLING_INTERVAL;
         AemConnection.login();
         AemConnection.open("/editor.html/content/etoolbox-authoring-kit-test/ignoreFreshness.html");
     }

it.tests/src/test/java/com/exadel/aem/toolkit/it/ListsTest.java

@@ -13,7 +13,6 @@
  */
 package com.exadel.aem.toolkit.it;
 
-import java.time.Duration;
 import java.util.Map;
 import java.util.function.Consumer;
 
@@ -26,6 +25,7 @@
 import com.google.common.collect.ImmutableMap;
 import com.codeborne.selenide.CollectionCondition;
 import com.codeborne.selenide.Condition;
+import com.codeborne.selenide.Configuration;
 import com.codeborne.selenide.ElementsCollection;
 import com.codeborne.selenide.Selenide;
 import com.codeborne.selenide.SelenideElement;
@@ -66,6 +66,8 @@ public class ListsTest {
 
     @BeforeClass
     public static void login() {
+        Configuration.timeout = AemConnection.TIMEOUT;
+        Configuration.pollingInterval = AemConnection.POLLING_INTERVAL;
         AemConnection.login();
     }
 
@@ -224,8 +226,9 @@ private static void displayDialog(
         }
 
         EditModeUtil.getInsertChildOptions(overlays.last()).first().click();
-        Selenide.Wait().withTimeout(Duration.ofSeconds(5)).until((webDriver) ->
-            Selenide.$$(SELECTOR_OVERLAY).size() == (existingItemProcessor != null ? 5 : 2));
+        Selenide.Wait()
+            .until((webDriver) ->
+                Selenide.$$(SELECTOR_OVERLAY).size() == (existingItemProcessor != null ? 5 : 2));
 
         listItemDialog = EditModeUtil.openComponentDialog("*/listitem", ACTION_CONFIGURE);
         newItemProcessor.accept(listItemDialog);

it.tests/src/test/java/com/exadel/aem/toolkit/it/OptionProviderTest.java

@@ -21,6 +21,7 @@
 import org.junit.Test;
 import com.codeborne.selenide.CollectionCondition;
 import com.codeborne.selenide.Condition;
+import com.codeborne.selenide.Configuration;
 import com.codeborne.selenide.ElementsCollection;
 import com.codeborne.selenide.Selenide;
 import com.codeborne.selenide.SelenideElement;
@@ -40,6 +41,8 @@ public class OptionProviderTest {
 
     @BeforeClass
     public static void login() {
+        Configuration.timeout = AemConnection.TIMEOUT;
+        Configuration.pollingInterval = AemConnection.POLLING_INTERVAL;
         AemConnection.login();
     }
 
@@ -57,7 +60,7 @@ public void shouldRenderFromSimpleList() {
             .stream()
             .map(option -> option.getAttribute("value"))
             .toArray(String[]::new);
-        Assert.assertArrayEquals(new String[] {"none", "en", "ja", "zh", "ww"}, greetingValues);
+        Assert.assertArrayEquals(new String[]{"none", "en", "ja", "zh", "ww"}, greetingValues);
         SelenideElement selectedGreetingOption = dialog.$("coral-select[name='./greeting'] coral-select-item[selected]");
         selectedGreetingOption.shouldHave(Condition.attribute("value", "en"));
     }
@@ -70,7 +73,7 @@ public void shouldRenderFromCustomList() {
             .stream()
             .map(option -> option.getAttribute(CoreConstants.PN_VALUE))
             .toArray(String[]::new);
-        Assert.assertArrayEquals(new String[] {"none", "linkedin", "myspace", "rss"}, socialChannelsValues);
+        Assert.assertArrayEquals(new String[]{"none", "linkedin", "myspace", "rss"}, socialChannelsValues);
         SelenideElement checkedSocialChannelOption = dialog.$("coral-radio[name='./socialChannel'][checked] input");
         checkedSocialChannelOption.shouldHave(Condition.attribute(CoreConstants.PN_VALUE, "rss"));
     }
@@ -83,7 +86,7 @@ public void shouldRenderFromNode() {
             .stream()
             .map(SelenideElement::getOwnText)
             .toArray(String[]::new);
-        Assert.assertArrayEquals(new String[] {"Nested-Container", "Empty"}, pageLabels);
+        Assert.assertArrayEquals(new String[]{"Nested-Container", "Empty"}, pageLabels);
     }
 
     @Test
@@ -94,7 +97,7 @@ public void shouldRenderFromTagFolder() {
             .stream()
             .map(option -> option.getAttribute(CoreConstants.PN_VALUE))
             .toArray(String[]::new);
-        Assert.assertArrayEquals(new String[] {"fruit:apple", "fruit:banana", "fruit:orange"}, fruitValues);
+        Assert.assertArrayEquals(new String[]{"fruit:apple", "fruit:banana", "fruit:orange"}, fruitValues);
         SelenideElement checkedFruitOption = dialog.$("coral-radio[name='./fruit'][checked] input");
         checkedFruitOption.shouldHave(Condition.attribute(CoreConstants.PN_VALUE, "fruit:banana"));
     }
@@ -108,7 +111,7 @@ public void shouldRenderFromEnum() {
             .map(SelenideElement::getOwnText)
             .toArray(String[]::new);
         Assert.assertArrayEquals(
-            new String[] {"None", "Blue", "Green", "Indigo", "Orange", "Red", "Violet", "Yellow"},
+            new String[]{"None", "Blue", "Green", "Indigo", "Orange", "Red", "Violet", "Yellow"},
             colorLabels);
         SelenideElement selectedColorOption = dialog.$("coral-select[name='./color1'] coral-select-item[selected]");
         Assert.assertEquals("Orange", selectedColorOption.getOwnText());
@@ -122,7 +125,7 @@ public void shouldRenderFromConstantsClass() {
             .stream()
             .map(option -> option.getAttribute(CoreConstants.PN_VALUE))
             .toArray(String[]::new);
-        Assert.assertArrayEquals(new String[] {"#FF0000", "#00FF00", "#0000FF", "#000000"}, colorValues);
+        Assert.assertArrayEquals(new String[]{"#FF0000", "#00FF00", "#0000FF", "#000000"}, colorValues);
     }
 
     @Test
@@ -139,29 +142,32 @@ public void shouldRenderLoadedAsync() {
         multisourceSelectionOptions.shouldBe(CollectionCondition.size(3));
 
         multisourceSelectionOptions.get(0).click();
-        Selenide.Wait().until(webDriver ->
-            dialog.$$(SELECTOR_MULTISOURCE_ITEM).shouldBe(CollectionCondition.size(3)));
+        Selenide.Wait()
+            .until(webDriver ->
+                dialog.$$(SELECTOR_MULTISOURCE_ITEM).shouldBe(CollectionCondition.size(3)));
         String[] optionValues = dialog.$$(SELECTOR_MULTISOURCE_ITEM)
             .asFixedIterable()
             .stream()
             .map(element -> element.getAttribute(CoreConstants.PN_VALUE))
             .toArray(String[]::new);
-        Assert.assertArrayEquals(new String[] {"carrot", "lettuce", "potato"}, optionValues);
+        Assert.assertArrayEquals(new String[]{"carrot", "lettuce", "potato"}, optionValues);
 
         multisourceSelectionOptions.get(1).click();
-        Selenide.Wait().until(webDriver ->
-            dialog.$$(SELECTOR_MULTISOURCE_ITEM).shouldBe(CollectionCondition.size(0)));
+        Selenide.Wait()
+            .until(webDriver ->
+                dialog.$$(SELECTOR_MULTISOURCE_ITEM).shouldBe(CollectionCondition.size(0)));
 
         multisourceSelectionOptions.get(2).click();
-        Selenide.Wait().until(webDriver ->
-            dialog.$$(SELECTOR_MULTISOURCE_ITEM).shouldBe(CollectionCondition.size(3)));
+        Selenide.Wait()
+            .until(webDriver ->
+                dialog.$$(SELECTOR_MULTISOURCE_ITEM).shouldBe(CollectionCondition.size(3)));
 
         optionValues = dialog.$$(SELECTOR_MULTISOURCE_ITEM)
             .asFixedIterable()
             .stream()
             .map(element -> element.getAttribute(CoreConstants.PN_VALUE))
             .toArray(String[]::new);
-        Assert.assertArrayEquals(new String[] {"apple", "banana", "orange"}, optionValues);
+        Assert.assertArrayEquals(new String[]{"apple", "banana", "orange"}, optionValues);
     }
 
     @After

it.tests/src/test/java/com/exadel/aem/toolkit/it/base/AemConnection.java

@@ -14,6 +14,7 @@
 package com.exadel.aem.toolkit.it.base;
 
 import java.nio.charset.StandardCharsets;
+import java.time.Duration;
 import java.util.Arrays;
 import java.util.Base64;
 
@@ -32,6 +33,9 @@
  */
 public class AemConnection {
 
+    public static final int TIMEOUT = 10000;
+    public static final int POLLING_INTERVAL = 1000;
+
     private static final String PROPERTY_HOST = "aem.host";
     private static final String PROPERTY_PORT = "aem.port";
     private static final String PROPERTY_LOGIN = "aem.login";
@@ -71,7 +75,10 @@ private AemConnection() {
     public static void open(String path, String... variants) {
         String url = getUrl(path);
         Selenide.open(url);
-        Selenide.Wait().until(webDriver ->
+        Selenide.Wait()
+            .withTimeout(Duration.ofMillis(TIMEOUT))
+            .pollingEvery(Duration.ofMillis(POLLING_INTERVAL))
+            .until(webDriver ->
             (webDriver.getCurrentUrl().equals(url)
                 || Arrays.stream(ArrayUtils.nullToEmpty(variants)).anyMatch(v -> webDriver.getCurrentUrl().equals(getUrl(v))))
                 && ((JavascriptExecutor) webDriver).executeScript("return document.readyState").equals("complete"));
@@ -90,7 +97,10 @@ public static void login() {
         Selenide.element(By.name("j_username")).setValue(getProperty(PROPERTY_LOGIN));
         Selenide.element(By.name("j_password")).setValue(getProperty(PROPERTY_PASSWORD));
         Selenide.element(By.id("submit-button")).click();
-        Selenide.Wait().until(webDriver -> webDriver.getCurrentUrl().contains(LANDING_PATH));
+        Selenide.Wait()
+            .withTimeout(Duration.ofMillis(TIMEOUT))
+            .pollingEvery(Duration.ofMillis(POLLING_INTERVAL))
+            .until(webDriver -> webDriver.getCurrentUrl().contains(LANDING_PATH));
     }
 
     /**

it.tests/src/test/java/com/exadel/aem/toolkit/it/base/PackageSuite.java

@@ -208,7 +208,7 @@ private static void run(RunNotifier notifier, Object testObject) {
                     e.getCause()
                 ));
                 if (method.isAnnotationPresent(Order.class)) {
-                    // It makes sense to terminate the testing flow  in case of failure for only ordered methods
+                    // It makes sense to terminate the testing flow in case of failure for only ordered methods
                     // (assuming that a subsequent method relies on the current one)
                     break;
                 }