This repository has been archived by the owner on Feb 11, 2022. It is now read-only.
Unable to run tests using Element lookup & Object Detection mode for iOS simulators #32
Comments
|
any updates about it? |
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Hi @jlipps ,
I tried to run the same tests on iOS Simulators which was successful on Android Emulators. But its unable to identify elements & below are the error details. Can you please guide & help resolve the issue here?
For Element Lookup Mode:
`
Code used ::
$("findElementByCustom :::----> Search icon using AI");
getAppDriver().findElement(MobileBy.custom("ai:search")).click();
Capabilities set
capabilities.setCapability("customFindModules", customFindModules);
capabilities.setCapability("testaiFindMode", "element_lookup");
capabilities.setCapability("testaiObjectDetectionThreshold", "0.1");
capabilities.setCapability("shouldUseCompactResponses", false);
2020-01-29 13:13:04,144 120104 [TestNG-tests-1] INFO com.walmart.asda.mobile.poc.TestSearch_01 - Executing : findElementByCustom :::----> Search icon using AI
[HTTP] --> POST /wd/hub/session/787a5a29-2972-4377-b5c1-d354b07c8b6b/element
[HTTP] {"using":"-custom","value":"ai:search"}
[debug] [W3C (787a5a29)] Calling AppiumDriver.findElement() with args: ["-custom","ai:search","787a5a29-2972-4377-b5c1-d354b07c8b6b"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
2020-01-29 13:13:06.149943: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-29 13:13:06.183955: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x102b8ec80 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-29 13:13:06.183991: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
[debug] [BaseDriver] Waiting up to 0 ms for condition
[ai] Retrieving current settings to check element response attributes
[ai] We will need to update settings to include element response attributes
[debug] [WD Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [WD Proxy] Proxying [POST /appium/settings] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/appium/settings] with body: {"settings":{"elementResponseAttributes":"rect"}}
[debug] [WD Proxy] Got response with status 200: {"value":{"screenshotQuality":1,"mjpegScalingFactor":100,"elementResponseAttributes":"rect","shouldUseCompactResponses":false,"mjpegServerFramerate":10,"mjpegServerScreenshotQuality":25},"sessionId":"C8C31F6A-57DD-4036-9457-297962A87A38","status":0}
[ai] Setting confidence threshold to default value of 0.1
[ai] Retrieving data for all leaf-node elements on screen
[debug] [BaseDriver] Valid locator strategies for this request: xpath, id, name, class name, -ios predicate string, -ios class chain, accessibility id
[debug] [BaseDriver] Waiting up to 0 ms for condition
[debug] [WD Proxy] Matched '/elements' to command name 'findElements'
[debug] [WD Proxy] Proxying [POST /elements] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/elements] with body: {"using":"xpath","value":"//[not(child::)]"}
[debug] [WD Proxy] Got response with status 200: {"value":[{"ELEMENT":"2F000000-0000-0000-8852-010000000000","rect":{"y":0,"x":0,"width":375,"height":40}},{"ELEMENT":"3C000000-0000-0000-8852-010000000000","rect":{"y":50,"x":150,"width":75,"height":32}},{"ELEMENT":"3D000000-0000-0000-8852-010000000000","rect":{"y":44,"x":286,"width":73,"height":44}},{"ELEMENT":"A4000000-0000-0000-8852-010000000000","rect":{"y":88,"x":0,"width":375,"height":62}},{"ELEMENT":"A5000000-0000-0000-8852-010000000000","rect":{"y":88,"x":16,"width":359,"height":62}},{"ELEMENT":"C1000000-0000-0000-8852-010000000000","rect":{"y":170,"x":16,"width":119,"height":27}},{"ELEMENT":"C2000000-0000-0000-8852-010000000000","rect":{"y":196,"x":16,"width":266,"height":49}},{"ELEMENT":"C3000000-0000-0000-8852-010000000000","rect":{"y":248,"x":16,"width":343,"height":3}},{"ELEMENT":"C4000000-0000-0000-8852-010000000000","rect":{"y":254,"x":16,"width":214,"height":25}},{"ELEMENT":"D0000000-0000-0000-8852-010000000000","rect":{"y":308,"x":16,"width":164,"height":40}},{"ELEMENT":"D1000000-0000-0000...
[ai] Getting window size in case we need to scale screenshot
[debug] [WD Proxy] Proxying [GET /window/size] to [GET http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/window/size] with no body
[debug] [WD Proxy] Got response with status 200: "{\n "value" : {\n "width" : 375,\n "height" : 812\n },\n "sessionId" : "C8C31F6A-57DD-4036-9457-297962A87A38",\n "status" : 0\n}"
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [WD Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [WD Proxy] Proxying [GET /screenshot] to [GET http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/screenshot] with no body
[debug] [WD Proxy] Got response with status 200: "{\n "value" : "iVBORw0KGgoAAAANSUhEUgAABGUAAAmECAIAAAAjR0TuAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAATCAAAAKAAABMIAAATCAAavDLBpvG4AAEAASURBVHgB\r\n7L0HfB7Fue9\/z\/3fc0\/LOScJ1V2W5YYxJUDoENKBAKElhBQIkBBOICS4qEuWuwED\r\nSSAhpN4AORwCBDf1atNC770GAqa6SZZkSd7\/95nnfcdrWRjZ4AL+7We9np2dnZn9\r\n7uyr57fPzOz\/+l9aREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAEREAE\r\nREAEREAEREA...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x812 but screenshot size is 1125x2436. Scaled screenshot to match screen size
[ai] Getting screenshot slices for each element
[ai] Resetting element response attribute setting to original value: "type,label"
[debug] [WD Proxy] Matched '/appium/settings' to command name 'updateSettings'
[debug] [WD Proxy] Proxying [POST /appium/settings] to [POST http://localhost:49049/session/C8C31F6A-57DD-4036-9457-297962A87A38/appium/settings] with body: {"settings":{"elementResponseAttributes":"type,label"}}
[debug] [WD Proxy] Got response with status 200: {"value":{"screenshotQuality":1,"mjpegScalingFactor":100,"elementResponseAttributes":"type,label","shouldUseCompactResponses":false,"mjpegServerFramerate":10,"mjpegServerScreenshotQuality":25},"sessionId":"C8C31F6A-57DD-4036-9457-297962A87A38","status":0}
[debug] [W3C (787a5a29)] Encountered internal error running command: TypeError: Cannot destructure property
xof 'undefined' or 'null'.[debug] [W3C (787a5a29)] at getCanvasByRect (/usr/local/lib/node_modules/test-ai-classifier/lib/image.js:57:9)
[debug] [W3C (787a5a29)] at map (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:171:28)
[debug] [W3C (787a5a29)] at Array.map ()
[debug] [W3C (787a5a29)] at asyncmap (/usr/local/lib/node_modules/test-ai-classifier/node_modules/asyncbox/lib/asyncbox.js:74:26)
[debug] [W3C (787a5a29)] at getElementImages (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:169:30)
[debug] [W3C (787a5a29)] at findViaElementScreenshots (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:303:32)
[debug] [W3C (787a5a29)] at Object.find (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:377:12)
[debug] [W3C (787a5a29)] at condition (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:159:16)
[debug] [W3C (787a5a29)] at wrappedCondFn (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:157:12)
[HTTP] <-- POST /wd/hub/session/787a5a29-2972-4377-b5c1-d354b07c8b6b/element 500 43510 ms - 611
[HTTP]
org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: Cannot destructure property
xof 'undefined' or 'null'.Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'm-c02xf25rjg5h', ip: 'fe80:0:0:0:1c5b:4ba:7301:ce39%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '1.8.0_192'
Driver info: io.appium.java_client.ios.IOSDriver
Capabilities {app: /Users/kverma1/Documents/mo..., automationName: XCUITest, browserName: , customFindModules: {ai: test-ai-classifier}, databaseEnabled: false, deviceName: iPhone X, fullReset: true, javascriptEnabled: true, locationContextEnabled: false, networkConnectionEnabled: false, newCommandTimeout: 120, noReset: false, platform: MAC, platformName: iOS, platformVersion: 12.0, shouldUseCompactResponses: false, takesScreenshot: true, testaiFindMode: element_lookup, testaiObjectDetectionThreshold: 0.1, udid: E18161C4-00ED-481A-9281-867..., useNewWDA: false, wdaLocalPort: 49049, webStorageEnabled: false}
Session ID: 787a5a29-2972-4377-b5c1-d354b07c8b6b
*** Element info: {Using=-custom, value=ai:search}
`
For Object Detection Mode:
`
Code used ::
$("findElementByCustom :::----> Search icon using AI");
getAppDriver().findElement(MobileBy.custom("ai:search")).click();
Capabilities set
capabilities.setCapability("customFindModules", customFindModules);
capabilities.setCapability("testaiFindMode", "element_lookup");
capabilities.setCapability("testaiObjectDetectionThreshold", "0.1");
capabilities.setCapability("shouldUseCompactResponses", false);
2020-01-29 13:42:08,349 120921 [TestNG-tests-1] INFO com.walmart.asda.mobile.poc.TestSearch_01 - Executing : findElementByCustom :::----> Search icon using AI
[HTTP] --> POST /wd/hub/session/8653e25f-6cd0-4603-b4c2-a942bf0c2f77/element
[HTTP] {"using":"-custom","value":"ai:search"}
[debug] [W3C (8653e25f)] Calling AppiumDriver.findElement() with args: ["-custom","ai:search","8653e25f-6cd0-4603-b4c2-a942bf0c2f77"]
[debug] [XCUITest] Executing command 'findElement'
[debug] [BaseDriver] Find plugin 'ai' requested; will attempt to use it from 'test-ai-classifier'
2020-01-29 13:42:10.518184: I tensorflow/core/platform/cpu_feature_guard.cc:142] Your CPU supports instructions that this TensorFlow binary was not compiled to use: AVX2 FMA
2020-01-29 13:42:10.544574: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x10c8c5770 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
2020-01-29 13:42:10.544601: I tensorflow/compiler/xla/service/service.cc:176] StreamExecutor device (0): Host, Default Version
[debug] [BaseDriver] Waiting up to 0 ms for condition
[ai] Will use object detection method for finding elements via AI
[ai] Setting confidence threshold to default value of 0.1
[ai] Getting window size in case we need to scale screenshot
[debug] [WD Proxy] Proxying [GET /window/size] to [GET http://localhost:46146/session/27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4/window/size] with no body
[debug] [WD Proxy] Got response with status 200: "{\n "value" : {\n "width" : 375,\n "height" : 812\n },\n "sessionId" : "27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4",\n "status" : 0\n}"
[ai] Getting screenshot to use for classifier
[debug] [XCUITest] Taking screenshot with WDA
[debug] [WD Proxy] Matched '/screenshot' to command name 'getScreenshot'
[debug] [WD Proxy] Proxying [GET /screenshot] to [GET http://localhost:46146/session/27188DFA-B718-41AD-AEF5-B4FCBAFDDBB4/screenshot] with no body
[debug] [WD Proxy] Got response with status 200: "{\n "value" : "iVBORw0KGgoAAAANSUhEUgAABGUAAAmECAIAAAAjR0TuAAAAAXNSR0IArs4c6QAA\r\nABxpRE9UAAAAAgAAAAAAAATCAAAAKAAABMIAAATCAAavbYrc7aAAAEAASURBVHgB\r\n7J0HnB5Vvfdf3\/fe973FexWlpSebTSOEIiBdRK4FFJAqoleqIioipGzfzaaRAAFU\r\nVNSLeilepQTStj9bEpr03qtECISQutnd7G7m\/f7P\/3lOnuxuQgrJRvKbz2Ry5syZ\r\nc\/7znTnP\/n9zyvyv\/6VFBERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERABERABERABERABERABERABERABERABERABERABERABERABERABERABERA\r\nBERABERAB...
[ai] Turning screenshot into HTML image for use with canvas
[ai] Screenshot and screen size did not match. Screen size is 375x812 but screenshot size is 1125x2436. Scaled screenshot to match screen size
[ai] Writing screenshot to /usr/local/lib/node_modules/test-ai-classifier/build-js/lib/Wed, 29 Jan 2020 08:12:11 GMT_classifiertest.png for use in object detection
[ai] Gathering object candidates from tensorflow native
2020-01-29 13:42:11.456773: I tensorflow/cc/saved_model/reader.cc:31] Reading SavedModel from: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model
2020-01-29 13:42:11.756360: I tensorflow/cc/saved_model/reader.cc:54] Reading meta graph with tags { serve }
2020-01-29 13:42:12.260225: I tensorflow/cc/saved_model/loader.cc:202] Restoring SavedModel bundle.
2020-01-29 13:42:12.261404: I tensorflow/cc/saved_model/loader.cc:212] The specified SavedModel has no variables; no checkpoints were restored. File does not exist: /usr/local/lib/node_modules/test-ai-classifier/model/obj_detection_model/variables/variables.index
2020-01-29 13:42:12.261778: I tensorflow/cc/saved_model/loader.cc:311] SavedModel load for tags { serve }; Status: success. Took 805000 microseconds.
[ai] Candidates retrieved
[ai] Making label predictions based on element images
This model execution did not contain any nodes with control flow or dynamic output shapes. You can use model.execute() instead.
[ai] Predictions for 34 element(s) took 0.686 seconds
[ai] Found 1 matching elements
[ai] Highest confidence of any element for desired label 'search' was 0.9994727373123169
[ai] Registering image elements with driver
[debug] [W3C (8653e25f)] Encountered internal error running command: TypeError: driver.registerImageElement is not a function
[debug] [W3C (8653e25f)] at map (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:365:19)
[debug] [W3C (8653e25f)] at Array.map ()
[debug] [W3C (8653e25f)] at findByObjectDetector (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:357:24)
[debug] [W3C (8653e25f)] at Object.find (/usr/local/lib/node_modules/test-ai-classifier/lib/classifier.js:375:12)
[debug] [W3C (8653e25f)] at condition (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/find.js:159:16)
[debug] [W3C (8653e25f)] at wrappedCondFn (/usr/local/lib/node_modules/appium/node_modules/appium-base-driver/lib/basedriver/commands/timeout.js:157:12)
[HTTP] <-- POST /wd/hub/session/8653e25f-6cd0-4603-b4c2-a942bf0c2f77/element 500 17780 ms - 587
[HTTP]
org.openqa.selenium.WebDriverException: An unknown server-side error occurred while processing the command. Original error: driver.registerImageElement is not a function
Build info: version: '3.141.59', revision: 'e82be7d358', time: '2018-11-14T08:17:03'
System info: host: 'm-c02xf25rjg5h', ip: 'fe80:0:0:0:1c5b:4ba:7301:ce39%en0', os.name: 'Mac OS X', os.arch: 'x86_64', os.version: '10.14.6', java.version: '1.8.0_192'
Driver info: io.appium.java_client.ios.IOSDriver
Capabilities {app: /Users/kverma1/Documents/mo..., automationName: XCUITest, browserName: , customFindModules: {ai: test-ai-classifier}, databaseEnabled: false, deviceName: iPhone X, fullReset: true, javascriptEnabled: true, locationContextEnabled: false, networkConnectionEnabled: false, newCommandTimeout: 120, noReset: false, platform: MAC, platformName: iOS, platformVersion: 12.0, shouldUseCompactResponses: false, takesScreenshot: true, testaiFindMode: object_detection, testaiObjectDetectionThreshold: 0.4, udid: E18161C4-00ED-481A-9281-867..., useNewWDA: false, wdaLocalPort: 46146, webStorageEnabled: false}
Session ID: 8653e25f-6cd0-4603-b4c2-a942bf0c2f77
*** Element info: {Using=-custom, value=ai:search}
`
The text was updated successfully, but these errors were encountered: