-
Notifications
You must be signed in to change notification settings - Fork 27
QA Testing
-
Start with a new install of the current release of macOS. Use Virtual Buddy to create a VM for this or a new macOS install on hardware if available.
-
Create an admin account on macOS and install the target build of XCreds.
-
Download the example profiles and begin by installing
xcreds preferences 1that sets the basic preferences for Azure authentication. -
The other example profiles include collections of preferences that can be tested together. They also include separate profiles to test basic authentication for Okta and Okta with ROPG, and for Google.
-
Use XCreds preference testing worksheet (Twocanoes internal link) to record testing status for each preference on the target build. The worksheet also includes information to help with the testing process.
-
Open the example Azure mobileconfig file in a text editor and find the section for
clientID. Replace the [redacted] value forclientIDwith the current value for the test Azure account. This value should either be provided for testing or can be found from Azure Portal (See Azure section below). Repeat the same for the Okta mobileconfig file. -
Double click the Azure mobileconfig file and go to macOS preferences > Privacy & Security > Profiles. Click on the new profile to install it.
-
Restart macOS and sign in using XCreds with the cloud provider user credentials provided separately. Confirm a macOS user account is created for the Azure user account.
-
When signed in to macOS using cloud user credentials, launch XCreds from the Applications folder in Finder. Confirm the XCreds menubar item is shown and that after a moment it shows a green indicator.
-
Click the menubar item, then check the About the dialog content.
-
Click the menubar item, then click "Sign In" and confirm re-entering cloud credentials shows no indication of failure.
-
Sign out of macOS and note the macOS username shown when signing out. Then from the XCreds cloud login window click "Switch Login Window" and confirm signing in using the macOS username for this account along with the same password entered earlier.
-
Turn off WiFi (if using Virtual Buddy, go to
macOS settings > Network > Ethernetand clickMake Inactive). While network access is disabled, sign out of macOS, then from the XCreds cloud login window click "Switch Login Window" and try signing in using an existing macOS username and password. Turn WiFi back on before proceeding further. -
Confirm that a trial status message is shown on the XCreds login window.
-
Install an XCreds license file, then sign out of macOS and confirm the trial status message is no longer shown and that the user can sign in again from the XCreds cloud login window.
-
Remove license from macOS settings > profiles. Confirm the trial status message is shown again and note the trial time duration it states. Update the macOS system time to simulate going beyond the expected trial end date. Retest XCreds menubar item refresh and XCreds login window sign in and confirm expected result. Update the macOS system time and/or install a different license to simulate that the license will be expiring in less than 14 days. Confirm that the login screen shows the correct number of days until license expiration (accounting for grace period). Reset macOS time to normal and reinstall XCreds license before proceeding further.
If the current mobileconfig file used for XCreds settings was recently created with Profile Creator, double check that Profile Creator has the General tab section for Payload Scope is set to System. If this is left as the default value of User the XCreds login window webview will display an error or waiting message.
It can be helpful to view the content of the installed settings profile in macOS Preferences > Privacy & Security > Profiles. When saving and exporting a settings profile from Profile Creator it can be easy to miss that a setting change was not included in the exported file. When in doubt go to macOS Preferences and view the content of the installed profile to confirm the settings values included.
When testing quick changes to a settings profile it can sometimes be better to use a text editor instead of Profile Creator. This requires care to edit XML keys manually but can be faster for some testing scenarios.
It may be necessary to update the Azure mobileconfig file value for clientID to match the value for the example Azure account. The clientID can be found from the Azure Portal by searching for App Registrations, then clicking All applications and finding the record for xcreds. This will show the value to use for XCreds for Application (client) ID.
It may be necessary to verify the Azure mobileconfig file value for redirectURI. This can be any value but must match configuration in Azure Portal. Go to Azure Portal > App Registrations > All applications > xcreds. Click on Redirect URIs and verify the correct value to use in the mobileconfig file or add a new redirect URI in Azure Portal for this.
If anything in testing requires reverting to the regular macOS login window to sign in, go to the XCreds cloud login window and hold down the Command key when clicking Switch Login Window. This is not intended for end users but can be used for testing to be able to log in using the regular macOS login window.
The setting for loginWindowBackgroundImageURL should change both the XCreds cloud login window background and the XCreds local login window. The image for this needs to be of type heic, png, or jpeg. Also to be able to see this background on the XCreds cloud login window, the settings for loginWindowWidth and loginWindowHeight need to be set in the profile as well to, for example, 500 each.
If a testing scenario results in an error rendering the login screen, or configuration is done to hide the login window option buttons and then these buttons become necessary to proceed, reboot to the recovery partition and run the following commands in Terminal to reset the XCreds auth DB and overlay. This will cause XCreds to self-heal and recreate the auth.db file, which should restore the access needed.
Note: after doing this and logging in it will be necessary to rename the com.twocanoes.xcreds-overlay.plist file back to the original name in order to restore showing the XCreds sign-in screen.
mv /Volumes/Macintosh \HD/var/db/auth.db /Volumes/Macintosh \HD/var/db/auth.db.aside
mv /Volumes/Macintosh \HD/Library/LaunchAgents/com.twocanoes.xcreds-overlay.plist /Volumes/Macintosh \HD/Library/LaunchAgents/com.twocanoes.xcreds-overlay.plist.aside
See the Admin Guide preferences section to set up Profile Creator. Profile Creator is used to prepare a mobileconfig file that will change the default XCreds settings.
When testing unreleased builds of XCreds it is necessary to update Profile Creator with a manifest file that contains information on what is new in XCreds. To get the manifest file for the current build, install XCreds and find the app file in Finder > Applications. Control-click the app file and choose Show Package Contents. Go to Contents > Resources > com.twocanoes.xcreds.plist and copy this plist file. After installing and launching Profile Creator, go to Profile Creator settings > Payloads. Uncheck the two items near the bottom under Payload Manifests that says Automatically check for updates (on launch) and Automatically download available updates. These must be disabled to allow overriding the manifest file for testing. Next go to Finder and press Ctrl-Shift-G and enter the following file path: ~/Library/Application Support/ProfilePayloads/Manifests/ManagedPreferencesApplications. Press enter to go to this location and paste the manifest plist file copied from the Applications folder. The copied file should replace the corresponding file in ~/Library/.... Relaunch Profile Creator after updating the manifest plist file.
In Profile Creator click the plus button to create a new profile and give it a name in the box for display name. Next a few lines below the box for profile name find Payload Scope and select System. Then from the toolbar at the left side find where it says General 1 Payload and below this find where there is a row with a checkmark, an Apple logo, and an App Store logo. Click the App Store logo, then below this scroll to the bottom and find the item named XCreds. Drag the XCreds item up to just under where is says General 1 Payload.
There should now be one item that says General 1 Payload and one that says XCreds 1 Payload. Click on the XCreds one and confirm that the current XCreds version and build numbers are shown near the top of the window.
The profile window will then show a list of Disabled Keys with a description and a plus button for each to add them to the profile. Start with creating the simplest profile for Azure authentication by adding the keys for: Client ID, Discovery URL, and Redirect URI. Set Client ID to the corresponding value for the Azure account used. The value for Discovery URL can be left at the default value. The value for Redirect URI should be set to https://127.0.0.1/xcreds (the current value configured for this in the Azure account for testing).
After adding keys and values to the profile, press Command-s to save, then Command-e to export. Choose a Finder location and confirm the file name used before clicking Save. Before applying the profile, go to macOS settings > Profiles. Remove any other profiles added for XCreds other than the license file. Then go back to the new profile in Finder and double click the file. Then go back to macOS settings and click on the new profile to install it. The profile should display the keys and values entered when installing it.
Test the profile by signing out of macOS and signing back in with Azure user account credentials. If unable to sign in, first check network access, then if necessary use the button to switch the login window and sign in with a local user account. Open Profile Creator again and verify that the profile created was set to have the section for General 1 Payload be set to have Payload Scope set as System.
Review information for the current build and use Profile Creator to test changing any settings that have been added, removed, or changed.
Use Azure Portal User section to manage user accounts for testing. Use for creating new user accounts, adding users to a group, and to reset passwords as needed for testing corresponding features of XCreds.
See what's new page
- shouldSetGoogleAccessTypeToOffline
- idpHostName
- idpHostNames
- map_firstname
- map_lastname
- map_fullname
- map_username
- redirectURI
- passwordElementID
- EnableFDE
- EnableFDERecoveryKey
- EnableFDERecoveryKeyPath
- EnableFDERekey
- shouldShowPreferencesOnStart
- KeychainReset
- shouldFindPasswordElement
Verify that an Active Directory server is currently running locally at 10.1.10.244. Configure the XCreds test machine in macOS settings by searching for DNS and selecting DNS Domains. Under DNS Servers click the plus button to add a new item with the value 10.1.10.244. Then in Profile Creator set the value of the test profile key ADDomain to be twocanoes.com. Check with someone maintaining the AD server to verify user account credentials to use for testing. On a machine with DNS configured and a profile installed with this setting for ADDomain it should be possible to go to the XCreds sign in screen, click Switch Login Window to go to the local XCreds sign in screen, and then sign into macOS using the relevant AD user credentials. After signing in go to macOS settings > Users & Groups. Verify that the username, first name, and last name values are as expected.
Launch XCreds from the Applications folder. Check if the cloud login window is shown. The cloud login window should be shown if XCreds settings are currently also configured to support cloud login (such as Azure). The cloud login window should not be shown if XCreds settings are currently only configured to support Active Directory login. If XCreds settings are configured to support both Active Directory and cloud login, the XCreds cloud login window will be shown when launching XCreds from the Applications folder but is not needed so close this window and confirm the XCreds menubar item remains running.
Next with the XCreds menubar item running, verify that a Kerberos ticket is created when signed in to macOS as the AD user created by XCreds. Use Spotlight to search for and open an app called Ticket Viewer. Ticket Viewer can also be found at /System/Library/CoreServices/Applications. When Ticket Viewer opens it should show an entry in the list shown. If nothing is shown in the Ticket Viewer list verify that the AD server machine is running. If the AD server is running and Ticket Viewer does not list any tickets, file a bug for further examination.
Next test changing the user password. When signed in to macOS as an AD user, open Ticket Viewer and use the button to change password. Then sign out of macOS and sign in again as the AD user using the new password. Make sure that no configuration is in place the would force offline authentication. Verify that the AD user can sign in using the new password. After signing in again, open Ticket Viewer and change the password back to the original value for future testing.
Note that this DNS configuration will disrupt normal behavior when going to twocanoes.com in a browser. If necessary go back to macOS settings for DNS and remove the DNS Servers item added to restore normal behavior for this.
From Azure Portal go to Users and verify two user accounts to use. Click on one of these user accounts to add them to a group. If necessary click Groups from the left navigation and verify a group to use or add a new one. Copy the Object Id for this group. Go back to the page for the target Azure user and assign them to the relevant group. Verify that this target user is assigned to this group and the other target user is not assigned to the group. Use Profile Creator to set the value of the test profile key CreateAdminIfGroupMember to be the object ID previously copied for the Azure group. Install the profile on a machine that does not yet have user accounts for the two target Azure users. Then use the XCreds cloud login window to sign in as each of these Azure users. After signing in as both, go to macOS settings > Users & Groups. Verify that the user assigned to the Azure group was created in macOS as Admin and the other user was created as Standard.
Use Profile Creator to set shouldSwitchToLoginWindowWhenLocked as checked or edit the file directly and set the value for this key as true. After installing the profile launch XCreds from the Applications folder. Then from the macOS Apple menu, lock the screen and press the escape key. Then sign back in and verify that the XCreds cloud login window was shown for this. After signing in verify that the XCreds menubar item is still present to indicate that the user session was maintained.
See the example plist content shown below. Copy this content into a text editor and save the file as override.plist. In Finder press Command-Shift-G and enter the path /usr/local and press enter. Create a new folder called xcreds and copy the plist file there.
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>loginWindowBackgroundImageURL</key>
<string>file://DESKTOPPICTUREPATH</string>
</dict>
</plist>
See the test script content shown below. Paste this content into a text editor and save the file as override.sh. Then copy the script file to the /usr/local/xcreds folder used for the plist file. Use Terminal to change ownership of the file to securityagent: sudo chown _securityagent /usr/local/xcreds/override.sh. Then change permissions of the file to 700 by running sudo chmod 700 /usr/local/xcreds/override.sh.
#!/bin/sh
dir="/System/Library/Desktop Pictures/Solid Colors"
desktoppicture=`/bin/ls -1 "$dir"/*.png | sort --random-sort | head -1`
cat /usr/local/xcreds/override.plist|sed "s|DESKTOPPICTUREPATH|${desktoppicture}|g"
Use Profile Creator to set the key settingsOverrideScriptPath to be /usr/local/xcreds/override.sh or what is the current full file path to the test script file if named or located differently than described here. Also in Profile Creator set the values for both loginWindowHeight and loginWindowWidth to be 500. Save, export, and install the profile, then sign out of macOS. Verify that the script sets a random background image for the XCreds login screen that changes every time the XCreds login screen is shown. Note that displaying a background image requires values be set for loginWindowHeight and loginWindowWidth. If the background image is not seen on the XCreds login window review the contents of the profile to confirm these were included.
These should not normally be set in the profile directly, however this should be done for testing. Verify a current admin account's credentials on the test machine and use this user name and password in Profile Creator to set these keys. Also set the setting PasswordOverwriteSilent to initially be either not set or set to false. Install the profile and remove any other profiles previously installed.
In Azure Portal go to the Users section and reset the password for a user that has previously signed into the XCreds test machine. Copy the new password shown for later use.
Sign out of macOS on the XCreds test machine and use the XCreds cloud login window to sign in as this Azure user. When prompted enter the temporary value for the current password and choose a new password that is different than the original password before it was reset. A prompt should be shown asking for the old local password. Do not enter the old local password and instead click the Reset button. The user should then be logged in to macOS without being prompted for an admin password to approve the password change.
After signing in to macOS as the user whose password was reset, verify that the user's Keychain was moved aside and a new one was created. This is done using Finder. In Finder press Command-Shift-G to go to the path ~/Library/Keychains/. There should be a new Keychain file in this folder along with the previous one that was renamed, both with names beginning with login and with the file extension keychain-db.
Next update the profile to set PasswordOverwriteSilent as true. Reinstall the profile, then reset the Azure user's password again and repeat the steps above to sign in. With PasswordOverwriteSilent set as true, expect that XCreds will not show the dialog that previously required clicking Reset. After signing in, confirm that the user now has an additional keychain file in the keychain folder.
Test the following new keys as described in the admin guide:
- shouldShowCloudLoginByDefault
- shouldShowMacLoginButton
Then check changing all other preferences from the admin guide except those listed above that should not be tested.
See what's new page
- ropgClientID
- ropgClientSecret
- shouldVerifyPasswordWithRopg
Testing for these should be skipped until testing materials for Okta authentication can be made available.
Set this key to true, apply the mobileconfig file, then sign out and confirm that the local login window is shown first instead of the cloud login window.
Set this key to true, apply the mobileconfig file, and turn off WiFi. Confirm no other XCreds settings are changed that would affect choice of login window, then sign out and confirm that the local login window is shown first instead of the cloud login window. Sign back in and turn on WiFi, then sign out again and confirm that the cloud login window is shown first.