Listing some of the common errors when running the UST and tips to resolve them
FileNotFoundError: [Errno 2]
Sample console error output:
FileNotFoundError: [Errno 2] No such file or directory: 'C:\\Users\\USER\\.pex\\install\\pycryptodome [...]'
May appear on Windows because of paths > 256 chars.
Tip: create an environment variable named PEX_ROOT with value C:\pex (if the script is run from the C: drive, or change the letter to match). Sometimes a system restart is required for it to take effect.
can't open file 'user-sync.pex': [Errno 2]
python: can't open file 'user-sync.pex': [Errno 2] No such file or directory
Tip: make sure you run the python command line from inside the folder where the user-sync.pex is located.
2018-01-01 11:49:42 28102 WARNING umapi - UMAPI timeout...service unavailable (code 429 on try 1)
2018-01-01 11:49:42 28102 WARNING umapi - Next retry in 42 seconds...
Tip: If the timeout is less than 30 minutes, these Warning messages are received when the quota of API calls to make within one minute was reached. UST comes with an exponential back-off mechanism to retry the call, increasing the time between retries. The script stops after three failed attempts.
The recommendaton here is to let the script run all the way to the end.
If the timeout is higher than 1000 seconds, this would be a throttling related to the run frequency of each User Sync Tool instance. If an instance is observed to be run more frequently it will be throttled anywhere between 30 and 75 minutes. Although the timeout just puts the UST on hold for a period of time (so not a critical stop kind of error), the UST will know to recover and continue the sync right after.
Since the script recognises when two instances are launched at same time, no new instance can be triggered until the initial one has finished running. In this case an error message might appear in the UST log referring to 'a process is already in progress'.
Here are some recommendations for best performance, in regards to the run frequency:
- revise the scheduled task frequency and make sure that it is set to repeat at least 2 hours apart
- revise the scheduled task trigger time so it does not start at :00 or :30 minute mark (avoid peak traffic activity)
- if it is required to run the tool more often, reflect on the possibility to use UST with push strategy (delta of changes) instead of full sync
- adapt the run time schedule of the UST to your organisation's work day when users need provisioned (eg if your organization doesn’t have a need to modify user provisioning at night, do not run syncing jobs during this time)
error.user.belongs_to_another_org
2018-01-01 11:49:42 28102 ERROR umapi.action - Error in requestID: action_1 (User: {'user': 'myuser@domain2.com', 'requestID':'action_1'}, Command: {'createFederatedID': {'email': 'myuser@domain2.com', 'country': 'US', 'option': 'ignoreIfAlreadyExists', 'firstname': 'fname', 'lastname': 'lname'}}): code: "error.user.belongs_to_another_org" message: "Illegal to invite user from another organization's owned auth src"
Tip: the domain used to create the account might not be a claimed/trusted inside your Organizations. A green flag or dot should appear for the active domains in Admin Console -> Settings. If that is not the case, completing the domain claim process can resolve it.
2018-01-01 12:34:23 13383 ERROR umapi.action - Error in requestID: action_6 (User: {'user': ‘user@domain.com’, 'requestID': 'action_6'}, Command: {'createEnterpriseID': {'email': 'user@domain.com', 'option': 'updateIfAlreadyExists', 'firstname': 'test', 'lastname': 'user', 'country': ‘US’}}): code: "error.user.type_mismatch" message: "The user type requested for the invite does not match the claimed domain type"
Tip: an attempt is made to create a federatedID type of account but the Directory is created for Enterprise use or vice-versa. Search the attribute user_identity_type inside user-sync-config.yml file. Modify the value according to the type of Directory shown in Admin Console (Settings > Identity > Domains > Directory type value for the given domain).
Sample console error output:
Failed to execute PEX file, missing compatible dependencies for:
pyyaml
cryptography
cffi
umapi-client
pycryptodome
pyldap
psutil
user-sync
Tips:
a) Check if the Python version you've installed on your system is the 32bit version. Uninstall the 32bit and install the 64bit version, to fix the issue.
b) Check whether the user-sync.pex version you've downloaded from GitHub is a match to your Python version and OS type. For example, user-sync-v2.3-win64-py365.zip is to be downloaded for Windows 64-bit and Python 3.
Using the latest version of Python is not always a good idea. Trying to match the Python version, the .pex was built in the first place, would be great. Follow the suffix of the downloaded .zip file to identify what Python version would work. For the example above (user-sync-v2.3-win64-py365.zip), Python 3.6.5 can be identified.
Error decrypting private key
2018-01-01 09:52:23 7920 DEBUG umapi - umapi: reading private key data from file C:\path\to\private.key
2018-01-01 09:52:23 7920 CRITICAL main - umapi configuration.enterprise: Error decrypting private key, either the password is wrong or: RSA key format is not supported
Tips:
a) if the problem can't be quickly identified, it's faster to re-issue the keypair
b) Do not use umapi_private_key_data attribute if you are running the script in Windows. Instead, encrypt the key and store that password inside Credential Manager.
c) Try an RSA256 /2048bits length private key, if you used a different format to issue the keypair
d) It's possible that you set up the secure_priv_key_pass_key: umapi_private_key_passphrase inside the connector-umapi.yml file. In this case, ensure that the entry in Credential Store for this and associated values match (see below).
No value in secure storage for user...
2018-01-01 09:52:23 7920 CRITICAL main - umapi CRITICAL main - umapi configuration.enterprise: No value in secure storage for user "someUUIDvalue@AdobeOrg", key "umapi_api_key"
Tips:
a) the Credentials Store entry for umapi_api_key may be missing. Create the entry in Credentials Store. Helping documentation can be found here.
b) It may be that the value has been added to Credentials Store under a different user account. However for the currently connected user this entry is missing. Add it, if necessary, or switch user accounts.
error.internal.exceptionflys / error.unauthorized
umapi_client.error.RequestError: Request Error (401): {"lastPage":false,"result":"error.internal.exceptionflys","message":"Failed to exchange token"}
OR
"umapi_client.error.RequestError: Request Error (401): {"lastPage":false,"result":"error.unauthorized","message":"Failed to authenticate provided token"}"
Tip: It may be that in Admin Console -> Settings -> Authentication Settings there is an option chosen other than Easiest for Users (Password never expires). If More Secure or Most Secure option is enabled, it can expire the password of the Technical Account linked to the integration.
To fix the issue, a new integration should be created. Ensure that the metadata is also renewed in the connector-umapi.yml file.
A fix was deployed for it, but it can affect integrations created before October 2018.
Failed to establish a new connection [Errno 10061]
ConnectionError: HTTPSConnectionPool(host='usermanagement.adobe.io', port=443): Max retries exceeded with url: /v2/usermanagement/users/someUUID@AdobeOrg/0?directOnly=True (Caused by NewConnectionError('<urllib3.connection.VerifiedHTTPSConnection object at 0x00000000027B9630>: Failed to establish a new connection: [Errno 10061] No connection could be made because the target machine actively refused it',))
Tip:
It is related to UST not being able to connect to the public API endpoints. Local settings can prevent access due to firewall rules, proxy blocking traffic, account settings for Internet access, and more.
In some situations it may help adding the https_proxy environment variable with a value such as:
http://<proxyAddress>:<port> OR https://<proxyAddress>:<port>
In other cases, allowing access to these endpoints might help:
ims-na1.adobelogin.com:443
usermanagement.adobe.io:443
It can only be resolved locally by clearing access to the above endpoints for the running account.
2017-07-07 09:01:37 4916 CRITICAL main - Connection to org [...] at endpoint https://usermanagement.adobe.io/v2/usermanagement failed: Unable to authorise against https://ims-na1.adobelogin.com/ims/exchange/jwt: Response Code: 400, Response Text: {"error_description":"The metascopes in the JWT are not a subset of the metascopes in the binding.","error":"invalid_scope"}
Tip: Access the integration you've created at https://developer.adobe.com/console/projects and check the left menu where the APIs are listed. Make sure that User Management API is added as a service (appears in the list).
No valid bindings were found for organization and technical account combination
Sample log entry (debug mode):
2017-07-07 09:01:37 4916 CRITICAL main - UMAPI connection to org id 'someUUIDvalue@AdobeOrg' failed: Unable to authorize against https://ims-na1.adobelogin.com/ims/exchange/jwt:
Response Code: 400, Response Text: {"error_description":"No valid bindings were found for organization and technical account combination","error":"invalid_token"}
Tips:
a) It can be caused because the tech_acct value inside connector-umapi.yml file corresponds to other value than technical account ID inside the integration at https://developer.adobe.com/console/projects. Verify the technical account ID value from the current integration and copy it to this file.
b) It can also be caused because the public certificate from the integration is expired. Renew the private and public key. Then, upload the public key, and replace the old private key with the new key. Verify the path inside connector-umapi.yml file to point to the right file.
c) Check if the Integration is made for the correct Organization. Select the Organization inside the drop-down list located in the upper-left corner at https://developer.adobe.com/console/projects. Then, verify the technical account ID value for the active integration along with the other metadata (Org ID, secret, and client ID).
SSL: CERTIFICATE_VERIFY_FAILED
2017-07-07 09:01:37 4916 CRITICAL main - UMAPI connection to org id 'someUUIDvalue@AdobeOrg' failed: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed (_ssl.c:661)
Tip: It is caused by the SSL inspection enabled on the local proxy server
Solution1: Obtain the proxy's root CA certificate in PEM format (assume that its name is thecert.crt). If DER format is used, transform it to PEM using openssl command:
openssl x509 -inform DER -in thecert.crt -out thecert.pem -outform PEM
Note: viewing the contents of the certificate and seeing a base64 encoded string in between
-----BEGIN CERTIFICATE----- and -----END CERTIFICATE----- lines would mean you have the PEM formatted file.
Next, create an environment variable named REQUESTS_CA_BUNDLE and set its value as the path to thecert.pem file.
Solution2: In some cases it was observed in Windows that if the tool is run from a different driver than the OS and Python is installed, this error may occur. Moving the entire script on the drive where the OS exists can be a solution. If it is not an option, the cacert.pem file containing all the trusted root CAs should could be copied to the other drive and its path to be used as input value for the REQUESTS_CA_BUNDLE environment variable. If a proxy inspects the SSL traffic as well, then the content of the root CA certificate must be copied inside the cacert.pem file to ensure the proxy cert is also trusted.
Note: A default Python install would have the bundle of certificates at C:\Python36\Lib\site-packages\certifi\cacert.pem.
Solution3: disable the SSL inspection on the proxy side for the API endpoints ims-na1.adobelogin.com and usermanagement.adobe.io
2018-01-01 09:01:37 4916 WARNING ldap - No group found for: Name_Of_The_Group
Tips:
a) This group does not exist in LDAP with that exact name; fix it by adding the correct LDAP name of the group
b) The group is not discoverable under the declared base_dn (see connector-ldap.yml file). Change the base_dn value to contain the mentioned group (it mainly occurs when base_dn points to some OU, instead of being as broad as possible).
2018-01-01 11:25:45 1 ERROR umapi.action - Error in requestID: action_1 (User: {'user': 'myuser@domain.com', 'useAdobeID': True, 'requestID': 'action_1'}, Command: {'add': {'product': ['group_name']}}): code: "error.group.not_found" message: "Group my_group_name was not found"
Tip: The user-group group_name in the output above does not exist on Adobe side, so go ahead and create it. If the intent was to set the name of a PLC instead one of a user-group, then check this documentation page for a more visual explanation.
2018-09-05 10:58:08 96329 CRITICAL main - Connection to org some_Org_UUID@AdobeOrg at endpoint https://usermanagement.adobe.io/v2/usermanagement failed: dlopen(/Users/user/.pex/install/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl.f77d5cc74b0deef9f1df7eacfe5f5ea57ed94a63/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl/cryptography/hazmat/bindings/_openssl.abi3.so, 2): Library not loaded: /usr/local/opt/openssl/lib/libssl.1.0.0.dylib
Referenced from: /Users/user/.pex/install/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl.f77d5cc74b0deef9f1df7eacfe5f5ea57ed94a63/cryptography-2.3-cp37-cp37m-macosx_10_12_x86_64.whl/cryptography/hazmat/bindings/_openssl.abi3.so
Reason: image not found
2018-09-05 10:58:08 96329 INFO main - ========== End Run (User Sync version: 2.3) (Total time: 0:00:00)
Tip: this error was recorded on a macOS High Sierra, when using UST v2.3, using Python 3.7.0. Running brew install openssl inside the Terminal fixed it for this particular scenario.
Could not create person for type 2 or 3 [...]
2019-07-28 07:17:51 2220 ERROR umapi.action - Error in requestID: action_1 (User: {'user': 'user@claimed-domain.com', 'requestID': 'action_1'}, Command: {'createFederatedID': {'email': 'user@claimed-domain.com', 'option': 'updateIfAlreadyExists', 'firstname': 'First', 'lastname': 'Last', 'country': 'US'}}): code: "error.internal.create_failed" message: "Could not create person for type 2 or 3. Renga result is NOT_ALLOWED, Resource is externally managed and token is missing the GROUP_SOURCE_UPDATE purpose."
Sometimes, the @claimed-domain.com is owned by a different Organization that has set up an Azure or Google connector to manage the synchronization of accounts to Admin Console. This same domain is then entrusted to a different Organization that uses the User Sync Tool to synchronize accounts of format @claimed-domain.com
Cause: The logged message appears when the Sync tool extracts the
user@claimed-domain.com account from an LDAP server to create it in the secondary organization. However, the account is not yet created/sycned into the main organization via Azure or Google connector.
Tip: try to create/sync the user@claimed-domain.com account in the Organization using thee Azure or Google connector, then retry syncing with the UST in the trustee Org.
Unexpected LDAP failure reading [...]
2018-09-05 10:58:08 96329 6348 CRITICAL main - Unexpected LDAP failure reading group info: {'desc': 'Referral', 'info': 'Referral:\nldap://domain.local/DC=sub,DC=domain,DC=local'}
Possible cause: The group(s) of interest are located in a subdomain, but the host value is one of the root domains.
Tip: Change the host value to one of the subdomains where user groups can be found. If users or groups of interest are located in both root domain and its sub-domain(s), use the global catalog port on the root domain. Modify the groups in the sub-domain(s) to be Universal instead of Global. Example of host value using global catalog: ldap://domain.local:3268 or ldaps://domain.local:3269
In this last scenario, if global catalog port is used, make sure base_dn value takes no value:
base_dn: ""
error.organization.invalid_id
2020-08-20 12:00:00 1892 ERROR main - Unhandled exception
raise RequestError(result)
umapi_client.error.RequestError: Request Error (401): {"lastPage":false,"result":"error.organization.invalid_id",
"message":"UNAUTHORIZED"}
This error was seen on older integrations.
The fix: create a new integration (or project) at https://developer.adobe.com/console/projects, next to the existing one, used for same purpose. The new integration will provide new credentials, so be sure to update them in the connector-umapi.yml file. It's very possible that the keypair (private/public key) be re-issued, therefore the new private key needs to replace the exiting one.
Could not create person of type createFederatedID
2021-01-01 18:00:00 14063 ERROR umapi.action - Error in requestID: action_19 (User: {'user': 'some_user', 'domain': some.domain', 'requestID': 'action_19'}, Command: {'createFederatedID': {'email': 'some_user@some.domain', 'option': 'ignoreIfAlreadyExists', 'firstname': 'FName', 'lastname': 'LName, 'country': 'GB'}}): code: "error.internal.create_failed" message: "Could not create person of type createFederatedID"
This is a generic error for multiple causes, but the usual problem is that the domain used in the create action is under an Azure or Google sync setup influence.
How to check: login in Admin Console with the System Administrator account, then click Settings menu, then click inside the Directory that holds the domain, then click the Sync menu tab
Is there a Sync Source card present in the Sync menu?
If Yes, the fix depends on how the synchronization should continue further:
-> if Azure or Google connector should be the one doing the sync, continue with the Sync Source setup and remove UST completely
-> if UST should be the one doing the sync, click the “Go to Settings” button and click the “Remove Sync” button at the bottom of the page; UST should start running as usual thereafter
If No, it may be that the current UST runs against a Console in which the domain in question is entrusted from a different Console (owning Org). This other Org may have the Azure or Google synchronization turned on and could lead to the current error. The fix in his case is to sync the account in the owning Org/Console first, then use UST to create/add the account in the current Console.
If none of the situations above fit the current scenario, it is best to contact Enterprise Support.
