aws_ec2 fails with 'can only concatenate str (not "bytes") to str' with session-manager-plugin 1.2.458.0

[aberkvam]

Summary

There was a "stealth" release of session-manager-plugin 1.2.458.0 today.

aws/session-manager-plugin#64

Upgrading to session-manager-plugin 1.2.458.0 causes a 'TypeError: can only concatenate str (not "bytes") to str' error when connecting to EC2 instances.

Reverting to session-plugin-manager 1.2.398.0 fixes the error

Issue Type

Bug Report

Component Name

aws_ec2

Ansible Version

$ ansible --version
ansible [core 2.14.3]
  config file = /Users/aberkvam/.ansible.cfg
  configured module search path = ['/Users/aberkvam/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /Users/aberkvam/ansible7/lib/python3.11/site-packages/ansible
  ansible collection location = /Users/aberkvam/.ansible/collections:/usr/share/ansible/collections
  executable location = /Users/aberkvam/ansible7/bin/ansible
  python version = 3.11.1 (main, Mar 15 2023, 16:48:41) [Clang 14.0.0 (clang-1400.0.29.202)] (/Users/aberkvam/ansible7/bin/python)
  jinja version = 3.1.2
  libyaml = True

Collection Versions

$ ansible-galaxy collection list
[DEPRECATION WARNING]: [defaults]callback_whitelist option, normalizing names to new standard, use callbacks_enabled
instead. This feature will be removed from ansible-core in version 2.15. Deprecation warnings can be disabled by setting
deprecation_warnings=False in ansible.cfg.

# /Users/aberkvam/.ansible/collections/ansible_collections
Collection      Version
--------------- -------
amazon.aws      5.1.0
community.aws   1.5.0
community.mysql 2.3.2

# /Users/aberkvam/ansible7/lib/python3.11/site-packages/ansible_collections
Collection                    Version
----------------------------- -------
amazon.aws                    5.1.0
ansible.netcommon             4.1.0
ansible.posix                 1.4.0
ansible.utils                 2.7.0
ansible.windows               1.12.0
arista.eos                    6.0.0
awx.awx                       21.8.0
azure.azcollection            1.14.0
check_point.mgmt              4.0.0
chocolatey.chocolatey         1.3.1
cisco.aci                     2.3.0
cisco.asa                     4.0.0
cisco.dnac                    6.6.0
cisco.intersight              1.0.20
cisco.ios                     4.0.0
cisco.iosxr                   4.0.2
cisco.ise                     2.5.9
cisco.meraki                  2.11.0
cisco.mso                     2.1.0
cisco.nso                     1.0.3
cisco.nxos                    4.0.0
cisco.ucs                     1.8.0
cloud.common                  2.1.2
cloudscale_ch.cloud           2.2.2
community.aws                 5.0.0
community.azure               2.0.0
community.ciscosmb            1.0.5
community.crypto              2.8.1
community.digitalocean        1.22.0
community.dns                 2.4.1
community.docker              3.2.1
community.fortios             1.0.0
community.general             6.0.1
community.google              1.0.0
community.grafana             1.5.3
community.hashi_vault         4.0.0
community.hrobot              1.6.0
community.libvirt             1.2.0
community.mongodb             1.4.2
community.mysql               3.5.1
community.network             5.0.0
community.okd                 2.2.0
community.postgresql          2.3.0
community.proxysql            1.4.0
community.rabbitmq            1.2.3
community.routeros            2.3.1
community.sap                 1.0.0
community.sap_libs            1.3.0
community.skydive             1.0.0
community.sops                1.4.1
community.vmware              3.1.0
community.windows             1.11.1
community.zabbix              1.8.0
containers.podman             1.9.4
cyberark.conjur               1.2.0
cyberark.pas                  1.0.14
dellemc.enterprise_sonic      2.0.0
dellemc.openmanage            6.3.0
dellemc.os10                  1.1.1
dellemc.os6                   1.0.7
dellemc.os9                   1.0.4
f5networks.f5_modules         1.20.0
fortinet.fortimanager         2.1.7
fortinet.fortios              2.1.7
frr.frr                       2.0.0
gluster.gluster               1.0.2
google.cloud                  1.0.2
hetzner.hcloud                1.8.2
hpe.nimble                    1.1.4
ibm.qradar                    2.1.0
ibm.spectrum_virtualize       1.10.0
infinidat.infinibox           1.3.7
infoblox.nios_modules         1.4.0
inspur.ispim                  1.2.0
inspur.sm                     2.3.0
junipernetworks.junos         4.0.0
kubernetes.core               2.3.2
lowlydba.sqlserver            1.0.4
mellanox.onyx                 1.0.0
netapp.aws                    21.7.0
netapp.azure                  21.10.0
netapp.cloudmanager           21.21.0
netapp.elementsw              21.7.0
netapp.ontap                  22.0.1
netapp.storagegrid            21.11.1
netapp.um_info                21.8.0
netapp_eseries.santricity     1.3.1
netbox.netbox                 3.8.1
ngine_io.cloudstack           2.2.4
ngine_io.exoscale             1.0.0
ngine_io.vultr                1.1.2
openstack.cloud               1.10.0
openvswitch.openvswitch       2.1.0
ovirt.ovirt                   2.3.1
purestorage.flasharray        1.14.0
purestorage.flashblade        1.10.0
purestorage.fusion            1.1.1
sensu.sensu_go                1.13.1
splunk.es                     2.1.0
t_systems_mms.icinga_director 1.31.4
theforeman.foreman            3.7.0
vmware.vmware_rest            2.2.0
vultr.cloud                   1.3.1
vyos.vyos                     4.0.0
wti.remote                    1.0.4

AWS SDK versions

$ pip show boto boto3 botocore
WARNING: Package(s) not found: boto
Name: boto3
Version: 1.26.92
Summary: The AWS SDK for Python
Home-page: https://github.com/boto/boto3
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/aberkvam/ansible7/lib/python3.11/site-packages
Requires: botocore, jmespath, s3transfer
Required-by:
---
Name: botocore
Version: 1.29.92
Summary: Low-level, data-driven core of boto 3.
Home-page: https://github.com/boto/botocore
Author: Amazon Web Services
Author-email:
License: Apache License 2.0
Location: /Users/aberkvam/ansible7/lib/python3.11/site-packages
Requires: jmespath, python-dateutil, urllib3
Required-by: boto3, s3transfer

Configuration

$ ansible-config dump --only-changed

OS / Environment

No response

Steps to Reproduce

ansible i-00000000000000000 -m ansible.builtin.setup  -i inventory/stg/

Expected Results

i-00000000000000000 | SUCCESS => {
    "ansible_facts": {
        "ansible_all_ipv4_addresses": [
[SNIP]

Actual Results

An exception occurred during task execution. To see the full traceback, use -vvv. The error was: TypeError: can only concatenate str (not "bytes") to str
i-00000000000000000 | FAILED! => {
    "msg": "Unexpected failure during module execution: can only concatenate str (not \"bytes\") to str",
    "stdout": ""
}

Code of Conduct

• I agree to follow the Ansible Code of Conduct

[tremble]

@aberkvam Thanks for taking the time to open this issue. I've moved it to the community.aws collection which is where the aws_ssm connection plugin lives.

Looking at the output of ansible-galaxy collection list you've got multiple copies of amazon.aws and community.aws installed. If possible, please could you uninstall them and install the latest release (5.3.0) of both amazon.aws and community.aws.

Specifically, I see release 5.0.0 and 1.5.0of community.aws, both of which have a number of known issues. There's been some substantial work done on this plugin between releases 5.0.0 and 5.3.0 of community.aws.

If you still see the issue (or another) after upgrading to release 5.3.0, please include:

1. The full output of ansible i-00000000000000000 -m ansible.builtin.setup -i inventory/stg/ -vvvvv (the -vvvvvis important, there's a lot of output, but it helps us track down what's going on with a connection plugin.)
2. Details of the host you're connecting to (SSM client version on the host, and Operating System)
3. The output of ansible-galaxy collection list again (to confirm the version that's installed)

[aberkvam]

I am hitting an unrelated issue when I upgrade community.aws past 3.4.0. For the moment, all I can do it confirm the issue still occurs with community.aws 3.4.0 and amazon.aws 5.3.0. I will work on resolving my unrelated issue and report back here when I can test with the latest community.aws.