When VHost has no tags v13.7.0 throws error: Could not prefetch rabbitmq_vhost provider 'rabbitmqctl': Cannot parse invalid vhost line: myvhost3 undefined

[LangJV]

Affected Puppet, Ruby, OS and module versions/distributions

• Puppet: 7.26.0
• Ruby: The one with Puppet above?
• Distribution: RabbitMQ 3.13.7
• Module version:13.7.0

How to reproduce (e.g Puppet code you use)

Upgrade from Module version 12.x to 13.7.0
Puppet will fail to run with an error:
Could not prefetch rabbitmq_vhost provider 'rabbitmqctl': Cannot parse invalid vhost line: myvhost3 undefined
I have traced this back (i believe) to the following snippet of code: https://github.com/voxpupuli/puppet-rabbitmq/blob/master/lib/puppet/provider/rabbitmq_vhost/rabbitmqctl.rb#L47

And the following equivalent CLI command:
sudo rabbitmqctl list_vhosts name,description,default_queue_type,tags -s

It looks like the "diff" here added tag and description support. In my case we are using neither descriptions, or tags, so those outputs are "empty"

My output of the above commands looks like:

myvhost1		undefined	[]
myvhost2		undefined	[]
myvhost3		undefined	
myvhost4		undefined	[]
myvhost5	undefined	[]

Though I've never used "tags" - all of my vhosts have an empty array [] in the output for "tags" with the rabbitmqcli command, except myvhost3 which is the one erroring.

I believe that is causing it to fail the regex matcher and trigger the error.

I was hunting around the Management UI and though I can add a column to view tag(s) on the vhost listing, it also shows all vhosts having no tags present. I cannot find a way to "edit" or "update" the tags to possibly get it to show an empty array instead of null/nil/"" that it is showing currently for this single vhost.

ALL of my vhosts have always been created using this puppet module idempotently - under 12.x it ran "with no changes" every 30 minutes without issue using:

rabbitmq_vhost { $name:
    ensure => $ensure,
  }

I'm not clear if i should be focusing on:

• Updating this regex in this code/module to also support the absence of even an empty array for tags, allowing it to work.
• Understanding "why" this single vhost doesn't have an empty array for tags even though it was created the same way as all the others, and address that
• Figuring out how in the UI or CLI to just "force" an empty tag list to the vhost to get it to "play nice" with this code and regex
• Bugfix rabbitmqcli in the main rabbitmq project itself to consistently return [] when there are no tags, never null/nil/""
• Something else entirely?

I'm at the edge of my depth, and really not understanding the context, reasoning, or "why" of this. Hoping someone with that context might be able to help direct me at the best path to resolving this for myself, and likely any others who may have a similar issue.

What are you seeing

What behaviour did you expect instead

I expected this to "work" instead. Puppet should run and complete and be happy.

Output log

Any additional information you'd like to impart

[wyardley]

(I edited your issue description to wrap some sections in backticks for easier reading)

• What does the JSON output of rabbitmqctl list_vhosts look like?
• Can you provide a longer code snippet of how the vhosts are defined etc. in Puppet?

A lot of these problems would probably be solved if / when this project drops support for older versions of RabbitMQ, and starts using JSON output for everything (or if a larger rewrite were undertaken to use the APIs instead, though guessing it's unlikely that someone will take the time / effort to do the latter). It seems like the --formatter json option shows up around 3.7; I'd be Ok to drop support for older versions of RabbitMQ (which are long since EOL) as a breaking release if someone wants to update the provider and unit tests to use json formatting intead.

Looking at the code you mentioned (also discussed in #1006, though that's a different issue), I think you're thinking along the right lines -- when the provider thinks supports_metadata is true, it will error unless the full regex matches, and the line with no tags array is confusing it.

Are you able to reproduce it from a "clean" setup? Do you have an environment where you can destroy / recreate the vhost that's having an issue? Have you tried adding tags with Puppet and then removing them all?

I have not been using / administering RabbitMQ for many years, but I tried some simple / obvious ways to try to create an empty vhost locally and wasn't able to. And the vhosts I added with no tags all had an empty array ([]) in the plaintext output.

[LangJV]

Afternoon @wyardley

Its funny you mention the JSON output. When originally scripting against RabbitMQ - the JSON return option was my go-to. Was actually surprised to see the code here parsing CLI outputs. But if its historical and wasnt introduced till RabbitMQ 3.7, that's good context to have that explains it :)

However the "difference" still shows up there as well. Though, id admit it probably be a bit easier to deal with in JSON object notation than regex...

sudo rabbitmqctl list_vhosts name,description,default_queue_type,tags -s --formatter json
[
{"name":"myvost1","description":"","default_queue_type":"undefined","tags":[]}
,{"name":"myvost2","description":"","default_queue_type":"undefined","tags":[]}
,{"name":"myvost3","description":"","default_queue_type":"undefined","tags":""}
,{"name":"myvost4","description":"","default_queue_type":"undefined","tags":[]}
,{"name":"myvost5","description":"","default_queue_type":"undefined","tags":[]}
]

I am "not" able to reproduce from a clean setup. I have several clusters that were built using RabbitMQ 3.8/3.9 and have been upgraded to 3.13 sequentially. A newly built cluster today (or even one built a few months ago starting at 3.11) does not have this "issue" - even when using the same puppet code. This caught us off guard as the lower-level environments are "more recently rebuilt" than our production and near-prod ones, meaning we got through a few upgrades of the module successfully before hitting this issue.

I have not tried adding a tag, as we aren't using them, but i wonder if simply adding a tag would help? I may give this a whirl, to see if it works around the issue and changes the "" to [tagname] across the board, which per the regex should cause it to work

My current puppetcode is (simplified a bit) something like this:

<create_vhost.pp>
define payx_rabbitmq::create_vhost(
  $ensure = present,
) {

  rabbitmq_vhost { $name:
    ensure => $ensure,
  }
}

<init.pp>
class mymodule () {

  mymodule::create_vhost { [
      'myvhost1',
      'myvhost2',
      'myvhost3',
      'myvhost4',
      'myvhost5',
    ]:
      ensure => present,
      use_federation => true,
  } 
}

I've left out a bunch of other unrelated parameters and code in the create_vhost definition - for example, creating all the federation policies, queue policies, exchange policies based on values passed in/defaulted in create_vhost, but that's the extent of it.

[wyardley]

That did tell us something: that somehow tags for that vhost seem to be set to a single empty string (and not, say, a space, or some representation of null or undefined). I wasn’t able to set that when I tried, but seems like that’s what has happened.

What if you modify the tags (outside of puppet) for that vhost to an empty list of tags (or delete it, if you can do that)? I would also try and upgrade to any available patch releases within the version you’re using, or even to a newer rmq version if possible.

Maybe nicely ask the rmq folks if they’ve got any ideas, though seems like you’re technically outside the community supported window.

And yes, agree that if we were parsing the json, we’d have some more options for more nuanced error handling

[LangJV]

We actually have a valid support contract, so I can always go to Broadcomm too. It "seems" like the root issue is an inconsistency between "" and [] as registered in the RabbitMQ

[wyardley]

Right, it seems as if there's a tag on that vhost that's an empty string vs no tags, even though it doesn't seem possible to set one (easily).

But you may want to see if resetting the tags on that vhost fixes it.