Support Terraform Workspaces

[MrJoy]

We use Terraform Workspaces, and it appears to be the case that there's no way to use inframap with that. It would be nice if we could integrate all the pertinent state data (from the central state file, and the per-workspace state files). As it stands, the views I can produce with inframap are... not terribly useful.

[xescugc]

Most likely if the State on the Workspace are formatted different than the normal State It'll not work, which I'm gonna guess it's the main issue here.

I don't have any current example on how does this specific state you mention looks like could I have an example to see how it looks like and potentially change the logic to support it?

How would your ideal use case look like?

[MrJoy]

The docs imply that the format is the same. The need is to combine non-workspaced state with workspaced state. I.E. merging the contents of two state files.

And, generally, if I want to build a unified view of all of my infrastructure, I have a need to combine multiple state files.

To give you a sense of how we're using Terraform that causes this need to arise:

We have 5 AWS accounts. 1 "meta" account, and 2 accounts for each of staging and production environments (one account per environment each for CI/CD, one for actual workload).

We have 5 centralized Terraform projects, one for each AWS account. These are used for shared/common infrastructure components.

We have per-service Terraform projects (colocated with the code for the service) that manages infrastructure objects unique to that service. These projects span both CI/CD and Workload AWS accounts. These projects use workspaces per-environment, so that their Terraform is guaranteed to be identical between staging and production.

So, to get a map of my production environment I need the following state files:

• Meta account (default workspace, shared repo)
• Prod workload account (default workspace, shared repo)
• Prod CI/CD account (default workspace, shared repo)
• Service A production (prod workspace, service A repo)
• Service B production (prod workspace, service B repo)
• ...

The way the accounts are structured and linked up (via heavy use of SSM parameters to public configuration information referenced as data objects by other accounts to know what to interact with) makes it challenging to look at things from just one state file and have any sort of meaningful view of things.

For example, the CI/CD process for a service runs image build tasks in the CI/CD AWS account, then publishes them to a list of AWS accounts provided in an SSM parameter published in the meta account. Its ability to do this publishing depends on IAM objects that exist in the workload accounts to which the images will be published.

Looking at one of the workspace files, it's pretty definitely a normal Terraform state file. I can't provide an example without having to remove sensitive information unfortunately, and the files are big enough for that to be kind of an involved process. This might be as simple as just doing a deep tree merge of the state files, and maybe correlating data references from one state file to resource instances in another. I'm not deep enough in the weeds to know for sure.

[DamonStamper]

You can work around this by navigating to the appropriate workspace. This means that the format is a non-issue. The breaking change is that the workspaces store their state files in nested directories named after the workspace. Thus if a parameter was added for the workspace name then inframap could look in the appropriate directory terraform.tfstate.d/${workspace} and this feature request would be completed.

In the below example the workspace name is 'dev'.
docker run --rm -v ${PWD}/terraform.tfstate.d/dev/:/opt cycloid/inframap generate /opt/terraform.tfstate

[xescugc]

Sorry for the delay, @MrJoy notification got lost and now I've seen @DamonStamper one hehe.

Ok, my main concern is that I want to try to avoid having to pull them, but for what @DamonStamper says you can have all the states pulled locally if that's the case then things are easier.

At the end the issue is basically to read multiple states and display one graph for them and to do that there are different solutions that come to mind.

Having as a requirement a list of states we want to se 1 graph for all of them.

1. We could basically read all of them as separated files and generate X(# of states) graphs for all of them but output them as 1, in the end images you would see X# of independent graphs (more or less).

2. Merge them all together into 1 big graph, similar to what @MrJoy is saying to do a deep tree merge, this has a potential issue/benefit of having repeated resources (provider+type+names, and module if any, have to be unique in a state file, merging state files can brake that and give erroneous shared nodes) but also the benefit of seeing connections between states, if they share some SecurityGroups and so one you would see the connections.

3. Mix between 1-2, basically try to have the states visually separated but with the shared connections/nodes visible

IDK which is the best one, potentially all 3 of them are valid and we could add a flag differentiate them, though 1,3 if you do not have shared resources would be the same, and for DOT to be able to represent shared resources we'll need to duplicate the Node and have it repeated on each one of the states but visually have a connection for this nodes representing that they are the same.

The easy and quick solution is the 1 but the most accurate would be the 3 I think.

The output from 3 would look something like

In this case there are no repeated nodes, but if they where they would be repeated and with an arrow connecting them that somehow would say that they are the same.

I can't provide an example without having to remove sensitive information unfortunately, and the files are big enough for that to be kind of an involved process.

InfraMap actually has a inframap prune which is intended for that purpose, to remove not needed information for states so they can be shared without leaking, nevertheless you should check the output if it's ok for you.

Once we have that well defined I don't know when we'll have time to do it on our side, but we'll try to allocate someone once we have the resources for it, but before I want to have it well defined.

[MrJoy]

It should be the case that the only node duplication one sees is having a given node show up as both a resource node and a data node. If one saw a duplicated resource node that would indicate some fairly dangerous misuse of Terraform, as it means that multiple state files are claiming ownership of the same object. I think merging these together would mask such a situation. It might be worth calling it out if it's ever observed to happen and possibly refusing to proceed -- or rendering a "merged" object in some special way to indicate the issue.

I think it's definitely the case that we would want to see separate-grouping-per-state-file, and edges between graphs indicating those connections. Both provide important information for understanding how an infrastructure is organized. I'll take a look at the prune command and see what I can come up with.

[MrJoy]

sample.zip

Sample of state files provided. global-stage.tfstate is the non-workspaced one, and workload-stage.tfstate is one of the relevant workspace state files.

In actuality, our infrastructure involves a meta account with a non-workspaced state file, two per-environment non-workspaced state files (each representing one AWS account; a CI/CD account and a workload account), and 2n workspace state files, where n is the number of distinct repositories defining components of our system. Currently n=4. So 2m+1 AWS accounts (where m is the number of environments), and 2mn workspace state files in total. We're not big on Terraform monoliths. But, this should provide a representative example with connections between state files.

I notice that inframap is excluding IAM roles and such. That's actually a pretty crucial thing we'd like to be able to represent. I see there's an open issue around that though, so I won't rehash it here. Notwithstanding that, there should be some interconnections represented in these state files...