CNI: use tmpfs location for ipam plugin

[tgross]

When a Nomad host reboots, the network namespace files in the tmpfs in /var/run are wiped out. So when we restore allocations after a host reboot, we need to be able to restore both the network namespace and the network configuration. But because the netns is newly created and we need to run the CNI plugins again, this create potential conflicts with the IPAM plugin which has written state to persistent disk at /var/lib/cni. These IPs aren't the ones advertised to Consul, so there's no particular reason to keep them around after a host reboot because all virtual interfaces need to be recreated too.

Reconfigure the CNI bridge configuration to use /var/run/cni as its state directory. We already expect this location to be created by CNI because the netns files are hard-coded to be created there too in libcni.

Note this does not fix the problem described for Docker in #24292 because that appears to be related to the netns itself being restored unexpectedly from Docker's state.

Ref: #24292 (comment)
Ref: https://www.cni.dev/plugins/current/ipam/host-local/#files

Testing & Reproduction steps

Run a cluster on a set of VMs, with at least one client. This can't be a server+client because we need to reboot the hosts. You should probably set the server.heartbeat_grace = "5m" to give yourself time to work.

• Run a non-Docker task with network.mode = "bridge". Wait for it to be healthy.
• Reboot the client host.
• Make sure the alloc is restored, the tasks are restarted, and networking works.

Contributor Checklist

• Changelog Entry If this PR changes user-facing behavior, please generate and add a
  changelog entry using the make cl command.
• Testing Please add tests to cover any new functionality or to demonstrate bug fixes and
  ensure regressions will be caught.
• Documentation If the change impacts user-facing functionality such as the CLI, API, UI,
  and job configuration, please update the Nomad website documentation to reflect this. Refer to
  the website README for docs guidelines. Please also consider whether the
  change requires notes within the upgrade guide.

Reviewer Checklist

• Backport Labels Please add the correct backport labels as described by the internal
  backporting document.
• Commit Type Ensure the correct merge method is selected which should be "squash and merge"
  in the majority of situations. The main exceptions are long-lived feature branches or merges where
  history should be preserved.
• Enterprise PRs If this is an enterprise only PR, please add any required changelog entry
  within the public repository.

[tgross]

I'm going to move this back into draft until we've had a bit more time to look into the CNI check command workflow being discussed in #24292 (comment)

[jrasell]

LGTM!

I tested this using an AWS environment with 1 server and 1 client instance. The client instance has CNI v1.3.0 plugins installed.

JobSpec:

job "example" {

  group "webserver" {
    network {
      mode = "bridge"
      port "http" {
        to = 80
      }
    }

    task "python3" {
      driver = "exec"

      config {
        command = "python3"
        args    = ["-m", "http.server", "80"]
      }

      resources {
        cpu    = 500
        memory = 256
      }
    }
  }
}

Running through the steps using a build from main f4529485563924462dbdccdd1b4cacbd9d68616e when I rebooted the instance the allocation failed with the error detailed below:

2024-12-13T10:57:44Z  Setup Failure          failed to setup alloc: pre-run hook "network" failed: failed to configure networking for alloc: failed to configure network: plugin type="bridge" failed (add): failed to allocate for range 0: 172.26.64.12 has been allocated to 0dc525f4-c491-7e25-a67d-0121069ad55e, duplicate allocation is not allowed
2024-12-13T10:57:41Z  Failed Restoring Task  failed to restore task; will not run until server is contacted

I then tested this patch 9e1a365ae3d22f04c6bb8aa0ce0fee6d1f83ae6f I performed the same steps as detailed in the PR and performed in the previous test. When the instance was rebooted the following task events are recorded:

2024-12-13T11:08:42Z  Started                Task started by client
2024-12-13T11:08:41Z  Failed Restoring Task  failed to restore task; will not run until server is contacted

The allocation HTTP server is accessible and responds as it did before the reboot.