Can not access client with many allocations via the web UI

[agarfer1]

Nomad version

Nomad v1.9.3
BuildDate 2024-11-11T16:35:41Z
Revision d92bf10

Operating system and Environment details

Ubuntu 24.04 x86-64

Issue

I have a host with many allocations and I can no longer access it via the web UI from the Clients page or directly using the URL of the node.

I have tested this both in Chrome and Brave.

Looking at the network traffic it seems that the request to retrieve the allocations fails if over 1.1MB of allocations are retrieved.

However, replaying that request manually loads the data correctly.

The screenshot is from Brave, the last request has been replayed a few seconds later and it goes through correctly.

P.S. Using nomad node status for that node, works as expected

Reproduction steps

Run over 600 allocations as parameterized jobs (they don't need to be running simultaneously, but before GC happens) in a client, so that the HTTP allocations request, https://$SERVER:4646/v1/node/$NODE_ID/allocations will be over 1.1MB and see that client from the Clients list, it will show 0 allocations from that list because the information is not properly loaded there either.

Visiting that specific client (both screenshots) will fail as well.

Expected Result

The client information should be visible just like with any other client.

Actual Result

[agarfer1]

When the size of HTTP allocations response dropped below 1-1.1MB (950 KB in the screenshot), it is again able to show the client correctly:

[tgross]

Hi @agarfer1! I've flagged this as a UI issue for investigation.

[philrenaud]

Thanks for raising this, @agarfer1 ; could you possibly pass me any dev console error messages that might pop up when you get to the Error screen?

[agarfer1]

This screenshot is from inside the client URL:

This screenshot is from the client list:

But the behavior is very inconsistent, now trying to replicate it sometimes I am able to see the page correctly, other times it fails.

Since the UI redesign/change of framework of version 1.7 or 1.8 I have been experiencing some strange problems. If I have more than around 5 tabs open, the next tab that is loaded doesn't load until I close one of the previous ones. And my server is configured to http_max_conns_per_client = 500, not sure if I would need a higher value.

As another weird behavior related-ish to the tab problem, I have just opened issue 24666.

[ocharles]

Most browsers have a limit on HTTP 1 connections, and to use HTTP 2 you need to be serving over https://. I just thought I'd raise this, but it looks like you are using https

[agarfer1]

Oh wow, @ocharles thank you for the information. https is configured correctly, but in the developer console I see that I am using http/1.1 for the requests .... any ideas on what else could be wrong that is making me use http/1.1?

Maybe the whole issue is due to that

[agarfer1]

In case somebody can help me, this is my server configuration:

data_dir  = "/opt/nomad/data"
bind_addr = "0.0.0.0"

advertise {
  http  = "REDACTED_DOMAIN_NO_PORT_SPECIFIED"
  rpc  = "REDACTED_DOMAIN_NO_PORT_SPECIFIED"
}

#Some telemetry changes, I guess those are irrelevant

server {
  enabled          = true
  bootstrap_expect = 1
 
  default_scheduler_config {
    scheduler_algorithm             = "spread"
    memory_oversubscription_enabled = true
    reject_job_registration         = false
    pause_eval_broker               = false # New in Nomad 1.3.2

    preemption_config {
      batch_scheduler_enabled    = true
      system_scheduler_enabled   = true
      service_scheduler_enabled  = true
      sysbatch_scheduler_enabled = true # New in Nomad 1.2
    }
  }
}

acl {
  enabled = false
}

limits {
  https_handshake_timeout   = "30s"
  rpc_handshake_timeout     = "30s"
  http_max_conns_per_client = 500
}


# Require TLS
tls {
  http = true
  rpc  = true

  ca_file   = "/etc/certs/ca.pem"
  cert_file = "/etc/certs/global-server-nomad.pem"
  key_file  = "/etc/certs/global-server-nomad-key.pem"

  verify_server_hostname = true 
  verify_https_client    = true 
}

The server is also using v1.9.3

[tgross]

Most browsers have a limit on HTTP 1 connections, and to use HTTP 2 you need to be serving over https://.

That's true. But the server itself also needs to support HTTP2. And for reasons having to do with our peculiar websocket implementation for alloc exec and the like, Nomad does not support HTTP2.

[ocharles]

Could you put a HTTP 2 reverse proxy in front of it?

[tgross]

Could you put a HTTP 2 reverse proxy in front of it?

Probably. Our tutorial for Nginx proxying is a HTTP/1.1 proxy though.

Nomad does not support HTTP2.

This actually isn't entirely accurate... Nomad's API clients don't use HTTP2. I'm pretty sure you can force the HTTP server to use HTTP2 for your own HTTP clients (if you're ok with broken alloc exec).