RPC works with LLama but not Qwen.

[ehtom]

Hello!

I have been experimenting using the following machine configuration:

• Machine1: 4090, WSL Linux on Win11, 2.5GbE wired. This machine runs rpc-server.
• Machine2: 2*2080Ti, Ubuntu Linux, 1GbE wired. This machine runs llama-cli or llama-server.

I have been attempting to run/test the following models. I had to comment out

llama.cpp/ggml/src/ggml-rpc/ggml-rpc.cpp

Line 467 in ebdee94

GGML_ASSERT(tensor->ne[0] % 512 == 0 && "unsupported quantized tensor");

to run these quantized models:

• Tulu3 70b Q4_K_M (bartowski quant) - WORKS 7tok/s all layers offloaded.
• Llama 3.3 70b Q4_K_M (bartowski quant) - WORKS 7tok/s all layers offloaded.
• Qwen2.5 72b Q4_K_M (official and bartowski quant) - RUNS BUT EMITS GARBAGE when all layers offloaded. WORKS but very slowly with about 40 layers offloaded.

Based on the line which I commented out, I suspect that this is because Qwen2.5-72b has a intermediate_size of 29568, which is not divisible by 512?

If this is the reason, is it possible to get Qwen2.5 working over RPC by implementing cuda-like padding of 512 in ggml-rpc.cpp?

I think this RPC functionality is extremely cool and its a lot more lightweight and configurable for enthusiasts than other options in other engines, which seem geared towards setting up production inference clusters as they all rely on docker + ray combo it seems.

[ehtom]

I have been trying to understand the RPC architecture and I've come up with this:

• The ggml_backend_rpc functions implement a sort of 'interface' backend which just calls an rpc server hosting the real backend. Some of the functions don't do what you might initially expect like the init function does not actually initialize anything on the server and the vast majority of action happens when doing graph computations.
• The rpc_server takes requests and passes everything to a CUDA backend.
• CUDA backends pad everything to 512.

It looks like RPC does not keep track of this padding which is actually going on, so it ends up not mirroring what the (assuming CUDA) backend is actually doing. In addition it will be passing wrong tensor information to whatever code is coordinating all the memory and tensor splitting. I guess this is the source of why this doesn't work for Qwen? Is this correct or am I barking up the wrong tree here?

[ehtom]

Thank you for your response, that is so much clearer now! If I understand correctly, this means that unless a tensor explicitly writes using set_tensor the memory will not be over-written, so any misaligned tensor will have some (potentially dangerous) leftover memory rather than zeros in the padding sections.

I see that ggml_backend_cuda_buffer_init_tensor accepts a ggml_tensor, which would add a lot of overhead if sent via RPC using the documented serialization format. However... ggml_backend_cuda_buffer_init_tensor does not actually use any of the data from what I can see, just needs the pointer for cudaMemset((char *)tensor->data + original_size, 0, padded_size - original_size). Would it be possible to implement some sort of lightweight metadata transmission for this operation instead?

Please let me know if I have misunderstood something.

[slaren]

The problem is not the amount of data that needs to be transferred - the tensor data is not transferred for these operations. The problem is the latency added by the network roundtrip required by each call of init_tensor. Every temporary tensor in the compute buffer (possibly thousands) has a call to init_tensor, and each call adds latency roughly equal to the network ping. So you can see that's a lot of roundtrips that happens before each graph evaluation.

There are ways to deal with this. For instance, the CUDA backend does not really need calls to init_tensor for the temporary tensors, only for the weights, and that only needs to be done once while loading the model. It may also be possible to bundle multiple calls in a single packet to avoid multiple roundtrips.

[ehtom]

Thank you, that really helped me understand! To verify that these padding issues are what is causing the result to become garbage I implemented init_tensor over RPC in https://github.com/matt23654/llama.cpp/tree/rpc-experimental . As you point out its very slow, I was intending to then try and figure out how to minimize calls...

However! This does not fix Qwen2.5-72b-Q4_K_M outputting garbage, it is very strange. Same behaviour as before with the model appearing to function normally with offloading, say, 30 or so layers, garbage with 70 layers. I will paste an example below.

[slaren]

You may also need to forward get_alloc_size, otherwise no memory will be allocated for the padding.

[ehtom]

Thanks! That works and I've made it somewhat performant too. From 0.02 tokens/sec -> 3.17 tokens/sec. I'll clean it up and turn it into a pull request.

[ehtom]

Example of this problem with RPC server accessed over SSH tunnel (it is not actually running on 127.0.0.1).

$ build2/bin/llama-cli -m [qwen gguf path] -p "You are a helpful assistant" -cnv -ngl 70 --rpc 127.0.0.1:50052
ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no
ggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no
ggml_cuda_init: found 2 CUDA devices:
  Device 0: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
  Device 1: NVIDIA GeForce RTX 2080 Ti, compute capability 7.5, VMM: yes
build: 4404 (0827b2c1) with cc (Ubuntu 13.2.0-23ubuntu4) 13.2.0 for x86_64-linux-gnu
main: llama backend init
main: load the model and apply lora adapter, if any
llama_load_model_from_file: using device RPC[127.0.0.1:50052] (RPC[127.0.0.1:50052]) - 22994 MiB free
llama_load_model_from_file: using device CUDA0 (NVIDIA GeForce RTX 2080 Ti) - 10846 MiB free
llama_load_model_from_file: using device CUDA1 (NVIDIA GeForce RTX 2080 Ti) - 10844 MiB free
llama_model_loader: loaded meta data with 35 key-value pairs and 963 tensors from [qwen gguf path] (version GGUF V3 (latest))
llama_model_loader: Dumping metadata keys/values. Note: KV overrides do not apply in this output.
llama_model_loader: - kv   0:                       general.architecture str              = qwen2
llama_model_loader: - kv   1:                               general.type str              = model
llama_model_loader: - kv   2:                               general.name str              = Qwen2.5 72B Instruct
llama_model_loader: - kv   3:                           general.finetune str              = Instruct
llama_model_loader: - kv   4:                           general.basename str              = Qwen2.5
llama_model_loader: - kv   5:                         general.size_label str              = 72B
llama_model_loader: - kv   6:                            general.license str              = other
llama_model_loader: - kv   7:                       general.license.name str              = qwen
llama_model_loader: - kv   8:                       general.license.link str              = https://huggingface.co/Qwen/Qwen2.5-7...
llama_model_loader: - kv   9:                   general.base_model.count u32              = 1
llama_model_loader: - kv  10:                  general.base_model.0.name str              = Qwen2.5 72B
llama_model_loader: - kv  11:          general.base_model.0.organization str              = Qwen
llama_model_loader: - kv  12:              general.base_model.0.repo_url str              = https://huggingface.co/Qwen/Qwen2.5-72B
llama_model_loader: - kv  13:                               general.tags arr[str,2]       = ["chat", "text-generation"]
llama_model_loader: - kv  14:                          general.languages arr[str,1]       = ["en"]
llama_model_loader: - kv  15:                          qwen2.block_count u32              = 80
llama_model_loader: - kv  16:                       qwen2.context_length u32              = 32768
llama_model_loader: - kv  17:                     qwen2.embedding_length u32              = 8192
llama_model_loader: - kv  18:                  qwen2.feed_forward_length u32              = 29568
llama_model_loader: - kv  19:                 qwen2.attention.head_count u32              = 64
llama_model_loader: - kv  20:              qwen2.attention.head_count_kv u32              = 8
llama_model_loader: - kv  21:                       qwen2.rope.freq_base f32              = 1000000.000000
llama_model_loader: - kv  22:     qwen2.attention.layer_norm_rms_epsilon f32              = 0.000001
llama_model_loader: - kv  23:                          general.file_type u32              = 15
llama_model_loader: - kv  24:                       tokenizer.ggml.model str              = gpt2
llama_model_loader: - kv  25:                         tokenizer.ggml.pre str              = qwen2
llama_model_loader: - kv  26:                      tokenizer.ggml.tokens arr[str,152064]  = ["!", "\"", "#", "$", "%", "&", "'", ...
llama_model_loader: - kv  27:                  tokenizer.ggml.token_type arr[i32,152064]  = [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, ...
llama_model_loader: - kv  28:                      tokenizer.ggml.merges arr[str,151387]  = ["Ġ Ġ", "ĠĠ ĠĠ", "i n", "Ġ t",...
llama_model_loader: - kv  29:                tokenizer.ggml.eos_token_id u32              = 151645
llama_model_loader: - kv  30:            tokenizer.ggml.padding_token_id u32              = 151643
llama_model_loader: - kv  31:                tokenizer.ggml.bos_token_id u32              = 151643
llama_model_loader: - kv  32:               tokenizer.ggml.add_bos_token bool             = false
llama_model_loader: - kv  33:                    tokenizer.chat_template str              = {%- if tools %}\n    {{- '<|im_start|>...
llama_model_loader: - kv  34:               general.quantization_version u32              = 2
llama_model_loader: - type  f32:  401 tensors
llama_model_loader: - type q5_0:   40 tensors
llama_model_loader: - type q8_0:   40 tensors
llama_model_loader: - type q4_K:  401 tensors
llama_model_loader: - type q5_K:   40 tensors
llama_model_loader: - type q6_K:   41 tensors
llm_load_vocab: special tokens cache size = 22
llm_load_vocab: token to piece cache size = 0.9310 MB
llm_load_print_meta: format           = GGUF V3 (latest)
llm_load_print_meta: arch             = qwen2
llm_load_print_meta: vocab type       = BPE
llm_load_print_meta: n_vocab          = 152064
llm_load_print_meta: n_merges         = 151387
llm_load_print_meta: vocab_only       = 0
llm_load_print_meta: n_ctx_train      = 32768
llm_load_print_meta: n_embd           = 8192
llm_load_print_meta: n_layer          = 80
llm_load_print_meta: n_head           = 64
llm_load_print_meta: n_head_kv        = 8
llm_load_print_meta: n_rot            = 128
llm_load_print_meta: n_swa            = 0
llm_load_print_meta: n_embd_head_k    = 128
llm_load_print_meta: n_embd_head_v    = 128
llm_load_print_meta: n_gqa            = 8
llm_load_print_meta: n_embd_k_gqa     = 1024
llm_load_print_meta: n_embd_v_gqa     = 1024
llm_load_print_meta: f_norm_eps       = 0.0e+00
llm_load_print_meta: f_norm_rms_eps   = 1.0e-06
llm_load_print_meta: f_clamp_kqv      = 0.0e+00
llm_load_print_meta: f_max_alibi_bias = 0.0e+00
llm_load_print_meta: f_logit_scale    = 0.0e+00
llm_load_print_meta: n_ff             = 29568
llm_load_print_meta: n_expert         = 0
llm_load_print_meta: n_expert_used    = 0
llm_load_print_meta: causal attn      = 1
llm_load_print_meta: pooling type     = 0
llm_load_print_meta: rope type        = 2
llm_load_print_meta: rope scaling     = linear
llm_load_print_meta: freq_base_train  = 1000000.0
llm_load_print_meta: freq_scale_train = 1
llm_load_print_meta: n_ctx_orig_yarn  = 32768
llm_load_print_meta: rope_finetuned   = unknown
llm_load_print_meta: ssm_d_conv       = 0
llm_load_print_meta: ssm_d_inner      = 0
llm_load_print_meta: ssm_d_state      = 0
llm_load_print_meta: ssm_dt_rank      = 0
llm_load_print_meta: ssm_dt_b_c_rms   = 0
llm_load_print_meta: model type       = 70B
llm_load_print_meta: model ftype      = Q4_K - Medium
llm_load_print_meta: model params     = 72.71 B
llm_load_print_meta: model size       = 44.15 GiB (5.22 BPW)
llm_load_print_meta: general.name     = Qwen2.5 72B Instruct
llm_load_print_meta: BOS token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOS token        = 151645 '<|im_end|>'
llm_load_print_meta: EOT token        = 151645 '<|im_end|>'
llm_load_print_meta: PAD token        = 151643 '<|endoftext|>'
llm_load_print_meta: LF token         = 148848 'ÄĬ'
llm_load_print_meta: FIM PRE token    = 151659 '<|fim_prefix|>'
llm_load_print_meta: FIM SUF token    = 151661 '<|fim_suffix|>'
llm_load_print_meta: FIM MID token    = 151660 '<|fim_middle|>'
llm_load_print_meta: FIM PAD token    = 151662 '<|fim_pad|>'
llm_load_print_meta: FIM REP token    = 151663 '<|repo_name|>'
llm_load_print_meta: FIM SEP token    = 151664 '<|file_sep|>'
llm_load_print_meta: EOG token        = 151643 '<|endoftext|>'
llm_load_print_meta: EOG token        = 151645 '<|im_end|>'
llm_load_print_meta: EOG token        = 151662 '<|fim_pad|>'
llm_load_print_meta: EOG token        = 151663 '<|repo_name|>'
llm_load_print_meta: EOG token        = 151664 '<|file_sep|>'
llm_load_print_meta: max token length = 256
llm_load_tensors: offloading 70 repeating layers to GPU
llm_load_tensors: offloaded 70/81 layers to GPU
llm_load_tensors: RPC[127.0.0.1:50052] model buffer size = 19668.07 MiB
llm_load_tensors:        CUDA0 model buffer size =  9039.54 MiB
llm_load_tensors:        CUDA1 model buffer size =  9064.88 MiB
llm_load_tensors:   CPU_Mapped model buffer size = 45213.44 MiB
..................................................................................................
llama_new_context_with_model: n_seq_max     = 1
llama_new_context_with_model: n_ctx         = 4096
llama_new_context_with_model: n_ctx_per_seq = 4096
llama_new_context_with_model: n_batch       = 2048
llama_new_context_with_model: n_ubatch      = 512
llama_new_context_with_model: flash_attn    = 0
llama_new_context_with_model: freq_base     = 1000000.0
llama_new_context_with_model: freq_scale    = 1
llama_new_context_with_model: n_ctx_per_seq (4096) < n_ctx_train (32768) -- the full capacity of the model will not be utilized
llama_kv_cache_init: kv_size = 4096, offload = 1, type_k = 'f16', type_v = 'f16', n_layer = 80
llama_kv_cache_init: RPC[127.0.0.1:50052] KV buffer size =   592.00 MiB
llama_kv_cache_init:      CUDA0 KV buffer size =   272.00 MiB
llama_kv_cache_init:      CUDA1 KV buffer size =   256.00 MiB
llama_kv_cache_init:        CPU KV buffer size =   160.00 MiB
llama_new_context_with_model: KV self size  = 1280.00 MiB, K (f16):  640.00 MiB, V (f16):  640.00 MiB
llama_new_context_with_model:        CPU  output buffer size =     0.58 MiB
llama_new_context_with_model: RPC[127.0.0.1:50052] compute buffer size =   584.00 MiB
llama_new_context_with_model:      CUDA0 compute buffer size =  1287.53 MiB
llama_new_context_with_model:      CUDA1 compute buffer size =   584.00 MiB
llama_new_context_with_model:        CPU compute buffer size =    24.01 MiB
llama_new_context_with_model: graph nodes  = 2806
llama_new_context_with_model: graph splits = 147 (with bs=512), 5 (with bs=1)
common_init_from_params: setting dry_penalty_last_n to ctx_size = 4096
common_init_from_params: warming up the model with an empty run - please wait ... (--no-warmup to disable)
main: llama threadpool init, n_threads = 14
main: chat template example:
<|im_start|>system
You are a helpful assistant<|im_end|>
<|im_start|>user
Hello<|im_end|>
<|im_start|>assistant
Hi there<|im_end|>
<|im_start|>user
How are you?<|im_end|>
<|im_start|>assistant


system_info: n_threads = 14 (n_threads_batch = 14) / 28 | CUDA : ARCHS = 750 | USE_GRAPHS = 1 | PEER_MAX_BATCH_SIZE = 128 | CPU : SSE3 = 1 | SSSE3 = 1 | AVX = 1 | AVX2 = 1 | F16C = 1 | FMA = 1 | AVX512 = 1 | LLAMAFILE = 1 | OPENMP = 1 | AARCH64_REPACK = 1 |

main: interactive mode on.
sampler seed: 3849300086
sampler params:
        repeat_last_n = 64, repeat_penalty = 1.000, frequency_penalty = 0.000, presence_penalty = 0.000
        dry_multiplier = 0.000, dry_base = 1.750, dry_allowed_length = 2, dry_penalty_last_n = 4096
        top_k = 40, top_p = 0.950, min_p = 0.050, xtc_probability = 0.000, xtc_threshold = 0.100, typical_p = 1.000, temp = 0.800
        mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
sampler chain: logits -> logit-bias -> penalties -> dry -> top-k -> typical -> top-p -> min-p -> xtc -> temp-ext -> dist
generate: n_ctx = 4096, n_batch = 2048, n_predict = -1, n_keep = 0

== Running in interactive mode. ==
 - Press Ctrl+C to interject at any time.
 - Press Return to return control to the AI.
 - To return control without starting a new line, end your input with '/'.
 - If you want to submit another line, end your input with '\'.

system
You are a helpful assistant

> hi
@,.(!.$C"BCBHC'5==@F<5%>@.F.HF/1.:=;<085H@@"CB>.';.-@*#8>5=)!,E#$4'E(HD6D24@3""C('A7
>
llama_perf_sampler_print:    sampling time =      14.32 ms /    95 runs   (    0.15 ms per token,  6633.15 tokens per second)
llama_perf_context_print:        load time =  213608.61 ms
llama_perf_context_print: prompt eval time =    7300.01 ms /    21 tokens (  347.62 ms per token,     2.88 tokens per second)
llama_perf_context_print:        eval time =   25119.07 ms /    83 runs   (  302.64 ms per token,     3.30 tokens per second)
llama_perf_context_print:       total time =   35510.62 ms /   104 tokens
Interrupted by user