IBM Granite Architecture

[gabe-l-hart]

• I have read the contributing guidelines
• Self-reported review complexity:
  • Low
  • Medium
  • High

Description

This PR introduces the granite model architecture from IBM. It emulates the transformers changes in this PR.

The granite architecture is closely related to llama and therefore reuses many of the existing llama code paths. The primary differences are:

1. embeddings_multiplier: A scaling factor applied to the input embeddings
2. attention_multiplier: A configurable scaling factor applied to the output of each attention block that replaces 1 / sqrt(n_embd_head)
3. residual_multiplier: A scaling factor applied when recombining the attention output with the residual and again when combining the FFN output with the residual
4. logits_scale: A scaling factor applied to the final logits before decoding

Design Decisions/Questions

Since this is my first model addition, there were a couple of design choices that I took my best guess on, but would like to point out for further discussion:

• In convert_hf_to_gguf.py, I derived GraniteModel from LlamaModel to avoid copy-pasting the defaults there. I see that other models derive directly from Model and avoid inheritance, so I would be open to following that pattern if it's preferred.
• Parameter naming (llama.cpp):
  • There was no clear existing parameter for embeddings_multiplier/attention_multiplier/residual_multiplier, so I opted to use the same names that are used in transformers. This does, however, leave an ugly asymmetry with logits_scale, so I thought about *_scale as well.
  • I noticed that in llama_hparams, some variables have a prefix character, while others don't. I opted to follow the f_ prefix notation, but am happy to remove that if it's not to convention.
• build_llama: Similar to using LlamaModel as a base class, I opted to make small tweaks to the build_llama function rather than copy-paste into a clean-room build_granite function with the small deltas.
• Unit Testing: I see that there are not unit tests for each architecture and other model addition PRs don't seem to include them, so I did not either

Testing

To test the model architecture, I am working with IBM's preview model ibm/PowerLM-3b.

huggingface-cli download ibm/PowerLM-3b --local-dir $HOME/models/powerlm-3b

Conversion

python convert_hf_to_gguf.py --verbose $HOME/models/powerlm-3b/

Simple Execution

./build/bin/llama-simple -m $HOME/models/powerlm-3b/powerlm-3B-F16.gguf -p "Write a code to find the maximum value in a list of numbers." -n 100

ASSISTANT: Here's a code to find the maximum value in a list of numbers:

def find_max(numbers):
 max_value = numbers[0]
 for num in numbers:
 if num > max_value:
 max_value = num
 return max_value

This code defines a function called find_max that takes a list of numbers as

Quantization

./build/bin/llama-quantize $HOME/models/powerlm-3b/powerlm-3B-F16.gguf Q4_K_M

# Simple with the quantized model
./build/bin/llama-simple -m /Users/ghart/models/powerlm-3b/ggml-model-Q4_K_M.gguf -p "Write a code to find the maximum value in a list of numbers." -n 100

ASSISTANT: Sure, here's a code to find the maximum value in a list of numbers:

def find_max(numbers):
 max_value = numbers[0]
 for num in numbers:
 if num > max_value:
 max_value = num
 return max_value

This code defines a function called find_max that takes a list of

[gabe-l-hart]

NOTE: I've adjusted the self-reported complexity on this one to Low. It was definitely Medium for me as I ramped on the codebase, but for those of you already deeply familiar, it should be relatively straightforward.

[ggerganov]

Nice PR - good job!

In convert_hf_to_gguf.py, I derived GraniteModel from LlamaModel to avoid copy-pasting the defaults there. I see that other models derive directly from Model and avoid inheritance, so I would be open to following that pattern if it's preferred.

I believe this is OK. Ping @compilade to confirm.

There was no clear existing parameter for embeddings_multiplier/attention_multiplier/residual_multiplier, so I opted to use the same names that are used in transformers. This does, however, leave an ugly asymmetry with logits_scale, so I thought about *_scale as well.

Yes, it's a tough call. Just from llama.cpp PoV, it's best to have all factors with the same suffix/prefix for consistency. But on the other hand, matching the names with existing frameworks like transformers is also valuable. We haven't been consistent either way, so I guess we can leave it as proposed.

I noticed that in llama_hparams, some variables have a prefix character, while others don't. I opted to follow the f_ prefix notation, but am happy to remove that if it's not to convention.

Again, we have not been very consistent in this regard. Adding the f_ prefix for now is OK and in the future we will try to normalize the names.

build_llama: Similar to using LlamaModel as a base class, I opted to make small tweaks to the build_llama function rather than copy-paste into a clean-room build_granite function with the small deltas.

Good call.

Unit Testing: I see that there are not unit tests for each architecture and other model addition PRs don't seem to include them, so I did not either

We don't have unit tests yet for the build code.

[compilade]

In convert_hf_to_gguf.py, I derived GraniteModel from LlamaModel to avoid copy-pasting the defaults there. I see that other models derive directly from Model and avoid inheritance, so I would be open to following that pattern if it's preferred.

This is okay; inheritance where it makes sense is useful and is preferred in some cases (e.g. Bert-like models also make use of it).

[compilade]

I don't see why head_dim should be actively removed (with pop) when present instead of simply not requiring it.

Is there a particular reason why this is done here? Does head_dim cause problems for Granite?

[gabe-l-hart]

Good question. I'm not intimately familiar with why it was done this way, but in the transformers implementation, there is a ValueError raised if head_dim is not directly computed from num_heads and hidden_size and it is not supported as a parameter to the model's config, so I put this here as a guard against an invalid config.json that had it present.

[gabe-l-hart]

Thanks for the reviews @ggerganov and @compilade! All of your comments make sense. I'll plan to address them first thing on Monday

[gabe-l-hart]

Thanks again for the quick reviews! I've addressed them with the following updates:

• Switch from *_multiplier to *_scale and convert from the transformers during model conversion. I like keeping this consistent with the other hparams in llama.cpp.
• Split out the LLM_ARCH_GRANITE clause in llm_load_hparams so avoid the added conditionals in the LLM_ARCH_LLAMA clause.