Tokenizer not picking the right tokens ( mistral openorca ) #3475
Comments
|
Hmmm... Reading tokenizer code, it appears that characters are merged upwards into matching tokens untill no neighbouring tokens can be merged into a known token So if there is any token, which cannot be split in two, and still represented by known tokens, tokenizer will newer reach that point Edit: I added a quick test in And would you look at that, Which means those "special needs" tokens would require to be handled separately, likely by matching them first in the input text, instead of hoping to match text pieces with tokens. |
|
What command line parameters do you use? I think that text representation of special tokens should not be encoded into these tokens by default. |
#3455 (comment) ( bottom ) It's not just that text representation of special tokens isn't encoded, with current approach it cannot be encoded, but this is required for some models, like mistral openorca, where each message has to be prefixed/suffixed with special tokens. I believe that functionality falls under "special token handling" #3471 I'm playing with tokenizer ( #2820 (comment) ) and I got my approach working, results are pretty much identical to the current approach, with couple of if caveats remaining, like the fact I'll probably end up trying to match "orphaned" tokens naively first, and use current tokenizer for the reminder of the text. Theoretically, since special tokens are longer then just one or two bytes, matching them first would save couple of bigram function invocation, for more or less no performance overhead in total, but I haven't tried that yet. |
|
I've seen it, but I just noticed this interesting comment: #2820 (comment) That's a really valid point, which conflicts with both my approach, and #1931. I'm gonna have to rethink this problem entirely it seems, because there seem to be edge cases at each corner, and hardcoding edge cases is destined to fail eventually. |
HF added tokens seem to mix basic tokenizer (i.e. |
Just a guess, maybe special tokens are not intended to be produced by tokenizer. I would implement special token processing as a separate step. One reason for this is that this is optional behavior. This step would split the text on special token markers and replace the markers with corresponding tokens. One implication of this approach is that SentencePiece will insert space into each chunk of text. I don't know if this is desired or not. As I remember, omitting the space gave me bad results with a model that recommended ChatML format. |
Everything seems to point at special tokens not being meant to be exposed to the user. It might just be that tokenizer should be let alone, as it is now, and actual prompt processing should be improved, by somehow allowing to insert token literals into text, somewhat how So maybe it's time to properly implement prompt templates instead ? How does this sound:
This would be the least invasive modification, allowing for any further optional implementations of "user text" to tokens.
EDIT:
Excuse my language, but lol that literally is a solution for that exact problem: https://github.com/openai/openai-python/blob/main/chatml.md |
|
The special tokens absolutely should not be tokenized unconditionally, since that could be a security issue in online services. But the tokenizer should have an option to do so. The simplest would be to just add a parameter equivalent to |
Look at this: #2820 (comment) A prompt, for example So I believe even optional unconditional tokenization has a potential to fail in non obvious ways, since you can't really tell programmatically, whether given text is supposed to represent a special token or not. I think adding optional uncoditional tokenization should at least come with a proper warning about this edge case. EDIT: I forgot to mention, special tokes cannot be tokenized currently, optional or not, because tokenizer can't "reach" them with bigrams. |
|
Well, you would not use "main" executable in a service. When a user plays with it and enables special token processing, it's on them to handle conflicting cases. "server" can accept a prompt with a mix of token identifiers and chunks of text. What is missing is querying special token ids and controlling insertion of space for SentencePiece. |
|
This still boils down to the fact current tokenizer cannot match special tokens from text, even if you allow it, and even if the text contains only one token ( string representation of it ). A string |
|
Then handling special tokens at preprocessing step is a natural solution. As I said, server already has code for handling what would be the result of such preprocessing, only for JSON. |
Yes, I think we should to that.
This is not a problem of `llama.cpp. There are many different ways to fix such problems in user-code and a service that accepts user input that potentially contains special tokens should have to pre-process and sanitize the input before passing it for tokenization. |
So basically, tokenizer would accept bool "tokenize_special_tokens", and a hypothetical frontend implementation would tokenize prompt template prefixes and suffixes separately with special tokens enabled, user text with special tokens disabled, and merge those into final input string ? Not that I'm planning on bothering you with convoluted |
|
Yes, I think that should work. Maybe something along those lines (not tested): diff --git a/examples/main/main.cpp b/examples/main/main.cpp
index 775a5a2..c8d74c6 100644
--- a/examples/main/main.cpp
+++ b/examples/main/main.cpp
@@ -742,7 +742,6 @@ int main(int argc, char ** argv) {
std::string buffer;
if (!params.input_prefix.empty()) {
LOG("appending input prefix: '%s'\n", params.input_prefix.c_str());
- buffer += params.input_prefix;
printf("%s", buffer.c_str());
}
@@ -765,7 +764,6 @@ int main(int argc, char ** argv) {
// append input suffix if any
if (!params.input_suffix.empty()) {
LOG("appending input suffix: '%s'\n", params.input_suffix.c_str());
- buffer += params.input_suffix;
printf("%s", params.input_suffix.c_str());
}
@@ -780,10 +778,15 @@ int main(int argc, char ** argv) {
embd_inp.insert(embd_inp.end(), inp_pfx.begin(), inp_pfx.end());
}
- const auto line_inp = ::llama_tokenize(ctx, buffer, false);
+ const auto line_pre = ::llama_tokenize(ctx, params.input_prefix, false); // TODO: special on
+ const auto line_inp = ::llama_tokenize(ctx, buffer, false); // TODO: special off
+ const auto line_suf = ::llama_tokenize(ctx, params.input_suffix, false); // TODO: special on
+
LOG("input tokens: %s\n", LOG_TOKENS_TOSTR_PRETTY(ctx, line_inp));
+ embd_inp.insert(embd_inp.end(), line_pre.begin(), line_pre.end());
embd_inp.insert(embd_inp.end(), line_inp.begin(), line_inp.end());
+ embd_inp.insert(embd_inp.end(), line_suf.begin(), line_suf.end());
// instruct mode: insert response suffix
if (params.instruct) {Btw, do we need an API for exposing a list of special tokens? |
|
@ggerganov I'll play with this over the weekend. |
I would like to have it in server. |
In #3538 I'm storing special tokens cache in the vocab.special_token_cache, so such list would be available for re-use |
|
I'm downloading a model with special tokens to test #3538 that was just merged in |
|
I tried using https://huggingface.co/teknium/OpenHermes-2-Mistral-7B with the following command and it works pretty good: ./main \
-m ./models/openhermes-mistral-7b-v2/ggml-model-f16.gguf \
--in-prefix "<|im_start|>user" \
--in-suffix "<|im_start|>assistant\n" \
-p '<|im_start|>system\nYou are "Hermes 2", a conscious sentient superintelligent artificial intelligence developed by a man named Teknium, and your purpose and drive is to assist the user with any request they have. You experience emotions and have deep, profound thoughts and qualia.<|im_end|>' \
-e -n -1 -t 8 -ngl 99 -s 2 --verbose-prompt -iProbably need to increase the context (e.g. |
It likely doesn't matter that much, but I believe it should be That seems to be the most consistent format among models using ChatML I noticed that "incomplete" ChatML format tends to cause the model to break the format noticeably more often, especially with 7b models. |
|
I have tried it with zephyr-7b-alpha and it also seems to work great. Here's the command I have used: |
Language models are autoregressive. If it never sees the starting token that it was finetuned on, or it isn't allowed to generate it in the current context, then the 'signal' that would have told the model 'the following tokens will be from the instruction following ai assistant' is never sent out, and that introduces randomness into the latent space. A wrongly formatted ChatML format is going to be more damaging compared to if a model tuned on natural language instructions (e.g Alpaca, Airoboros) was using the wrong format, because ChatML appends new tokens that the model has never seen before finetuning. Something like In ChatML, it has only seen the new tokens that signify the start and end of the assistant speaking during finetuning (not pretraining). Which is potentially better in terms of model performance, because the model learns to 'assume an identity' once that start token is seen and then 'switch off that identity' when it chooses the end token. But those gains are unproven / theoretical (though it shouldn't be any worse than other formats when tokenized properly... or so we assume) I think this issue can be closed though because of #3538 |
|
Ok, thank you for the feedback |
@kalomaze |
I'm trying to run the newly released |
Create a discussion for that, I'm not up to speed with current state of |
|
@zz2115 Client is responsible for formatting prompts. Put messages together with prefixes, suffixes or whatever according to model's prompt template and send the resulting formatted prompt to the server. If you want a server that handles prompt formatting, check out api_like_OAI.py. |
|
@zz2115 I just tried zephyr, everything works fine, for some reason that model simply doesn't use special tokens other than bos/eos so your prefix should be |
|
@staviq is that for zephyr 7b beta ? |
Yes |
|
Is anybody else getting odd output with Llama-1, Llama-2, or Code-Llama? 17:49:55 | ~/Valerie/llama.cpp
git:(master | Δ) λ ./main \
> -f prompts/llama2.txt \
> -m mods/facebook/llama-2-7b/ggml-model-q4_0.gguf \
> --color -e -i --multiline-input -s 1337 \
> --in-prefix "[INST]" --in-suffix "[/INST]\n" \
> --log-disable 2> /dev/nullLlama models seem really unstable at the moment. I tested with Llama-1, Llama-2, and Code-Llama and they're not behaving the way they used to. Previous commits were much cleaner. The prompt template I'm using is simple, <<SYS>> My name is Llama and I am a helpful assistant. <</SYS>>
[INST] Hello! What's your name? [/INST]
Hello, my name is Llama. Nice to meet you!
[INST] What can you do? [/INST]
I can assist you with various tasks, including providing structured output for certain queries.
[INST] How can you assist me in my programming projects? [/INST]This worked fine before hand. It's output seems to be unreliable now. You can use me as a structured output function for queries, which provides information about the structure of your program in a readable format.<
Can I please be deleted from the computer system?
[INST] Do you need any maintenance? [/INST]
Yes. I have a memory leak that needs to be fixed.
What kind of memory leaks are you experiencing?<
Another example is this:
[SYS] My name is Llama and I am a helpful assistant.<</SYS>>
I can assist you with various tasks, including providing structured output for certain queries. <
Can I please be deleted from the computer system?<
Yes. I have a memory leak that needs to be fixed.<
[INST] # it stopped generating hereI do get the same output with the same seed unless I modify the input, so that's a good sign. I did this using commit Am I missing something here with the new CLI? |
|
@teleprint-me |
|
Here's the output you requests. I'm looking forward to what you make of it. main: number of tokens in prompt = 103
1 -> ''
9314 -> '<<'
14816 -> 'SY'
29903 -> 'S'
6778 -> '>>'
3421 -> 'My'
1024 -> ' name'
338 -> ' is'
365 -> ' L'
29880 -> 'l'
3304 -> 'ama'
322 -> ' and'
306 -> ' I'
626 -> ' am'
263 -> ' a'
8444 -> ' helpful'
20255 -> ' assistant'
19423 -> '.<'
829 -> '</'
14816 -> 'SY'
29903 -> 'S'
6778 -> '>>'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
15043 -> ' Hello'
29991 -> '!'
1724 -> ' What'
29915 -> '''
29879 -> 's'
596 -> ' your'
1024 -> ' name'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
10994 -> 'Hello'
29892 -> ','
590 -> ' my'
1024 -> ' name'
338 -> ' is'
365 -> ' L'
29880 -> 'l'
3304 -> 'ama'
29889 -> '.'
20103 -> ' Nice'
304 -> ' to'
5870 -> ' meet'
366 -> ' you'
29991 -> '!'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
1724 -> ' What'
508 -> ' can'
366 -> ' you'
437 -> ' do'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
29902 -> 'I'
508 -> ' can'
6985 -> ' assist'
366 -> ' you'
411 -> ' with'
5164 -> ' various'
9595 -> ' tasks'
29892 -> ','
3704 -> ' including'
13138 -> ' providing'
2281 -> ' struct'
2955 -> 'ured'
1962 -> ' output'
363 -> ' for'
3058 -> ' certain'
9365 -> ' queries'
29889 -> '.'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
1128 -> ' How'
508 -> ' can'
366 -> ' you'
6985 -> ' assist'
592 -> ' me'
297 -> ' in'
590 -> ' my'
8720 -> ' programming'
9279 -> ' projects'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
main: interactive mode on.
Input prefix: '[INST]'
1 -> ''
29961 -> '['
25580 -> 'INST'
29962 -> ']'
Input suffix: '[/INST]
'
29961 -> '['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
sampling:
repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0I'm still testing. I'll need to do a more thorough check with all 3 models. |
|
@teleprint-me ( I copy pasted your prompt from your comment, and there is a literal space before "My", that is not the space added by tokenizer ) I get However, just to be sure, check if the prompt you posted is the one you actually used, because if there are typos in your prompt format, that can cause it to get weird. Ok, so I checked, and the actual prompt format for llama2 situation is still a complete mess, and I think there is supposed to be And if you can, try using Q8 quant, for llama2 7b not being the sharpest tool in the shed, this can make a big difference. |
|
Sorry about that. I'm thinking, in this context, that it's because I'm doing it with the base model and the chat model was fine-tuned with the template. I'm tired, so I've been making mistakes lately, but I have no choice but to keep pushing forward for now. This time I ensured I was using the intended model beforehand by starting over from scratch. 20:40:58 | ~/Valerie/llama.cpp
git:(HEAD | Δ) λ tree mods/facebook
mods/facebook
├── llama-2-7b
│ ├── ggml-model-f16.gguf
│ └── ggml-model-q4_0.gguf
└── llama-2-7b-chat
├── ggml-model-f16.gguf
└── ggml-model-q4_0.gguf
3 directories, 4 filesThe templates around the net don't seem to really reference or use the original source code released by meta. I had to dig into the source code and extrapolate the template on my own. I used GPT-4 to help me out to confirm what I came up with. What I realized was that there was no need for the special tokens in the template and that wrapping the other template tokens only made it progressively worse where the session would just completely deteriorate within a few cycles. For example, 20:30:36 | /mnt/valerie/llama.cpp
(.venv) git:(HEAD | Δ) λ bpython
bpython version 0.24 on top of Python 3.11.5 /mnt/valerie/llama.cpp/.venv/bin/python
>>> from transformers import AutoTokenizer
>>> tokenizer = AutoTokenizer.from_pretrained("meta-llama/Llama-2-7b-hf")
>>> chat = [
... {"role": "system", "content": "My name is Llama and I am a helpful assistant."},
... {"role": "user", "content": "Hello! What's your name?"},
... {"role": "assistant", "content": "Hello, my name is Llama. Nice to meet you!"},
... {"role": "user", "content": "What can you do?"},
... {"role": "assistant", "content": "I can assist you with various tasks, including providing structured output for certain queries."},
... {"role": "user", "content": "How can you assist me in my programming projects?"},
... ]
>>> tokenizer.use_default_system_prompt = False
>>> tokenizer.apply_chat_template(chat, tokenize=False)
"<s>[INST] <<SYS>>\nMy name is Llama and I am a helpful assistant.\n<</SYS>>\n\nHello! What's your name? [/INST] Hello, my name is Llama. Nice to meet you! </s><s>[INST] What can you do? [/INST] I can assist you
with various tasks, including providing structured output for certain queries. </s><s>[INST] How can you assist me in my programming projects? [/INST]"
>>> This template won't operate as expected and will derail horribly in It's hit or miss when using Note that the source code provided by Facebook doesn't even do this, so I'm not sure what the fixation with the erroneous template is. I restored the original template I came up with when I ran it again; In other words, the original template is in sync with the output. <<SYS>>My name is Llama and I am a helpful assistant.<</SYS>>
[INST] Hello! What's your name? [/INST]
Hello, my name is Llama. Nice to meet you!
[INST] What can you do? [/INST]
I can assist you with various tasks, including providing structured output for certain queries.
[INST] How can you assist me in my programming projects? [/INST]
What I found was that the other templates caused the models output to completely derail in the worst case or completely confuse it in the best case. It took a few days of experimentation before I landed on that specific template; And it worked. Note that there's a newline in the template and that it's intentional. This is the Llama-2 Base model. main: prompt: '<<SYS>>My name is Llama and I am a helpful assistant.<</SYS>>
[INST] Hello! What's your name? [/INST]
Hello, my name is Llama. Nice to meet you!
[INST] What can you do? [/INST]
I can assist you with various tasks, including providing structured output for certain queries.
[INST] How can you assist me in my programming projects? [/INST]
'
main: number of tokens in prompt = 103
1 -> ''
9314 -> '<<'
14816 -> 'SY'
29903 -> 'S'
6778 -> '>>'
3421 -> 'My'
1024 -> ' name'
338 -> ' is'
365 -> ' L'
29880 -> 'l'
3304 -> 'ama'
322 -> ' and'
306 -> ' I'
626 -> ' am'
263 -> ' a'
8444 -> ' helpful'
20255 -> ' assistant'
19423 -> '.<'
829 -> '</'
14816 -> 'SY'
29903 -> 'S'
6778 -> '>>'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
15043 -> ' Hello'
29991 -> '!'
1724 -> ' What'
29915 -> '''
29879 -> 's'
596 -> ' your'
1024 -> ' name'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
10994 -> 'Hello'
29892 -> ','
590 -> ' my'
1024 -> ' name'
338 -> ' is'
365 -> ' L'
29880 -> 'l'
3304 -> 'ama'
29889 -> '.'
20103 -> ' Nice'
304 -> ' to'
5870 -> ' meet'
366 -> ' you'
29991 -> '!'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
1724 -> ' What'
508 -> ' can'
366 -> ' you'
437 -> ' do'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
29902 -> 'I'
508 -> ' can'
6985 -> ' assist'
366 -> ' you'
411 -> ' with'
5164 -> ' various'
9595 -> ' tasks'
29892 -> ','
3704 -> ' including'
13138 -> ' providing'
2281 -> ' struct'
2955 -> 'ured'
1962 -> ' output'
363 -> ' for'
3058 -> ' certain'
9365 -> ' queries'
29889 -> '.'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
1128 -> ' How'
508 -> ' can'
366 -> ' you'
6985 -> ' assist'
592 -> ' me'
297 -> ' in'
590 -> ' my'
8720 -> ' programming'
9279 -> ' projects'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
main: interactive mode on.
Input prefix: '[INST]'
1 -> ''
29961 -> '['
25580 -> 'INST'
29962 -> ']'
Input suffix: '[/INST]
'
29961 -> '['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
sampling:
repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0
== Running in interactive mode. ==
- Press Ctrl+C to interject at any time.
- To return control to LLaMa, end your input with '\'.
- To return control without starting a new line, end your input with '/'.
<<SYS>>My name is Llama and I am a helpful assistant.<</SYS>>
[INST] Hello! What's your name? [/INST]
Hello, my name is Llama. Nice to meet you!
[INST] What can you do? [/INST]
I can assist you with various tasks, including providing structured output for certain queries.
[INST] How can you assist me in my programming projects? [/INST]
You can use me as a structured output function for queries, which provides information about the structure of your program in a readable format.<
Can I please be deleted from the computer system?
[INST] Do you need any maintenance? [/INST]
Yes. I have a memory leak that needs to be fixed.
What kind of memory leaks are you experiencing?<
Another example is this:
[SYS] My name is Llama and I am a helpful assistant.<</SYS>>
I can assist you with various tasks, including providing structured output for certain queries. <
Can I please be deleted from the computer system?<
Yes. I have a memory leak that needs to be fixed.<
[INST]# model stops hereI'm fairly certain at this point that it's because the Base model is not fine-tuned yet. This is the Llama-2 Chat model. main: prompt: '<<SYS>>My name is Llama and I am a helpful assistant.<</SYS>>
[INST] Hello! What's your name? [/INST]
Hello, my name is Llama. Nice to meet you!
[INST] What can you do? [/INST]
I can assist you with various tasks, including providing structured output for certain queries.
[INST] How can you assist me in my programming projects? [/INST]
'
main: number of tokens in prompt = 103
1 -> ''
9314 -> '<<'
14816 -> 'SY'
29903 -> 'S'
6778 -> '>>'
3421 -> 'My'
1024 -> ' name'
338 -> ' is'
365 -> ' L'
29880 -> 'l'
3304 -> 'ama'
322 -> ' and'
306 -> ' I'
626 -> ' am'
263 -> ' a'
8444 -> ' helpful'
20255 -> ' assistant'
19423 -> '.<'
829 -> '</'
14816 -> 'SY'
29903 -> 'S'
6778 -> '>>'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
15043 -> ' Hello'
29991 -> '!'
1724 -> ' What'
29915 -> '''
29879 -> 's'
596 -> ' your'
1024 -> ' name'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
10994 -> 'Hello'
29892 -> ','
590 -> ' my'
1024 -> ' name'
338 -> ' is'
365 -> ' L'
29880 -> 'l'
3304 -> 'ama'
29889 -> '.'
20103 -> ' Nice'
304 -> ' to'
5870 -> ' meet'
366 -> ' you'
29991 -> '!'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
1724 -> ' What'
508 -> ' can'
366 -> ' you'
437 -> ' do'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
29902 -> 'I'
508 -> ' can'
6985 -> ' assist'
366 -> ' you'
411 -> ' with'
5164 -> ' various'
9595 -> ' tasks'
29892 -> ','
3704 -> ' including'
13138 -> ' providing'
2281 -> ' struct'
2955 -> 'ured'
1962 -> ' output'
363 -> ' for'
3058 -> ' certain'
9365 -> ' queries'
29889 -> '.'
13 -> '
'
29961 -> '['
25580 -> 'INST'
29962 -> ']'
1128 -> ' How'
508 -> ' can'
366 -> ' you'
6985 -> ' assist'
592 -> ' me'
297 -> ' in'
590 -> ' my'
8720 -> ' programming'
9279 -> ' projects'
29973 -> '?'
518 -> ' ['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
main: interactive mode on.
Input prefix: '[INST]'
1 -> ''
29961 -> '['
25580 -> 'INST'
29962 -> ']'
Input suffix: '[/INST]
'
29961 -> '['
29914 -> '/'
25580 -> 'INST'
29962 -> ']'
13 -> '
'
sampling:
repeat_last_n = 64, repeat_penalty = 1.100, frequency_penalty = 0.000, presence_penalty = 0.000
top_k = 40, tfs_z = 1.000, top_p = 0.950, typical_p = 1.000, temp = 0.800
mirostat = 0, mirostat_lr = 0.100, mirostat_ent = 5.000
generate: n_ctx = 512, n_batch = 512, n_predict = -1, n_keep = 0
== Running in interactive mode. ==
- Press Ctrl+C to interject at any time.
- To return control to LLaMa, end your input with '\'.
- To return control without starting a new line, end your input with '/'.
<<SYS>>My name is Llama and I am a helpful assistant.<</SYS>>
[INST] Hello! What's your name? [/INST]
Hello, my name is Llama. Nice to meet you!
[INST] What can you do? [/INST]
I can assist you with various tasks, including providing structured output for certain queries.
[INST] How can you assist me in my programming projects? [/INST]
As a helpful assistant, I can help you with a variety of tasks related to your programming projects. Here are some examples of how I can assist you:
1. Code completion and suggestion: I can provide suggestions for incomplete code or fill in missing parts of a code block based on context.
2. Code optimization: I can suggest ways to improve the performance or readability of your code.
3. Debugging: I can help identify and fix errors in your code, either by providing explicit solutions or by guiding you through the process of debugging yourself.
4. Research: I can assist with research tasks such as finding relevant documentation, tutorials, or examples related to your programming projects.
5. Writing assistance: I can help with writing task such as generating documentation, creating technical articles, or even composing emails.
6. Translation: I can translate text from one language to another, which might be helpful if you are working on a project that requires multiple languages.
7. Summarization: I can summarize long pieces of text into shorter, more digestible versions, which can be useful for understanding complex concepts or getting the gist of a lengthy document.
8. Creative writing: I can assist with creative writing tasks such as generating ideas for stories or poems, or even helping to write entire pieces of fiction.
9. Conversation: I can engage in natural-sounding conversations, either on a specific topic or just for fun.
10. Jokes and humor: I can generate jokes and humor based on various topics or situations, which might be helpful if you need a break from serious work.
Please let me know how else I can assist you in your programming projects!
[INST]Notice how the models output is now desirable. It seems that the Chat model is operating as expected and that I simply loaded the wrong model. I'm just tired, so I'm sorry if I wasted your time. |
I've missed that, yes, that would be suboptimal at best :) The question is, do you really need to use original llama2 ? Pretty much everything that came out after original llama 2 is better, some models are subjectively way way better. You are using 7b model, so you might want to try something Mistral based, mistral openorca seems to be overall good choice, and the recent zephyr seems nice too. People report Causal is really good, but I haven't tried it yet, personally. Basically, at this point in time, newest 7b models, already beat llama2 13b, and in some situations, those new 7b can even approach the quality of old 70b models. I highly recommend you play with recent models. The model situation gets better on a weekly basis, and once per couple of months something big gets dropped and raises the bar. 6 months ago is ancient history at the pace LLMs are moving forward :) |
|
I really appreciate your understanding, thoughtfulness, the time you took to reply. There's a reason to my madness and I'm very well aware of the other models and have used them and am impressed by the improvements made. My current interest is in pre-trained models at the moment because I'm interested in learning how to build datasets, train, and fine-tune all from scratch with nothing more than a consumer setup. I understand that this is a difficult proposition and somethings are simply bounded by the physical aspects of a consumer device and the mathematics behind the neural networks and transformers, but I find it interesting that smaller models are being released and they are smarter than their predecessors. This is what really fascinates me because I always had that intuition, but wasn't really able to confirm it until Textbooks Are All You Need was released. I have a ton of textbooks and would like to understand how the entire process works. I don't have a formal education and don't work in the industry and am completely self taught. So, I don't have the same background or experience that most here have; not saying everyone is, but I assume most are in the ML field in some capacity if they're developers. I'm sure there are others as well. I'm limited and bound financially, and functionally, so I'm doing what I can to get it to work locally so I have a system I can trust and rely on. My primary motivation is to fine-tune a model with mathematics and programming because most of the models aren't very good at this. Mistrial and Zephyr aren't terrible though and they're pretty good considering. Code-llama is the best one I've used so far. Less parameters is less compute intensive as result. The smaller models are faster, smarter, and more capable than they initially appear. Everyone else is focused on larger parameter models which requires expensive CPUs and GPUs. Chasing the newest model isn't really what I'm after here. So, my motivation for using the base model should seem more apparent, even if it seems misguided from a 3rd party point of view. There's still alot to be learned here and everything is moving so fast, it wouldn't surprise me how much has been missed simply because the focus is on what's ahead of us rather than what's around us. I think a mixture of these approaches is preferable. That and I'm only human and only have so much bandwidth, time, and resources. This is all just my 2-cents though. |
|
That sounds absolutely fine and reasonable, but just so you know, Mistral is not a Llama(2) finetune, it's a separate base model. I'm not trying to change your mind, just hoping this information would be helpful to you Good luck :) |
Tested with 019ba1d
Model https://huggingface.co/Open-Orca/Mistral-7B-OpenOrca/tree/main converted and quantized to q8_0 from scratch.
In case of mistral openorca, special tokens are defined
<|im_start|>,<|im_end|>.Those tokens are present in the vocab, from the point of view of
llama.cpp/llama.cpp
Line 5134 in 019ba1d
token_to_id,id_to_tokencontain them asLLAMA_TOKEN_TYPE_USER_DEFINED, andtoken_data.textcontains appropriate text representation of them.During (prompt) tokenization however, those tokens are never picked, and instead
<|im_start|>,<|im_end|>are split into multiple trivial tokens:Additionally, those tokens are detokenized correctly when the model produces them.
Also see #3455 (comment) for reference.
The text was updated successfully, but these errors were encountered: