invalid_api_key using OpenAI API

[natbutter]

I am trying to use the OpenAI API to access a local model, but cannot get the API key working.

If I start my server with:

bash start_linux.sh --api --ssl-keyfile key.pem --ssl-certfile cert.pem --api-key 1234

Using curl works as expected:

curl -k https://127.0.0.1:5000/v1/chat/completions  -H "Authorization: Bearer 1234" -H "Content-Type: application/json"   -d '{"messages": [{"role": "user","content": "What is the latest iphone model?"}],"mode": "chat","character": "Example"}'

{"id":"chatcmpl-1714106093700253440","object":"chat.completions","created":1714106093,"model":"TheBloke_CodeLlama-34B-AWQ","choices":[{"index":0,"finish_reason":"stop","message":{"role":"assistant","content":"The latest iphone is the iPhone 11 pro max. Its been released in 2019-"}}],"usage":{"prompt_tokens":239,"completion_tokens":23,"total_tokens":262}}

But If I try to use the OpenAI API, i.e.

from dotenv import load_dotenv
load_dotenv()
import openai
llm = openai.OpenAI()
response = llm.chat.completions.create(
    model='TheBloke_CodeLlama-34B-AWQ',
    messages=[{"role": "user","content": "What is the latest iphone model?"}]
    )
print(response)

and run it with

python test_localAPI.py 

Traceback (most recent call last):
File "tests/test_localAPI.py", line 10, in
response = llm.chat.completions.create(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "miniconda3/envs/llmlocal/lib/python3.11/site-packages/openai/_utils/_utils.py", line 277, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "miniconda3/envs/llmlocal/lib/python3.11/site-packages/openai/resources/chat/completions.py", line 579, in create
return self._post(
^^^^^^^^^^^
File "miniconda3/envs/llmlocal/lib/python3.11/site-packages/openai/_base_client.py", line 1232, in post
return cast(ResponseT, self.request(cast_to, opts, stream=stream, stream_cls=stream_cls))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "miniconda3/envs/llmlocal/lib/python3.11/site-packages/openai/_base_client.py", line 921, in request
return self._request(
^^^^^^^^^^^^^^
File "/miniconda3/envs/llmlocal/lib/python3.11/site-packages/openai/_base_client.py", line 1012, in _request
raise self._make_status_error_from_response(err.response) from None
openai.AuthenticationError: Error code: 401 - {'error': {'message': 'Incorrect API key provided: 1234. You can find your API key at https://platform.openai.com/account/api-keys.', 'type': 'invalid_request_error', 'param': None, 'code': 'invalid_api_key'}}

I have set a bunch of environment variables that do seem to be read correctly by Python.

OPENAI_API_KEY=1234
OPENEDAI_KEY_PATH=/home/user/key.pem
OPENEDAI_CERT_PATH=/home/user/cert.pem
OPENAI_API_BASE=https://127.0.0.1:5000/v1
SD_WEBUI_URL=https://127.0.0.1:7860/
OPENAI_API_HOST=https://127.0.0.1:5000
OPENEDAI_PORT=5000

I was on an older version of text-generation-webui, but updated to the latest and get the same issue. And using openai==1.23.6
Any suggestions for how to use the local model using the OpenAI API??

[natbutter]

FYI old version of openai also works fine, albeit the code is a little different.
pip install openai==0.28.1

from dotenv import load_dotenv
load_dotenv()
import openai
completion = openai.ChatCompletion.create(model="TheBloke_CodeLlama-34B-AWQ", messages=[{"role": "user", "content": "What is the latest iphone model"}])
print(completion)

[natbutter]

Also, ollama works out of the box.

from dotenv import load_dotenv
load_dotenv()
from openai import OpenAI

client = OpenAI(
    base_url = 'http://localhost:11434/v1',
    api_key='ollama', # required, but unused
)

response = client.chat.completions.create(
  model="llama3",
  messages=[
    {"role": "system", "content": "You are a helpful assistant."},
    {"role": "user", "content": "Who won the world series in 2020?"},
    {"role": "assistant", "content": "The LA Dodgers won in 2020."},
    {"role": "user", "content": "Where was it played?"}
  ]
)
print(response.choices[0].message.content)

[natbutter]

Okay.... it must not have been finding the correct base.
This works. Start with the api and set a key:

bash start_linux.sh --api  --api-key 1234

Then call from Python with:

from openai import OpenAI
llm = OpenAI(api_key='1234',base_url='http://127.0.0.1:5000/v1')
response = llm.chat.completions.create(
    model='deepseek-coder-33b-instruct.Q5_K_M.gguf',
    messages=[{"role": "user","content": "What is the latest iphone model?"}]
    )
print(response)