sem_extract operator fails with Ollama/Phi-3 model due to LiteLLM transformation error

[sekharmalla]

sem_extract operator fails with Ollama/Phi-3 model due to LiteLLM transformation error

Description

When trying to use the sem_extract operator with an Ollama-hosted Phi-3 model, the operation fails with a KeyError in LiteLLM's response transformation layer. This appears to be due to assumptions about function calling capabilities that may not be supported by all models.

Environment

• LOTUS version: latest
• Python version: 3.10
• LiteLLM version: latest
• Model: phi3:3.8b-mini-128k-instruct-q3_K_M via Ollama
• OS: macOS

Steps to Reproduce

import pandas as pd
from datetime import datetime
import lotus
from lotus.models import LM

# Configure LOTUS with Phi-3 through Ollama
lm = LM(
    model="ollama/phi3:3.8b-mini-128k-instruct-q3_K_M",
    api_base="http://localhost:11434",
    max_ctx_len=2048,
    max_tokens=500
)

# Configure LOTUS settings
lotus.settings.configure(lm=lm)

# Create sample dataset
data = {
    'Id': [1, 2, 3, 4, 5],
    'Name': ['Alice Johnson', 'Bob Smith', 'Charlie Brown', 'Diana Prince', 'Eve Torres'],
    'DOB': ['1990-05-10', '1985-03-15', '1992-07-20', '1988-11-25', '1995-12-30'],
    'Country': ['USA', 'Canada', 'India', 'UK', 'India'],
    'Salary': [80000, 70000, 50000, 90000, 40000]
}

employees = pd.DataFrame(data)
employees['Age'] = employees['DOB'].apply(lambda x: datetime.now().year - int(x.split('-')[0]))

# Attempt sem_extract operation
input_cols = ["Name", "Country", "Salary", "Age"]
output_cols = {
    "salary_bracket": "The salary bracket (Low: <50K, Medium: 50-75K, High: >75K)",
    "experience_level": "Inferred experience level based on age (Junior: <30, Mid: 30-40, Senior: >40)",
    "location_type": "Whether the employee is domestic (USA) or international"
}

extracted_df = employees.sem_extract(
    input_cols=input_cols,
    output_cols=output_cols,
    extract_quotes=True
)

Error Message

Traceback (most recent call last):
  File "/Users/m/Documents/mlx/lib/python3.10/site-packages/litellm/main.py", line 2648, in completion
    response = base_llm_http_handler.completion(
  [...]
KeyError: 'name'

litellm.exceptions.APIConnectionError: litellm.APIConnectionError: 'name'

[StanChan03]

Hi @sekharmalla Can you specify which Ollama version you have. It is required to have an Ollama version >= 0.5.0

[sekharmalla]

Hello @StanChan03 thanks for the reply, I am using Ollama 0.5.4.

[sidjha1]

Hey @sekharmalla I opened an issue with the LiteLLM folks. I was able to get your example to work with a one line change to the LiteLLM codebase so lets see what the maintainers say.

[sekharmalla]

@sidjha1 thanks so much. Appreciate for the support. My and my colleague working on building a search engine for a sample database using lotus. When we test given examples they are working very well but when working with actual database as data frame we are not seeing expected results. Would really appreciate if provided any resources/Documentation other than the ones we have or if possible for a google meet where we can explain our business use case. Thanks for the hard work the team has been doing.

[liana313]

Hi @sekharmalla, happy to find sometime to help with this -- do you mind joining our slack here (https://join.slack.com/t/lotus-fnm8919/shared_invite/zt-2tnq6948j-juGuSIR0__fsh~kUmZ6TJw) and sending me a message so we can coordinate there