Small Models, Big Impact: Enterprise Use Cases for Lightweight LLMs

File uploads are also supported, but you need to handle it in the callback. Visit the Panel ChatFeed and ChatInterface documentation to learn more. Deploying this interface is straightforward: see the deployment docs for details.

Use Case: Escalate

In the previous examples, the focus was on free-form text. However, free-form output is difficult for downstream code to consume reliably, even with regular expressions (regex). For instance, LLMs can add additional comments or format the output differently than expected.

To address this, I’d like to introduce two libraries, instructor and pydantic, to transform LLM outputs into well-defined, structured models that can be used safely and consistently downstream.

import os
import openai
import instructor
from pydantic import BaseModel, Field

SYSTEM_PROMPT = """
Determine if the user message requires escalation. 
For example, users that need to change profile information can be handled by redirecting to self-service options,
but users that want to report an issue or cancel their subscription should be escalated to a human agent.
"""  # 1
USER_CONTENT = "I think my subscription cost is too high, can you help me with that?"  # 2

class EscalationResponse(BaseModel):  # 3
    escalate: bool = Field(
        description="Indicates whether the message requires escalation to a human agent."
    )
    reason: str = Field(
        description="A brief explanation of why the message does or does not require escalation."
    )

raw_client = openai.Client(
    base_url=os.getenv("AI_CATALYST_BASE_URL"), api_key=os.getenv("AI_CATALYST_API_KEY")
)
client = instructor.from_openai(raw_client, mode=instructor.Mode.JSON)
response = client.chat.completions.create(
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_CONTENT},  # 4
    ],
    model="ai_catalyst",
    response_model=EscalationResponse,  # 5
)

if response.escalate:  # 6
    print(f"User needs human agent. {response.reason!r}")
else:
    print(f"User does not need human agent. {response.reason!r}")

Here, we use print statements as placeholders. In production scenarios, you would redirect the user to a self-service page or trigger an alert for human intervention.

With this approach, you can easily adapt this pattern for other use cases like triaging or sentiment analysis.

Use Case: Extract

Structured outputs are also powerful for data extraction tasks. Imagine your sales team returned from a conference with lead information scattered across notes, napkins, and business cards, names, emails, phone numbers, and companies all jumbled together in an inconsistent format.

While you could upload photos of these materials directly, for this example we’ll use text content to demonstrate the extraction.

import os
import openai
import instructor
import pandas as pd
from pydantic import BaseModel, Field

SYSTEM_PROMPT = """
Extract contact information from sales lead notes.
Parse out names, emails, company names, and phone numbers even if the format is inconsistent.
"""

# Pretend user content (could also be extracted from photos of business cards/notes)
USER_CONTENT = """
john smith - Acme Corp [email protected] 111-1234
[email protected], TechStart Industries Phone: (222) 987-6543
[email protected] Mike Brown - 333.222.3333 Innovations LLC
Jane Doe | Global Solutions | [email protected] | tel: 4448881234
[email protected] Robert Wilson Enterprise Systems (555)-777-9999
"""

class Person(BaseModel):
    email: str = Field(description="Email address of the contact")
    name: str = Field(description="First and last name of the contact in title case")
    company: str = Field(description="Company or organization name")
    phone: str = Field(description="Phone number in (XXX) XXX-XXXX format")

class People(BaseModel):
    contacts: list[Person] = Field(description="List of extracted contacts")

raw_client = openai.Client(
    base_url=os.getenv("AI_CATALYST_BASE_URL"), 
    api_key=os.getenv("AI_CATALYST_API_KEY")
)
client = instructor.from_openai(raw_client, mode=instructor.Mode.JSON)

response = client.chat.completions.create(
    messages=[
        {"role": "system", "content": SYSTEM_PROMPT},
        {"role": "user", "content": USER_CONTENT},
    ],
    model="ai_catalyst",
    response_model=People,
)

# Convert to pandas DataFrame.
df = pd.DataFrame([contact.model_dump() for contact in response.contacts])

# Save to CSV.
df.to_csv("conference_leads.csv", index=False)
df

The result is a cleanly formatted DataFrame that you can write to CSV and upload to your CRM or sales system.