Provider Setup Guide

Detailed setup instructions for each provider supported by OneLLM.

OpenAI
Anthropic
Google AI Studio
Mistral
Groq
Together AI
Fireworks
Anyscale
X.AI
Perplexity
DeepSeek
Cohere
OpenRouter
Azure OpenAI
AWS Bedrock
Google Vertex AI
Ollama
llama.cpp

OpenAI

1. Get API Key

Go to platform.openai.com
Sign up or log in
Navigate to API Keys
Create new secret key

2. Set Environment Variable

export OPENAI_API_KEY="sk-..."

3. Test Connection

from onellm import OpenAI

client = OpenAI()
response = client.chat.completions.create(
    model="openai/gpt-4o-mini",
    messages=[{"role": "user", "content": "Hello!"}]
)
print(response.choices[0].message['content'])

Available Models

openai/gpt-4o - Most capable
openai/gpt-4o-mini - Faster, cheaper
openai/gpt-4-turbo - Previous generation
openai/gpt-3.5-turbo - Legacy, fast

Anthropic

1. Get API Key

Go to console.anthropic.com
Sign up for access
Navigate to API Keys
Generate key

2. Set Environment Variable

export ANTHROPIC_API_KEY="sk-ant-..."

3. Test Connection

response = client.chat.completions.create(
    model="anthropic/claude-3-5-sonnet-20241022",
    messages=[{"role": "user", "content": "Hello!"}]
)

Available Models

anthropic/claude-3-5-sonnet-20241022 - Latest, most capable
anthropic/claude-3-opus-20240229 - Powerful, expensive
anthropic/claude-3-sonnet-20240229 - Balanced
anthropic/claude-3-haiku-20240307 - Fast, cheap

Google AI Studio

1. Get API Key

Go to makersuite.google.com
Sign in with Google account
Get API key from settings

2. Set Environment Variable

export GOOGLE_API_KEY="..."

3. Test Connection

response = client.chat.completions.create(
    model="google/gemini-1.5-flash",
    messages=[{"role": "user", "content": "Hello!"}]
)

Available Models

google/gemini-1.5-pro - Most capable, 1M context
google/gemini-1.5-flash - Fast, efficient
google/gemini-pro - Previous generation

Mistral

1. Get API Key

Go to console.mistral.ai
Create account
Navigate to API Keys
Create new key

2. Set Environment Variable

export MISTRAL_API_KEY="..."

3. Test Connection

response = client.chat.completions.create(
    model="mistral/mistral-small-latest",
    messages=[{"role": "user", "content": "Hello!"}]
)

Available Models

mistral/mistral-large-latest - Most capable
mistral/mistral-medium-latest - Balanced
mistral/mistral-small-latest - Fast
mistral/mistral-tiny - Fastest

Groq

1. Get API Key

Go to console.groq.com
Sign up for access
Get API key from dashboard

2. Set Environment Variable

export GROQ_API_KEY="gsk_..."

3. Test Connection

response = client.chat.completions.create(
    model="groq/llama3-70b-8192",
    messages=[{"role": "user", "content": "Hello!"}]
)

Available Models

groq/llama3-70b-8192 - Llama 3 70B
groq/llama3-8b-8192 - Llama 3 8B
groq/mixtral-8x7b-32768 - Mixtral MoE
groq/gemma-7b-it - Google Gemma

Together AI

1. Get API Key

Go to api.together.xyz
Sign up
Get API key from account

2. Set Environment Variable

export TOGETHER_API_KEY="..."

3. Test Connection

response = client.chat.completions.create(
    model="together/meta-llama/Meta-Llama-3.1-70B-Instruct-Turbo",
    messages=[{"role": "user", "content": "Hello!"}]
)

Available Models

Various Llama models
Mistral models
CodeLlama models
50+ open source models

Fireworks

1. Get API Key

Go to app.fireworks.ai
Create account
Get API key

2. Set Environment Variable

export FIREWORKS_API_KEY="..."

3. Test Connection

response = client.chat.completions.create(
    model="fireworks/accounts/fireworks/models/llama-v3p1-70b-instruct",
    messages=[{"role": "user", "content": "Hello!"}]
)

Anyscale

1. Get API Key

Go to anyscale.com
Sign up for Endpoints
Get API key (starts with esecret_)

2. Set Environment Variable

export ANYSCALE_API_KEY="esecret_..."

3. Test Connection

response = client.chat.completions.create(
    model="anyscale/llama3-70b",
    messages=[{"role": "user", "content": "Hello!"}]
)

X.AI

1. Get API Key

Go to x.ai
Request access
Get API key when approved

2. Set Environment Variable

export XAI_API_KEY="..."

3. Test Connection

response = client.chat.completions.create(
    model="xai/grok-2-latest",
    messages=[{"role": "user", "content": "Hello!"}]
)

Perplexity

1. Get API Key

Go to perplexity.ai/settings/api
Sign up
Generate API key

2. Set Environment Variable

export PERPLEXITY_API_KEY="pplx-..."

3. Test Connection

response = client.chat.completions.create(
    model="perplexity/llama-3.1-sonar-small-128k-online",
    messages=[{"role": "user", "content": "What's happening today?"}]
)

DeepSeek

1. Get API Key

Go to platform.deepseek.com
Register account
Get API key

2. Set Environment Variable

export DEEPSEEK_API_KEY="..."

3. Test Connection

response = client.chat.completions.create(
    model="deepseek/deepseek-chat",
    messages=[{"role": "user", "content": "你好!"}]
)

Cohere

1. Get API Key

Go to dashboard.cohere.com
Sign up
Get API key

2. Set Environment Variable

export COHERE_API_KEY="..."

3. Test Connection

response = client.chat.completions.create(
    model="cohere/command-r-plus",
    messages=[{"role": "user", "content": "Hello!"}]
)

OpenRouter

1. Get API Key

Go to openrouter.ai
Sign up
Get API key from dashboard

2. Set Environment Variable

export OPENROUTER_API_KEY="sk-or-..."

3. Test Connection

# Access 100+ models
response = client.chat.completions.create(
    model="openrouter/meta-llama/llama-3.2-3b-instruct:free",
    messages=[{"role": "user", "content": "Hello!"}]
)

Azure OpenAI

1. Setup Azure Resources

Create Azure account
Create OpenAI resource
Deploy models
Get endpoint and key

2. Create Configuration File

Create azure.json:

{
    "endpoint": "https://your-resource.openai.azure.com",
    "api_key": "your-key",
    "api_version": "2024-02-01",
    "deployments": {
        "gpt-4": "your-gpt4-deployment-name",
        "gpt-35-turbo": "your-gpt35-deployment-name"
    }
}

3. Test Connection

client = OpenAI(azure_config_path="azure.json")
response = client.chat.completions.create(
    model="azure/gpt-4",
    messages=[{"role": "user", "content": "Hello!"}]
)

AWS Bedrock

1. Setup AWS

Create AWS account
Enable Bedrock models
Configure IAM permissions

2. Create Configuration File

Create bedrock.json:

{
    "region": "us-east-1",
    "aws_access_key_id": "AKIA...",
    "aws_secret_access_key": "..."
}

3. Test Connection

response = client.chat.completions.create(
    model="bedrock/claude-3-sonnet",
    messages=[{"role": "user", "content": "Hello!"}]
)

Google Vertex AI

1. Setup GCP

Create GCP project
Enable Vertex AI API
Create service account
Download credentials JSON

2. Set Environment Variable

export GOOGLE_APPLICATION_CREDENTIALS="path/to/service-account.json"

3. Test Connection

response = client.chat.completions.create(
    model="vertexai/gemini-1.5-flash",
    messages=[{"role": "user", "content": "Hello!"}]
)

Ollama

1. Install Ollama

# macOS/Linux
curl -fsSL https://ollama.com/install.sh | sh

# Windows
# Download from ollama.com

2. Pull Models

ollama pull llama3
ollama pull mistral

3. Test Connection

# Default localhost
response = client.chat.completions.create(
    model="ollama/llama3",
    messages=[{"role": "user", "content": "Hello!"}]
)

# Custom endpoint
response = client.chat.completions.create(
    model="ollama/llama3@192.168.1.100:11434",
    messages=[{"role": "user", "content": "Hello!"}]
)

llama.cpp

1. Install Dependencies

pip install llama-cpp-python

2. Download Models

# Using OneLLM's utility
onellm download llama-3-8b

# Or manually download GGUF files
wget https://huggingface.co/model.gguf

3. Set Model Directory

export LLAMA_CPP_MODEL_DIR="/path/to/models"

4. Test Connection

response = client.chat.completions.create(
    model="llama_cpp/llama-3-8b-instruct.gguf",
    messages=[{"role": "user", "content": "Hello!"}]
)

GPU Acceleration

# Configure GPU layers
client = OpenAI(
    llama_cpp_config={
        "n_gpu_layers": 35,  # Number of layers on GPU
        "n_ctx": 4096        # Context window
    }
)

Troubleshooting

API Key Issues

Ensure key is set correctly
Check for extra spaces/quotes
Verify key hasn’t expired
Try regenerating key

Connection Issues

Check internet connection
Verify firewall settings
Try different regions (if applicable)
Check service status pages

Model Access

Ensure model is available in your region
Check if model requires special access
Verify billing is set up
Check quota limits

Next Steps

Provider Capabilities - Compare features
Examples - Provider examples
Configuration - Advanced config

Provider Setup Guide

Table of Contents

OpenAI

1. Get API Key

2. Set Environment Variable

3. Test Connection

Available Models

Anthropic

1. Get API Key

2. Set Environment Variable

3. Test Connection

Available Models

Google AI Studio

1. Get API Key

2. Set Environment Variable

3. Test Connection

Available Models

Mistral

1. Get API Key

2. Set Environment Variable

3. Test Connection

Available Models

Groq

1. Get API Key

2. Set Environment Variable

3. Test Connection

Available Models

Together AI

1. Get API Key

2. Set Environment Variable

3. Test Connection

Available Models

Fireworks

1. Get API Key

2. Set Environment Variable

3. Test Connection

Anyscale

1. Get API Key

2. Set Environment Variable

3. Test Connection

X.AI

1. Get API Key

2. Set Environment Variable

3. Test Connection

Perplexity

1. Get API Key

2. Set Environment Variable

3. Test Connection

DeepSeek

1. Get API Key

2. Set Environment Variable

3. Test Connection

Cohere

1. Get API Key

2. Set Environment Variable

3. Test Connection

OpenRouter

1. Get API Key

2. Set Environment Variable

3. Test Connection

Azure OpenAI

1. Setup Azure Resources

2. Create Configuration File

3. Test Connection

AWS Bedrock

1. Setup AWS

2. Create Configuration File

3. Test Connection

Google Vertex AI

1. Setup GCP

2. Set Environment Variable

3. Test Connection

Ollama

1. Install Ollama

2. Pull Models

3. Test Connection

llama.cpp

1. Install Dependencies

2. Download Models

3. Set Model Directory