All Otoroshi LLM Extension Features

Try LLM extension with our Otoroshi Managed Instances

Read the documentation
The logo of the authify

Unified interface

  • Simplifies integration with multiple LLM providers
  • Reduces development and maintenance overhead
Learn More about Unified interface

Use multiple providers

  • Supports 10+ LLM providers, with more coming
  • Increases flexibility and choice
Learn More about Use multiple providers

Load balancing

  • Distributes workload across multiple LLM providers
  • Ensures optimal performance
Learn More about Load balancing

Fallbacks

  • Automatically switches LLMs during failures
  • Delivers uninterrupted and accurate performance
Learn More about Fallbacks

Automatic retries

  • Handles inexplicable LLM API failures
  • Rescues a substantial number of failed requests
Learn More about Automatic retries

Semantic cache

  • Speeds up repeated queries
  • Enhances response times
Learn More about Semantic cache

Custom quotas

  • Manage LLM token quotas per consumer
  • Optimize and control costs
Learn More about Custom quotas

Key vault

  • Securely store LLM API keys
  • Integrates with Otoroshi vault or other secret vaults
Learn More about Key vault

Observability and reporting

  • Audits every LLM request with detailed metadata
  • Exportable audit events for further reporting
Learn More about Observability and reporting

Fine grained authorizations

  • Advanced control over model usage
  • Constrains usage based on user identity, API key, metadata, etc.
Learn More about Fine grained authorizations

Prompt Fences

  • Validates prompts and responses
  • Prevents sensitive or personal information leakage
Learn More about Prompt Fences

Prompt engineering

  • Provides contextual information to prompts
  • Stores prompts in a reusable library
Learn More about Prompt engineering

Model Context Protocol (MCP)

  • Standardizes tool and resource exposure for LLMs
  • Enables interoperability between LLM providers and agents
Learn More about Model Context Protocol (MCP)

Guardrails

  • Enforces safety and compliance constraints
  • Reduces risk of inappropriate or harmful outputs
Learn More about Guardrails

Cost Tracking

  • Monitors resource usage and costs per request
  • Enables budgeting and optimization of LLM operations
Learn More about Cost Tracking

MCP Connectors

  • Facilitates integration with external systems
  • Expands LLM capabilities via third-party APIs
Learn More about MCP Connectors

MCP Server Exposition

  • Makes tools and features discoverable to LLMs and agents
  • Supports standardized tool calling via MCP
Learn More about MCP Server Exposition

Wasm Functions (Tool Calls)

  • Runs sandboxed WebAssembly modules securely
  • Enables cross-platform, language-agnostic tool execution
Learn More about Wasm Functions (Tool Calls)

HTTP Functions (Tool Calls)

  • Allows LLMs to call external HTTP APIs
  • Enables integration with web services and cloud functions
Learn More about HTTP Functions (Tool Calls)

Embedding Models

  • Generates vector representations for text
  • Enables semantic search and retrieval
Learn More about Embedding Models

Reporting

  • Generates detailed usage and performance reports
  • Supports compliance and auditing requirements
Learn More about Reporting

Ecological Impact LLM

  • Estimates and tracks the environmental footprint of LLM usage
  • Helps organizations meet sustainability goals
Learn More about Ecological Impact LLM

Personal Information Guardrail

  • Prevents leakage of personally identifiable information (PII)
  • Ensures compliance with privacy regulations
Learn More about Personal Information Guardrail

Auto Secrets Leakage Guardrail

  • Detects and prevents accidental sharing of secrets or credentials
  • Reduces risk of security breaches
Learn More about Auto Secrets Leakage Guardrail

Characters Count Guardrail

  • Enforces limits on input or output length
  • Prevents prompt or response overflow
Learn More about Characters Count Guardrail

Prompt Contains Guardrail

  • Checks for presence of restricted or required terms in prompts
  • Improves prompt quality and safety
Learn More about Prompt Contains Guardrail

Gender Bias Guardrail

  • Detects and mitigates gender bias in outputs
  • Promotes fairness and inclusivity
Learn More about Gender Bias Guardrail

Gibberish Guardrail

  • Detects and blocks nonsensical or incoherent outputs
  • Improves response quality
Learn More about Gibberish Guardrail

LLM Guardrail

  • Applies general safety and compliance checks to LLM interactions
  • Provides a framework for extensible guardrail enforcement
Learn More about LLM Guardrail

Language Moderation Guardrail

  • Blocks toxic, offensive, or inappropriate language
  • Ensures outputs are suitable for all audiences
Learn More about Language Moderation Guardrail

Secrets Leakage Guardrail

  • Prevents exposure of confidential or sensitive information
  • Protects organizational and user data
Learn More about Secrets Leakage Guardrail

Connect more than 10 providers

and access an infinite number of models

OpenAI

OpenAI

Add OpenAI to your Gateway
Anthropic

Anthropic

Add Anthropic to your Gateway
Google Gemini

Google Gemini

Add Google Gemini to your Gateway
Ollama

Ollama

Add Ollama to your Gateway
Mistral

Mistral

Add Mistral to your Gateway
Scaleway

Scaleway

Add Scaleway to your Gateway
Deepseek

Deepseek

Add Deepseek to your Gateway
Grok

Grok

Add Grok to your Gateway
Cloudflare AI

Cloudflare AI

Add Cloudflare AI to your Gateway
Groq

Groq

Add Groq to your Gateway
OVH Cloud AI Endpoints

OVH Cloud AI Endpoints

Add OVH Cloud AI Endpoints to your Gateway
Cohere

Cohere

Add Cohere to your Gateway
Azure AI Foundry

Azure AI Foundry

Add Azure AI Foundry to your Gateway
Tester maintenant avec nos instances managées