Connecting AnythingLLM to an OpenAI-Compatible Multi-Model Gateway

Description and use

Configuring AnythingLLM to use the CometAPI gateway centralizes model management, billing, and interfaces without altering existing workflows.

Problem

Integrating a multi-model gateway into AnythingLLM without altering existing workflows or code.

Outcome

By configuring AnythingLLM’s Generic OpenAI provider to point at CometAPI, users can route all chat, embedding, and agent calls through a single API endpoint, simplifying model management, billing, and experimentation without modifying existing workflows.

Workflow

• 1. Verify gateway accessibility with curl or Python to ensure API key, network, and model name are correct.
• 2. In AnythingLLM, navigate to Settings → AI Providers → LLM Preferences.
• 3. Add a new provider: select "Generic OpenAI" and set Base URL to "https://api.cometapi.com/v1", API key to your secret, and model name to one published by the gateway (e.g., gpt-4o).
• 4. Configure token context window and max tokens to match the chosen model’s limits.
• 5. Save the provider and start a new chat or workspace.
• 6. Select the Generic OpenAI provider and the desired model, then send a test prompt to confirm response.
• 7. For RAG or agent workflows, ensure the same provider is selected; no code changes are needed.
• 8. Monitor usage, adjust token limits, and enable streaming or disable it based on provider support.
• 9. Resolve any UI/.env conflicts by checking /app/server/.env for overridden values.

Evidence

AnythingLLM already supports a Generic OpenAI provider, so it can connect to any service that exposes an OpenAI-style API.

Source

The useful part is not the UI integration itself—it is the ability to change models without reworking the AnythingLLM setup.

Source