LangChain kit

ChatOpenAI accepts a custom base URL — both patterns from the OpenAI SDK kit apply.

JS/TS

npm i @langchain/openai @langchain/core

import { ChatOpenAI } from "@langchain/openai";

// Through the gateway: cascade + cooldowns included
const llm = new ChatOpenAI({
  model: "auto",
  apiKey: "local",
  configuration: { baseURL: "http://localhost:8787/v1" },
});

const reply = await llm.invoke("One-line summary of CRDTs");

Python

pip install langchain-openai

from langchain_openai import ChatOpenAI

llm = ChatOpenAI(
    model="auto",
    api_key="local",
    base_url="http://localhost:8787/v1",
)
print(llm.invoke("One-line summary of CRDTs").content)

Direct-provider fallback without the gateway

LangChain has .with_fallbacks() — pair it with the catalog's ranked routes:

groq = ChatOpenAI(base_url="https://api.groq.com/openai/v1",
                  api_key=os.environ["GROQ_API_KEY"],
                  model="llama-3.3-70b-versatile")
gemini = ChatOpenAI(base_url="https://generativelanguage.googleapis.com/v1beta/openai",
                    api_key=os.environ["GEMINI_API_KEY"],
                    model="gemini-2.5-flash")
llm = groq.with_fallbacks([gemini])

The gateway remains the better default: LangChain's fallbacks don't track key state, so a rate-limited key gets retried every call; ModelHubby's pool cools it down once.

Gotchas

• Agents/tool-calling: verify tool_calling per provider in the catalog ("unknown" is common) before building agent loops on a free rung.
• Streaming through the gateway works with llm.stream(...) — SSE is passed through.