Vercel AI SDK kit

npm i ai @ai-sdk/openai-compatible

Through the gateway (recommended — cascade + cooldowns for free)

import { createOpenAICompatible } from "@ai-sdk/openai-compatible";
import { generateText, streamText } from "ai";

const modelhubby = createOpenAICompatible({
  name: "modelhubby",
  baseURL: "http://localhost:8787/v1",
  apiKey: "local",
});

const { text } = await generateText({
  model: modelhubby("auto"),
  prompt: "Summarize this in one line: ...",
});

// Streaming works — the gateway passes SSE through untouched
const stream = streamText({
  model: modelhubby("auto"),
  prompt: "Write a haiku",
});

Direct provider (no gateway) — Groq example

const groq = createOpenAICompatible({
  name: "groq",
  baseURL: "https://api.groq.com/openai/v1",
  apiKey: process.env.GROQ_API_KEY!,
});
await generateText({ model: groq("llama-3.3-70b-versatile"), prompt: "hi" });

Swap baseURL/model per the catalog (pnpm mh show <provider> --json); the fallback chain from pnpm mh recommend --snippet drops into a route handler unchanged.

Next.js route handler

// app/api/chat/route.ts
import { streamText } from "ai";

export async function POST(req: Request) {
  const { messages } = await req.json();
  const result = streamText({ model: modelhubby("auto"), messages });
  return result.toDataStreamResponse();
}