Generative Engine Optimization (GEO) & AI

Integrating LLM APIs with Next.js & Serverless Nodes: A Performance & Latency Guide

Streaming, token budgets, caching layers, timeouts and observability patterns for shipping LLM features on serverless and edge runtimes.

Thushara SanjeewaLead Systems Architect & Media Buying Director8 min read

Large language model features break the latency assumptions web engineering is built on. A database query returns in milliseconds; a model call returns a token stream over seconds. Wrapping that in a conventional request/response handler produces the pattern users hate most — a spinner with no progress, followed by a platform timeout. Meanwhile the infrastructure bill grows in proportion to tokens generated, not requests served.

The engineering objective is therefore twofold: make perceived latency low and make cost predictable. Time to first token (TTFT) is the metric that governs user perception; total tokens governs spend. These are optimised by different means — TTFT by streaming, colocation and prompt size discipline; cost by caching, model routing and output caps.

This guide covers the production patterns for shipping LLM features on Next.js and serverless or edge runtimes: streaming transport, runtime selection, timeout and abort handling, caching layers, structured output validation, rate limiting and observability. It is written for engineering leaders who need AI features that survive real traffic, not demos.

Stream Everything User-Facing

Any generation longer than roughly one sentence must stream. Streaming converts a multi-second wait into sub-second feedback and, critically, keeps the connection active so platform response timeouts do not terminate the request.

Implementation rules that matter in production:

  • Return the upstream stream directly. Do not buffer the full completion server-side before responding; buffering destroys the entire benefit.
  • Propagate cancellation. Wire the client AbortSignal through to the provider call so a user navigating away stops token billing immediately.
  • Emit heartbeats for slow first tokens. A comment frame or keep-alive event prevents intermediate proxies from closing an idle connection.
  • Send structured events, not raw text, when the UI needs status, tool-call progress or citations alongside content.
  • Never stream through a serverless function configured for buffered responses — verify the runtime supports chunked transfer end to end.

Runtime Selection

Edge runtimes give low network overhead and fast cold starts but restrict Node APIs and CPU time; Node serverless offers full APIs and longer execution but higher cold-start cost. The decisive factor is usually provider proximity: routing a request through an edge node geographically distant from the model provider adds a full round trip to TTFT. Colocate the function with the provider region and treat the edge as a proxy for cheap work only.

Persistent background jobs — batch summarisation, embedding backfills, long agent runs — belong on queues with worker processes, not in request handlers.

Cost and Latency Controls

  1. Cap output tokens explicitly. Unbounded max_tokens is the most common source of runaway spend and tail latency.
  2. Shrink prompts before upgrading models. Retrieval that returns three relevant chunks beats one that returns twenty; prompt size affects both TTFT and price.
  3. Route by task complexity. Classification, extraction and routing run on small fast models; only synthesis needs a frontier model. Model routing typically cuts spend by half or more.
  4. Exploit prompt caching. Keep static instructions and schemas at the head of the prompt and variable input at the tail, so provider-side prefix caching applies.
  5. Cache at the semantic layer. Hash normalised inputs for exact-match caching; for near-duplicates, use an embedding similarity threshold with a strict cutoff to avoid serving wrong answers.

Reliability Patterns

  • Timeouts per attempt, not per request, with jittered retries limited to idempotent calls.
  • Fallback provider or model on 429 and 5xx, behind a circuit breaker so a degraded provider does not consume your entire concurrency budget.
  • Idempotency keys on any generation that triggers a side effect such as an email or a database write.
  • Validated structured output. Request JSON schema mode, then parse with a runtime validator; on failure, retry once with the validation error appended rather than shipping malformed data downstream.

Architecture Options Compared

PatternTTFTMax durationCost profileBest for
Edge streaming handlerLowest when colocatedShort to mediumLow compute, token-dominatedChat, inline assistance
Node serverless streamingLowMediumModerateTool use, DB-backed RAG
Queue plus workerNot applicableHoursCheapest per unit workBatch, embeddings, agents
Synchronous buffered APIWorstPlatform limitSame tokens, worse UXMachine-to-machine only
Client-direct to providerVery lowProvider limitKey exposure riskNot recommended

Implementation Blueprint

A streaming handler with abort propagation, output caps and cache-key hashing:

ts
// src/routes/api/public/assist.ts style handler
const encoder = new TextEncoder();

export async function POST(request: Request) {
  const { prompt } = (await request.json()) as { prompt: string };

  if (typeof prompt !== "string" || prompt.length > 4000) {
    return new Response(JSON.stringify({ error: "invalid_prompt" }), { status: 400 });
  }

  const controller = new AbortController();
  request.signal.addEventListener("abort", () => controller.abort());

  const upstream = await fetch("https://ai.gateway.lovable.dev/v1/chat/completions", {
    method: "POST",
    signal: controller.signal,
    headers: {
      "Content-Type": "application/json",
      Authorization: `Bearer ${process.env["LOVABLE_API_KEY"]}`,
    },
    body: JSON.stringify({
      model: "google/gemini-2.5-flash",
      stream: true,
      max_tokens: 700,
      messages: [
        { role: "system", content: "You are a concise technical assistant." },
        { role: "user", content: prompt },
      ],
    }),
  });

  if (!upstream.ok || !upstream.body) {
    return new Response(JSON.stringify({ error: "upstream_unavailable" }), { status: 502 });
  }

  const stream = new ReadableStream({
    async start(sink) {
      const reader = upstream.body!.getReader();
      sink.enqueue(encoder.encode(": open\n\n")); // heartbeat opens the pipe
      try {
        while (true) {
          const { done, value } = await reader.read();
          if (done) break;
          sink.enqueue(value);
        }
      } finally {
        sink.close();
      }
    },
    cancel() {
      controller.abort();
    },
  });

  return new Response(stream, {
    headers: {
      "Content-Type": "text/event-stream",
      "Cache-Control": "no-cache, no-transform",
      Connection: "keep-alive",
    },
  });
}

Log per request: model, prompt tokens, completion tokens, TTFT, total duration, cache outcome and retry count. Without token-level telemetry, cost regressions are invisible until invoicing.

Strategic Action Plan: 90 Days

  1. Days 1–10 — Baseline the pipeline. Instrument TTFT, total latency, token counts and error classes per feature. Identify which calls are user-facing and which can move to queues.
  2. Days 11–30 — Convert to streaming. Replace buffered handlers with streamed responses, propagate abort signals, and add heartbeats. Verify no proxy in the path buffers responses.
  3. Days 31–50 — Install cost controls. Cap output tokens, route simple tasks to small models, restructure prompts for prefix caching, and add exact-match response caching.
  4. Days 51–70 — Harden reliability. Add per-attempt timeouts, jittered retries, a fallback model behind a circuit breaker, schema-validated structured output, and per-user rate limits.
  5. Days 71–90 — Operationalise. Publish a cost-per-feature dashboard, set alerts on TTFT p95 and spend per thousand requests, and run a quarterly model-and-price review.

Frequently Asked Questions

Why do our LLM API calls time out on serverless platforms?

Because a buffered handler holds the connection idle while the model generates, and platform response timeouts or intermediate proxies close it. Streaming the response keeps data flowing, which both satisfies the timeout and improves perceived latency. Work that genuinely exceeds request limits — long agent runs or batch jobs — belongs on a queue with a worker and a status endpoint.

How do we reduce time to first token?

Colocate the function with the provider region, keep the prompt short by retrieving fewer and better context chunks, avoid pre-call work such as sequential database reads, reuse HTTP connections, and select a lower-latency model for interactive paths. Streaming does not reduce TTFT itself, but it makes the remainder of generation invisible to the user.

Is semantic caching safe for production AI features?

It is safe with strict controls: normalise inputs, require a high embedding similarity threshold, scope cache keys per tenant and per prompt version, and exclude any request containing personal or transactional data. Invalidate the cache whenever the system prompt, model or retrieval corpus changes, and log cache hits so incorrect answers can be traced to a specific entry.

Should the browser call the model provider directly?

No for production. Direct calls either expose a provider key or require short-lived tokens plus client-side rate limiting, and they remove your ability to enforce prompt policy, apply caching, validate output and record cost telemetry. Proxy through your own streaming endpoint so authorisation, quotas and observability stay server-side.