The hype

When OpenAI released ChatGPT, every startup rushed to put "serverless" on their pitch decks. The promise is simple: upload a model, write a function, and let the cloud provider auto‑scale to zero. No ops, no capacity planning, pure pay‑as‑you‑go. On X it reads like a meme: "Deploy a 2‑billion‑parameter model on Lambda and never worry about servers again." The reality is messier.

When serverless works for AI inference

Serverless shines when the workload is bursty, cheap, and latency tolerant. Typical use‑cases:

Tiny models (< 50 MB) for text classification, sentiment analysis, or image thumbnail tagging.

Event‑driven inference – e.g., a new image lands in an S3 bucket, a function classifies it, and stores the result.

Proof‑of‑concepts – spin up a demo in minutes without provisioning EC2 or GPUs.

Because the function runs in a sandbox, you get built‑in security isolation and automatic versioning. Providers like AWS Lambda now offer up to 10 GB of memory and 15 minutes of execution time, enough for many CPU‑only models.

Example: a Lambda that runs a tiny BERT for sentiment

python
import json, os
from transformers import pipeline

Load model once, cold start penalty only on first request

sentiment = pipeline("sentiment-analysis", model="distilbert-base-uncased-finetuned-sst-2-english")

def handler(event, context):
body = json.loads(event["body"])
text = body.get("text", "")
result = sentiment(text)[0]
return {
"statusCode": 200,
"body": json.dumps({"label": result["label"], "score": result["score"]})
}

Deploy with the AWS SAM CLI, set the memory to 4 GB, and you’ll see cold start latency around 2‑3 seconds – acceptable for a chat widget but not for real‑time voice assistants. If you enable provisioned concurrency at 10 instances, the latency drops to sub‑500 ms, but you now pay roughly $0.07 per hour per instance for warm containers.

When it blows up

Serverless fails when any of these conditions appear:

Large models – Anything over a few hundred megabytes exceeds Lambda’s /tmp storage or memory limits. Even if you use container images (up to 10 GB), cold starts become minutes‑long.

GPU requirements – Most providers only offer CPU for FaaS. Inference on a 7B LLM needs a GPU, and the cost of a GPU‑backed serverless offering (e.g., AWS Lambda with Graviton2) is still higher than a reserved instance for sustained load.

Strict latency – A cold start of 2 seconds is fine for batch jobs, but not for sub‑100 ms SLAs in gaming or finance.

High QPS – Serverless pricing is per‑invocation + duration. At 10 k RPS, the per‑invocation cost dwarfs the compute cost, and you quickly hit concurrency limits unless you request a limit increase.

Vendor lock‑in – Each platform has its own packaging, environment variables, and IAM quirks. Porting a serverless AI pipeline from AWS to GCP or Azure is non‑trivial.

Cold start nightmare

Even with provisioned concurrency, you pay a premium for keeping "warm" containers. For a 4‑GB Lambda, provisioned concurrency at 100 instances costs roughly $0.07 per hour per instance. At scale, that can outpace a modest EC2 spot instance, turning what looked like a cheap solution into a hidden expense.

Real‑world stories

Netflix moved its thumbnail generation to Lambda and saved 30% of its compute budget, because each thumbnail is a 2 MB image processed in < 200 ms.

Scale AI tried to serve a 1.2 GB object detection model from Cloud Functions and hit a 512 MB memory ceiling. They fell back to a Kubernetes pod with a GPU, adding 2 minutes of dev time but achieving sub‑50 ms latency.

GitHub Copilot runs inference on a fleet of dedicated GPU servers, not on serverless. The team explicitly said serverless was "not an option for the latency and cost profile we need".

Alternatives & patterns

If you love serverless ergonomics but need more horsepower, consider hybrid patterns:

Async queue + container – Push requests to SQS, have a Fargate task pull, run the model, and push results back. You keep auto‑scale without cold starts.

Edge functions – Cloudflare Workers or Fastly Compute@Edge can run tiny models at the edge, cutting round‑trip latency to < 10 ms for static inference.

Knative on a managed K8s – Gives you "serverless" scaling (scale‑to‑zero) with the ability to attach GPUs to pods when needed.

Model as a Service (MaaS) – Use hosted inference APIs (OpenAI, Cohere, etc.) for large models, and reserve serverless for glue code.

Bottom line

Serverless is not a universal silver bullet for AI inference. It excels for small, bursty, and cost‑sensitive workloads, but it crumbles under large models, strict latency, or massive QPS. The smart move in 2024 is to match the right tool to the right slice of your pipeline: serverless for preprocessing, edge for ultra‑low latency, containers or dedicated GPU servers for the heavy lifting.

Hot take: If you’re still betting on "run any model on Lambda" as a long‑term strategy, you’re ignoring the economics of cold starts and the hard limits on memory and GPUs. Embrace a polyglot architecture now, or you’ll be paying for "serverless" while secretly running a private EC2 farm behind the scenes.