A model that works in a notebook is not a service yet. To deploy an AI API to a server, you need something that loads the model once, answers HTTP requests, restarts when it crashes, and sits behind a proper web server with HTTPS. This guide walks through that setup on Ubuntu 26.04 using Python, FastAPI, Uvicorn, systemd, and Nginx.
This guide also covers the problems that only show up with machine learning models, such as workers that each load their own copy of the model, timeouts on slow inference, and streamed replies that arrive all at once.
Quick Outline
- What You Need to Deploy an AI API on Ubuntu
- Can You Deploy an AI API for Free?
- Deploying an AI API with Python and Nginx on Ubuntu 26.04
- When Should You Deploy an AI API on Kubernetes?
- How to Troubleshoot Common AI API Deployment Errors
- Conclusion
- FAQs
What You Need to Deploy an AI API on Ubuntu
Requests come in from the internet to Nginx on ports 80 and 443. Nginx handles HTTPS and rate limiting, then forwards each request to your Python app listening only on 127.0.0.1:8000. The app keeps the model in memory and returns predictions. Because the app never faces the internet directly, nobody can bypass your TLS or rate limits.
You will need an Ubuntu server with sudo access, a domain with an A record pointing to it, and enough RAM for your model. For a small transformer model, plan on at least 2 GB. Our guide on how much RAM and CPU AI models need covers sizing in more detail.
Can You Deploy an AI API for Free?
Yes, within limits. The cheapest route is to deploy the AI locally on your own computer while you build, then move to a server when others need to use it. A free-tier cloud VM can host the finished service. Oracle’s Always Free Arm instances are the usual choice, but allowances and capacity change, so read the current terms.
Check that your libraries publish Arm builds, and expect 1 GB servers to be too small for most models. Free hosting suits a small classifier or a demo, not a large language model. You can explore our guide, How to Self-Host AI Applications on a Linux VPS.
Deploying an AI API with Python and Nginx on Ubuntu 26.04
To deploy an AI API with Python and Nginx on Ubuntu, build the API using a Python framework such as FastAPI, run it with a production server like Uvicorn, and configure it as a systemd service. Configure Nginx as a reverse proxy to forward HTTP requests to the API, enable HTTPS with an SSL certificate, and secure the server for production use.
Step 1: Prepare Ubuntu for Python AI API Deployment
Update the system and install Python tooling and Nginx:
sudo apt install -y python3 python3-venv python3-pip nginx |
|---|
Ubuntu blocks system-wide pip install with an externally managed environment error, which is intentional. Use a virtual environment for everything, and give the app its own unprivileged user:
sudo useradd –system –home /opt/aiapi –shell /usr/sbin/nologin aiapi sudo mkdir -p /opt/aiapi && sudo chown aiapi:aiapi /opt/aiapi sudo -u aiapi python3 -m venv /opt/aiapi/venv |
|---|
Install the packages. On a server without a GPU, use the CPU build of PyTorch.
sudo -u aiapi /opt/aiapi/venv/bin/pip install fastapi “uvicorn[standard]” transformers |
|---|
The default Linux wheel pulls in several gigabytes of CUDA libraries that a CPU server never uses, and that alone can fill a small disk:
sudo -u aiapi /opt/aiapi/venv/bin/pip install torch –index-url https://download.pytorch.org/whl/cpu |
|---|
Step 2: Build an AI API with Python and FastAPI
This example serves a small sentiment model with FastAPI. Let’s build an AI API with Python and FastAPI.
Load the Machine Learning Model at Startup
Create /opt/aiapi/main.py:
import os import secrets from contextlib import asynccontextmanager from fastapi import Depends, FastAPI, Header, HTTPException from pydantic import BaseModel, Field from transformers import pipeline models = {} @asynccontextmanager async def lifespan(app: FastAPI): models[“clf”] = pipeline( “sentiment-analysis”, model=”distilbert/distilbert-base-uncased-finetuned-sst-2-english”, ) yield models.clear() app = FastAPI(title=”AI API”, lifespan=lifespan) API_KEY = os.environ.get(“API_KEY”, “”) def check_key(x_api_key: str = Header(default=””)): if not API_KEY or not secrets.compare_digest(x_api_key.encode(), API_KEY.encode()): raise HTTPException(status_code=401, detail=”Invalid API key”) class TextIn(BaseModel): text: str = Field(min_length=1, max_length=2000) @app.get(“/health”) def health(): return {“status”: “ok”} @app.post(“/predict”, dependencies=[Depends(check_key)]) def predict(body: TextIn): out = models[“clf”](body.text)[0] return {“label”: out[“label”], “score”: round(out[“score”], 4)} |
|---|
Three choices here matter for any service that serves an AI model with FastAPI. The model loads once at startup inside lifespan, never inside the request handler. The endpoint is a plain def, not async def, because FastAPI runs plain functions in a thread pool, while a blocking model call inside an async function freezes the whole event loop. And the endpoint checks an API key, since an open inference endpoint invites abuse and a surprise bill.
Create and Test the Prediction Endpoint
Run it once by hand to download the model and confirm it works:
cd /opt/aiapi sudo -u aiapi env API_KEY=test-key /opt/aiapi/venv/bin/uvicorn main:app –host 127.0.0.1 –port 8000 |
|---|
In a second terminal, test it:
curl -s http://127.0.0.1:8000/health curl -s -X POST http://127.0.0.1:8000/predict \ -H “x-api-key: test-key” -H “Content-Type: application/json” \ -d ‘{“text”:”This deployment was easier than I expected”}’ |
|---|
Stop the server with Ctrl+C. Downloading the model now, instead of on first service start, avoids a confusing timeout later.
Step 3: Run the Python AI API as a systemd Service
Store the real API key in a root-owned file:
echo “API_KEY=$(openssl rand -hex 32)” | sudo tee /etc/aiapi.env sudo chmod 600 /etc/aiapi.env |
|---|
Create /etc/systemd/system/aiapi.service that configure Linux server network settings:
[Unit] Description=AI API (FastAPI with Uvicorn) After=network.target [Service] User=aiapi Group=aiapi WorkingDirectory=/opt/aiapi EnvironmentFile=/etc/aiapi.env ExecStart=/opt/aiapi/venv/bin/uvicorn main:app –host 127.0.0.1 –port 8000 –workers 1 Restart=always RestartSec=5 NoNewPrivileges=true PrivateTmp=true [Install] WantedBy=multi-user.target |
|---|
Enable and start it:
sudo systemctl daemon-reload sudo systemctl enable –now aiapi |
|---|
Why Should You Use One Worker for a Machine Learning API?
Every worker process loads its own full copy of the model. Four workers with a 1.5 GB model need roughly 6 GB before the operating system takes anything. Start with one worker, measure memory and latency, and add more only if RAM allows. Heavy models rarely get faster with more processes anyway, because PyTorch already uses multiple CPU cores for a single prediction.
Step 4: Configure Nginx as a Reverse Proxy for Your AI API
You installed Nginx in Step 1. Confirm it is running and open the firewall for web traffic and SSH:
sudo systemctl status nginx sudo ufw allow OpenSSH sudo ufw allow ‘Nginx Full’ sudo ufw enable |
|---|
Do not open port 8000. The app listens on localhost only. Now create the reverse proxy at /etc/nginx/sites-available/aiapi:
limit_req_zone $binary_remote_addr zone=aiapi:10m rate=5r/s; server { listen 80; server_name api.example.com; client_max_body_size 20m; location / { limit_req zone=aiapi burst=10 nodelay; proxy_pass http://127.0.0.1:8000; proxy_http_version 1.1; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Forwarded-Proto $scheme; proxy_connect_timeout 5s; proxy_read_timeout 120s; proxy_send_timeout 120s; } } |
|---|
Enable the site and reload:
sudo ln -s /etc/nginx/sites-available/aiapi /etc/nginx/sites-enabled/ sudo nginx -t && sudo systemctl reload nginx |
|---|
Three settings in that file exist because of AI workloads. proxy_read_timeout defaults to 60 seconds, and a slow model can exceed that, which produces a 504 error. client_max_body_size defaults to 1 MB, so image or audio uploads fail with a 413 error until you raise it. And limit_req caps how often one client can hit an expensive endpoint.
Streaming Replies
If your API streams tokens, Nginx will buffer the whole response by default and deliver it at the end. Add a separate location for the streaming route:
location /stream { proxy_pass http://127.0.0.1:8000; proxy_http_version 1.1; proxy_buffering off; proxy_read_timeout 300s; } |
|---|
You can also send the response header X-Accel-Buffering: no from FastAPI’s StreamingResponse, and Nginx will honor it for that response.
Step 5: Enable HTTPS for Your Python AI API with Nginx
Install Certbot and request a certificate:
sudo apt install -y certbot python3-certbot-nginx |
|---|
Now, request a certificate.
sudo certbot –nginx -d api.example.com |
|---|
Certbot edits the Nginx config to serve HTTPS and redirect HTTP, and the dry run confirms automatic renewal works. Test the live endpoint:
curl -s https://api.example.com/health |
|---|
That is all from deploying an API with Python and Nginx on Ubuntu.
When Should You Deploy an AI API on Kubernetes?
A single server is enough for many projects. If you later deploy Nginx on Kubernetes, note that the community ingress-nginx controller stopped receiving updates in March 2026, and the Kubernetes project recommends moving to the Gateway API.
Pick a current controller or Gateway implementation for new clusters. The lessons above still apply, including timeouts, body limits, and buffering for streams; only the configuration syntax changes. If you want to package the API and its dependencies into containers before moving toward orchestration, start with our guide to install Docker on Ubuntu 26.04 LTS.
How to Troubleshoot Common AI API Deployment Errors
To fix Linux server errors, you can explore our guide on Using ChatGPT to Troubleshoot Linux Server Errors.
- 502 Bad Gateway. The app is down or listening on a different port. Run systemctl status aiapi and curl http://127.0.0.1:8000/health to troubleshoot Nginx 502 Bad Gateway errors on Linux.
- 504 Gateway Timeout. The model is still loading, or inference exceeds proxy_read_timeout. Check the logs, then raise the timeout.
- 413 Request Entity Too Large. Raise client_max_body_size.
- 401 Invalid API key. Check the header name. FastAPI reads x-api-key, and the key must match /etc/aiapi.env.
- The service restarts in a loop. Look for an out-of-memory kill with sudo dmesg -T | grep -i oom. Use a smaller model or fewer workers.
- Permission denied writing the model cache. The aiapi user must own its home directory, /opt/aiapi.
Conclusion
Python API deployment for a machine learning model comes down to a few habits. Load the model once, keep the app on localhost, let systemd restart it, and let Nginx handle the public side. Then tune the parts that are specific to AI, namely worker count, timeouts, upload size, and buffering. Get those right and a modest Ubuntu server will serve a real model reliably.
FAQs
1. Can I deploy an AI API for free?
Yes, for small models. Run it locally while developing, or use a free-tier cloud VM for a light classifier or demo. Free servers are usually too small for large models, and their limits and terms change, so check them before you commit.
2. Should I use FastAPI or Flask for an AI model API?
FastAPI is the better default. It validates requests, generates interactive docs, and runs plain functions in a thread pool. Flask is simpler and works well with Gunicorn if you only need a small endpoint. Both sit behind the same Nginx setup.
3. Why does Nginx return 504 Gateway Timeout when my model is slow?
Nginx waits 60 seconds by default for the app to respond. If inference takes longer, raise proxy_read_timeout. If you use Gunicorn, also raise its worker timeout, which defaults to 30 seconds.
4. How many workers should a machine learning model API use?
Start with one. Each worker loads a separate copy of the model, so memory use multiplies. Add workers only after confirming you have the RAM and that the extra processes actually improve throughput.
5. Do I need Nginx in front of Uvicorn or Gunicorn?
For anything public, yes. Nginx handles HTTPS, rate limiting, request size limits, and slow clients, and it keeps your app server off the open internet. Running Uvicorn directly on a public port skips all of that.
Leave feedback about this