How to Monitor Resource Usage for Self-Hosted AI Applications

Self-hosted AI applications can consume significant CPU, RAM, GPU memory, and disk space, especially when running large language models or handling multiple requests. Without proper monitoring, resource exhaustion can slow down inference, cause application crashes, and affect other services running on your server.

This guide explains how to monitor resource usage for self-hosted AI applications on Linux using tools such as htop, nvidia-smi, and Docker statistics. You will learn how to track resource consumption, identify performance bottlenecks, and optimize your server to keep AI applications running reliably.

Quick Outline

• How Do You Monitor Resource Usage for Self-Hosted AI?
• Why Resource Monitoring Matters for Self-Hosted AI Applications
• Which Resource Metrics Matter Most for Self-Hosted AI?
• How to Troubleshoot Common AI Resource Usage Problems
• How to Monitor Self-Hosted AI Applications Regularly
• Which AI Resource Monitoring Setup Should You Choose?
• Conclusion
• Frequently Asked Questions
  

How Do You Monitor Resource Usage for Self-Hosted AI?

Start with nvidia-smi and ollama ps for a live look at GPU and VRAM. Add docker stats or btop for CPU and RAM. When you want history and alerts, run Prometheus with the NVIDIA DCGM exporter, node exporter, and cAdvisor, then build dashboards in Grafana.

That stack covers GPU utilization, GPU memory usage, CPU, system RAM, and container limits in one place. The rest of this guide shows each step, what the numbers mean, and which ones actually tell you something is wrong.

Why Resource Monitoring Matters for Self-Hosted AI Applications

When you call a hosted API, someone else worries about hardware. When you run self-hosted AI applications, the machine becomes your problem. A few things tend to go wrong:

• A model quietly spills out of VRAM and half of it runs on the CPU, so answers arrive five to ten times slower.
• Two services share one GPU and fight over memory until one crashes with an out-of-memory error.
• A container with no memory limit eats all the RAM, and the Linux OOM killer takes out something unrelated.
• The GPU throttles because of heat, and nobody notices until response times double.

Good AI resource monitoring answers three questions fast. Is the hardware the bottleneck? Which process is responsible? Is it getting worse over time?

Which Resource Metrics Matter Most for Self-Hosted AI?

Before picking tools, know what you are looking for. For local LLMs, these four signals explain most problems.

1. GPU memory usage (VRAM). This is the hard limit. A model that does not fit either fails to load or gets split between GPU and system memory. Watch used versus total, and leave a margin for the KV cache, which grows with context length.

2. GPU utilization. Shows how busy the compute cores are. During generation, it is normally high. If it sits low while requests are slow, the bottleneck is somewhere else, often CPU offload, slow storage during model load, or a single-threaded pre-processing step.

3. System RAM and swap. Models that spill past VRAM, large embedding jobs, and vector databases all use RAM. Any sustained swap use on an AI box is a warning sign.

4. Temperature and power draw. Thermal throttling reduces clock speeds without any error message. A GPU holding 83 C or higher under load deserves a look at airflow or fan curves.

Step 1: Monitor GPU Usage on Ubuntu with nvidia-smi

The quickest way to monitor GPU usage on Ubuntu is nvidia-smi, which ships with the NVIDIA driver.

nvidia-smi

This prints a snapshot of utilization, memory, temperature, power, and the processes using the GPU.

To refresh every two seconds:

watch -n 2 nvidia-smi

Log GPU Utilization and Memory Usage

For a log you can read later or feed into a spreadsheet, query only the fields you care about:

nvidia-smi –query-gpu=timestamp,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw –format=csv -l 5 >> gpu_log.csv cat gpu_log.csv

This appends one line every five seconds. It is crude, but it has solved more “why was it slow at 3 a.m.” mysteries than any fancy dashboard.

Identify Processes Using GPU Memory

To see which process owns which slice of memory:

nvidia-smi –query-compute-apps=pid,process_name,used_memory –format=csv

Best Command-Line Tools for GPU Monitoring

If you prefer a live terminal view, these GPU monitoring programs are worth installing:

• nvtop: sudo apt install nvtop gives you an htop-style view of GPU load, memory, and per-process usage. It also supports AMD and Intel GPUs in recent versions.
• btop: sudo apt install btop combines CPU, GPU monitoring, RAM, disk, and network in a single screen. GPU panels depend on the version and build, so check yours.
• rocm-smi: the equivalent of nvidia-smi for AMD cards on ROCm.

For quick checks, nvtop is the one most people keep open in a second terminal.

Step 2: Monitor Ollama Resource Usage

Ollama monitoring is simpler than it looks because Ollama tells you where a model is running.

ollama ps

The output lists loaded models, their size in memory, and a PROCESSOR column. Read that column carefully:

• 100% GPU means the whole model sits in VRAM. This is what you want.
• A split such as 40%/60% CPU/GPU means part of the model was offloaded to system RAM. It still works, but generation speed drops noticeably.
• 100% CPU means the GPU is not being used at all, which usually points to a driver, container, or runtime problem.

This one command answers the question “why is my local model slow” more often than anything else.

Monitor VRAM While Loading a Model

Open two terminals. In the first, run watch -n 1 nvidia-smi. In the second, send a prompt with ollama run. You will see VRAM jump when the model loads and climb a bit more as the context fills. Try the same model with a longer prompt. If VRAM nearly doubles its growth, your context window is a major memory consumer, and lowering num_ctx may let the model fit fully on the GPU.

Check Ollama Logs for GPU Offloading Issues

On a standard Linux install, Ollama runs as a systemd service:

journalctl -u ollama -f

Log lines at model load time show how many layers were offloaded to the GPU and how much memory was requested. If you use Docker, run docker logs -f ollama instead.

One practical note: Ollama does not give you a full metrics history by itself, so pair it with the Prometheus setup below if you want trends over days or weeks.

Step 3: Monitor CPU and RAM Usage in Docker AI Containers

If Docker is not installed on your server yet, follow our step-by-step guide to install Docker on Ubuntu 26.04 LTS before monitoring your AI containers. Most self-hosted AI stacks run in containers, such as Ollama, Open WebUI, a vector database, and an embedding service. Start with the built-in tool:

docker stats

It shows live CPU percentage, memory used against the limit, network and disk I/O for each container. Two habits will save you pain:

1. Set memory limits. A container without a limit can consume all host RAM. In Docker Compose, add mem_limit: 16g or use deploy.resources.limits for the service.
2. Reserve GPU access explicitly. Use –gpus all or the Compose deploy.resources.reservations.devices block so the container actually sees the GPU. Without the NVIDIA Container Toolkit installed, containers silently fall back to CPU.

If docker stats shows high CPU and low GPU utilization for an inference container, check ollama ps or your server logs for CPU fallback before blaming the model. For more container management and troubleshooting commands, see our guide to essential Docker commands.

Step 4: Monitor AI Resource Usage with Prometheus and Grafana

Command-line tools show you the present. Prometheus and Grafana show you the past, which is where most real diagnosis happens. Here is a practical way to use Prometheus and Grafana for AI workload monitoring.

Understand the AI Monitoring Stack

Component	Role	Default port
NVIDIA DCGM exporter	Exposes GPU utilization, VRAM, temperature, and power	9400
node exporter	Exposes host CPU, RAM, swap, disk, and network	9100
cAdvisor	Exposes per-container CPU and memory	8080
Prometheus	Scrapes and stores the metrics	9090
Grafana	Dashboards and alerts	3000

Configure the NVIDIA DCGM Exporter, Node Exporter, and cAdvisor

The DCGM exporter runs as a container and needs the NVIDIA Container Toolkit. Check NVIDIA’s documentation for the current image tag, then run something like:

docker run -d –gpus all –restart unless-stopped -p 9400:9400 nvcr.io/nvidia/k8s/dcgm-exporter:<current-tag>

Install node exporter from your package manager or run it as a container, and run cAdvisor the same way.

Configure Prometheus to Collect AI Resource Metrics

Add this to prometheus.yml:

scrape_configs: – job_name: gpu static_configs: – targets: [“localhost:9400”] – job_name: node static_configs: – targets: [“localhost:9100”] – job_name: containers static_configs: – targets: [“localhost:8080”]

If Prometheus itself runs in Docker, replace localhost with the host IP or the service names on your Compose network. This is the single most common reason a target shows as down.

Build a Grafana Dashboard for GPU and System Metrics

Add Prometheus as a data source in Grafana, then import community dashboards instead of building from scratch. The NVIDIA DCGM Exporter dashboard (ID 12239) and Node Exporter Full (ID 1860) are widely used starting points.

For GPU monitoring in Grafana, these are the metrics worth putting on one panel:

• DCGM_FI_DEV_GPU_UTIL for GPU utilization
• DCGM_FI_DEV_FB_USED and DCGM_FI_DEV_FB_FREE for monitoring GPU memory usage
• DCGM_FI_DEV_GPU_TEMP for temperature
• DCGM_FI_DEV_POWER_USAGE for power draw

Place VRAM used next to GPU utilization on the same time axis. When a slowdown happens, you can usually see whether memory was full, compute was saturated, or neither.

Configure Alerts for GPU Memory, Temperature, and RAM

Keep alerts few and meaningful:

• VRAM used above 90% for more than 5 minutes
• GPU temperature above 85 C for more than 10 minutes
• Host available memory below 10%
• Any swap growth on an inference server
• GPU utilization at zero for hours while containers are running (often a sign a service fell back to CPU or crashed)

How to Troubleshoot Common AI Resource Usage Problems

Symptom	Likely cause	Where to look
Responses suddenly 5x slower	Model partly offloaded to CPU	ollama ps processor column
Out of memory on model load	Model plus context exceeds VRAM	VRAM graph, lower context or use a smaller quant
High CPU, GPU idle	Container cannot see the GPU	docker run –gpus all, container toolkit install
Speed drops after 10 minutes	Thermal throttling	Temperature and clock panels
Random service crashes	Host RAM exhausted, OOM killer	node exporter memory, dmesg
Two apps slow each other down	Shared GPU contention	Per-process memory in nvidia-smi

How to Monitor Self-Hosted AI Applications Regularly

For broader Linux VPS monitoring and automated alerts beyond AI-specific metrics, you can also monitor Linux server resources with Zabbix.

You do not need to stare at dashboards. A light routine is enough:

1. After loading any new model, run ollama ps and confirm it says 100% GPU.
2. Check VRAM headroom at your longest realistic context length, not just a short test prompt.
3. Glance at the Grafana dashboard once a week for slow trends such as rising idle VRAM or temperature creep.
4. Re-check after every driver, CUDA, or container image update, because these are the changes most likely to break GPU access.
   

Which AI Resource Monitoring Setup Should You Choose?

Your situation	Recommended setup
One machine, one or two models	nvidia-smi, nvtop and ollama ps
Several containers on one host	Add docker stats and btop
Homelab you leave running 24/7	Prometheus, DCGM exporter, node exporter, Grafana
Small team sharing a GPU server	Full stack with alerts and per-container metrics

Conclusion

Good monitoring for self-hosted AI is mostly about catching three things early: a model that no longer fits in VRAM, a container that lost GPU access, and a machine that is running too hot. Start small with nvidia-smi and ollama ps, then add Prometheus and Grafana once you want history and alerts. Even a single dashboard with VRAM, GPU utilization, RAM, and temperature will answer most of the questions you will have.

Frequently asked questions

1. How do I monitor GPU usage for local LLMs on Ubuntu?

Run nvidia-smi for a snapshot or watch -n 2 nvidia-smi for a live view. For a friendlier interface, install nvtop. For history over time, use the DCGM exporter with Prometheus and Grafana.

2. How can I check whether Ollama is using my GPU or CPU?

Run ollama ps while a model is loaded and read the PROCESSOR column. It shows how much of the model sits on the GPU versus the CPU. You can also confirm by watching nvidia-smi while you send a prompt, since VRAM and utilization should rise.

3. What is the best GPU monitoring software for AI workloads?

For quick checks, nvidia-smi and nvtop are enough. For continuous AI application performance monitoring, the most common choice is Prometheus with the NVIDIA DCGM exporter and Grafana, because it keeps history and supports alerts.

4. How do I monitor RAM and CPU usage in Docker AI containers?

Use docker stats for a live view of CPU and memory per container. For long-term data, run cAdvisor and node exporter and scrape them with Prometheus. Set memory limits on every container so one service cannot exhaust the host.

5. Why is GPU memory usage high even when no model is running?

Ollama and similar servers keep models loaded for a period after the last request, so VRAM stays reserved until the keep-alive timer ends. Other processes, such as a desktop session or another container, can also hold memory. Check nvidia-smi –query-compute-apps=pid,process_name,used_memory –format=csv to see who owns it.