{"id":14671,"date":"2026-10-12T01:01:00","date_gmt":"2026-10-12T05:01:00","guid":{"rendered":"https:\/\/www.both.org\/?p=14671"},"modified":"2026-10-04T17:41:20","modified_gmt":"2026-10-04T21:41:20","slug":"running-ollama-in-a-container-a-beginners-guide","status":"publish","type":"post","link":"http:\/\/www.both.org\/?p=14671","title":{"rendered":"Running Ollama in a Container: A Beginner\u2019s Guide"},"content":{"rendered":"
\r\n
\r\n \r\n <\/i>\r\n <\/a>\r\n <\/span>\r\n<\/div><\/div>\n

Containers aren\u2019t new, but I am new to using them. I\u2019m particularly interested in locally hosted AI. In the past, I wrote several articles about utilizing Ollama, GenAI, and other locally installed large language models (LLMs) on my Linux desktop. In this article, I will share my recent experience setting up Ollama in a container. Running Ollama in a container simplifies dependency management, prevents conflicts on your workstation, and helps keep your development environment consistent.<\/p>\n\n\n\n

An old friend recently began experimenting with locally hosted AI, and since I\u2019ve been working with containers, I thought, why not give Ollama another try in a container environment. I found a good \u2018how-to\u2019 from Fedora Magazine and adopted it for my use. First, install Podman. I used Flatpak to install it, but you can use dnf<\/strong> on Fedora or apt<\/strong> on Ubuntu and Debian.<\/p>\n\n\n\n

Once Podman is installed, open a terminal and enter the following:<\/p>\n\n\n\n

podman volume create ollama_storage<\/code><\/pre>\n\n\n\n
This creates the storage volume for the LLMs. LLM weights are large. A persistent volume ensures your downloaded models aren\u2019t lost when the container stops.<\/p>\n\n\n\n
Now you are ready to run the Ollama container. Enter the following command in a terminal:<\/p>\n\n\n\n
podman run -d \\\n  --name ollama \\\n  -v ollama_storage:\/root\/.ollama \\\n  -p 11434:11434 \\\n  docker.io\/ollama\/ollama:latest\n<\/code><\/pre>\n\n\n\n
I couldn\u2019t run a container on my setup using the specified ports. I changed the port setting to 4134, and the container ran flawlessly. I don\u2019t have an NVIDIA GPU, which would be ideal for this setup, but if you do, here is the code you should run instead of the code I used, which was for CPU-only installations.<\/p>\n\n\n\n
podman run -d \\\n  --name ollama \\\n  --security-opt=label=disable \\\n  --device nvidia.com\/gpu=all \\\n  -v ollama_storage:\/root\/.ollama \\\n  -p 11434:11434 \\\n  docker.io\/ollama\/ollama:latest\n<\/code><\/pre>\n\n\n\n
Now you want to open Podman and ensure that your newly created container is running.<\/p>\n\n\n\n
\"\"
Screen picture by Don Watkins CC by SA 4.0<\/figcaption><\/figure>\n\n\n\n
With the container started you will want to pull your first model down so that you can begin experimenting with your locally hosted large language model. Do that by entering the following command:<\/p>\n\n\n\n
 podman exec -it ollama ollama run llama3<\/code><\/pre>\n\n\n\n
Depending on your internet speed, this might take a few minutes, and then you will be ready to use the Llama 3 model in Ollama. In the prompt, I asked the model for a simple fact about Linux Mint. This was the result:<\/p>\n\n\n\n
>>> tell me a fun fact about Linux Mint\nHere's a fun fact:\n\nLinux Mint is a Linux distribution that was created by Clement Lefebvre, a \nFrench programmer, in 2006. What's interesting is that Linux Mint was \noriginally intended to be a Ubuntu-based distribution, but it quickly \ngained popularity and became a standalone project in 2009. Today, Linux \nMint is one of the most popular Linux distributions in the world, with \nover 3 million users!\n<\/code><\/pre>\n\n\n\n
In conclusion, setting up Ollama in a container provides a streamlined way to experiment with locally hosted AI models while managing dependencies and avoiding conflicts. By utilizing tools like Podman and adopting a few configuration tweaks, you can easily create a consistent development environment.<\/p>\n","protected":false},"excerpt":{"rendered":"
Containers aren’t new, but I am new to using them. I’m particularly interested in locally hosted AI. In<\/p>\n","protected":false},"author":32,"featured_media":14675,"comment_status":"open","ping_status":"closed","sticky":false,"template":"","format":"standard","meta":{"_lmt_disableupdate":"","_lmt_disable":"","footnotes":"","_members_access_role":[],"_members_access_error":""},"categories":[5],"tags":[],"class_list":["post-14671","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-linux"],"modified_by":"David Both","_links":{"self":[{"href":"http:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/14671","targetHints":{"allow":["GET"]}}],"collection":[{"href":"http:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"http:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"http:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/users\/32"}],"replies":[{"embeddable":true,"href":"http:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=14671"}],"version-history":[{"count":2,"href":"http:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/14671\/revisions"}],"predecessor-version":[{"id":14674,"href":"http:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/posts\/14671\/revisions\/14674"}],"wp:featuredmedia":[{"embeddable":true,"href":"http:\/\/www.both.org\/index.php?rest_route=\/wp\/v2\/media\/14675"}],"wp:attachment":[{"href":"http:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=14671"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"http:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=14671"},{"taxonomy":"post_tag","embeddable":true,"href":"http:\/\/www.both.org\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=14671"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}