How to Run Local AI on Linux Mint with Ollama

Want to use AI without sending every question to an online chatbot?

With Ollama, you can download an AI model and run it on your own computer. Once everything is installed, you can chat with that model without an internet connection.

In this guide, I’ll show you how to set it up on Linux Mint using Llama 3.2, then test it offline.

Watch the Video Walkthrough

Follow along with my video below, or work through the written steps at your own pace.

What You’ll Need

Before you start, have these ready:

What You’ll Need

• A computer running Linux Mint.
• An internet connection for the initial downloads.
• Your administrator password.
• Several gigabytes of free storage.

Performance depends on your computer’s memory and processor, as well as any supported graphics hardware. Start with a small model rather than downloading the biggest one available.

The default Llama 3.2 model is approximately 2 GB to download. It needs additional memory while running, so that download size isn’t the total amount of RAM it requires.

Step 1: Open the Terminal

Press Ctrl + Alt + T to open a terminal window.

We’ll use it to install Ollama and start the model. You don’t need to be an experienced Linux user, just work through one command at a time.

Step 2: Install Ollama

Visit the official Ollama Linux download page to check the installation command.

The official command is:

curl -fsSL https://ollama.com/install.sh | sh

Paste it into the terminal and press Enter.

This command downloads and runs Ollama’s installation script. It installs software on your computer, so make sure the address matches the official site before running it.

If you’re asked for your administrator password, enter it and press Enter again. The terminal won’t display characters while you type your password, that’s normal.

Wait for installation to finish before moving on.

Step 3: Download and Run Llama 3.2

Ollama is the software that runs the AI model. Llama 3.2 is the model we’ll use in this walkthrough.

Enter:

ollama run llama3.2

The first time you run this command, Ollama downloads the model. Keep your internet connection on until the download finishes.

When you see the chat prompt, type a question and press Enter.

Try something simple:

Explain what a browser cookie is in plain English.

Ollama’s default Llama 3.2 download is the 3B model. If your computer struggles with it, try the smaller 1B model instead:

ollama run llama3.2:1b

The smaller version downloads separately and may give less capable answers, but it’s another option for more limited hardware.

Step 4: Test It Without the Internet

Once the model has downloaded and answered your first question:

1. Turn off Wi-Fi.
2. Disconnect any wired internet connection too.
3. Return to the Ollama chat.
4. Ask another question.

For example:

Write a simple HTML page with a heading and a paragraph.

If the model responds while your computer is disconnected, you’re running it locally.

You’ll need the internet again to download other models or updates, but the downloaded model can answer questions offline.

Step 5: Open It Again Later

To leave the chat, type:

/bye

The next time you want to use the same model, open the terminal and run:

ollama run llama3.2

If you chose the smaller version, use:

ollama run llama3.2:1b

You don’t need to download the model again each time.

What Does “Local AI” Mean for Privacy?

For the setup in this guide, your computer processes the questions and generates the answers. Ollama states that it doesn’t see your prompts or data when you run models locally.

However, Ollama also supports cloud features. Running a local model and using a cloud-hosted model are different things. Stick to the downloaded model shown here if you want to follow this local setup.

Running AI locally also doesn’t protect information from someone who can access your computer. Keep your device and any saved conversations secure.

What Can You Use It For?

Start with everyday tasks such as:

• Brainstorming ideas.
• Rewriting a paragraph.
• Explaining a technical term.
• Drafting simple code.
• Summarising text you paste into the chat.

The model won’t automatically inspect your computer or read your documents. In this basic terminal chat, you provide the information it works with.

Check its answers before relying on them. Local AI can make mistakes, and a small model may struggle with complex questions or coding tasks.

Your First Local AI Setup

You now have a straightforward way to try AI on your own Linux Mint computer.

Start with a few simple questions, see how your machine performs and use the offline test to confirm that the downloaded model works without an internet connection.