Run Qwen Locally: Ollama, LM Studio & Hardware Guide

Run Qwen locally by installing a local inference runtime such as Ollama or LM Studio and downloading an explicit Qwen model or compatible quantization that fits your hardware. Qwen is a family rather than one downloadable model, so the right choice depends on available RAM or VRAM, the task, context length, and whether you need text, vision, coding, or multi-user serving. For most first-time users, qwen3.5:4b through Ollama is a practical starting point; it is small enough to test on mainstream systems while retaining current Qwen text-and-image capabilities.

Last verified: October 8, 2026. Model tags, package sizes, runtime support, licenses, and installation commands can change. Verify the exact model page and runtime documentation before downloading a large checkpoint or deploying it commercially.

The fastest way to run Qwen locally

Choose the setup route according to how much control you need:

Route	Best suited to	Model format	Main advantage	Main limitation
Ollama	A fast terminal setup and a simple local API	Managed Ollama packages based on compatible quantized models	One explicit command can download and run a model	Less direct control over individual model files and low-level loading options
LM Studio	Users who prefer a graphical interface	GGUF and supported local model formats	Search, download, load, chat, and start a local server from one application	Model compatibility and features still depend on the exact file and runtime engine
llama.cpp	Direct GGUF control, CPU inference, and detailed GPU offloading	GGUF	Fine-grained control over model files, layers, context, and server options	Requires more command-line knowledge
Transformers	Python development using original Qwen checkpoints	Safetensors	Direct access to official model classes, processors, and generation settings	Original-precision weights can require considerably more memory
vLLM or SGLang	Higher-throughput local or private servers	Official checkpoints and supported quantizations	Continuous batching, OpenAI-compatible serving, and multi-user workloads	Best suited to supported GPUs and more advanced deployment environments
MLX	Compatible Apple Silicon workflows	MLX-converted models	Designed around Apple unified memory and Metal acceleration	Requires a compatible conversion and Apple Silicon Mac

Use Ollama for the quickest terminal route. Use LM Studio when you want a desktop interface. Move to llama.cpp, Transformers, vLLM, or SGLang when you need greater control, original checkpoints, or a service for several users.

Choose a Qwen model before installing anything

“Qwen” alone is not a sufficient model choice. Current official collections include small dense models, larger multimodal models, Mixture-of-Experts models, coding-specialized checkpoints, and server-scale releases. Their storage, memory use, modalities, and licenses differ.

Use our Qwen model guide to compare model families and sizes before downloading.

The following packages provide useful starting points in the current official Ollama Qwen3.5 library and related official model pages. Package size is the downloaded quantized model size—not a complete RAM or VRAM requirement.

Explicit model or tag	Approximate Ollama package size	Suitable starting use	Important qualification
`qwen3.5:0.8b`	About 1.2–1.3 GB	Basic testing on constrained hardware	Small size improves accessibility but limits difficult reasoning and coding quality
`qwen3.5:2b`	About 2.7–3.1 GB	Light chat, extraction, classification, and experimentation	Still intended for relatively modest tasks
`qwen3.5:4b`	About 3.3–4.0 GB	A balanced first local model for chat and image understanding	The runtime needs additional memory for context, buffers, and the vision component
`qwen3.5:9b`	About 6.6–7.6 GB	Stronger general reasoning, coding, and multimodal use	Requires substantially more runtime headroom than its download size alone suggests
`qwen3.8`	About 18 GB	Higher-capability local multimodal work	The current Ollama package represents the 27B Qwen3.8 model and is aimed at high-memory systems
`qwen3.6:35b`	About 23–24 GB	Advanced reasoning, agent, and multimodal workloads	The underlying model is a 35B-A3B MoE; 3B active parameters do not mean only 3B parameters must be stored
`qwen3-coder-next`	About 52 GB for the current Q4 package	Large local coding agents and repository-level work	The model has 80B total and 3B activated parameters and is intended for very high-memory systems

Use an explicit tag such as qwen3.5:4b rather than an unspecified latest tag. The current qwen3.5:latest tag maps to the 9B package, while the current qwen3:latest tag maps to Qwen3 8B. An explicit tag prevents an unexpected model size from being downloaded if a default changes later.

Official checkpoints for advanced deployments

Qwen/Qwen3.5-4B: dense 4B post-trained model with text, image, and video understanding and a native 262,144-token context.
Qwen/Qwen3.5-9B: dense 9B multimodal model for stronger local general-purpose use.
Qwen/Qwen3.8-27B: dense 27B model with a vision encoder and configurable reasoning behaviour.
Qwen/Qwen3.6-35B-A3B: multimodal MoE with approximately 35B total and 3B activated parameters.
Qwen/Qwen3-Coder-Next-GGUF: official coding-focused GGUF release with an official Q4_K_M package.

Do not select the largest model automatically. A smaller model that remains fully in fast memory can provide a better interactive experience than a larger model that repeatedly moves data between GPU memory and system RAM.

Estimate RAM and VRAM requirements correctly

The model download size is only one part of local inference memory. A complete estimate must account for model weights, quantization metadata, KV cache, activations, runtime buffers, vision components, GPU-driver overhead, context length, batch size, and concurrent requests.

A rough weight-only estimate for an uncompressed dense model is:

weight_memory ≈ parameter_count × bits_per_parameter ÷ 8

BF16 or FP16 ≈ 2 bytes per parameter
INT8         ≈ 1 byte per parameter
4-bit        ≈ 0.5 byte per parameter

This formula is not a complete system requirement. Quantized package sizes can also differ because of metadata, mixed precision, tensor alignment, and conversion choices.

Leave memory for the KV cache

The KV cache stores attention information for the active context. Its memory use rises as context length, batch size, or concurrency increases. A model that loads successfully at a short context can run out of memory after the context is raised substantially.

At the time of verification, Ollama selected its default context according to available VRAM:

Detected VRAM	Current Ollama default context
Less than 24 GiB	4,096 tokens
24–48 GiB	32,768 tokens
48 GiB or more	262,144 tokens

This means a Qwen model can advertise a 262K native context while the local runtime initially allocates only 4K. Ollama recommends at least 64K for many coding-agent and tool-heavy workflows, but increasing the context should be treated as a hardware decision rather than a free setting change.

Dense and MoE memory are different concepts

A dense model generally uses all of its model layers for every token. A Mixture-of-Experts model activates a subset of experts for each token, which can reduce active computation. It still needs access to the complete stored expert weights.

For example, 35B-A3B means approximately 35 billion total parameters and 3 billion activated parameters per token. It does not mean the complete model has the storage or memory footprint of a dense 3B model.

Run Qwen locally with Ollama

Ollama currently supports Windows, macOS, and Linux and provides both an interactive CLI and a local HTTP API. Local model requests do not require an API key.

1. Install Ollama

The safest general option for Windows and macOS is the installer on the official Ollama download page. The following are also current official installation commands.

Windows PowerShell:

irm https://ollama.com/install.ps1 | iex

macOS or Linux shell:

curl -fsSL https://ollama.com/install.sh | sh

These commands download and execute installation scripts from Ollama’s official domain. Security-conscious or managed environments can instead download and review the official installer or package before execution.

2. Verify that the CLI is available

Open a new terminal after installation and run:

ollama ls

A successful command confirms that the CLI is on the current shell’s PATH. No model needs to appear in the list yet.

3. Download and run an explicit Qwen model

Use the current 4B Qwen3.5 package for the first test:

ollama run qwen3.5:4b

Ollama downloads the model if it is not already present, loads it, and opens an interactive session. Test it with a bounded request whose result is easy to inspect:

Return exactly three bullet points explaining why the sky appears blue.
Do not include a heading or conclusion.

Check whether the answer follows the requested structure and whether the speed is acceptable. Type /bye to leave the interactive session.

4. Check whether Ollama is using the GPU

While the model is loaded, run this in another terminal:

ollama ps

The PROCESSOR column identifies whether the model is loaded entirely on the GPU, entirely on the CPU, or split between them. A split is not necessarily an error, but extensive CPU offloading can reduce generation speed.

Ollama’s current GPU documentation lists support routes for compatible NVIDIA GPUs, selected AMD GPUs through ROCm or Vulkan, and Apple GPUs through Metal. Support depends on the exact device, operating system, driver, and Ollama version.

5. Stop, list, or remove the model

ollama ps
ollama stop qwen3.5:4b
ollama ls
ollama rm qwen3.5:4b

ollama stop unloads the model from memory. ollama rm deletes the downloaded package and should be used only when the model is no longer needed.

6. Find the downloaded model files

macOS: ~/.ollama/models
Linux: /usr/share/ollama/.ollama/models under the standard service installation
Windows: C:\Users\%username%\.ollama\models

Do not edit model blobs directly. Use the runtime’s supported pull, copy, and remove commands so its manifest and cache remain consistent.

Configure Qwen context length in Ollama

First inspect the active allocation with ollama ps. Increase it only when the use case requires more retained text and the system has sufficient memory.

When starting the Ollama server manually on macOS or Linux, the current documentation gives this pattern:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Stop an existing manually started server before launching another instance on the same port. On desktop installations, use the application setting or configure the environment variable for the service, then restart Ollama.

Do not set the model’s theoretical maximum automatically. Begin with the smallest context that handles the workload, then monitor memory, processing speed, and truncation.

Control Qwen thinking mode

Current Qwen3.5 and Qwen3.8 Ollama packages expose thinking capabilities, but support and accepted values are model-specific. Check the model metadata instead of assuming one universal parameter.

Query the installed model:

curl http://localhost:11434/api/show \
  -H "Content-Type: application/json" \
  -d '{"model":"qwen3.5:4b"}'

If the metadata confirms Boolean thinking control, a native Ollama request can disable it for a simple task:

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:4b",
    "messages": [
      {
        "role": "user",
        "content": "Return the ISO 8601 date format in one line."
      }
    ],
    "think": false,
    "stream": false
  }'

Thinking can improve difficult reasoning, but it can also increase latency and consume more context. Use direct mode for extraction, rewriting, formatting, and other bounded tasks when extended reasoning adds little value.

Use Qwen through a local API

Ollama exposes its native API at http://localhost:11434/api and an OpenAI-compatible interface at http://localhost:11434/v1. Local requests currently require no API key.

Native Ollama chat request

curl http://localhost:11434/api/chat \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:4b",
    "messages": [
      {
        "role": "system",
        "content": "Answer concisely and do not invent missing facts."
      },
      {
        "role": "user",
        "content": "Explain the difference between RAM and VRAM in four sentences."
      }
    ],
    "stream": false
  }'

OpenAI-compatible request

curl http://localhost:11434/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "qwen3.5:4b",
    "messages": [
      {
        "role": "user",
        "content": "Return a valid JSON object with the keys model and status."
      }
    ],
    "stream": false
  }'

The server binds to 127.0.0.1:11434 by default, which limits access to the same machine. Do not bind it to 0.0.0.0 or open the port to the internet without authentication, firewall restrictions, TLS, and a secured reverse proxy.

Run Qwen locally with LM Studio

LM Studio is a graphical alternative for searching, downloading, loading, and serving local models. Its current application requirements recommend at least 16 GB of RAM. The Windows guidance recommends at least 4 GB of dedicated VRAM, while the current macOS application requires Apple Silicon and macOS 14 or later. These are application-level recommendations, not guarantees that a particular Qwen model will fit.

Download LM Studio from its official site for the supported operating system.
Open the model search interface and search for the exact Qwen family and size.
Inspect the publisher, source checkpoint, quantization, file size, context support, and modality labels.
Download one file that leaves sufficient memory headroom.
Load the model and begin with a modest context rather than the architectural maximum.
Open the Developer tab when a local API is needed.
Start the local server and keep it bound to the local machine unless remote access is secured deliberately.

LM Studio’s OpenAI-compatible server currently uses the base URL:

http://localhost:1234/v1

The server can be started from the Developer interface or with:

lms server start

After the application and model files are downloaded, LM Studio documents offline operation for local chats and local inference. Features that depend on model search, updates, external tools, or remote services still require a network connection.

Alternative local Qwen routes

Docker Model Runner

The official Qwen3.5 model cards currently document Docker Model Runner as a direct route:

docker model run hf.co/Qwen/Qwen3.5-4B

This is useful when the Docker model runtime is already part of the development environment. Confirm current Docker Model Runner support, GPU access, storage, and context configuration before relying on it for production.

Transformers

Use Transformers when you need the official Safetensors checkpoint, direct Python control, custom preprocessing, or model internals. Follow the exact installation instructions on the selected model card because newer Qwen architectures can require a recent Transformers release.

Original BF16 or FP16 weights require much more memory than a 4-bit GGUF package. Do not copy an Ollama hardware expectation into a Transformers deployment.

vLLM or SGLang

Use vLLM or SGLang when the objective is an OpenAI-compatible server, several users, continuous batching, or multi-GPU deployment. Current Qwen model cards provide model-specific examples and minimum framework versions. Verify the exact Qwen architecture, vision support, reasoning parser, tool parser, tensor-parallel configuration, and context allocation before starting the server.

llama.cpp and official Qwen GGUF

llama.cpp is appropriate when you want direct GGUF control or CPU and GPU layer offloading. Qwen publishes an official GGUF repository for Qwen3-Coder-Next, including an official Q4_K_M package. Its current compressed package is still very large, so it is not the appropriate beginner download for a typical laptop.

Choose the right quantization

Quantization reduces model-weight precision to lower storage and memory requirements. It can also change quality, speed, feature compatibility, and the hardware on which a model performs well.

Format or level	Typical purpose	Main trade-off
BF16 or FP16	Original or near-original checkpoint quality	Highest weight-memory requirement
FP8	Reduced GPU memory on compatible accelerators and runtimes	Requires specific hardware and software support
Q8	High-fidelity quantized GGUF inference	Large file and memory footprint compared with lower-bit packages
Q5	A middle ground between size and retained quality	Not every Q5 variant behaves identically
Q4	A common practical choice for local GGUF inference	Greater quality loss than higher-bit packages can appear on difficult tasks
Q3 or lower	Severely constrained memory	Higher risk of degraded reasoning, coding, instruction following, and factual consistency
AWQ or GPTQ	Quantized GPU inference in compatible runtimes	Runtime and architecture support varies
MLX	Apple Silicon-optimized model packaging	Requires a compatible MLX conversion and workflow

GGUF is a file format, not one quantization level. A filename should identify the actual quantization, such as Q4_K_M or Q8_0. Confirm the publisher, original source model, file size, license, chat template, and any required vision projector before downloading a community conversion.

Run Qwen locally with images

Current Qwen3.5, Qwen3.6, and Qwen3.8 model families include multimodal checkpoints, and their current Ollama packages are labelled for text and image input. That does not mean every Qwen GGUF, conversion, or runtime supports images correctly.

Confirm that the exact source checkpoint includes a vision encoder.
Confirm that the selected Ollama tag, LM Studio model, or GGUF package is marked as multimodal.
Verify whether an additional projector or vision file is required.
Use the current runtime-specific image syntax rather than syntax copied from another application.
Allow additional memory for image processing and longer multimodal prompts.
Test OCR, small text, charts, and document layouts independently before relying on them.

A model’s ability to accept an image does not guarantee accurate OCR, chart interpretation, or document reasoning. Verify important values against the original file.

Is local Qwen fully offline and private?

Local inference, offline operation, and complete privacy are different claims.

Claim	What it means	What can still use the network
Local inference	The model calculation runs on the user’s hardware	Downloads, updates, telemetry, external tools, and connected applications
Offline operation	The required runtime and model can generate without an active network connection	Web search, remote MCP servers, cloud embeddings, and updates stop working
Self-hosted service	The user or organization operates the inference endpoint	Connected clients, reverse proxies, monitoring, logging, and external tools can still transmit data
Private workflow	Data handling is controlled according to a defined architecture and policy	Local logs, chat history, backups, telemetry, tools, and other clients must still be reviewed

Ollama’s current FAQ states that prompts and data sent to its local runtime are not sent to ollama.com. Cloud features use a different route. Ollama also provides a local-only setting:

OLLAMA_NO_CLOUD=1

Review how the operating system or service applies that environment variable, then restart Ollama. Also inspect browser extensions, Open WebUI instances, MCP servers, web-search tools, cloud fallback, and remote embedding providers before describing the complete workflow as offline.

Secure a local Qwen server

A local model is not automatically safe. The model can still generate incorrect commands, follow prompt injections from files, or receive excessive access through agents and tools.

Keep personal servers bound to 127.0.0.1 unless remote access is deliberately configured.
Do not expose an unauthenticated Ollama or LM Studio endpoint directly to the public internet.
Use firewall rules, authentication, TLS, and a secured reverse proxy for approved remote access.
Run coding agents as a non-administrator user.
Work on a Git branch, review commands, inspect diffs, and keep backups.
Restrict file, shell, database, and network tools to the minimum necessary scope.
Review third-party GGUF files, extensions, custom code, and MCP servers before installing them.
Never place credentials, private keys, or production secrets in model prompts or persistent context.

Binding a service to localhost protects it only while the service is genuinely limited to loopback. Verify the actual listening address rather than relying on the URL used by one client.

Qwen licenses and commercial local use

Do not apply one license statement to the complete Qwen family. Check the exact model card and license file for the checkpoint being downloaded.

The official Qwen3.5-4B, Qwen3.5-9B, Qwen3.6-35B-A3B, Qwen3.8-27B, and Qwen3-Coder-Next cards linked in this guide currently display Apache-2.0. Other Qwen releases can use research, community, or custom licenses. A community quantization normally remains subject to the original model license and may include additional conversion files or terms.

Review the exact license for the checkpoint, quantization, runtime, derivative, and intended use. The runtime’s software license does not replace the model-weight license.

Common local Qwen problems

Problem	Likely cause	Fastest safe check	Corrective action
`ollama` command is not found	The terminal was opened before installation completed, or the executable is not on `PATH`	Open a new terminal and run `ollama ls`	Restart the terminal, verify the official installation, and reinstall through the supported method if necessary
Model tag is not found	The tag is misspelled, removed, or unsupported by the installed runtime	Check the current official Ollama model page	Use an explicit current tag and update Ollama through the official process
The wrong model was downloaded	An unspecified `latest` tag resolved to a larger default	Run `ollama ls` and inspect the exact model name and size	Remove it if unnecessary and pull an explicit tag such as `qwen3.5:4b`
Download fails or stops	Insufficient disk space, an interrupted connection, proxy restrictions, or corrupted local data	Check free storage and network access to the official model source	Free sufficient space, correct proxy or certificate settings, then retry the official pull
Model will not load	The model plus runtime overhead exceeds available memory	Compare the package size with free RAM or VRAM and check system memory pressure	Use a smaller model or quantization, shorten context, close other applications, or reduce GPU offloading
Qwen uses the CPU instead of GPU	The GPU is unsupported, drivers are missing, or the runtime cannot initialize its accelerator backend	Run `ollama ps` and check the current official GPU compatibility page	Install a supported driver or backend, update the runtime, or use a supported device
Only part of the model uses the GPU	The complete model and cache do not fit in VRAM	Inspect the `PROCESSOR` split in `ollama ps`	Use a smaller package, reduce context, or accept CPU offloading with lower performance
Generation is very slow	CPU fallback, a model too large for fast memory, long context, or heavy thinking	Check processor placement, active context, model size, and thinking mode	Use a smaller model, lower context, disable thinking for simple tasks, or improve GPU acceleration
Earlier messages are forgotten	The active runtime context is shorter than the conversation	Use `ollama ps` to inspect the allocated context	Increase context carefully or summarize and start a new conversation
Increasing context causes an out-of-memory failure	The larger KV cache consumed the remaining memory	Return to the earlier context and monitor memory	Use a smaller model, quantized KV cache when supported, or a more modest context
An image is ignored	The checkpoint, quantization, or runtime package is text-only or lacks required vision files	Inspect the exact model’s modality labels and runtime documentation	Use a verified multimodal package and the current runtime-specific image syntax
Local API connection is refused	The server is not running, the port is wrong, or another process uses the port	Confirm the runtime process and test its documented loopback URL	Start the server, correct the port, or stop the conflicting process
The model remains in memory	The runtime keeps it loaded for faster subsequent requests	Run `ollama ps`	Use `ollama stop MODEL_NAME` or adjust the runtime’s keep-alive behavior

Local Qwen vs the Qwen API

Decision factor	Local Qwen	Managed Qwen API
Initial setup	Runtime, model download, hardware, and configuration	Account, API key, endpoint, and model selection
Cost structure	Hardware, electricity, storage, engineering, and maintenance	Usage-based or subscription billing
Idle cost	Hardware may remain underused	No dedicated local hardware required
Model size	Limited by available memory and compute	Can provide access to models too large for a workstation
Data control	Inference can remain inside controlled infrastructure	Requests are processed under the provider’s terms
Offline use	Possible after required files are downloaded, subject to external tools	Requires network access
Scaling	The user operates concurrency, batching, updates, and failover	The provider manages infrastructure within account limits
Context and tools	Limited by runtime defaults, memory, and local integrations	Can expose larger managed context and provider-built tools
Best fit	Sustained use, controlled environments, offline needs, and experimentation	Occasional use, large models, rapid production deployment, and managed scaling

Local inference is not automatically cheaper. Compare total monthly API spend with hardware depreciation or cloud-GPU rental, electricity, storage, engineering time, monitoring, security, and expected utilization.

Frequently asked questions

What is the easiest way to run Qwen locally?

Ollama is the fastest terminal-based route for most first-time users. Install it from the official source and run an explicit package such as ollama run qwen3.5:4b. LM Studio is the easier option for users who prefer a graphical interface.

Can Qwen run on Windows, macOS, and Linux?

Yes. Ollama currently supports all three operating systems. LM Studio currently supports Windows, Linux, and Apple Silicon Macs; its current macOS release does not support Intel Macs.

Can Qwen run without a GPU?

Smaller quantized models can run on a supported CPU with sufficient RAM, but functional loading does not guarantee acceptable interactive speed. CPU architecture, instruction support, memory bandwidth, threads, model size, quantization, and context all affect performance.

How much RAM or VRAM does local Qwen need?

There is no one number for the entire Qwen family. The requirement depends on the exact model, quantization, runtime, context, KV-cache format, modality, batch size, and GPU offloading. The downloaded model file must fit with additional headroom for runtime memory and the active context.

Which Qwen model should I start with?

Start with qwen3.5:4b for a balanced current test. Use qwen3.5:0.8b or qwen3.5:2b on constrained systems. Evaluate qwen3.5:9b when more memory is available and stronger output is needed.

Should I choose Q4, Q5, or Q8?

Q4 is a common starting point when memory matters. Q5 can retain more quality at a larger size, while Q8 is much larger and closer to the original weights. The best option depends on the task and hardware; compare representative outputs rather than relying on the label alone.

Can local Qwen analyze images?

Yes, when the exact checkpoint and runtime package support vision. Current Qwen3.5 and Qwen3.8 Ollama packages are labelled for text and image input. A text-only GGUF or incomplete conversion will not gain vision support merely because another checkpoint in the family has it.

Can I use Qwen through a local OpenAI-compatible API?

Yes. Ollama currently exposes an OpenAI-compatible interface at http://localhost:11434/v1, while LM Studio uses http://localhost:1234/v1 by default. vLLM, SGLang, and llama.cpp can also provide compatible local servers for supported models.

Is local Qwen completely private?

Local inference can keep model processing on the machine, but the complete workflow may still use model downloads, telemetry, updates, web search, remote MCP servers, cloud embeddings, connected applications, and local logs. Review every component before making a privacy claim.

Can I use local Qwen commercially?

Commercial permission depends on the exact checkpoint and license. Several current Qwen checkpoints use Apache-2.0, while other Qwen releases use different licenses. Review the precise model card, license file, quantization source, runtime license, and intended deployment.

Is running Qwen locally free?

There is no hosted per-token charge for a downloaded local model, but local use still consumes hardware, storage, electricity, engineering time, and maintenance. Cloud GPU use creates a separate infrastructure bill.

Sources checked

The following primary sources were checked on October 8, 2026: