What Local and Open AI Mean for Knowledge Work

Future signal

AI no longer has to live in someone else’s data centre

Smaller models can run on laptops, phones, or company servers, changing who controls the tool, where business information goes, and what AI costs.

Open Source and Local Are Different

Local AI describes where a model runs: on infrastructure controlled by the organisation rather than through a third-party cloud service.

Open-source AI describes the freedom to use, study, modify, and share a system. Many models described this way are more accurately called open-weight because their parameters can be downloaded, but their data, code, or licence may not provide every open-source freedom.

These are separate choices. An open model can run in the cloud, while a proprietary model can run locally. A business must ask both: Can we inspect and change it? And where does our information go?

Why This Is Becoming Practical

Useful models are becoming smaller and cheaper. Stanford’s 2025 AI Index reported that the cost of running a system at roughly GPT-3.5 performance fell more than 280-fold between November 2022 and October 2024. On some benchmarks, the gap between open-weight and closed models narrowed from 8% to 1.7% in one year.

Tools such as MLX, Foundry Local, Ollama, and llama.cpp have made local experimentation easier. A laptop will not match the best cloud model on every task, but it no longer needs to for many routine jobs.

Training Is Not Inference

Training creates or changes a model. Post-training specialises an existing model. Inference runs the finished model to answer a request. Different companies can provide each layer.

Cursor’s Composer 2.5 started with Moonshot AI’s downloadable Kimi K2.5 checkpoint, then added training for long-running coding work. Fireworks provides its inference infrastructure. Composer is customised but not local: users access it through Cursor. The open base let Cursor specialise a model without creating a foundation model or running the full infrastructure itself.

In a Harvey and Fireworks experiment on 100 legal tasks, fine-tuning Kimi K2.6 increased fully passed tasks from 11 to 15 while inference cost rose from $75 to $84. It was not cheaper overall, but cost per fully passed task fell from about $6.82 to $5.60. Training cost was excluded.

Another setup used GLM 5.1 for most work and Claude Opus 4.7 only as an adviser. It fully passed 18 tasks versus 14 for Claude alone, at about 39% of the cost. These company-run results are not independent proof, but they show the potential of specialising a model or reserving an expensive model for difficult work.

What Changes for Knowledge Work

The first change is the data boundary. A local system can transcribe meetings, classify documents, search internal files, or check drafts against private policies without uploading the source material.

The second is that AI becomes a utility rather than a destination. A small model can sit inside a repeated workflow instead of waiting in a chatbot.

The third is choice. A business can select a model for each job and change suppliers. Local inference removes some token charges but adds hardware, support, security, and maintenance costs.

The Open Source Codex CLI Can Use Other Models

Codex CLI is open source; Codex cloud is not. The CLI can connect to compatible providers and local models through Ollama or LM Studio, letting a team route work without changing its agent workflow.

Options include DeepSeek, Kimi, GLM, Qwen, Mistral, and OpenAI’s gpt-oss. Large models may still need cloud or server-grade hardware. For smaller local jobs, Google’s open Gemma models can run on laptops, phones, and tablets. Gemini Nano also runs on-device but is separate from the open Gemma family.

For example, high-volume document classification or extraction could go to DeepSeek, with difficult cases sent to a frontier model. At current V4 Flash prices, 100 million input and 20 million output tokens would cost about $19.60 before retries, tools, storage, and review.

A cheap model becomes expensive if it creates retries or corrections. Measure cost per accepted document, including review, rather than cost per token.

A hosted API still sends documents to its provider. Sensitive material may require self-hosting. Quality, tool use, structured output, context limits, and Codex compatibility all need testing. Codex also says the older Chat Completions protocol is deprecated, making Responses-compatible providers the more durable option.

The likely result is hybrid AI:

Local models handle private or offline work.
Cheaper hosted models handle tested, repetitive tasks.
Frontier models handle difficult exceptions.
People control routing, sensitive data, and important decisions.

Stanford researchers tested a similar pattern in which an on-device model reduced the document context and paid processing sent to a frontier model.

Local Does Not Mean Safe

A local application may still make network calls or expose files through weak permissions. Downloaded models can also be compromised. The UK government’s AI Playbook warns that neither open nor closed models are inherently safer. Both require access controls, testing, monitoring, updates, and review.

Start With One Workflow

Section’s analysis of 4,500 workplace AI use cases found that only 15% were likely to generate a business return. Running weak use cases locally will not improve them.

A better starting point is one repeated, reviewable task:

Map its information and decide what must stay local.
Test local, cheaper hosted, and frontier models on the same examples.
Define what AI may draft, what people must decide, and who reviews it.
Measure time, corrections, failures, cost, and maintenance.

Choose work that happens often, has stable inputs, and can be checked quickly.

The Bottom Line

Local and open AI will not replace the cloud. The advantage comes from routing work well: local models for private tasks, cheaper hosted models for tested routine work, and frontier models for difficult judgement.

Measure cost per accepted result, not model price alone, and keep people in control of sensitive data and important decisions.

Sources

Read another article