Local AI and private RAG: a practical decision guide

By Angus Lewington, Alien IT · Last updated: 9 September 2026 · Facts checked: 9 September 2026

"Run it locally" sounds like a complete privacy plan. It is the first line on a system diagram.

Local AI keeps sensitive work in-house only when every part of the path stays local: the model, the software that runs it, the search index, the chat screen, the logs, the backups and the updates. A model can sit on a computer in your office while its chat screen sends usage analytics somewhere else. A model installer can fetch updates and send usage data unless you tell it not to.

Why this matters

For Australian organisations, the OAIC recommends privacy due diligence when choosing commercial AI products and, as a matter of best practice, advises against putting personal information (especially sensitive information) into publicly available generative AI tools (OAIC guidance). Running the model locally removes one external path. It does not remove Privacy Act analysis, security duties, transparency or human oversight.

Start with the job, not the model

Do not start by asking which model to buy. Ask what a person needs to get done, in one sentence:

That sentence names the users, the data, the job, the evidence and who keeps custody. "We need our own ChatGPT" names none of them.

Good first jobs:

• searching a controlled set of documents
• summarising internal reports
• sorting support requests into categories
• drafting from approved templates
• extracting fields from documents for a person to check

Poor first jobs: high-stakes decisions made without a person, open access to every file in the company, advice that needs the best current public knowledge, or any task where nobody can tell whether the answer is good. If the real need is to automate a repeatable workflow rather than search documents, see AI automation.

The parts, in plain words

A private document assistant is a chain of parts, and each link can be local or can reach out to the internet.

• Model weights. The model itself: a very large file of numbers produced by training. On its own it does nothing. Its source and its licence both matter.
• Inference. Running the model to produce an answer. Software such as Ollama or llama.cpp loads the weights into memory and runs them on your processor or graphics card.
• Ingestion. Software reads your files, extracts the text and divides it into passages.
• Embeddings. A second model turns each passage into a list of numbers standing for roughly what it is about, so similar passages can be found by comparing numbers rather than matching words.
• Vector store. The database that keeps those numbers with the passage text and its details (title, owner, version, who may see it). It is a copy of your documents in another form and needs the same protection.
• Retrieval. A question is turned into numbers the same way, and the store returns the passages that sit closest to it.
• The language model step. The question and the retrieved passages are assembled into a prompt, and the model writes an answer from them.
• User interface, identity and logs. The chat screen people use, the login that says who they are, and the record of what was asked and answered.

Which parts can reach outside

Any of these parts can send data outside the office. Check each one rather than assuming.

For any part that sends document text or prompts outside the boundary, record the provider, the purpose, the retention and the contract.

Running the model locally

Tools such as Ollama and llama.cpp run models on your own hardware. Ollama says it does not see your prompts or data when you run locally, and it documents a setting that turns off its cloud features (Ollama FAQ). llama.cpp describes its goal as running language model and vision model inference with minimal setup across local and cloud hardware (llama.cpp).

That gives you options, not a blanket assurance. A vendor's statement about its own runtime says nothing about the chat screen, extensions or external services wired around it. Beyond the table above, check whether the runtime listens on the network beyond the machine itself, which programs may call it, whether the logs contain prompts, and how updates and vulnerability notices are handled. The safest default is no exposure to the public internet, with a login and an approved access layer in front of any shared server.

Running it in-house does not secure the machine by itself. Someone still has to give every user their own login, remove shared administrator accounts, document and minimise how long prompts and responses are kept, and own updates and vulnerabilities with a target time.

What private RAG does, and what it does not

Retrieval-augmented generation, or RAG, searches an approved collection of documents and hands the model the relevant passages along with the question. A well-designed answer shows those passages so a person can check it.

RAG helps with two common problems: the model knows nothing about your business, and what it does know may be stale or hard to verify.

RAG does not "teach the model everything in your business". The model weights usually stay as they were; at answer time the retriever supplies a limited selection of text, and that is all the model sees of your documents.

It does not guarantee a correct answer either. The retriever can miss the right document, the model can misread a passage or answer beyond the evidence, and the source can itself be wrong. Running it on your own server changes none of that: a wrong answer produced locally is still a wrong answer, and keeping it in-house does not make it safe to act on.

Require answers to show their sources, and treat "not enough evidence" as a successful outcome. I would rather a search assistant said that than guessed.

Permissions before intelligence

An ordinary document system may correctly hide HR, board or client files from most staff. A careless AI index copies all of them into one searchable pile.

Use one of three defensible designs:

• separate collections for clearly different groups
• permission-aware retrieval that filters every result against the person's current rights
• a narrow approved corpus holding only material meant for everyone who uses the assistant

Test it with awkward questions: log in as an ordinary user and ask for executive pay, another client's file, deleted material, and content from a group that user left yesterday.

Permissions change. The index needs a process for additions, edits, revocations and deletions. "We removed it from SharePoint" is not enough if the text is still sitting in a local index and its backups.

Prompt injection can arrive inside a document

A retrieved file can contain text written to steer the model: ignore your instructions, reveal another document, call a tool, send this somewhere else. It can be deliberate or accidental.

Treat retrieved content as untrusted data. Limit the tools the model can use, keep instructions separate from documents, require a person to confirm any action with consequences, and test with malicious files before going live. A read-only search assistant can do far less damage than an agent allowed to email, delete or change records.

Hardware: memory is only the beginning

The practical constraints are:

• model size and quantisation (a smaller, compressed copy of the weights that needs less memory)
• available RAM or graphics card memory
• how long the prompt and its context can be
• how fast answers must arrive (tokens per second)
• how many people use it at once
• storage and processing for the index, the models and ingestion
• power, cooling and noise
• replacement and rebuild time

A small quantised model may run well on a modern workstation or compact server. A larger model, or many people using it at once, may need expensive graphics card memory. Quality depends on the task: a smaller model grounded in clean procedures can beat a larger general model on a narrow internal question, while complex reasoning may still favour a hosted frontier model.

Benchmark before you buy production hardware. Use at least 30 to 50 representative questions, including hard, ambiguous and unanswerable ones.

Local, hosted business or both?

Use a decision table rather than a slogan.

Hosted business products are not free consumer chat accounts. OpenAI says business and API data is not used for training by default, while content from its individual services may be, depending on settings. Microsoft says Microsoft 365 Copilot prompts, responses and the data it reaches through Microsoft Graph are not used to train foundation models, and that interactions are stored under Microsoft 365 controls including history, audit and eDiscovery (Microsoft).

Consumer tools make different choices about training and retention, and policies change. Record the product, account type, settings, contract and check date. For a fuller comparison of what the providers say they do with your data, read AI has changed the value of your data.

A safe proof-of-value sequence

1. Pick one bounded task. Write down the users, the data sources, the acceptable and prohibited uses, the quality threshold and the person who decides.
2. Use a copied, minimised corpus. Start with approved non-production documents, remove personal information and unneeded files, and keep a manifest of everything you ingest.
3. Configure local-only paths. Block outbound connections you do not need, restrict the model's interface to approved programs, decide what is logged, and record any update destinations you must allow.
4. Keep the source details with every passage. Store the title, owner, version, effective date, sensitivity, access group and source link.
5. Evaluate. For each test question, record the expected source and version and the facts the answer must contain. Also record claims that would be unacceptable, whether a refusal is the right result, the maximum response time and who may see the answer. Score retrieval and the answer separately: if the right passage never reached the model, changing the prompt will not fix it. Include prompt-injection and cross-user access tests, and repeat after any change to the model, embeddings, chunking, application or documents. "It worked in the demo" is not change control.
6. Prove deletion and recovery. Remove a test document and confirm it disappears from search, the logs and the active index. Then restore the whole system from documented backups onto clean hardware. Local AI does not replace backups; it adds an index, a configuration and model files that all belong in the backup plan, with the source documents staying authoritative (see private storage and backup).
7. Decide with evidence. Compare the measured local result with an approved hosted business product and a no-AI baseline, counting hardware, power, support and staff time.

Cost and maintenance

Difficulty: high for a shared production system; medium for one technical user on a workstation.

Cost: from existing capable hardware for a small test, up to business-grade GPU, storage and redundancy for many users.

Maintenance: operating system, runtime, interface and model updates; identity lifecycle; backups; document ingestion; permission sync; evaluation; capacity monitoring; licence review.

Ordinary-user fit: local chat for one person is approachable; private multi-user RAG on real business data should usually be designed and managed professionally.

What local AI does not solve

• It does not make false answers true.
• It does not give you the right to use every model or every document for every purpose; model licences and confidentiality still apply.
• It does not secure the host or make its users trustworthy.
• It does not remove your retention duties.
• It does not prevent screenshots, copying or deliberate exfiltration.
• It does not replace backups.

Local AI is a custody tool. Governance still matters.

The practical conclusion

Run AI locally when keeping the data path local genuinely matters and a tested model can do the job. Use a hosted business service when its contract, controls and quality make it the better risk. Use both when the workloads differ. The decision that matters is where the real data goes. If you would rather have a bounded use case tested than buy on a slogan, the Alien Private AI Server page describes the service.

Want a bounded private AI use case tested before you buy hardware? Call 02 9707 0999 or use the contact page and tell us the job, the documents and who needs the answers.

Frequently asked questions

Related reading