Onshore hosting vs onshore inference — the distinction most Australian vendors won't clarify

A pattern appears in Australian AI vendor documentation often enough that it deserves its own article. A vendor markets themselves as “Australian” or “hosted in Australia” or “compliant with Australian data residency”. The technical detail, once you read it closely, reveals that while some component of the system runs in Australia, the actual AI processing happens elsewhere — typically on infrastructure operated by a large overseas model provider.

For most compliance questions, the location that matters is where the inference occurs, not where the login page loads.

The two things that get conflated

Hosting is where the application infrastructure lives — the servers running the web application, the database holding your account and settings, sometimes the storage layer where documents are held between operations.

Inference is where the AI model actually runs — where the prompt is processed and the response is generated. For large models, this typically requires substantial GPU infrastructure that most application vendors do not run themselves.

For a large number of Australian AI products, particularly those in the SaaS/wrapper category, the hosting is in Australia and the inference is not. The vendor operates their application on Australian cloud infrastructure (usually AWS Sydney or Azure Australia East) and calls out to an overseas model provider (OpenAI, Anthropic, Google) for the actual AI processing.

This is a legitimate architectural choice with real benefits: the vendor can offer world-class model capability without operating their own model infrastructure. It is also a distinction that changes the privacy analysis.

Why the distinction matters for privacy

Under APP 8 and section 16C of the Privacy Act, disclosure to an overseas recipient carries with it accountability for what happens to the information overseas. The question is not “where does my account live” — it is “where does the personal information actually go”.

For a product that hosts in Australia but processes overseas:

• The account and settings are in Australia
• The customer data storage may be in Australia
• The AI processing sends prompts (and any grounding data) to the overseas model provider
• The prompts and grounding data include, by definition, whatever is being processed — often personal or sensitive information

The APP 8 analysis attaches to the second bullet, not the first. Marketing that emphasises the first can obscure this.

What to look for in vendor documentation

Vendors handling this properly will be explicit. The technical documentation or security whitepaper will say something like:

“Application infrastructure is hosted in AWS Australia (Sydney). Inference is performed by OpenAI in the United States. Prompts and context sent to OpenAI are subject to OpenAI’s data handling terms; no training on customer data is performed.”

That is honest and lets you assess. You can decide whether OpenAI’s terms are acceptable for your use, whether the cross-border disclosure is covered by APP 8 exceptions, and whether the residual risk is acceptable.

Vendors handling this less clearly may say:

“Our platform is hosted in Australia and compliant with Australian data residency requirements.”

That statement can be technically true — the platform is hosted in Australia — while being materially misleading about the flow of your data. It is not necessarily bad faith; often it reflects marketing teams simplifying without the technical review that would catch the elision. But the effect is the same: the buyer forms an impression that does not match the architecture.

The specific questions to ask a vendor

When evaluating an AI vendor claiming Australian residency, ask them, in writing:

1. Where is your application infrastructure hosted?
2. Where is inference performed for our workload — the same location, or elsewhere?
3. If inference is elsewhere, which provider and which jurisdiction?
4. What is transmitted to that provider — prompts only, or prompts plus grounding data / retrieved content?
5. What are the terms under which that provider handles our data?
6. What retention applies at the inference provider, as opposed to at your hosting layer?
7. If we require inference to occur only in Australia, is that available on your product?

Get the answers in writing. A vendor confident in their architecture will produce these answers quickly and unambiguously. A vendor whose sales team has been marketing “Australian” without checking the technical detail will need time — and you will find out something useful either way.

Where this most commonly matters

The distinction is most consequential in two specific categories.

AI scribes for medical practices. Some Australian medical AI products run the entire pipeline in Australia — audio ingestion, transcription and summarisation all on infrastructure operated within Australian jurisdiction. Others host the practice-facing application in Australia and send audio or text to an overseas model provider for the actual processing. Both may market themselves as “Australian”. For patient information under the Privacy Act (and often state health privacy legislation), the distinction is not cosmetic.

“Australian” AI chat products for business. Several products aimed at Australian professional firms present as domestic alternatives to OpenAI. Some are — running open models on Australian infrastructure they operate themselves. Some are wrappers around overseas models with an Australian branding layer. The user experience is similar; the privacy analysis is quite different.

Where on-premise sits in this framing

The reason on-premise avoids this entire discussion is straightforward. If the AI runs on hardware in your building, there is no hosting location versus inference location — there is one location and it is yours. The vendor questions above do not apply because there is no vendor whose downstream arrangements you need to inspect.

That is not an argument that on-premise is always the right answer — for many workloads, an Australian-hosted-and-processed vendor is entirely appropriate. It is an argument that if you find yourself needing to conduct careful due diligence on the difference between hosting and inference for every AI product you evaluate, on-premise removes that ongoing overhead by removing the vendor.

The Australian AI market is developing quickly

To be fair to Australian AI vendors, several are actively moving toward full onshore inference — running open models on Australian GPU infrastructure they operate, precisely because they understand this distinction matters to buyers. The market is maturing.

But the maturity is uneven, and marketing has moved faster than technical delivery. When you are told a product is “Australian”, the appropriate follow-up is “Australian in what specific sense” — and the vendors who take the question in good faith are usually the ones worth working with.