Local AI inference is a latency and cash-flow decision

“Host it locally” can mean a server in the company's office, rented capacity in a domestic data centre, a regional cloud zone or an edge appliance at a customer site. Each option changes delay, capital, operations and who carries spare capacity.

Console Connect's 2025 report shows carrier-neutral data-centre presence concentrated in 24 African countries, with planned facilities weighted toward markets that already have capacity. GSMA's AI work identifies compute, connectivity, skills and data among deployment constraints. Geography matters, but a country label cannot select an inference architecture.

The worksheet starts with the workload.

Requests and response time

Record requests by hour, input and output size, model, context length and acceptable response. Separate interactive work from batch processing. A chatbot waiting on a user needs a different latency budget from overnight document classification.

Measure current end-to-end time from the user's device. Split network lookup, connection, data transfer, queue, model computation and application work. Moving the model cannot repair a slow database or overloaded interface.

Use median and tail latency. A good average can hide timeouts during peak traffic. Test from the actual networks and locations customers use, not the engineering office alone.

Leave local and remote latency cells blank until measured. A nearby facility with poor routing can be slower than a well-connected distant one.

Compute and utilisation

Owned accelerators require capital before demand. Calculate useful inference capacity, expected utilisation, power draw, maintenance and replacement. Hardware can be idle during low traffic and unavailable during repair.

Rented compute turns part of the cost variable and may carry reservations or minimum commitments. Record instance or service price, storage, data movement, support and currency. A managed model API includes model operations while limiting control.

Batching and caching improve utilisation under suitable workloads. They can add waiting time or return stale results. Quantisation can reduce compute and change accuracy. Test the task after optimisation.

Model size is not a business outcome. A smaller model that meets the verified threshold can improve cost and delay.

Data movement and residency

Remote inference sends inputs across networks. Calculate upload and download volume, transit or cloud egress charges and retries. Audio, images and long documents differ from short text.

Local hosting can reduce international movement. It does not remove privacy and security obligations. The facility, operator, support vendors and backups determine where data and logs go.

Residency requirements need legal interpretation. Storing data in a country may not satisfy restrictions if administrators elsewhere can access it. Conversely, a policy may permit transfer under safeguards. Obtain current advice for the data and jurisdictions involved.

Minimise prompts and retention in either design. Encryption protects transport and storage while the model still needs usable data during computation.

Power and facility operations

An office server depends on building power, cooling, fire protection, physical security and connectivity. Add backup runtime and the staff who respond at night. A low electricity tariff does not capture downtime or generator fuel.

A colocation provider packages some controls and charges for space, power, cross-connects and services. Ask whether quoted capacity is live, redundant and available for the required rack density.

Cloud or managed providers shift hardware operation and retain service dependence. Check region availability, support, outage history and exit tools.

The worksheet needs actual supplier quotes. This article will not invent a price per accelerator-hour or kilowatt because those figures change by hardware, contract and market.

Reliability and fallback

Define the consequence of unavailable inference. A recommendation feature may degrade gracefully; a fraud control or support channel may stop a transaction. The architecture should match the consequence.

Local capacity can fail with the facility. Remote capacity can fail with international connectivity or the provider. A hybrid design adds resilience and complexity. Test failover, data consistency and the cost of duplicate capacity.

Models and dependencies need patching. Identify who validates a new version, schedules rollout and restores the prior model after regression. An appliance with no local skills can remain vulnerable or stale.

Support response needs a clock and jurisdiction. A global provider's premium plan and a local engineer on call solve different incidents.

Cash timing

Owned infrastructure creates an upfront cash outflow and depreciation under the company's accounting policy. Rented capacity creates recurring foreign- or local-currency bills. Reservations trade commitment for unit price.

Compare monthly cash under base and peak traffic. Include implementation, monitoring, staff, security review and unused capacity. If customer revenue arrives in local currency and compute is billed in dollars, stress the exchange rate.

A local provider can offer local billing while importing equipment and indexing price. Read escalation terms. “Local currency” at invoice does not guarantee fixed local cost.

Set a volume at which one option appears cheaper only after filling every material cell. Then test lower demand and a delayed launch. The crossover is an assumption-dependent output, not a universal threshold.

Decide per workload

A company can host sensitive low-latency inference locally and send non-sensitive batch work elsewhere. It can use retrieval in one country and a model in another. Architecture doesn't need one flag for the whole business.

Document the choice, evidence, unresolved gaps and a review trigger such as sustained volume, a new facility or regulation. Preserve benchmark scripts so future quotes can be compared on the same task.

Local inference can improve user experience and control. It can also buy scarce hardware before customers arrive. The workload sheet decides which risk the company can carry.

Sources

• Console Connect, 2025: Africa Interconnection Report 2025
• GSMA, July 2024: AI for Africa: Use cases delivering impact