A laptop can sometimes load a model that still feels painfully slow or runs out of memory when a real coding task arrives. The file containing model weights is only one part of the budget. The runtime, context window, key-value cache, operating system, editor, build tools, and graphics workload all compete for memory and power. Microsoft’s Foundry Local documentation says it selects compatible CPU, GPU, or NPU variants where available; it also requires enough memory and disk for the selected model. The right question is not “Can it open?” but “Can it finish my task comfortably?”
Quick scan
In brief
Model weights do not include all runtime, context, cache, editor, and operating-system memory needs.
Hardware acceleration and performance depend on the exact model, device, execution provider and drivers.
Test with a representative coding task, not only a short prompt that fits a tiny context window.
Start with the full memory budget
Model file size is a rough lower bound, not a promise that the model will fit comfortably in system memory. Once loaded, the runtime may create additional buffers, and a long prompt or conversation needs working space for the context and its cache. Your editor, browser, language server, build system, and operating system remain active at the same time. A configuration that uses nearly all available RAM can begin swapping to storage and feel far slower than the advertised token speed.
For a laptop with unified memory, CPU and graphics workloads share the same pool. On a machine with a discrete GPU, some model data may need to fit in dedicated video memory or be split across devices; the exact behavior depends on the runtime. Keep a margin for the rest of your normal tools. If the model’s own documentation lists a minimum, treat it as a launch threshold rather than a comfortable daily-use target.
Quantization and context have trade-offs
Quantization stores model values with fewer bits, reducing weight memory and sometimes increasing the range of devices that can run the model. It can also alter output quality, and the trade-off varies by model, quantization method, runtime, and task. Compare the quantized variant on work you actually do: code completion, explaining a test failure, editing a small function, or planning a larger change. A benchmark score from a different model or machine does not settle whether your task succeeds.
Context is another hidden cost. A model that handles a short snippet may struggle when given a repository, a long error log, or multiple files. Increasing context can use more memory and may reduce responsiveness. Test a realistic prompt length, include the relevant files, and confirm the model can identify what it did not inspect. A concise answer produced after silently truncating the important part is not a useful local coding result.
Action map
What to do next
| Layer | Focus | Working instruction |
|---|---|---|
| Check | Start with the full memory budget | Model file size is a rough lower bound, not a promise that the model will fit comfortably in system memory. |
| Action | Quantization and context have trade-offs | Quantization stores model values with fewer bits, reducing weight memory and sometimes increasing the range of devices that can run the model. |
| Verify | The accelerator label is not a speed guarantee | Microsoft documents execution providers for supported CPU, GPU, and NPU hardware, with compatibility depending on the model variant and device. |
The accelerator label is not a speed guarantee
Microsoft documents execution providers for supported CPU, GPU, and NPU hardware, with compatibility depending on the model variant and device. Foundry Local can select a compatible hardware variant using an alias; the chosen provider may be CPU, GPU, or NPU. That abstraction is convenient, but it does not make every model compatible with every chip or make an NPU automatically faster for every task. Driver versions, thermal limits, memory bandwidth, and model support matter.
Check the runtime’s active execution provider rather than assuming an accelerator is in use. On a Windows machine, Microsoft says Task Manager can show GPU and NPU activity; the runtime’s model listing can also expose which provider is active. Watch for a silent CPU fallback, sustained fan noise, high temperatures, and battery drain. A laptop that runs a small local request for a few minutes may behave differently during a longer code-generation session.
Test the coding workflow, not the model demo
Pick a bounded job that resembles your daily work: explain a failing unit test, draft a small function, or summarize a known module. Keep a reference result so you can judge accuracy. Measure whether the answer arrives quickly enough, whether the context covers the files you supplied, and how much editing is needed. Then run a second task that contains an edge case or a misleading error to see whether the model admits uncertainty.
Do not grant a new coding model access to a production repository, credentials, or shell execution just to see if it works. Begin with a disposable copy and read-only access. If the model can invoke tools, isolate the project and limit its allowed directories and network access. A faster benchmark is not worth granting an assistant broad authority before you understand what the tool can read or change.
Calculate the real cost of local use
A model that fits may still have a high operating cost: longer battery sessions, fan noise, warm surfaces, reduced capacity for other apps, or extra time waiting for output. Compare that with a smaller model, a remote service allowed by your privacy rules, or a hybrid workflow. Record the model identifier, quantization, context, runtime, device, and execution provider so the comparison is repeatable after an update.
Hardware upgrades have their own trade-offs. More memory can improve headroom but does not automatically improve inference speed; a faster GPU may be constrained by memory capacity, heat, or power. Before buying a device for local AI, check that the exact model is supported and find measurements from independent tests using the same runtime and quantization. Vendor performance numbers are useful only when their workload and conditions are visible.
A practical go / no-go test
Call the setup usable if it can load the intended model, keep the needed context, finish a representative task at an acceptable pace, and leave enough memory for your editor and tools. Repeat the test on battery and while plugged in. Check whether a disconnected network changes behavior if offline operation matters. If it passes only by closing everything else, shrinking context below your normal use, or accepting frequent poor answers, it is a lab demo rather than a reliable daily workflow.
Write down a simpler fallback: use a smaller model, shorten the context, switch to an approved service, or stop local execution for that task. The best choice is the one that meets your privacy and reliability requirements, not the largest parameter count that fits in a product table. Revisit the test after runtime, driver, or model-catalog changes because hardware support and compatible variants evolve.
