VRAM is King: Why Running LLMs on Consumer Hardware is a Near-Impossibility Without Renting GPU Time
For serious Local LLM training, consumer hardware budget is irrelevant; paying to rent compute time via services like vast.ai is the only viable path, according to artifex.
Opinions fracture on initial build strategy. Some push for APUs like the 8700G as a cost-saving entry point (Robin), while others argue that modern iGPUs already punch above their weight class and might be enough for current needs. A major dividing line exists over the optimal GPU: while Nvidia maintains status as the standard due to CUDA support, the general consensus among serious builders is that VRAM, specifically 24GB from the RTX 3090 or 4090, trumps raw clock speed (GenderNeutralBro).
The community is split between pure optimization and necessary compromises. While some advise building strictly around the highest VRAM ceiling, others suggest hybrid workflows, like the one proposed by Agility0971—using a dedicated, connected server for heavy lifting while keeping the laptop quiet for portability. The clear technical directive, however, is NVIDIA/high VRAM, and professional ML work requires external compute rental.
Key Points
Local LLM training requires external, rented GPU time, making consumer purchases inadequate.
artifex stated that training is 'practically impossible on a budget' and pointed directly to services like vast.ai.
VRAM capacity is the single most critical factor for Machine Learning builds.
GenderNeutralBro asserted that VRAM is 'king' and more important than raw clock speed for training.
Nvidia's CUDA support maintains its dominance in the ML hardware recommendation.
The consensus points to the RTX 3090 or 4090 as the top recommendation despite AMD's general Linux strengths.
The value of NPU cores is highly questionable for immediate use.
iturnedintoanewt cautioned users that NPU capability is unreliable unless the specific application is proven to use those libraries.
Starting with an APU is a reasonable compromise for users on tight budgets.
Robin suggested the 8700G for a small form factor start, but this implies a later GPU upgrade.
System component selection must prioritize PSU calculation accuracy.
Mistic advised users to calculate the PSU wattage by multiplying the total power draw by 1.3 or 1.4.
Source Discussions (4)
This report was synthesized from the following Lemmy discussions, ranked by community score.