Methodology

Release reporting

We separate what a developer announced from what the available evidence demonstrates. Each material claim records its source, verification state, applicable version, and any caveat that changes how a reader should interpret it.

Benchmark and hardware requirements

For new controlled benchmarks, we record the exact model artifact and checksum, runtime version or commit, backend, command arguments, operating system, CPU or GPU, memory, workload, context length, generated-token target, repetitions, aggregation, raw artifacts, and power mode when relevant. For historical or sanitized evidence, we disclose missing fields or withheld artifacts and narrow the published conclusions to what the retained evidence supports.

Results from different hardware, versions, quantizations, or prompt sets are not treated as directly comparable without a clear warning. A single run is labelled as a single run.

Comparisons

Comparisons define the decision first: device, workload, privacy needs, latency target, quality requirements, and operating constraints. There is no universal “best local LLM” independent of those conditions.

Uncertainty

We use four claim states: verified, partially verified, unverified, and disputed. Missing evidence is not converted into a confident conclusion. Material updates trigger a visible correction or update note.