POCKET-35B-GGUF: Running a 35-Billion-Parameter LLM on CPU Without a GPU

说明与用途

POCKET‑35B‑GGUF 允许在无 GPU 的 CPU 上运行 350 亿参数的 LLM，使用 GGUF 格式和 llama.cpp。

问题

Deploying a 35‑billion‑parameter LLM on consumer or on‑premise CPU hardware without GPU access.

结果

The workflow enables local inference of a 35B LLM on CPU, expanding accessibility for developers and organizations lacking GPU resources.

工作流

• Download the GGUF model via Hugging Face CLI: huggingface-cli download VIDRAFT/POCKET-35B-GGUF --local-dir ./pocket-35b
• Run inference with llama.cpp CLI: ./llama-cli -m ./pocket-35b/<model-file>.gguf -p "Explain transformer attention in one paragraph."
• Optionally start an OpenAI‑compatible server: ./llama-server -m ./pocket-35b/<model-file>.gguf --port 8080

证据

VIDRAFT's POCKET-35B-GGUF is a 35-billion-parameter large language model quantized and packaged in GGUF format for CPU-only inference — no GPU required.

来源

The practical tradeoff is latency: CPU inference at 35B scale is slower than GPU inference, but for many use cases — document analysis, batch summarization, local coding assistance, air-gapped environments — throughput-per-dollar or data-sovereignty requirements make this a worthwhile exchange.

来源

The Mindverse source article focuses on the model's availability and CPU-inference positioning rather than publishing specific benchmark scores.

来源