Project EREN: My Journey to Building a CUDA-Powered LLM Inference Engine

Description and use

A foundational CUDA LLM inference engine setup with project structure, CMake, basic kernels, error handling, and tests, enabling further development of a GPT‑2 reference implementation.

Problem

Understand and implement the components that run large language models on GPU, rather than relying on existing inference frameworks.

Outcome

Established a reliable C++/CUDA development foundation with basic kernels, error handling, and testing, enabling progression to a GPT‑2 reference implementation.

Workflow

• Set up repository structure and CMake build system
• Implemented basic CUDA vector addition kernel to validate memory allocation and kernel launch
• Implemented shared‑memory multi‑stage reduction kernel
• Created reusable CUDA error‑checking helper
• Added CTest smoke tests and initial reduction benchmark
• Verified clean build and passing tests via terminal output

Evidence

My first milestone was establishing a reliable C++/CUDA development foundation.

Source

Implemented a basic vector addition kernel to validate device memory allocation, host-to-device transfers, kernel launches, and result verification.

Source

Implemented a shared-memory, multi-stage reduction baseline.

Source

Created a reusable CUDA error-checking helper to make runtime failures easier to identify and debug.

Source

Added smoke tests using CTest and an initial reduction benchmark target.

Source