Choosing an Open-Source LLM for Our Waste‑Sorting Assistant

Description and use

Choosing Qwen3‑4B‑Instruct‑2507 enabled a waste‑sorting assistant to run on free Colab T4 GPUs, delivering accurate, consistently formatted responses without extra API keys.

Problem

Selecting a free, instruction‑following LLM that fits within a 16 GB GPU and can provide concise, format‑consistent answers for a waste‑sorting assistant.

Outcome

The assistant now delivers consistent, instruction‑following responses within memory limits, enabling accurate waste sorting and rule explanation.

Workflow

• 1. Identify project constraints (budget, GPU memory, output format).
• 2. Search for open‑source LLMs ≤4 B parameters that fit 16 GB.
• 3. Evaluate candidates on size, license, context window, instruction following, and ease of use.
• 4. Test Qwen3‑4B‑Instruct‑2507 on Colab T4: download, load in float16, confirm memory usage.
• 5. Verify output format consistency and instruction compliance.
• 6. Deploy model in the assistant pipeline.
• 7. Monitor for memory limits and fallback to Qwen3‑1.7B if needed.

Evidence

Our project is an assistant that tells people which bin an item belongs in, and why.

Source

Our team has no budget and uses the free Google Colab T4 GPU, which has 16 GB of memory.

Source

I chose Qwen3-4B-Instruct-2507 for four reasons.

Source

It fits on the free Colab T4. I downloaded and loaded it on a free Colab T4. The model files are 8.04 GB, which matches my estimate of 4 billion × 2 bytes.

Source

It follows instructions and gives clean answers. Our assistant must answer in the same format every time.

Source

Limits. Qwen3-4B is the biggest of the four, so it is a little slower and uses the most memory.

Source