LLM JSON output: one setting dropped valid JSON from 75% to 4%

Description and use

Small LLMs frequently fail to produce schema‑valid JSON when anti‑repetition settings are enabled, especially no_repeat_ngram_size: 3, dropping accuracy from 75 % to 4 %.

Problem

Small language models frequently fail to produce schema‑valid JSON when anti‑repetition settings are enabled, even though the models can generate JSON under greedy decoding.

Outcome

Turning on no_repeat_ngram_size 3 in the chat settings caused a dramatic drop from 75 % to 4 % valid JSON for Llama 3.2 1B, and 0 % for SmolLM2 360M. Temperature and repetition_penalty had minor effects, but the n‑gram ban was the primary cause of failure.

Workflow

• 1. Load models in BEINGS chatroom via Transformers.js on WebGPU.
• 2. Prompt each model with 8 inputs for three tasks (tagging, extraction, shopping list).
• 3. Run each prompt twice: plain and example schema prompt.
• 4. Sample twice: greedy (do_sample false) and chat (do_sample true with repetition_penalty 1.12 and no_repeat_ngram_size 3).
• 5. Record whether each reply parses, matches schema, and is correct.
• 6. Aggregate results per setting.
• 7. Analyze impact of each sampling knob on JSON validity.

Evidence

Llama 3.2 1B returned schema‑valid JSON 75% of the time with plain greedy decoding.

Source

Turning on one anti‑repetition setting, no_repeat_ngram_size: 3, dropped that to 4%.

Source

With all the chat settings on, it was 0 out of 24.

Source