ReplyDesk: a small tuned model for Google review replies, and the test it lost

Description and use

ReplyDesk uses a small Qwen3.5‑4B model fine‑tuned on real shop owner replies to generate personalized Google review responses. The app displays similar past replies and flags unbacked promises, enabling owners to quickly edit and post responses at a cost of about $0.0001 per reply.

Problem

Small shop owners have many unanswered Google reviews; writing personalized replies is time‑consuming and generic chat‑assistant replies feel impersonal.

Outcome

Fine‑tuning reduced stock phrases from 19.3% to 2.0% and specific promises from 18.0% to 11.3%, matching real owner reply length (34 words). Cost per reply dropped to $0.00010. However, a blind‑test judge favored the prompted model 40.3% of the time, indicating the app’s value is still uncertain.

Workflow

• 1. Owner logs into the app and sets up shop facts and past replies.
• 2. New reviews appear in the inbox.
• 3. Owner clicks a review to view a draft reply generated by the fine‑tuned model.
• 4. The app displays similar past replies and a promise‑check flag.
• 5. Owner edits the draft if needed, copies it, and posts it to Google Business Profile.
• 6. The app records the reply and updates the blind‑test database for future evaluation.

Evidence

ReplyDesk writes replies to Google reviews for small shops: garages, salons, dentists, cafes, plumbers, small stores.

Source

One reply costs about $0.0001 on the fine‑tuned model. A shop with 60 reviews a month costs less than a cent to serve.

Source

Fine‑tuning did exactly what it was trained to do. The replies moved from where the prompted model sits to where real owners sit: the same length as real replies, far fewer stock phrases than even real owners, and fewer made‑up promises.

Source

I wrote replies to 300 held‑out reviews with four writers: the same 4B with a careful prompt, my fine‑tuned 4B, Qwen3.8‑27B with the same prompt, and the real owner's reply.

Source