Automated ML vs LLM Task Router using Oxlo.ai

Description and use

A Python-based automated task router that leverages traditional ML (scikit-learn) and LLMs (via Oxlo.ai) to determine the optimal approach for a given task, saving time on architectural debates.

Problem

Teams often debate whether a new project requires a classical ML pipeline with labeling or simply an LLM prompt, leading to potential over-engineering or under-utilization of resources.

Outcome

The Task Router successfully distinguishes between tasks suitable for traditional ML (e.g., structured classification) and those requiring LLMs (e.g., open-ended generation), providing clear reasoning and confidence levels for each recommendation.

Workflow

• Set up the Oxlo.ai client using the OpenAI SDK with the Oxlo.ai base URL and API key.
• Train a traditional ML baseline (e.g., RandomForest with TF-IDF) on a small labeled dataset to serve as a comparison point.
• Define a strict system prompt for the LLM to output JSON recommendations based on task characteristics (structured vs. generative).
• Create a function to call the Oxlo.ai LLM (Llama 3.3 70B) with the task description and ML baseline context.
• Implement a comparison harness that runs both the traditional ML prediction and the LLM router side-by-side.
• Execute the router on test scenarios (e.g., classification vs. email generation) to validate the recommendations.

Evidence

It compares the problem against a live traditional ML baseline, then uses an LLM hosted on Oxlo.ai to explain which paradigm fits and why, saving engineering hours and preventing over-engineering.

Source

I use Oxlo.ai here because the flat per-request pricing... means I can iterate on the prompt and run many evaluations without watching token meters spin up.

Source