Jev vs. Five LLMs: Selection Quality, Latency, and Cost for Building RAG Systems

Why a RAG system needs a filter before the model

In RAG and analytical pipelines, the final model is rarely the only constraint. Quality depends on what enters the index and reaches deeper analysis. Irrelevant material in the vector database means extra context tokens, misleading sources in answers, and recurring costs on subsequent requests.

In IdeaRadar, projects pass through a sequence: collection from GitHub, a PHP filter for obvious noise, a short AI assessment, and deeper analysis with DeepSeek. This article asks which model should handle that short assessment, and what the choice means for quality, latency, and cost.

For RAG, this is corpus preparation: deciding which sources are worth storing and indexing. Here, we test project selection based on READMEs. The experiment does not measure embedding quality, retrieval, ranking of retrieved passages, or final answers. Those stages require separate tests using your system’s documents and questions.

How we ran the experiment

1. One prompt for every model

   The same semantics as Jev: 1 means reject, 2 means worth considering; when uncertain, choose 2.

2. Identical input

   The complete normalized README, name, description, topics, and repository metadata.

3. A strict response format

   JSON: {"verdict":1|2,"confidence":0-100,"reason":"..."}, without streaming.

4. Consistent latency measurement

   Requests run sequentially within each model, with latency measured only around the HTTP call. We save the raw response, token counts, status, and input hash.

A key limitation. Agreement with Jev is not accuracy. We do not yet have a manually labeled ground-truth dataset, so we examine agreement between models and manually review disputed cases.

How an aggregator helps choose a model for RAG

A comparison quickly becomes less useful if each model needs a separate client, different error handling, and incompatible metrics. Through Apixly, we sent requests to one OpenAI-compatible endpoint and changed the model using identifiers from GET /v1/models. This kept the prompt, JSON format, timeouts, logging, and latency measurement consistent.

A shared gateway does not make model responses identical or remove differences between providers. It removes an extra integration variable: the experiment compares model behavior rather than five different SDKs. In production, it also simplifies fallback routing: if one model fails, the application can send the task to another without rewriting the entire pipeline.

The practical benefit of aggregators such as apixly.ai is being able to select a model for each RAG stage instead of building the whole system around the first LLM you connect. For filtering, compare false rejections and cost per check; for interactive answers, latency and grounding in retrieved sources; for background analysis, completeness and processing cost. The last two tasks need their own evaluation sets: the best filter is not necessarily the best answer generator.

A useful workflow is to prepare a manually labeled evaluation set, run candidates through a shared client, compare quality and p95 latency at the required load, and assign a model to each task. A shared API simplifies repeated evaluations and model substitutions, but each model’s limits, parameters, and response format still need checking.

The first problem: 24 empty responses out of 500

In the first run, GLM-5.3 returned 14 invalid responses and Kimi K3 returned 10. There was no API error: HTTP 200, but the JSON was truncated.

The cause was the max_tokens limit. Both models perform hidden reasoning even when not explicitly asked to. All 14 GLM responses hit exactly 1,024 output tokens, with 991–1,024 spent on reasoning. Kimi hit exactly 256. There was no room left for a one-digit verdict and a short explanation.

After increasing the limits to 4,096 for GLM and 2,048 for Kimi and repeating the same requests with the same inputs, we obtained 500 valid responses out of 500. One retried GLM response used 3,875 output tokens.

For reasoning models, max_tokens covers both reasoning and the answer. Setting it too low can turn a paid request into unusable output rather than save money. Track invalid responses as a separate metric instead of silently discarding them.

Test 1: how LLMs judge the projects Jev accepted

100 projects that Jev rated 2.

All five models unanimously rejected 19 projects. We checked them manually and agreed: a tutorial ML pipeline using a public dataset, a personal portfolio, a company website, a shader collection, school projects explicitly described as such in their READMEs, and Docker Compose wrappers around other services. A majority of three or four models rejected another 19 projects.

Overall, a majority of LLMs rejected 38 of the 100 projects Jev considered worthwhile. Those projects would otherwise add unnecessary work to the next stage, entering expensive analysis or the RAG index.

Models differ significantly in strictness

Pairwise agreement shows that these models are not interchangeable judges. GLM-5.3 and Kimi K3 agreed in 90% of cases; Claude and GLM, and DeepSeek and GLM, agreed in 85%; Claude and Grok agreed in only 60%.

Choosing a model here means choosing a selection policy. If missing a good idea costs more than reviewing an extra project, use a permissive model. If the next stage is expensive, consider a stricter one.

Test 2: can we trust Jev’s rejections?

100 new projects that Jev rejected.

91 projects were rejected unanimously. Seven received one “worth considering” vote, and two received two votes. No project received a majority of positive votes.

Reviewing the disagreements supported Jev’s decisions. Five of Claude’s eight “finds” were nearly identical forks of a Discord blocking-circumvention tool under different anonymous accounts. Such clone sequences suggest mass-generated repositories rather than independent products. The remaining disputed cases were a personal knowledge base, a marketing website without product code, and a Vite template without an implementation.

Latency: RAG needs predictable tails, not just a low median

If the filter is part of an interactive flow where a user uploads documents and waits for an answer, the 95th percentile and maximum matter more than the average. These slow requests are what users experience as the system hanging.

1. An order-of-magnitude difference

   Jev is roughly six times faster than the fastest LLM and 15–20 times faster than the reasoning models.

2. Hidden reasoning adds latency

   On obvious rejection cases, GLM used an average of 278 reasoning tokens and responded in 4 s; on borderline projects, it used 556 tokens and took 11 s. The model spends more time thinking when the decision is difficult, and the user feels the delay.

3. Long latency tails

   Kimi K3 reached 52 s, and an earlier run on 500 projects included a 149 s response. A synchronous RAG request needs a timeout and fallback strategy to handle this.

4. The most predictable model

   Claude Sonnet 5 ranged from 3.4 to 4.9 s on clear-cut cases.

How long would the full workload take?

At the time of the experiment, IdeaRadar contained 26,634 projects that had passed the PHP filter. Estimates based on mean latency:

These are approximate estimates: gateway and provider latency may increase under concurrent load, and each provider has its own rate limits.

Cost per 1,000 checks

These were the actual gateway rates in the groups used for our requests, per million input/output tokens: Claude Sonnet 5 — $0.40/$2.00; DeepSeek V4 Pro — $0.858/$2.574; GLM-5.3 — $0.70/$2.20; Grok 4.7 — $0.20/$0.60; Kimi K3 — $1.95/$9.75. Our token-based calculation closely matched billing: $4.62 versus $4.54 for all approximately 3,000 requests across the experiment, with the difference attributable to cache discounts.