TypeSafe and Jev in Laravel: Lessons from Evaluating 4,344 GitHub Projects

The task we gave Jev

In IdeaRadar, projects go through several stages: collection from GitHub, checks in PHP, an initial assessment through Ollama, and detailed analysis through DeepSeek. The early stage needs to answer one question: does this repository contain an independent product idea worth exploring?

For this experiment, Jev ran separately over the accumulated database. It saved its own assessment without controlling which projects went to DeepSeek. This let us compare results while keeping the existing routing in place.

1. Prepare the input

   The name, description, README, repository metadata, and available homepage information.

2. Get a short decision

   Jev chooses between “not worth considering” and “worth considering.”

3. Save the result

   The decision, probabilities, model version, input hash, request duration, and token usage.

4. Compare with Ollama

   Check which projects each approach would send to the next stage.

TypeSafe: decisions with a defined response type

The TypeSafe HTTP API accepts input data as state and a set of questions. Our task suited Choice: selecting from predefined options. The response includes the selected option, a probability distribution, and confidence.

We called POST https://api.typesafe.ai/v1/systemone with the model set to jev-latest. Every saved response in this run reported the actual version as jev-1.13.0. This ties the experiment to a specific model, even if the alias later points to another version.

Running the evaluation from Laravel

We added a separate service using Laravel’s built-in HTTP client and an Artisan command for the batch run. The same input builder used by the short Ollama gate prepared the data. The API received information about each project, without other models’ verdicts.

The command fixed the upper project ID for the batch and saved each result immediately. Subsequent runs skipped processed records, with a separate option to retry errors. Temporary API failures triggered a limited number of retries with increasing delays. Authentication problems or a series of failures stopped the batch.

php artisan idearadar:jev-evaluate --max-id=111618
php artisan idearadar:jev-evaluate --report --max-id=111618

The projects table gained a line such as Jev - 1 or Jev - 2. We also stored the detailed response: a single number is not enough to investigate classification errors later.

4,344 requests: speed and results

The run took place on September 21, 2026, from 18:42:39 to 19:29:33 UTC — approximately 47 minutes. Requests were processed sequentially.

The median request time was 611 ms, the 95th percentile was 734 ms, and the mean was approximately 640 ms. Total usage reported by the API was 4,764,842 input tokens and 139,008 output tokens. We are not claiming financial savings from these figures: that requires accounting for the API price and the cost of subsequent analysis.

The saved short Ollama requests had a median duration of approximately 25 seconds. These are operational measurements taken at different times, not a simultaneous benchmark under identical load. Nor can they be compared directly with the longer, full analysis in the previous article: this task is much narrower.

The first mistake was in our selection query

For the batch, we selected projects with screening_decision=accept. During the audit, we found that this status did not mean they had passed the PHP pre-filter: the 4,344 records included 875 with a reject decision and 20 with no result from that filter.

The appropriate pass/review group for the next stage contained 3,449 projects. Jev rejected 1,299 and accepted 2,150. Both AI assessments were available for 3,448 projects; these form the comparison below.

Before evaluating a model, check which records actually entered the experiment. Otherwise, the metrics describe a different sample from the one you intended to study.

Why agreement with Ollama was 59.8%

The two approaches agreed in 2,062 cases and disagreed in 1,386. 59.8% is the agreement rate, not Jev’s accuracy. Ollama’s answer is not a ground-truth label.

Of the 1,291 cases where Jev accepted a project but Ollama did not advance it, 1,053 had an Ollama decision of MAYBE with confidence below 75. The code does not send those projects onward. Jev’s instructions, meanwhile, explicitly told it to choose consider when uncertain. Different policies for handling uncertainty explain much of the disagreement.

The saved input hashes matched in 3,437 of the 3,448 pairs. The remaining 11 pairs cannot be treated as comparisons on identical input.

The audit also uncovered a UI inconsistency: the table displayed every MAYBE as a positive Ollama result, even though the actual routing applied a threshold. Our report used the rule in the code, rather than the number shown on screen.

Where Jev helped and where it went wrong

The clearest errors appeared in FroggyAwesome/FroggyAwesome and jishan2001/jishan2001. Both are developer profile READMEs, yet both received a 2. Their consider probabilities were 91% and 78%, respectively.

A likely explanation is that the model attributed the value of projects mentioned in the README to the profile repository itself. A reference to a custom operating system, for example, deserves a separate investigation, but does not turn an “about me” page into a software product.

The sample contained 71 repositories whose names matched their owners’ names; Jev accepted 37. This is a reason to review potential profiles, not proof of 37 errors: matching names alone are insufficient for a final decision.

Model confidence does not replace a semantic check

Across the full run, 516 responses had confidence below 0.3. The current implementation still saved the chosen option as 1 or 2. The same number on screen could therefore represent either a confident decision or nearly equal probabilities.

According to the TypeSafe documentation, confidence is calculated from the probability distribution. It helps distinguish a clear answer from an uncertain one, but does not prove that the classification is correct. The profile that received a 91% consider probability illustrates the limits of relying on a threshold alone.

Two separate changes are needed: define the subject of the assessment more precisely in the question, and decide how to handle uncertainty. For profiles, an ordinary code-based check before calling AI is also useful.

A fast filter can increase costs further down the pipeline

On the shared sample of 3,448 projects, the Ollama approach advances 953 candidates, while Jev advances 2,149. Simply replacing one filter with the other would increase the flow to DeepSeek by a factor of 2.25.

This is a projection based on saved decisions. The experiment did not automatically launch those additional DeepSeek requests. More candidates may mean fewer missed ideas, but also more noise and expense. Choosing the right balance requires labeled examples.

What to check before enabling it in the main pipeline

1. Fix the sample and the rules

   Select the intended PHP filter results and apply the same interpretation of uncertainty to both models.

2. Evaluate the current repository

   Explicitly distinguish its own product from other projects merely mentioned in the README.

3. Create a manually labeled sample

   Review 100–200 examples: useful products, profiles, learning exercises, templates, and borderline cases.

4. Measure the whole pipeline

   Track missed ideas, unnecessary admissions, and DeepSeek costs alongside the speed of the initial request.

At the time of this experiment, Jev remains a separate assessment used for comparison. Its speed suits large-scale preliminary screening, and the error analysis produced a concrete list of improvements. These results do not yet justify automatically replacing Ollama.