Running a Local LLM in Production: Integrating Ollama with Laravel

The task was more difficult than a regular chat

In this project, AI is built into the backend conveyor. The system collects GitHub projects, reads the description, README, site and metadata, and then determines the product type, target audience, problem, commercial potential, implementation complexity, data availability and competition.

The answer cannot be replaced by a free text. The next stage awaits the structured object: 21 mandatory field, specified data types and estimates in acceptable ranges. The model error here breaks not the beauty of the answer, but the automatic processing.

1. Data collection

   The project description, README and metadata form the original dataset.

2. Preparation of entrance

   ProjectAnalysisInput Provides information to a single format for analysis.

3. Choice of AI provider

   The request is processed by DeepSeek or local model via Ollama.

4. Testing JSON

   The validator checks the 21 mandatory field, data types and acceptable values.

5. Calculation of the result

   The system calculates the final indicators and scores on a scale from 0 to 10.

The provider is isolated by the interface: business logic and validation do not depend on the specific model.

Where did you start?

7B vs. 14B and cloud model

Same thing. ProjectAnalysisInput compared DeepSeek, Qwen2.5 7B and Qwen2.5 14B. The larger local model scored closer to the DeepSeek, but the CPU took more than six minutes to launch.

The conclusion is not that 14B is “bad.” For the interactive CPU conveyor, its time price was higher than the quality gain.

Why eight CPUs are more practical than twelve

The next test is to limit the number of logical CPU for the same query. The main increase occurred between four and six streams. After eight, the speed almost stopped changing.

If you give the models all 12 streams, MySQL, PHP, Redis and queues will be left without stock, and inference will not accelerate. Therefore, the upper limit was set at 8 CPU: this is not an absolute maximum, but the balance of the entire production server.

Structured output gave more than an increase in the model

Initially, the JSON Schema was located right in the prompt next to the instructions, README and project data. For the cloud-based API, the extra tokens are almost invisible, but the local model handles each of them on CPU.

The scheme was transmitted through the native parameter format Ollama. After reducing the instruction and input, the laboratory analysis time decreased from about 159 to about 95-100 seconds, almost by 40%, while the response retained all required fields.

For a local inference, context optimization is often better than switching to a larger model or highlighting additional CPU.

Valid JSON does not mean correct analysis

The Qwen2.5 7B consistently returned the 21 field with no extra keys. But after an overly aggressive reduction of the system prompt, a semantic error appeared: a book repository. Python Data Science Handbook The model was classified as ecommerce because of the printed version.

Formally, the answer was perfect. Meaning to the unbelievers. After clarifying the rules, the classification was corrected. This example showed that it is impossible to check the replacement of the provider with the phrase “JSON has arrived”.

1. Scheme.

   Mandatory fields, types, ranges and no unexpected keys.

2. Semantics

   Control projects, borderline cases and comparisons with historical results.

3. Operation

   Timeout, repetitions, queues, logging, load and impact on other services.

Laravel was not tied to Ollama.

Integration has already been built through AiProviderInterface. We added another implementation and left provider choice in the configuration. The main analyzer still collects the input, calls the contract, validates DTO and calculates the totals.

AiProviderInterface
├── OpenAiProvider
├── DeepSeekProvider
└── OllamaProvider

OLLAMA_API_URL=http://…/v1
OLLAMA_MODEL=qwen2.5:7b
OLLAMA_TIMEOUT=600

Ollama provides OpenAI-compatible /v1/chat/completionsInfrastructure difference does not seep into business logic. DeepSeekProvider remains affordable: the provider and model can be changed regardless of the conveyor.

How to close the network access model

Laravel and Ollama reside on different physical servers. The inference API was not publicly disclosed. An encrypted SSH tunnel operates between hosts, and the individual system user key is limited to a single direction. n permitopen="127.0.0.1:11434".

Control production test

The integration was checked from the same Horizon container where the background tasks work. For the real project, the input was formed by a regular builder, after which OllamaProvider was called - without saving the result in the database.

Final architecture: Ollama and DeepSeek together

The experiment began with the question “Can you replace the cloud API?”, but the more useful option was to distribute the work by cost and complexity.

1. Regular filters

   The code discards projects that do not pass formal criteria.

   metadata · heuristics · rules

2. Ollama makes initial assessment

   Qwen2.5 7B handles mass flow locally and distributes projects into basic categories.

   ignore · watch · interesting

3. DeepSeek gets only strong candidates

   The cloud model performs more expensive in-depth analysis where quality is more important than the cost of a single request.

   investigate · build

As the cost of the cloud inference does not grow linearly with the overall flow, the initial analysis data remains inside the infrastructure, and the system does not become dependent on a single provider.

When CPU-only LLM fits