Building an Intelligent News Pipeline

Development of the intelligent news pipeline (crmfeed)

Project type: Automated AI Content Aggregator Role: Software Architect, Full-Stack Developer

Stack: PHP, Laravel, MySQL, Nginx, GigaChat API, OpenAI API (ChatGPT), Mistral, Gemini, Chrome Headless

Project description

In a constant flow of information manual processing, rewriting and visual design of news require a huge investment of time. Project crmfeed It was designed as a fault-tolerant conveyor (pipeline) to fully automate the process.

The system independently collects incoming news, conducts their multi-stage processing through neural networks and provides the editor with a minimalist, convenient dashboard for final moderation and publication.

Architecture and technical solutions

The project is based on a monolithic architecture based on the Laravel framework. The main priority in the development was pragmatic: the system should work autonomously, correctly handle the timeouts of third-party API and do not require constant developer intervention.

Key nodes of the system:

• Isolated background processes: Heavy task processing is carried out in Artisan-commands running on schedule (Cron). This eliminates the blocking of the web interface and allows you to flexibly manage the load on the server.

• Smart content parsing: A custom scraper has been developed that can bypass protections (browser imitation) and extract clean text from sites of any complexity, including adaptive algorithms for specific domains (for example, rbc.ru).

• Integration of GigaChat API In the first stage, the AI analyzes the original news. A custom service is written that not only rewrites, but also extracts metadata: determines the tone (positive / negative), geo-reference, tags and calculates «wes» (relevance score) news for the target audience.

• System of finite machines (State Machine): The life cycle of the article is strictly controlled by a set of statuses.new -> ai_processing -> ai_rewrite -> ai_image_processing -> pending). This protects against duplication of requests to API and makes it easy to catch errors.ai_failed, ai_image_failed).

User interface and dashboard

Instead of overloaded corporate interfaces, a minimalistic admin panel was implemented. It solves one main problem of — to give the moderator the opportunity to evaluate the finished news in seconds.

Implemented a smart filtering system on the fly: articles can be sorted by value (weight), tonality or current status. Color coding status and AI metric allows you to instantly read the context of the data array.

R&D and alternative image generation (PHP)

In addition to the basic flow of image generation through neural networks (YandexART / OpenAI), the customer was offered and implemented a more predictable solution — dynamic assembly of news infographics on the server side.

With the help of PHP and headless-mode Google Chrome, the system on the fly compiles HTML/CSS templates into ready-made news cards, overlaying AI-generated short theses and icons over thematic backgrounds. This guarantees 100% readability of text (unlike « hallucinations of » fonts in neural networks) and strict adherence to the corporate identity of the project.

In addition, R&D (Research & Development) conducted a large-scale test of various AI models for computer vision (Vision) and generation. Integrations and console tests were written to assess the quality of image and text analysis through:

• Google Gemini 1.5 Flash (Vision API)

• Mistral Vision

• GigaChat API (image generation)

• OpenAI (GPT-4.1-mini)

This approach allowed us to choose the most optimal price / quality / speed tools for the final product.

Overcoming technical challenges

When integrating several heavyweight LLM models into one process, problems arose with the consumption of RAM (OOM) and the stability of the connection with the database. The problem was solved at the level of server infrastructure (SWAP configuration, swappiness optimization) and software — by limiting samples (chunking) and tight control of script execution time. Additionally, the mechanisms of follback (fallback) are implemented when parsing responses from AI, if the neural network returns the invalid JSON, as well as backup image search (Google/DuckDuckGo Image Search) in case of unavailability of YandexART.

The result

Created a stable, scalable tool that reduced the time to prepare a publication from tens of minutes to a few clicks. The project demonstrates the ability to build fault-tolerant backend systems, write complex parsers, conduct independent R&D and integrate modern AI tools into real business processes.