Autonomous Intelligence Radar & Self-Evolving Knowledge Base
The headline
A person reads 20 articles a day, seriously. This pipeline screens 500 and puts the 5–10 that score 8 or above on your desk — your time moves from screening to judging, at a few thousandths of a dollar an item.
20 = what one person can read closely in a day, an experience-based baseline. 500/day = the daily throughput ceiling the pipeline cadence is designed for, not a measurement. 8 and above = the scoring threshold route; 5–10 items usually reach the desk. A few thousandths of a dollar = model unit price × average tokens ÷ items, on a locally deployed Qwen2.5-7B, which bills electricity plus GPU depreciation.
What you are buying
You are not buying another feed reader. You are buying a pipeline that runs by itself: a Producer that scrapes and cleans, a Consumer that polls on its own cadence, a Dify agent that scores each item cold — so the board you open in the morning holds the few items that scored 8 or above, not the 200 unread links behind them. Signal-to-noise is decided by the architecture, not by keywords.
What this replaces
- It is 8:40am and the industry group chat has dropped another few hundred links. You click from the first one to the 47th before you notice it is clickbait; the piece that actually explains the mechanism is the one you find at 3pm. It was there the whole time, just further down. Someone senior spends a fixed 3–4 hours a day on skim, filter and tag — self-measured, n=1 — and that is not a glance at a feed. It is a standing daily cost.
- A person can read about 20 articles a day properly, an experience-based baseline for deep reading. Multiply your sources by ten and your reading does not multiply by ten. Whatever you cannot screen is the opportunity you never saw.
- Scrape, analyse and file all sit on one thread. One long article that times out takes the whole chain down with it; there is no async buffer and no independent consumption cadence. Scoring also depends on a frozen prompt — the definition of a good article is written into the prompt and does not move as the industry moves. The model is not too weak. The pipeline is not decoupled, and the standard it judges by cannot evolve.
Architecture
Four stages left to right: Producer scrapes and cleans, PostgreSQL holds the buffer, the Consumer and Agent A score each item, Action routes it — with the state machine spine pending → scored → routed drawn underneath as dashes. Two things are drawn as hard structures, not notes: the manual review dead-letter queue, so nothing the model cannot judge disappears quietly, and the feedback arrow from Action back into the Dify knowledge base that becomes the next round’s scoring baseline.
Guardrails
- The model’s coat comes off before the parse. A code node strips the surrounding ```json wrapper and the backticks with a regex, trims, and only then parses. The model likes to dress its output up; the pipeline does not accept the outfit.
- A parse failure does not stop the line. A fatal JSON error does not throw — the item takes a default score, is labelled as entering the default channel, and keeps moving. It also lands in the manual review dead-letter queue, so it does not vanish in silence.
- One config stream holds the secrets. Auth tokens such as dify_feed_token and the model parameters live in a separate global flow, Get Master Config (SSOT), which every Consumer node reads hot. Rotating a key or adjusting a parameter touches neither the workflow nor a redeploy.
- content_id is a hash of the source URL. Producer runs crypto.createHash("sha256") over the URL to produce the primary key and writes through n8n Auto-Map Upsert, which removes the fragile check-then-insert pair and keeps the relational table unique and clean under concurrent pressure.
- Async buffering is structure, not a patch. Producer only scrapes, cleans and writes; Consumer polls independently. The PostgreSQL buffer pool decouples them, so a long article timing out costs a single item of consumption and never the scraping side.
- Exploration is a routed window, not a dial left alone. High scores go straight to the core board, and the 3–6 band keeps a random exploration window (is_exploration: true) — controlled randomness against the filter bubble, instead of a threshold nailed shut.
- The scoring standard comes back through the same door. Anything 8 or above is vectorised automatically and written back to the Dify vector knowledge base over a webhook, where it becomes the baseline for the next round. The judgement criteria grow with the industry instead of sitting frozen in a prompt.
What you end up holding
- Distributed decoupled automation blueprint — the n8n workflow DSL files, deployable into your own network in one step, split into a separate Producer scraping flow and a Consumer batch flow.
- Dify cognitive agent configuration — the cold-arbiter system prompt, the structured output schema, and the RAG orchestration that mounts the knowledge base.
- Relational database and vector engine configuration — the PostgreSQL DDL for feed_items and rss_sources state transitions, plus the matching vector persistence schema.
2 to 3 weeks for a standard modular deployment, sequenced as source-matrix mapping, async queue decoupling, then the RAG write-back path. A planning range, not a contract commitment.
Stack
- Orchestration
- n8n — Schedule Trigger · HTTP Request · a heavy JS code node for regex stripping and fallback handling
- Data
- PostgreSQL buffer pool and state base (feed_items / rss_sources) · crypto.createHash("sha256") over the source URL yields the content_id primary key · writes through Auto-Map Upsert
- Cognition & knowledge
- Dify · Qwen2.5-7B deployed locally (API compatible), so marginal cost per item is electricity plus GPU depreciation (= model unit price × average tokens ÷ items) · vector knowledge base on Qdrant
The figures, with their arithmetic
- 3–4 hours a day screening — the manual baseline
- Self-measured, n=1: skim, filter and tag. This is our own baseline, not a client measurement. It is a standing daily cost for whoever owns the beat, and after automation that time goes to judging instead of screening.
- 20 articles a day — what one person can actually read
- An experience-based baseline for deep reading, an outside magnitude rather than our measurement. Against the 500 a day the pipeline is designed for, the input side is an order of magnitude higher.
- 500 articles a day — design capacity
- A design capacity, not a measurement: the daily throughput ceiling the pipeline cadence is built around. A person reads 20 deeply. The bottleneck is the server, not the reader.
- A few thousandths of a dollar per item filed
- Model unit price × average tokens ÷ items, on a locally deployed Qwen2.5-7B, so it is electricity plus GPU depreciation. Your own bill is the authority on the amount. Set it against the time cost of a person reading one item.
- Nothing ships below a score of 8 — usually 5–10 items reach the desk
- The scoring threshold route: 8 or above enters the push queue, below the threshold goes to the manual review queue. Your attention moves from screening to judging.
The same shape, three other problems
This is the part that matters if the featured use case is not yours: the same architecture pointed at three unrelated situations, none of them a variation on the case above, and none of them requiring a rebuild. These are transfer directions rather than delivered engagements — the system above is what we have run, and the three below are what its shape solves next. We label the difference rather than blur it.
Primary-market deal sourcing
watch the startups in a sector for press releases, hiring changes and patent filings, score each one against the firm’s own investment logic, and put only the projects carrying a moat and a funding signal on the partner’s desk — the judgement that held up is written back to the knowledge base.
Always-on B2B sentiment monitoring
scrape the long-tail reviews competitors and the company itself collect on platforms and forums, filter out the meaningless review-farm comments, distil what is left into “material defects” and “logistics pain in a specific country”, and generate the R&D iteration board.
Enterprise legal and policy compliance radar
scrape policy documents from regulators in several countries, check them against the company’s business red lines, and distil only the changes that could trigger a fine into three plain sentences for the legal owner.
Have a system that needs building?
Write down the situation. You get a written, specific response: no calendar, no discovery call, no fifteen minutes of introductions.
Status
Built in-house. This is a capability we built for ourselves — not a client case and not a client delivery, and no external organisation runs it. It proves we can build our own tools. It is running in our own environment.
Built in-house. Running in our own stack.