Where AI Belongs in a Business Data Pipeline

Most of what I learned building a tender-intelligence pipeline came from deciding where not to put a model.

Fig. 1: The end-to-end chain from external collection to outbound channels; the model sits at the ambiguous edge, while the core is deterministic code and named people.

The product distributes paid tender notices to industry users. Two problems surfaced before any AI discussion was reasonable. First, an Excel import framework silently dropped paid rows: when a project number was blank and a nested subtable was present, it treated the row as a continuation of the previous record and discarded the main fields without raising an error. Fixing it meant defining row semantics at the business level and producing a row-level import report, not changing which column to check. Second, a historical governance pass found 284 duplicate rows among 335 records — an 84.8% rate at that point — and the duplicates traced back to imports that ran before a three-layer deduplication key existed. I recorded the remaining concurrency gap as debt instead of claiming a full cure.

AI entered in one narrow place. A colleague’s collection workflow supplies the data; I built the entry point, the deduplication and the downstream chain. Rather than hand that workflow a database account, we exposed three endpoints behind a 39-field whitelist, idempotent routing, an audit log, a daily quota and explicit failures. The deployment smoke test passed 10 of 10, and a later inspection confirmed several days of real writes with no duplicates or collisions.

To match notices against company profiles, we deliberately avoided calling a large model per query. Tags are assigned by a maintained dictionary at write time, and queries are plain index lookups. Roughly 35,000 historical award records — a figure from a September 2026 inventory — were tagged once in a batch. The trade-off is explicit: dictionary upkeep, uneven coverage and weaker semantic understanding, in exchange for predictable cost and a smaller failure surface.

Outbound messages pass through a human. Machine hits land in a pending queue where a person approves, deletes or edits them, and all channels share one review gate. Some channels’ production send settings and first live delivery were still unconfirmed at that moment, so I make no claim about real reach. Nor do I claim commercial success, high concurrency, or that a finished pipeline proves validated demand.

Fig. 2: Six stages in the order the narrative presents them; dates appear only where the article gives one.

The lesson is a division of responsibility: actions that affect authoritative data or real users — writes, deduplication, outbound delivery — belong to deterministic systems and named people. Models belong where inputs are ambiguous and downstream checks can catch mistakes.

This is a summary; the full case study is in Chinese: read the Chinese version.