Content revision 4e771eaf0390cc87b6c7c6fd8469f3508f6ceeec7119472e6c1ccf17e2e1e5d2

## Processing and maintaining source material

AI for IT Operations
Source ID ai-it-operations:from-a-source-to-maintained-knowledge
Original https://www.henryw.me/#/ai-it-operations#from-a-source-to-maintained-knowledge

The crawler research covered Crawl4AI Adaptive, SiteOne, Mdream/crawl, Browsertrix, and Firecrawl OSS across Azure, SAP, ServiceNow, and Oracle documentation. The results informed the separation between deciding what knowledge to acquire and the mechanics of fetching and processing pages. For each candidate page, the acquisition agent assesses relevance to the objective, source authority, overlap with material already collected, and likely information gain. Seed discovery supplies new starting URLs when the existing crawl cannot cover a question. The agent retains the original questions alongside the broader objective, allowing coverage to be checked as the search proceeds. Approved material passes through a shared ingestion and update pipeline for extraction, normalisation, deduplication, chunking, and indexing. Provenance records connect the searchable chunks to their source.

The pipeline updates the Knowledge Base and Source Registry together for additions, replacements, and removals. The accepted snapshot is the baseline for monitoring. It advances when a reviewed update is applied to the knowledge base, so later comparisons remain tied to the guidance the assistant is using. Source fetching, normalisation, comparison, and record updates use conventional system logic. AI assesses what a detected change means for the linked knowledge. Knowledge Base Source Registry Searchable knowledge stays linked to its accepted source material 02 Browser agent