Content revision 4e771eaf0390cc87b6c7c6fd8469f3508f6ceeec7119472e6c1ccf17e2e1e5d2

## What to automate, and how

AI for IT Operations
Source ID ai-it-operations:where-judgement-changes-hands
Original https://www.henryw.me/#/ai-it-operations#where-judgement-changes-hands

Across both parts of the platform, I approached automation at the level of individual decisions within a task. Researching a question, proposing an update, accepting the guidance and applying the change each require different information and judgement. In knowledge maintenance, I assigned investigation and preparation to AI, gave the reviewer a concrete proposal with its evidence, and used the ingestion pipeline to apply the accepted change. Deciding to delegate a step to AI also means deciding how much of its method to specify. A prompt can give a model an objective, evidence and criteria for judging its next step. A more structured workflow can prescribe stages and available tools while leaving particular decisions to the model. Where the inputs and procedure are already well defined, a script can execute the same logic each time.

One workflow can use all of these, with different amounts of freedom in different decisions. In knowledge acquisition, I specified what the search should achieve before knowing which pages it would need. The original questions define the coverage required, and the broader research objective gives the agent room to follow related evidence. Relevance, source authority and information gain guide its choices. This structure keeps the research connected to the missing knowledge while allowing the route through the documentation to emerge as the agent reads. I used prompts to give the model criteria for weighing evidence, and system logic to enforce fixed conditions. Tool schemas define the actions the system can accept, and page and time limits bound execution. Predictable operations such as comparing normalised snapshots and updating source records belong in system logic.

In source monitoring, that comparison determines whether there is a change for AI to assess at all. The model then reasons about its implications for the linked knowledge. This concentrates reasoning on the decision that needs interpretation. Human-in-the-loop design adds a choice about responsibility and context. A person can authorise a well-understood change, supply information the agent lacks, or correct its interpretation. Each requires the relevant evidence and a way to continue from the person's decision. I kept source changes connected to their affected passages, and retained task history and current page state when an operator intervened. Those records make review and correction part of the workflow, so handing work back to AI preserves what the person has just contributed. How much structure to provide can also change as the work becomes better understood.

Repeated cases may reveal a rule that can be written into guidance. If applying it still requires interpreting the situation, it can guide the model's judgement. If its conditions and exceptions can be stated precisely, it becomes a candidate for a decision table or script. I see this as part of developing the automation over time, making established knowledge reusable while keeping reasoning available where the next decision still depends on context.