Content revision ea31a3343577b30cee4337cf2fef2b0864a5741341f27598e9108766c486fb2d

## Function-Vector Heads Are Two Populations: Writers and Cancellers in In-Context Learning

Research
Source ID research:functionVector
Original https://www.henryw.me/#/research#functionVector

I studied how a language model carries out a task inferred from examples in its prompt. Function vectors capture this task information through the combined outputs of attention heads. Among candidate function-vector heads, I identified writers that support the answer specified by the demonstrated rule and cancellers that systematically suppress it. Interventions on individual heads established these roles, and tests on new prompts showed that the groups had opposing effects on prediction. Removing cancellers improved accuracy by 2.4 to 6.8 percentage points across two rule families and three Pythia models. The dominant suppressive contributions depended on the task content the heads read. These roles also depended on the task, as some of the same components supported correct answers in a different setting.

Examining the groups separately revealed substantial support and suppression that could nearly cancel in the combined effect. This explains how a population with a small combined effect can still contain components that strongly affect prediction.