TL; DR
- Learning to learn operators (self-attention + cross-attention)
- ICON is trained so that the operator lives in the prompt, not in the weights.
- Training repeatedly hands the network small few-shot tasks, each with a different hidden operator, so the weights have to learn how to read an operator off examples.
The data: many operators, many pairs per operator
- For ODE 1, u′ = a₁c + a₂, one operator is one (a₁, a₂).
- The network never sees the parameters (a₁, a₂). It sees only function samples.
Turning functions into tokens
[ term indicator, t, x | value | index ]
└──── key ────┘
The index is needed because the transformer has no positional encoding, so the prompt is effectively a bag of points.
The index is the only thing telling the model "this point belongs to example 2's output.