Ask what the next input adds
A feature is an input variable used by a model. Two useful-looking features can repeat information, while a weak-looking one may help in combination with another. Google Research’s February 4, 2026 explanation presents Sequential Attention as a way to make selection depend on the features already chosen. [1]
- Keep the selected inputs
- Rank what remains in that context
- Add one and reassess
Choose, then reassess
The method maintains a selected set and uses learned attention weights to rank remaining candidates. It adds a candidate and updates the selection process in light of the expanded set. The authors use these weights as a practical proxy for importance, avoiding a separate full training exercise for every possible addition. [1] [2]
A useful shortcut is not a universal guarantee
The linked paper first appeared in September 2022 and was revised in April 2023. Its theoretical connection to Orthogonal Matching Pursuit concerns a linear-regression adaptation. Neural-network results are empirical. A theorem for that simpler setting should not be read as a promise that every deep model will retain its accuracy after pruning. [2]
Go a little deeper
Optional reading · about 1 more minute
A pair of almost-identical readings
Hypothetical example: A building model can afford two sensor inputs. Once one room-temperature reading is selected, a nearly identical reading might add less than a different kind of measurement. This illustrates the question of added information; it does not claim which sensors an actual building should use.
Ask what was kept and what was tested
Our interpretation: The interesting comparison is not simply “large model versus small model.” Ask which input budget was imposed, how the subset was selected and whether the resulting predictions were checked on the intended task. A smaller input set is a design choice whose value needs evidence.
Original sources
Attributed synthesis, not original reporting. Examples labeled hypothetical or illustrative are explanatory. Reviewing a source does not independently validate its findings.
- Google Research: Sequential Attention ↗
February 4, 2026 developer explanation; feature interactions and sequential mechanism read September 27. No speed multiplier or universal accuracy claim adopted.
- Yasuda and colleagues: Sequential Attention for Feature Selection ↗
September 29, 2022 submission, April 25, 2023 revision; ICLR2023. Abstract, introduction and algorithm explanation read September 27; not independently replicated here.