#self-attention

Mila & Universite de Montreal researchers introduce FoX, a novel Transformer variant with learnable forget gates that improve long-context language modeling efficiency and accuracy without computational trade-offs.