Go Forth and Prosper: Language Modeling with Ancient Textual History
Rik Koncel-Kedziorski and Noah A. Smith
Published in Pre-Print, 2021
“We introduce a simple technique for improving language modeling of long documents by effectively extending the LM’s accessible history beyond the architecture-specified context window and into the “ancient history”—text which comes before the beginning of the context window. We train an auxiliary function to select the parts of the ancient history that are most predictive of the future text…”