Mar 3, 2025
New Paper - On the Power of Context-Enhanced Learning in LLMs
Check out our new paper on context-enhanced learning! When training LLMs, putting helpful information purely in-context can exponentially improve sample efficiency while leading to little verbatim memorization of the in-context materials!
See twitter/X thread here.
Kids use open textbooks for homework. Can LLM training benefit from "helpful textbooks" in context with no gradients computed on these tokens?
— Xingyu Zhu (@XingyuZhu_) March 5, 2025
We call this Context-Enhanced Learning – it can exponentially accelerate training while avoiding verbatim memorization of “textbooks”!… pic.twitter.com/NnxZIBeRNN