Yarn was created by Facebook (now Meta) in 2016 to fix npm's then-real problems with install speed and lockfile reproducibility. npm caught up over the next few years, and Yarn split into two: Yarn Classic (v1), the original, now in maintenance-only mode; and Yarn Berry (v2-v4), a near-total rewrite by the same team with radical new ideas. If you're starting a new project today, you mean Yarn Berry — never Yarn Classic.
Yarn Berry's headline feature is Plug'n'Play (PnP): it eliminates node_modules entirely. Packages are stored as zip files in .yarn/cache/, and a single generated file (.pnp.cjs) maps every import to the right zip entry. No traversing 80,000 node_modules files; no phantom deps; install times drop dramatically. The trade-off is that some tooling (older bundlers, some IDEs) doesn't understand PnP without a plugin.
Berry's other big idea is zero-installs: commit .yarn/cache/ to Git. New teammates clone and immediately run code — no yarn install step at all. CI pipelines skip the install. The trade-off is bigger Git repos, but for teams where install time dominates CI, it's a 5-10x speedup.
Yarn v4 (current, requires Node 18+) folded all the previously-separate plugins into core, made Berry 3.9x faster than v3 on cold installs, and replaced the Prolog constraints engine with a JavaScript-based one. It's the most ambitious of the JS package managers — and the most opinionated.