• Transformers has plateaued. Hence the pursuit of alternative architectures.

• We know how it works. It’s built and rebuilt from scratch, it’s one of the most heavily studied systems on the planet. The research is open.

• Scaling has plateaued, hence we have a pretty good trajectory for LLMs specifically; towards increasingly efficient tool use. It’s clear that “AGI” research will go down a different path.

See this interview from a GLM dev for a more grounded take on what the labs are feeling now:

https://www.chinatalk.media/p/the-zai-playbook

https://m.youtube.com/watch?v=Q0TXO8BBqhE

You make a good point in how much the applications change each decade. What we have 10 years from now will be unreal… That being said, I think a lot of past gains were facilitated by picking low hanging hardware/framework fruit.

In 2005, we had Pentium 4s.

In 2015, researchers were hacking stuff onto GTX 780s with doubled up VRAM, no specialized blocks, frankly primitive tooling/APIs and few libraries. PyTorch was a shell of its current self.

In 2025, we have now scaled up to massive interconnects and dedicated datacenter accelerators with mature software frameworks with tons of libraries. We have wafer sized inference accelerators and NPUs for deployment.

But shrinks are slowing, we’ve already scaled up past diminishing returns. In 2035… I don’t see the scale or software environment being significantly different? It seemed like bitnet was going to change everything for a hot moment (turning expensive matmuls into addition, and blowing up the whole software/ASIC pipe), but that hasn’t panned out.