I don’t disagree with any of that. But I think we’re talking about different things here. My point is that it’s not clear that capability will continue to scale in a useful way just because you make the model bigger. If you keep getting diminishing returns while needing vastly more resources, then it’s not economically viable to run these huge models.
So, I expect that labs focusing on more efficient architectures will outcompete those that are trying to brute force the problem. Like sure, DeepSeek isn’t small in a sense that you can run it locally, but it is small compared to other models in its class, and much more energy efficient. Whatever hardware we get down the road is going to benefit more efficient models the same way meaning that they will always have a competitive advantage.
From what I see in the latest releases from Anthropic, Fable isn’t a huge leap ahead from Opus. There is an improvement, but it’s not a definitive jump in capability the way it was from Sonnet to Opus. So, they managed to make a bigger model, but got diminishing returns, and it’s evidently so expensive to run right now that they can’t even offer it as a default.
The real progress will almost certainly be happening in hybrid architectures where people start coming up with algorithms that complement LLMs and augment their capabilities. These will be like different brain regions responsible for different tasks. For example, memory formation is an obvious example here, another would be to have a built in mathematics engine. A real huge win would be to figure out how to do few shot learning on the fly as well, for which memory is a prerequisite. So, there are plenty of things we already know that can be done much better.
I don’t disagree with any of that. But I think we’re talking about different things here. My point is that it’s not clear that capability will continue to scale in a useful way just because you make the model bigger. If you keep getting diminishing returns while needing vastly more resources, then it’s not economically viable to run these huge models.
So, I expect that labs focusing on more efficient architectures will outcompete those that are trying to brute force the problem. Like sure, DeepSeek isn’t small in a sense that you can run it locally, but it is small compared to other models in its class, and much more energy efficient. Whatever hardware we get down the road is going to benefit more efficient models the same way meaning that they will always have a competitive advantage.
From what I see in the latest releases from Anthropic, Fable isn’t a huge leap ahead from Opus. There is an improvement, but it’s not a definitive jump in capability the way it was from Sonnet to Opus. So, they managed to make a bigger model, but got diminishing returns, and it’s evidently so expensive to run right now that they can’t even offer it as a default.
The real progress will almost certainly be happening in hybrid architectures where people start coming up with algorithms that complement LLMs and augment their capabilities. These will be like different brain regions responsible for different tasks. For example, memory formation is an obvious example here, another would be to have a built in mathematics engine. A real huge win would be to figure out how to do few shot learning on the fly as well, for which memory is a prerequisite. So, there are plenty of things we already know that can be done much better.