On the limits of reasoning
I'll keep it short. I think at a certain point in the future, the bottleneck of capabilities won't be on the ability of models to reason quickly. There are physical limits to how fast one can reason, but I believe models can (and arguably already are) far surpass humans in that regard. I believe the real bottleneck will be context. And not just context in the form of like conversation context, or something like that. I'm referring to context in the broadest term. Everything you feed that model to get it at the point it's at; everything data; everything that's not just model architecture.
To an extent, we don't even understand the context of our own mind. At this very point in time, we're experiencing what could otherwise be described (to the extent of lossless regeneration) in potentially like, I don't know, millions of tokens. Most of this information will probably never be used, and will be lost to time. No one cares what I ate for breakfast today - not even me. Sometimes information like that is simply lost to time. But at other times, I might get reminded of these thoughts at random points in time. I'll be washing the dishes and suddenly remember when so and so's birthday is. These are things which are quite literally untrainable to a given frontier model.
The point I'm making is that frontier models will always be fundamentally differently trained than a human individual. We just were based on different contexts. And that's the bottleneck. That's the baseline difference causes divergences in reasoning chains and thought processes between human and model.
So now here's the problem I pose: I believe information is lost between almost every conversation you have with codex or chatgpt. Irretrievable context. This is inevitable - the assumption is that codex or chatgpt was trained to have similar enough priors as you did entering that message. But of course it can't. It's impossible to have it fully aligned like that. There's too much variance in human experience and the beliefs we form.
Because of that, we face a tradeoff. Give more priors, ensure more accurate performance out of these models, at the expense of well spending more time creating that context for the model. OR, you can get around this by simply giving it smaller problems requiring less context or shifted context, OR you can well... just reason about it yourself.
But you're always making a(n unconscious) bet when you prompt it. You're taking the risk of that misalignment of priors, because you believe that the objective is better achieved through such communication rather than reasoning under your own sole priors.
I think it's a good idea to keep this kind of tradeoff in mind. It's always going to be there - that information theoretic bottleneck - no matter how good models get. The energy/objective cost it takes to get a good working conversation out of the llm, versus the energy/objective cost it takes for you to just reason through things yourself.
Under this kind of thought model, anything is teachable given enough aligned priors. In fact, reasoning itself is one such act of modifying these priors. It's not the only one, but it's one we understand relatively well. It's just a matter of how much you truly need to specify. No model will ever perfectly fit your priors aside from yourself, and you won't ever even perfectly know the fullness of a given model's priors either (not even your own). So the choice is yours: modifying and using your own priors for the task, or the model's priors. This is ultimately an open problem - one that only reality can empirically measure (although people might disagree with me even here) for sure after we make such choices. Ultimately, just as with all models, if you take enough assumptions out of this mental model, you just end up with the question "how do we know anything anyways?"
Fortunately, there are safety groups specifically designated to going up against model capabilities to find these kinds of misalignments. I believe one is best off finding value over AI in areas they specify. Regardless, I think this is a good, surprisingly practical underlying model to use to approach working with AI.