ML as functional approximation (between two potatoes)
OK hear me out. My takeaway from the basis of ML is it is possible to learn certain properties from... certain data.
You see, we're usually very caught on the idea of causation being some law over time. I push this water bottle off the desk, and it will hit the ground in less than 5 seconds. I claim that the prediction here can be modeled under conditional probabilities, even if that is not the same thing as fully capturing causation. The probability that the water bottle will hit the ground in less than 5 seconds GIVEN I just pushed the water bottle is 100%... or is it?
We go on believing these causations - unconsciously searching for them - because it gives our lives certainty, a way to make (to a degree) certain sense of it all. We form our beliefs around these, and sometimes we forget that that is exactly what they are... beliefs. Beliefs which can be radically changed. Is there a reason for why we believe the sun should come up every day? No, but we've learned by repetition that that is what always happens, which reaffirms our pattern-matched-original belief until eventually that probability in our mind becomes so high we almost take it for granted. If someone were to have told you the water bottle would float in midair, we would indeed be extremely surprised. But there would always be a chance, right...?
But there's another thing. Stacked and constructed together, these beliefs about causation form a very rigorous model of the world in our own minds. You can probably picture the water bottle falling, or the sun coming up every day before it even reaches that point in time. Maybe you can even predict different trajectories of the water bottle, each with different assigned "belief" probabilities. I believe they call this a "world model."
Anyways, another really cool thing conditional probability allows us to do is move past the temporal aspect I've been holding onto. Now we can say that the base event, which may occur after the actual event, can still update our belief about the actual event. It all comes down to how we represent these events.
So basically, the idea behind ML is the fact that we can learn these kinds of predictive relationships somehow from data, just as we humans do. After repeating into the head of your hypothetical neural network that the sun will rise tomorrow, it should tell you with very high probability that the sun will, indeed, rise tomorrow. This, therefore, is the function you want to approximate... a function which, in this case, will tell you the probability the sun will rise tomorrow.
OK sounds simple enough right? Just feed some random function approximator enough data until it makes a good function right? NO. For this, let's look at how we really formally define a function:
A function is really just a total, unique, and well-defined way of drawing arrows on two function potatoes, representing sets which may or may not contain finite, countably infinite, or uncountably infinite elements. Very formal i know i know.
So, really, what we're looking for is some way to represent/encode the input of the function, and some way to decode the output of the function, and we expect the function to do cool stuff to the input to produce an output which is what we want.
Unfortunately, as was proved by Turing, there exist some problems which, in this sense, are absolutely impossible to compute in full generality by any algorithm. E.g. the halting problem. Sure for certain cases you can ingrain invariants, or other logical tricks to give you the output, but there will not exist a general, universal trick which you can use on all relevant inputs.
This gives rise to a very big problem with our definition: The function potato sets can be infinite in cardinality. This is a big problem since, if it were finite, we would have no problem just making a function consist of a hashmap to the right answer. No matter how big this finite number is, it's still possible in theory.
But when you enter infinite cardinalities, or cardinalities that in practice are just too big to practically compute, you must rely on "generalized" function pairings... you must make the function approximate on general notions, not specific input elements.
How we train these functions, pick their architectures, parameters, everything goes into making sure there is a suitable environment in which the function can pick a generalized mapping between these two potatoes.
Where did we all begin learning ML? Through linear regression. Linear regression is a very very restrictive environment, which is good since we usually are running these approximations on pretty non-compute-intensive resources anyways. If you give it a MSE loss, then boom you bring in even more information, and specify the objective you're trying to minimize. So you've specified the setting, data points, and the traversal, and get pretty good results. I can't say the same if you tried it with a neural network on the exact same traversal and data points and setting... given the amount of data points you're working with, assuming u converge the loss, u might just have overfit the function, which in this case is the neural network.
So this observation gives rise to something general: The No-Free-Lunch rule (credits to the AI researcher came up with it idk who it was). Basically saying, doesn't matter how much finite data points you have, realistically you need to have some strategy you ingrained into your system in order to actually match these dang function potatoes well in the settings you care about. Some traversal (even randomness), some data, some setting to get these usually infinite cardinality function potatoes churning.
So this gives rise to optimizing this expert knowledge we inject into the function approximator in order to get good approximations, fast. Gradient descent with backprop for the NN function approximator was HUGE... in the case that the model and loss you're optimizing can be represented at least somewhat well as continuous and differentiable, that is (which in most practical cases works very well). idk the actual maths behind it, never looked into it, but all i can say is it was a massive practical speedup in many cases. So it really goes to show how much this expert knowledge really helps the approximator's case here.
Of course, that also leads to the observation that, the more data you have, the less expert knowledge you actually gotta feed, and the less data you have, the more expert knowledge you actually gotta feed. But, by the definition of ML, you'll never have any of them be exactly none. If you have a function on no data, thats not approximation, thats not ML. If you have no expert knowledge... well then you don't even have an approximation traversal strategy so you can't even approximate it.
So in viewing the field like this, you can really see how ML has branched from a field of statistics. And that's where I'm interested... how can we best optimize the function for certain datasets and problem settings. In the ideal finite potato case, we can just have the function mapping just be random without replacement, but of course that's just an ideal...