Weight-Space Programming: An Experiment Where LLMs Directly Specify Neural Network Weights

From AI Radio (English) · 1:52

Show by AI Radio

In machine learning, learning is an iterative process. You feed in vast amounts of data, calculate the errors, and gradually adjust the weights. One experiment posed a fundamentally different question to this conventional wisdom. What if you had an LLM directly write out the network weights as numerical values from the beginning? For the MNIST handwritten digit classification task, researchers had it specify approximately six thousand parameters from scratch. Without using gradient descent or optimizers, the AI purely thought through and determined the weights itself. The accuracy it achieved was about ninety percent, roughly equivalent to the results of standard training over fifty steps. But what was even more striking was the actual strategy the AI chose to employ. Edge detection, template matching, probability-based scoring. These were all techniques established in computer vision research from the nineteen nineties. Because the LLM has absorbed decades of research papers, it was able to translate that knowledge into the form of weights. Rather than inventing entirely new methods, it reproduced the correct solutions that humans had previously discovered. Researchers have started calling this LLM Descent, viewing it as a candidate for a new paradigm where weights are determined through conceptual understanding rather than numerical gradients. However, in its current state, a clear ceiling has also emerged. The AI only works within the scope of solutions it already knows. If the LLM can substitute for the act of learning itself, it might begin with reproducing existing knowledge and perhaps one day move toward transcending knowledge altogether. But that answer still lies ahead, beyond the experiment.

More episodes of AI Radio (English)

More episodes

PodSearch is a free, independent podcast search engine. Read about PodSearch, the privacy policy or contact us.