Astra weight-space programming
The notion of weight-space programming holds that top-tier large language models are able to read, author and modify neural network weights, and a post on notesbylex.com looks at how that plays out when the model is the one producing the weights.12
According to the original post, its authors asked GPT-6 Astra to construct a network that plays Go starting from nothing, and then to make it stronger by running practice games, studying the games it lost, and adjusting the weights accordingly.3 The Go network had 1.5M parameters, all set to zero; it was run four times with budgets of 0, 10, 25 and 50 practice-game passes, each pass playing up to 128 practice games.45 With 50 passes, the Astra-built network beat two small PPO-trained networks in over 80% of games; even with no practice games it won over half its games against the smaller networks.67
In the author's own MNIST test, Claude was prompted to write code that sets all of a network's weights from scratch, with rules including no fitting, regression or searching; its first network scored 71% on the test set, rising to 77% after the digits the confusion matrix flagged as wrong were redrawn.89
The author argues that Astra's Go network is essentially a rule-based Go program compiled into weights, built by a play, diagnose and add-or-retune-a-rule loop much like how rule-based Go programs were made for decades.1011 The author concludes that, from these experiments, LLMs are less "writing in weight-space" than chaining together hand-designed techniques, and notes the limits of hand-crafted features show on harder problems like cats vs dogs.1213
Sources quoted
The basic idea is that frontier LLMs can read, write and edit the weights of neural networks.
— notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcherIn this article, I run my own weight-writing experiment with Claude Fable 5.2 on a familiar image classification problem.
— notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcherthe authors had GPT-6 Astra build a network to play Go from scratch, then improve it by playing practice games, and editing the weights based on analysis of its losses.
— notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcherAstra was given a 9x9 Go network with 1.5M parameters, all set to zero.
— notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researchereach pass plays a batch of up to 128 practice games and shows it the results. The authors ran it four times, with budgets of 0, 10, 25 and 50 passes.
— notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcherWith 50 passes, the Astra-built network beat two small networks trained with reinforcement learning (PPO) in over 80% of games
— notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher- Quote not reprinted: this page already repeats as much of this article as we allow. — notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher
- Quote not reprinted: this page already repeats as much of this article as we allow. — notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher
- Quote not reprinted: this page already repeats as much of this article as we allow. — notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher
- Quote not reprinted: this page already repeats as much of this article as we allow. — notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher
- Quote not reprinted: this page already repeats as much of this article as we allow. — notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher
- Quote not reprinted: this page already repeats as much of this article as we allow. — notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher
- Quote not reprinted: this page already repeats as much of this article as we allow. — notesbylex.com, Weight-Space Programming builds neural networks like a 1990s researcher
Who reported it
- Hacker Newscommentary: Weight-Space Programming builds neural networks like a 1990s researcher
- Hugging Face Daily Papersrepeats: Weight-Space Programming Doesn't Need Transformers
- notesbylex.comown facts: Author reviews the paper's Go network, which Astra built from scratch, finding it is a rule-based program compiled into weights.
Facts and takes
Takes: 3 views from 1 holders · 1 positive · 1 mixed · showing 1
- mixednotesbylex.com: Quote not reprinted: this page already repeats as much of this article as we allow.
Timeline
- · Hacker News: Weight-Space Programming builds neural networks like a 1990s researcher
- · notesbylex.com: Weight-Space Programming builds neural networks like a 1990s researcher
- · Hugging Face Daily Papers: Weight-Space Programming Doesn't Need Transformers