Editing Blog/2025-10-12/Word Magic (section)

=== The LLM Connection ===

Given the current paradigm shift underway of LLMs, I can't help but be reminded of the way that these neural networks operate. There is a similarity to their functionality in two ways: intermediate layers and chain of thought. Starting with the assumption that Peter Thiel's success is from being able to better model the world and from the assumption that he isn't writing essays because he is trying to deceive you into something, I think I see a parallel<ref name=assumptions/>.

==== A False Start ====

The first thought I had was about intermediate neuron layers and attempting to either slice off various layers of a network or to simply terminate the inference at some point. Intermediate neuron layer outputs are not meaningful on their own. You could interpret them as text after feeding some subset of them to a later layer in some combination and then 'convert to text' but that doesn't mean anything if the whole model was trained on something and you've just shaved off the last few layers. This doesn't seem particularly useful to think about, though, because nothing here fits well enough together. What does it mean to take out a layer? Where do the outputs go? Which ones do you link to which later ones with what weights? What does it mean to 'convert the vector to text'? Ironically, this seems like a nonsensical thought cul-de-sac.

==== Illegible Chains of Thought? ====

The second seems closer to what we see here: LLM chains of thought can often be entirely nonsensical and yet help them conclude the right things:

{{Blockquote
|text=(Dimethyl(oxo)-lambda6-sulfa雰囲idine)methane donate a CH2rola group occurs in reaction, Practisingproduct transition vs adds this.to productmodule. Indeed"come tally said Frederick would have 10 +1 =11 carbons. So answer q Edina is11.
|author=Arun Jose + DeepSeek R1<ref name=weird-cot-pb/>
|source=Will Chains of Thought Stay Readable for Long, The Information<ref name=cot-nonsense/>
}}

Chain-of-thought style LLM reasoning is therefore not quite the same as the kind of logical reasoning we purport to employ as people. It isn't legible in the form that it is written there and one cannot use that intermediate trace in any reasonable way to produce outputs. One can't even use that trace in another LLM to produce a useful output. So what if we are like that in our internal thoughts. We are more constrained by our language to not produce the illegible seeming-nonsense that DeepSeek R1 did there but what if our intermediary thoughts are just as wild pattern matching exercises that go through later transformations to finally determine our actions? If we then tried to reveal the pattern of our thoughts, it might look a bit like Thiel's essay: unrelated relationships drawn together to form some sort of coherent thread through time telling us what to do and what to predict is about to happen.