nic 🔹
@nmk
mathematician, quant intern, reader of things I give 10% of my income to GiveWell and so should you
I have now been able to further bring down validation loss by optimizing the number of heads, swapping softmax for softmax1 and normalizing the Q and K matrices.
I also tried every other combination of wiring. It turns out you really want QK from the residual stream, and V from the input.
This way the model doesn't have to actively remember everything that was in the input because it will get another look at it later. You can see this in the logit lens, where relative to a vanilla model the model delays the actual prediction.
Over the past week I trained a bunch of picoGPTs (~50M params) on single GPUs trying to get minimal loss with a fixed number of epochs. The biggest architectural impact I found was replacing some of the transformer Value tensors with Value tensors that read from the original input.
While its a good meme, I do think its important to point out that anthropic has a more precise policy position that this.
"My undergraduates can not only prove √2 and √3 are irrational but also that √4 is irrational."
In this model the cars are allowed only to dodge against the flow of the other traffic.
Switching between the deterministic and randomized update rule is extremely mesmerizing!
Woah, just played around with both, and it turns out their behavior is shockingly different!
It is also not just underpowered models. I trained Neural Networks and they exhibit the same failure mode.
This is also robust to the training period of the data. (Note that with the larger training period we don't have House YoY, so you will notice that mortgage becomes more important)