We also test out-of-distribution generalization. Even when a token is spatially restricted during training, the network can bind it to new OOD positions at test time. This points to a flexible binding operation, rather than reliance on fixed label-location co-occurrences. 8/

Interventions show that the in-context memory is both plastic and stable. The network can add new token-location bindings late in a scene sequence, even after extensive exposure. Yet, overwriting an existing binding is gradual, suggesting richer memory dynamics than a simple dictionary. 7/

Decoding analyses reveal two key ingredients: Path integration: the network represents absolute position despite receiving only relative displacements. Object-location binding: token identity and position are represented together, not merely as separately decodable variables. 6/

Scenes contain a few letter tokens placed in continuous 2D space. A recurrent neural network (GRU) sees the current token plus a saccade-like displacement, and must predict the token it will encounter next. 3/

If a system learns to predict what it will sense next, conditioned on its own actions, can it acquire a structured model of its ‘world’? We study this in a minimal setting using small recurrent neural networks: no semantics, no visual statistics, just in-context action-conditioned prediction. 2/

Today I will present our work on CorText and how to fuse neural data with LLMs in the MedARC Journal Club! I’m thankful for the invitation and looking forward! 🧠🌸 Join online: 2:15 PM UTC meet.google.com/bof-ikcz-ygh

My first time at #Cosyne2026 in beautiful Lisbon! I’ll be presenting my work on CorText (brain-language fusion) in the “AI for Interpretable Model Discovery in Neuroscience” workshop on Tuesday 17th (09:30). Looking forward! Reach out to me if to chat about neuroAI and brain foundation models 🧠

Returning home inspired after a great visit to KU Leuven, where I presented our work on CorText in the Brain & Cognition group. Thank you for the invitation and great discussions! @hansopdebeeck.bsky.social @costantinoai.bsky.social (pictured: the magnificent architecture of the Liège station)

CorText also responds to in-silico microstimulations in line with experimental predictions: For example, when amplifying face-selective voxels for trials where no people were shown to the participant, CorText starts hallucinating them. With inhibition we can "remove people”. 7/n

Following Shirakawa et al. (2025), we test zero-shot neural decoding: When entire semantic categories (e.g., zebras, surfers, airplanes) are withheld during training, the model can still give meaningful descriptions of the visual content. 6/n

What can we do with it? For example, we can have CorText answer questions about a visual scene (“What’s in this image?” “How many people are there”?) that a person saw while in an fMRI scanner. CorText never sees the actual image, only the brain scan. 5/n

Introducing CorText: a framework that fuses brain data directly into a large language model, allowing for interactive neural readout using natural language. tl;dr: you can now chat with a brain scan 🧠💬 1/n

Warren McCulloch

Wow, peak library experience at Princeton! Looking forward to a week of the “Automated Scientific Discovery of Mind and Brain” workshop - where I will also present my work on CorText and brain-language fusion 🧠