We're excited to see how the community will continue to use MSA Pairformer as a tool to study and engineer biology. Please reach out if there's anything we can help out with! Credit to Olivia Tang for this cool illustration of our project👩‍🎨 (9/10)

Need additional proof that MSA Pairformer can adapt to sequences outside of the training set? Here we replaced A-U-C-G with 4 amino acids in a tRNA alignment and find that MSA Pairformer can accurately predict contacts between ribonucleic acids. (8/10)

Here, we show that MSA Pairformer more accurately predicts contacts than ESMC 6B, which became available after revisions. Rather than scaling model size and training data (à la bitter lesson), MSA Pairformer leverages the rapidly growing scale of sequence data during inference. (7/10)

A key highlight of MSA-based pLMs is their ability to adapt to newly available sequences at inference time. With @eunbelivable.bsky.social, we demonstrate large improvements in modeling the evolution of low-homology sequences by searching the Logan database. (6/10)

MSA Pairformer weights every sequence's contribution to the pairwise representation by its inferred evolutionary relevance to the query sequence. This mechanism enables it to accurately model the evolution of homo-oligomerization by disentangling subfamily-specific binding interfaces. (5/10)

Modeling the evolution of PPIs requires accurately pairing interacting sequences between protein families. This is confounded by paralogous genes. With Olivia Tang, we explored using MSA Pairformer to pair sequences by optimizing their pseudolikelihoods. (4/10)

Its improvements in modeling the coevolution of interacting proteins leads to improvements in its ability to predict the effects of mutations at the interface of protein complexes. (3/10)

Although MSA Pairformer trained on multiple sequence alignments (MSAs) of individual protein families, it generalizes to protein-protein interactions (PPIs) at inference time. We see this in highly accurate contact prediction at PPI interfaces. (2/10)

Special thanks to all members of our team! Their mentorship and support are truly world-class. And a huge shoutout to the entire solab! I'm so grateful to work with these brilliant and supportive scientists every day. Keep an eye out for exciting work coming out from the team!

Thanks for tuning in--we've already received incredibly valuable feedback from the community and will continue to update our work! We're excited for all of MSA Pairformer's potential applications for biological discovery and for the future of memory and parameter efficient pLMs

We made some updates to MSA pairing with MMseqs2 for modeling protein-protein interactions! Mispairing sequences leads to contamination of non-interacting paralogs. We use genomic proximity to improve pairing, and find that MSA Pairformer's predictions reflect pairing quality

We also looked into how perturbing MSAs effects contact prediction. Interestingly, unlike MSA Transformer, MSA Pairformer doesn't hallucinate contacts after ablating covariance from the MSA. Hints at fundamental differences in how they extract pairwise relationships

We ablate triangle updates and replace it with a pair updates analog. As expected, contact precision deteriorates, and the false positives are enriched in indirect correlations. These results suggest the role of triangle updates in disentangling direct and indirect correlations

Whereas the ESM2 family models show an interesting trade-off between contact precision and zero-shot variant effect prediction, MSA Pairformer performs strongly in both P.S. this figure slightly differs from what's in the preprint and will be updated in v2 of the paper!

Using a library of mutants at four key ParD3-ParE3 toxin-antitoxin interface residues from Aakre et al. (2015), we find that MSA Pairformer's pseudolikelihood scores better discriminate binders and non-binders, directly related to its ability to model the interaction

Beyond monomeric structures, accurate prediction of protein-protein interactions is crucial for understanding protein function. MSA Pairformer substantially outperforms all other methods in predicting residue-residue interactions at hetero-oligomeric interfaces

On unsupervised long-range contact prediction, it outperforms MSA Transformer and all ESM2 family models, suggesting that its representations more accurately capture structural signals from evolutionary context

We introduce MSA Pairformer, a 111M parameter memory-efficient MSA-based protein language model that builds on AlphaFold3's MSA module to extract evolutionary signals most relevant to the query sequence via a query-biased outer product