Predicting and designing proteins (4): What protein language models learn from sequences

16 Jul, 2026

Calling proteins a language is useful only if the analogy is handled carefully. Amino acids are not words with dictionary meanings, and evolution does not compose sentences with an intention. A protein sequence is a physical polymer that must be synthesized, folded, and selected in an organism. Yet sequences do have context-dependent regularities. Some residue combinations are common, some are incompatible, and a residue that is acceptable in one region may be destructive in another. A model trained to predict missing residues can learn those regularities without being given a separate label for every structure or function.

A protein language model (PLM) is trained on amino-acid sequences using objectives adapted from natural-language processing. In masked modelling, some residues are hidden and the model predicts them from their context. In autoregressive modelling, the model predicts the next residue from the preceding sequence. Training at large scale produces internal numerical representations, or embeddings, that can be reused for other tasks.

The model treats each amino-acid symbol as a token, the basic item it receives or predicts. An embedding converts that discrete symbol into a list of numbers, much as coordinates locate a point in a multidimensional space. During training, residues that participate in similar contexts tend to acquire representations that downstream calculations can use in similar ways. The individual dimensions usually have no simple dictionary definition; the information is distributed across the whole vector and across layers of the network.

Why sequence databases invited self-supervised learning

Experimental structures are expensive and comparatively scarce. Sequences are vastly more abundant because genome sequencing reads an organism’s DNA, while metagenome sequencing reads the mixed DNA in an environmental sample; both can reveal encoded proteins without purifying and imaging each one. This imbalance suggested a self-supervised strategy: construct training labels from the sequences themselves. Every masked or next-residue prediction becomes a small learning problem.

“Self-supervised” means that no scientist needs to annotate each example with a fold or function before training. The original sequence supplies both the question and the answer: hide a residue to create the question, then use the actual residue as the answer. Billions of such automatically constructed exercises can teach broad regularities, although they cannot by themselves tell the model which regularities are causal for a particular biological function.

The labels are cheap, but the constraints they reflect are not trivial. Natural sequences have passed through evolution. They must usually be expressible, compatible with a cellular environment, and adequate for reproduction in a particular organism. A language model therefore learns a mixture of structural, functional, phylogenetic, and sampling regularities. It does not learn a pure law of biophysics.

UniRep provided an early influential demonstration that representations learned from unlabeled protein sequences could support engineering tasks with limited labelled data.¹ TAPE then supplied benchmark tasks for evaluating transfer learning across secondary structure, contacts, remote homology, fluorescence, and stability.² These studies helped turn “learn a general protein representation” into an experimentally comparable research programme.

UniRep used a recurrent architecture rather than a Transformer. A multiplicative long short-term memory network read the chain from left to right. At position $i,$ it combined the current residue with a hidden state $h_{i-1}$ carrying information from the preceding positions, produced a new state $h_i,$ and learned to predict the next residue. Averaging the hidden states across a completed sequence produced a fixed-length representation that a comparatively small supervised model could use for stability or activity prediction. The recurrence made sequence order explicit, but information from two distant positions had to pass through every intervening update. Transformer attention later supplied a more direct route between them.

What emerged as models and datasets scaled

The Evolutionary Scale Modeling project trained transformers on hundreds of millions of sequences. ESM-1b showed that structural and functional features emerged as masked-language modelling scaled, even though the training objective did not explicitly require three-dimensional coordinates.³ ProtTrans found similar transfer across several transformer families and large protein corpora.⁴

A transformer is a neural-network architecture built around repeated attention and feature-transformation layers. Unlike a method restricted to a short window, it can allow a residue near one end of the sequence to influence the representation of a residue far away. That ability is relevant because distant sequence positions may become neighbours after folding. It is still not a direct simulation of folding: the model learns statistical dependencies in its training sequences.

“Emergence” here means that a relatively simple readout could recover information from the model’s internal states. It does not mean that the model independently discovered every biological concept. The training sequences already embody evolution’s constraints, and the model architecture is designed to combine context across a chain.

The relation to an MSA is especially important. An alignment collects the relatives of one query and computes family-specific evidence at inference time. A single-sequence PLM compresses patterns across many families into model parameters during training. The PLM is fast to apply and useful when explicit relatives are scarce, but its finite parameters cannot preserve every detail that a deep query-specific alignment may contain. MSA Transformer explored a middle position by applying transformer learning directly to aligned families.⁵

The two approaches are therefore complementary forms of evolutionary inference. An MSA resembles consulting a dossier assembled for one family. A PLM resembles consulting a researcher who has internalized recurring patterns from many dossiers. The second is faster to ask, but the first may contain unique evidence.

What a transformer layer computes

Before entering the transformer, a sequence of length $L$ becomes a matrix $H\in {ℝ}^{L\times d}.$ Each row is the current $d$-number representation of one residue, augmented with information about its position. In one attention head, three learned matrix multiplications turn $H$ into queries $Q,$ keys $K,$ and values $V.$ The central operation is

The product $Q{K}^{\mathsf{T}}$ is an $L\times L$ table: every residue receives a compatibility score with every other residue. Softmax converts each row into weights that sum to one, and the final multiplication forms a weighted mixture of the value vectors. Multiple heads perform this calculation with different learned projections; a feed-forward network then transforms each position separately. Residual connections add earlier features back into later ones, and normalization prevents numerical scale from drifting. Stacking these operations lets a residue representation incorporate increasingly indirect context.

For masked training, a set $M$ of selected residues is replaced by mask symbols and the last layer produces twenty amino-acid logits—unnormalized scores—at each selected position. Softmax converts them into probabilities. The training loss is the sum of the negative log-probabilities assigned to the residues that were actually hidden:

Here $s_i$ is the true amino acid at position $i,$ $s_{⧵M}$ is the sequence with all selected positions masked, and $\theta$ denotes the learned network parameters. Minimizing the loss is therefore concrete: change the attention and feed-forward weights so that the correct residue receives more probability whenever similar contexts recur. Because the model can inspect residues on both sides of a mask, this is not the same factorization as predicting a sequence from left to right; the conditionals at separate masked positions do not by themselves constitute one autoregressive probability for the complete sequence.

The sequences used by ESM and related projects came largely from clustered UniProt/UniRef and metagenomic collections: enormous public archives, but not an even sample of life. Clustering reduces repetition, while taxonomic and research biases remain.³⁴ Training billions of such predictions stores recurring sequence contexts in the matrices of the network. It does not store an explicit alignment for a new query.

This implementation creates two important limits. The attention table grows with $L^2,$ so long proteins consume much more memory. More fundamentally, the loss rewards reconstruction of natural sequences. A model can reduce it by learning ancestry, cellular localization signals, or database composition as well as three-dimensional constraints. Structural features that emerge are useful because structure helps explain sequences, but the objective never asks the network to separate those causes.

How ESMFold connected language to coordinates

ESMFold tested whether a large sequence model’s representations could support atomic structure prediction without a query-time MSA. It connected ESM-2 embeddings to an AlphaFold-inspired folding trunk and structure module.⁶ For many proteins, the resulting single-sequence predictions were useful and much faster to obtain than alignment-heavy predictions. This enabled structural annotation at metagenomic scale.

Concretely, ESMFold reads the per-residue embeddings from several ESM-2 layers and also uses the language model’s attention maps as pairwise evidence. Learned projections convert these outputs into a single-residue array and an $L\times L$ pair array. A folding trunk adapted from AlphaFold/OpenFold repeatedly updates the two arrays, and a structure module turns them into residue frames, torsion angles, atoms, and confidence estimates. The language model is therefore not asked to emit coordinates directly: it replaces the query-time MSA as the source of evolutionary features, while a geometric network performs the final folding computation.⁶

ESMFold is often compared with AlphaFold2 as if one should replace the other. Their trade-off is more informative than a simple ranking. AlphaFold2 can exploit a deep, target-specific MSA and is usually stronger on difficult targets for which that evidence is informative. ESMFold pays the cost of evolutionary compression during pretraining, then offers rapid inference from one sequence. The appropriate choice depends on target difficulty, available homologues, throughput, and the consequences of error.

The model also changed how PLMs could participate in design. A fast structure predictor can filter thousands of generated sequences for agreement with an intended fold. That use is valuable, but predictor agreement is not an independent experiment. If both the generator and evaluator learned from overlapping sequence and structure distributions, correlated errors may appear convincing.

How likelihood became a biological score

The masked model already produces the quantity needed for a mutation comparison: a conditional distribution over amino acids at a hidden position. ESM-1v masks every mutated position in a set $M$ and compares the log-probability of the mutant residues with that of the wild-type residues:⁷

A positive term means that, in the same masked context, the language model prefers the mutant amino acid; a negative term means that it prefers the wild type. Summing the terms assumes that the mutation contributions can be combined additively at the scoring stage, even though the Transformer used the shared surrounding sequence to produce them. This masked log-odds score made zero-shot mutation prediction possible across diverse proteins without task-specific training, but it remains a ranking derived from sequence compatibility rather than a calibrated change in activity, stability, or fitness.

“Zero-shot” means that the model is applied to the new prediction task without first being trained on labelled examples from that task. It uses regularities learned during broad pretraining to compare the original and mutated sequences. This is attractive when experiments are scarce, but the score is an extrapolation from general sequence statistics; it is not evidence that the model has observed the mutation’s cellular consequences.

Related family-specific generative models preceded the large PLMs. EVmutation used an evolutionary statistical energy, while deep latent-variable models learned family distributions and captured mutation effects. EVE scaled the latter approach to disease-associated genes.⁸ AlphaMissense later combined protein-language representations, structural context, and supervised calibration for proteome-wide missense-variant classification.⁹

These outputs should be named precisely. A language-model score measures compatibility with a learned sequence distribution. A pathogenicity model estimates a clinical label under its training data. Neither alone identifies the causal mechanism of disease. Paralogue context, cell type, dominance, compensation, and ascertainment all intervene between a residue substitution and a patient phenotype.

When a sequence model becomes a generator

Before large autoregressive PLMs, family-specific latent models showed that sequence distributions could be sampled and navigated. DeepSequence used a variational autoencoder. Its encoder produced a distribution $q_{\varphi }(z\mid s)$ over a continuous latent vector $z$ for an aligned family sequence $s,$ while its decoder assigned probabilities $p_{\theta }(s\mid z)$ to sequences generated from that vector. Training maximized an evidence lower bound:

The first term rewards a latent code that lets the decoder reconstruct family members. The second prevents the encoder from assigning every sequence an isolated code unrelated to a shared prior $p(z).$ That trade-off creates a continuous family-level landscape from which sequences can be compared or sampled, and the learned distribution also captured mutation effects.¹⁰

ProteinGAN used a different training signal to expand an enzyme family and yielded active generated variants in experiments.¹¹ A generator proposed sequences while a discriminator learned to distinguish generated examples from natural training examples; the generator improved by becoming harder to distinguish. Unlike the variational autoencoder, this adversarial arrangement did not require an encoder to map each natural sequence into a regular latent distribution or an explicit likelihood for every sequence. The competition encourages realistic-looking samples, although looking natural to the discriminator does not guarantee physical function. These systems offered relatively narrow family expertise rather than one model spanning protein space, but they established that novelty and function could coexist in generated sequences.

An autoregressive PLM uses a third factorization. It assigns the probability of a complete sequence by multiplying next-residue probabilities:

where $s_{<i}$ contains only the residues already generated. This makes training and generation use the same direction of information flow: during training the true prefix supplies the context, while during generation a sampled residue is appended to the prefix before the next prediction. ProtGPT2 used this mechanism to generate diverse protein-like sequences without conditioning on a specified structure.¹² ProGen added control tags associated with properties or families and produced experimentally functional generated sequences.¹³ ESM3 later brought sequence, discretized structure, and function annotations into one generative transformer. A user can provide some of these modalities and ask the model to complete the others. The experimentally characterized fluorescent protein esmGFP demonstrated generation at considerable evolutionary distance from known fluorescent proteins.¹⁴

This progression illustrates a general conversion from prediction to generation. A model trained to predict the next element defines a probability distribution; sampling from that distribution generates new sequences. The difficult question is control. A statistically plausible sequence is not guaranteed to adopt a precise backbone, bind a specified epitope, position catalytic residues correctly, or remain soluble at manufacturing concentrations.

Sampling means repeatedly drawing one allowed residue according to the probabilities the model assigns, appending it, and asking for the next set of probabilities. A temperature $T$ changes the sharpness of this choice without retraining the model. If $z_{i,a}$ is the logit for amino acid $a$ at position $i,$ then

For $T<1,$ the largest logits dominate and sampling is more conservative. For $T>1,$ the distribution flattens, so lower-ranked residues are selected more often and diversity rises together with the risk of failure. Temperature changes the sampling policy, not what the network learned. These probabilities still summarize resemblance to the learned data distribution; they are not folding free energies, fitness values, or laboratory success probabilities unless a separate study shows that they are calibrated for that purpose.

ESM3 addresses some of that gap by coupling modalities, but discretized structure tokens still approximate continuous atomic geometry. Its impressive examples establish what the system can do; broad prospective experiments are needed to establish how often it does so across targets.

What ESMFold2 suggests about the next scale

The ESMC–ESMFold2 release reported in 2026 extends this lineage. ESMC was trained at scales up to six billion parameters on approximately 2.8 billion sequences. ESMFold2 uses its representations to build pair features and employs atom-level diffusion for coordinate generation. The accompanying work reports protein–protein and antibody–antigen prediction, optional MSA augmentation, a 6.8-billion-protein atlas, and prospective binder experiments.¹⁵

The system is historically interesting because boundaries have blurred. A protein language model supplies evolutionary representations; a pair trunk performs structural reasoning; a diffusion module generates coordinates; and experimental design closes part of the loop. The label “language model” no longer describes the whole system, but it identifies the pretrained substrate from which the other capabilities grow.

Because the release is recent, its developer-reported benchmark advantages and design rates require independent reproduction. Training-cutoff analysis is particularly important when billions of sequences and predicted structures are involved. Scale increases coverage, but it also makes it harder to know what counts as a genuinely new test.

What the language analogy contributes

Protein language modelling changed the economics of learning from evolution. It made one pretrained model useful across mutation scoring, annotation, structure prediction, and generation. It also made sequence generation possible without choosing a backbone first.

Its limits point toward the next stage. Many design problems begin with a desired geometry: an interface that must fit a target, a pocket that must hold a ligand, or a catalytic arrangement that must position several atoms. For such tasks, the model must generate a sequence conditional on structure. This inverse problem became the practical bridge between backbone design and a gene that can be synthesized.

References

#Biology