Greater than the Sum of Its Parts: Building Substructure into Protein Encoding Models
Abstract
Protein representation learning has advanced rapidly with the scale-up of sequence and structure supervision, but most models still encode proteins either as per-residue token sequences or as single global embeddings. This overlooks a defining property of protein organization: proteins are built from recurrent, evolutionarily conserved substructures that concentrate biochemical activity and mediate core molecular functions. Although substructures such as domains and functional sites are systematically cataloged, they are rarely used as training signals or representation units in protein models. We introduce Magneton, an environment for developing substructure-aware protein models. Magneton provides (1) a dataset of 530,601 proteins annotated with over 1.7 million substructures spanning 13,075 types, (2) a training framework for incorporating substructures into existing protein models, and (3) a benchmark suite of 13 tasks probing representations at the residue, substructural, and protein levels. Using Magneton, we develop substructure-tuning, a supervised fine-tuning method that distills substructural knowledge into pretrained protein models. Across state-of-the-art sequence- and structure-based models, substructure-tuning improves function prediction, yields more consistent representations of substructure types never observed during tuning, and shows that substructural supervision provides information that is complementary to global structure inputs. The Magneton environment, datasets, and substructure-tuned models are all openly available (https://github.com/rcalef/magneton/).
Summary
This paper addresses the limitation of current protein representation learning models that primarily encode proteins as per-residue sequences or single global embeddings, overlooking the crucial role of recurrent, evolutionarily conserved substructures (domains, functional sites) in protein organization and function. The authors introduce Magneton, a comprehensive environment designed to develop substructure-aware protein models. Magneton includes a large-scale dataset of 530,601 proteins annotated with over 1.7 million substructures of 13,075 types. They also provide a training framework and a benchmark suite of 13 tasks to evaluate representations at residue, substructure, and protein levels. The core of their approach is "substructure-tuning," a supervised fine-tuning method that distills substructural knowledge into pre-trained protein models. This involves classifying evolutionarily conserved substructures using residue-level embeddings pooled to construct substructure representations. They systematically evaluate this tuning method across various state-of-the-art sequence- and structure-based models. Key findings include improved function prediction, more consistent representations of substructure types (even those unseen during tuning), and the demonstration that substructural supervision is complementary to global structure inputs. This research matters because it provides a novel framework and methodology to explicitly incorporate biological knowledge about protein substructures into protein encoding models, potentially leading to more accurate and interpretable protein representations.
Key Insights
- •Magneton provides a large-scale, curated dataset of 530,601 proteins with over 1.7 million substructural annotations spanning 13,075 types, addressing the lack of curated datasets for substructure-aware protein modeling.
- •Substructure-tuning, a supervised fine-tuning method, improves protein function prediction in downstream tasks such as Enzyme Commission (EC) prediction (e.g., with ESM-C 300M, Fmax improves from 0.688 to 0.815) and Gene Ontology molecular function prediction (e.g., with ESM-C 300M, Fmax increases from 0.429 to 0.525).
- •Substructure-tuning results in models that produce more consistent representations of substructures of the same type, even for substructure types *never* observed during training, as shown by increased silhouette scores (Table 7). For example, the silhouette score for "unseen" Homologous Superfamilies with ESM-C 300M increases from 0.180 to 0.584.
- •Substructure-tuning provides information complementary to global protein structure, as evidenced by performance improvements even in models like SaProt and ProSST-2048 that already incorporate structural information.
- •The gradient conflict analysis suggests that the task-specific effects of substructure-tuning might be due to biasing the model against fine-grained residue-level distinctions by encouraging similar representations for residues within the same substructure.
- •The study found that aggressive task-specific finetuning can attenuate the benefits of substructure-tuning, suggesting that a more integrated approach is needed to incorporate substructural information effectively.
- •Elastic Weight Consolidation (EWC) moderates the performance improvements of substructure-tuning while reducing performance degradation in tasks where substructure-tuning has negative effects.
Practical Implications
- •Magneton provides a valuable resource for researchers and engineers working on protein representation learning, offering a standardized dataset, training framework, and benchmark suite.
- •The substructure-tuning method can be directly applied to existing pre-trained protein models to improve their performance on function-related tasks, such as enzyme function prediction and gene ontology term prediction.
- •The findings suggest that future research should focus on developing novel architectures and training objectives that more deeply integrate substructural information into protein models, potentially through hierarchical or graph-based approaches.
- •The observed complementarity between substructural and global structural information suggests that combining these sources of information could lead to even more powerful protein representations.
- •The limitations of the current substructure-tuning approach highlight the need for methods that can better balance the representation of residue-level details and substructural information, possibly through multi-task learning or regularization techniques.