中文版本

Opportunity  

Self-supervised pre-training, particularly using masked language modeling (MLM) strategies, has become a cornerstone in fields like natural language processing (NLP) and has been adapted for graph neural networks (GNNs) to learn from molecular graph data. However, existing masking strategies, such as Attribute Masking (AttrMask), often employ random masking, which leads to a critical problem: the imbalanced distribution of atoms within molecular datasets. In typical molecular datasets, common atoms like carbon, oxygen, and nitrogen constitute the vast majority (e.g., ~96%), while trace elements (e.g., chlorine, fluorine) appear infrequently. Random masking disproportionately selects high-frequency atoms for masking, causing the pre-trained model to focus excessively on learning representations of these common atoms while neglecting rare but potentially chemically significant elements. This imbalance limits the model's ability to capture comprehensive chemical knowledge, as trace elements can play crucial roles in molecular properties and functions. Although some model-centric approaches, like MOLE-BERT and GraphMAE, attempt to address this by adding complex modules (e.g., tokenizers or reconstruction layers), they increase computational burden and parameter count. Thus, there is a clear need for a more efficient, data-centric solution that directly tackles the atom imbalance issue without adding model complexity, thereby improving the pre-training of GNNs for downstream molecular prediction tasks like drug discovery and property forecasting.

Technology  

The invention introduces a data-centric weighted random masking strategy for each molecule (WMM) to address atom imbalance during pre-training. Instead of masking atoms randomly, the method assigns a weight to each atom based on its atomic type's frequency within the specific molecule. The weight is calculated using a defined equation: \( w_a(i) = \frac{\ln(k(n_a(i) + 1))}{n_a(i)} \), where \( n_a(i) \) is the number of atoms of type \( a(i) \) in the molecule, and \( k \) is a hyperparameter (typically ≥0.8). This weighting ensures that atoms of types with higher counts (e.g., carbon) receive lower weights, reducing their probability of being masked, while rarer atom types receive higher weights, increasing their masking likelihood. This approach balances the learning signal by encouraging the model to predict less frequent atoms, thereby capturing more diverse chemical information. The method integrates seamlessly with existing self-supervised learning frameworks like AttrMask, GraphMAE, or Masked Atoms Modeling (MAM) without requiring additional model parameters. During pre-training, the GNN processes the masked molecular sample, predicts the masked atoms, and compares predictions with the original atoms through iterative training. This strategy maximizes learning from each molecule's unique atomic composition, enhancing the model's ability to generalize across varied molecular structures and improving performance on downstream tasks.

Advantages  

• Addresses atom imbalance directly through a data-centric weighting strategy, improving learning of rare but critical elements.
• Enhances pre-training performance without increasing model parameters or computational complexity, maintaining efficiency.
• Compatible with various existing self-supervised learning models (e.g., AttrMask, GraphMAE, MAM), offering flexibility and ease of integration.
• Improves overall accuracy in downstream molecular property prediction tasks, with experimental results showing performance gains (e.g., 1.38% improvement with AttrMask and 0.89% with GraphMAE).
• Provides a tunable hyperparameter (k) to control masking probabilities, allowing optimization for different datasets and tasks.
• Reduces overfitting to high-frequency atoms by making pre-training tasks more challenging, leading to better generalization.
  
  

Applications  

• Drug discovery and design: Pre-trained GNNs can predict molecular properties, aiding in identifying potential drug candidates.
• Molecular property prediction: Applications include toxicity (e.g., ClinTox), solubility, and bioactivity forecasting for compounds.
• ADMET (Absorption, Distribution, Metabolism, Excretion, Toxicity) profiling: Enhancing models for pharmacokinetic and safety assessments in pharmaceutical development.
• Chemical informatics and material science: Accelerating the discovery of new materials with desired properties through improved molecular representations.
• Protein sequence pre-training: The method can be extended to biological sequences (e.g., proteins) to handle imbalanced amino acid distributions.
• Educational and research tools: Providing robust pre-trained models for academic and industrial research in chemistry and biology.