NaFM: Pre-training a Foundation Model for Small-Molecule Natural Products [@ding_nafm_2025]

---
author: Jonas Schaub
date: 2025-04-23
bibliography: ../references.bib
---

# NaFM: Pre-training a Foundation Model for Small-Molecule Natural Products [@ding_nafm_2025]

## Why did I choose this paper?
- Was curious about what a "Foundation Model" in Natural Product Chemistry 
would be exactly, what could it be used for?

## Abstract
- Statement: existing deep learning methods for Natural Products (NP) research 
primarily rely on supervised learning approaches designed for specific 
downstream tasks
- Statement 2: one-model-for-a-task paradigm often lacks generalizability and 
leaves significant room for performance improvement
- Statement 3: existing models for chemical problems do not work well for NP
- This article describes NaFM ("Natural product Foundation Model"), a 
"pre-trained foundation model" for natural products
- Pre-training was done using masked graph learning and contrastive learning (see below)
- "Validation" of the approach was done in 3 experiments using NaFM as a (kind-of) encoder
and a Multi-Layer Perceptron (MLP) as a (kind-of) decoder to solve the following tasks:
  - NP taxonomy classification
  - Predicting biological sources of NP
  - Predicting bioactivity of NP

## Introduction
- NP-based drug discovery/design with traditional methods (NP extracts from (cultivated) organisms, 
bioactivity-guided fractionation, target validation, structure elucidation, etc.) is costly and time-consuming
- With today's computing resources and databases like COCONUT, LOTUS, and Supernatural II,
NP drug discovery can be sped up, perspectively
- Prior work:
  - [@xu_composite_2024]: predicting kingdom of NP origin species
  - [@stokes_deep_2020]: predicting antibiotic activities
  - [@kim_npclassifier_2021]: predicting structural classes and source pathways of NP
  - Mostly use traditional fingerprints like Morgan/ECFP (Extended Connectivity FingerPrints)
  - "these methods could achieve satisfying performance
    but always suffer when applied to out-of-distribution test data or less-related downstream tasks."
  - They exclusively employ supervised learning but this is problematic because we do not
    have a lot of labelled data, and it is often biased (e.g. plant NP are usually overrepresented in
  NP databases)
- Pre-training on unlabelled data:
  - Historically, often done with SMILES as data, RNN or Transformers as model architectures, and 
  masked language modelling as a pre-training strategy
  - Lately, Graph Neural Networks (GNN) as model architectures, hence graphs as input data, and 
  pre-training strategies specifically for molecular graphs seem promising
- But no pre-trained foundation model for NP has been developed so far
- Some thoughts on NP structures in general:
  - NP structures in general are more complex and diverse than synthetic compounds, 
  have specific scaffolds (= ring systems) and functional groups; which is why models trained on synthetic 
  compounds usually struggle with NP
  - But the advantage of NP structures is that they are "evolved", i.e. they often carry 
  "preserved" substructures like scaffolds which depend on / relate to the source pathway or source organism
  - And these scaffolds are related to the exhibited bioactivity
  - A NP-specific pre-training strategy should therefore focus on scaffolds

\
![Figure 1 D. The "central dogma" of natural products. The biological source, biosynthetic gene clusters,
biosynthetic pathways, and bioactivity of natural products are interconnected through the scaffold, which acts as a bridge linking
these three key aspects.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig1d.png){width="600" fig-align="center"}
\

- Contrastive learning, masked learning and activity cliffs
  - Contrastive learning: in the learned embeddings, similar structures should have similar embeddings,
  and dissimilar structures should have dissimilar embeddings 
  - Masked learning: substructures are "masked", i.e. bond or atom attributes are omitted and have to be predicted by 
  the model based on the context
  - Existing approaches use the masking of structures simply as data augmentation 
  and define similarity/dissimilarity for contrastive learning with pairs of the original structures
  - Activity cliffs: very similar structures can have completely different (bio-)activities
  - Activity cliffs contradict the principles of contrastive learning
  - Idea pursued to avoid this here: use the masked structures as "similar" 
  structures to their original, unmasked structures
  - Also: mask whole substructures, not just single atoms and bonds, to make pre-training more 
  complex and hence more successful

## Model Architecture and Pre-Training
- NaFM: a pre-training framework ("pre-trained foundation model"?) specific to NP
- Two novel pre-training approaches: scaffold-subgraph reconstruction and scaffold-aware contrastive learning.

\
![Figure 1 B. Details of scaffold-subgraph reconstruction. First, a subgraph is randomly selected from the scaffold, consisting
of multiple atoms and chemical bonds. In the subgraph, both node and edge attributes are masked, and all nodes within the
subgraph are fully connected (i.e., expanded into a fully connected graph), thereby masking the topological information. During the
reconstruction process, the model needs to predict both node and edge attributes, while also distinguishing between real edges and
those artificially added virtual edges.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig1b.png){width="600" fig-align="center"}
\

- scaffold-subgraph reconstruction:
  - (not really evident in the figure, but) masking employed only to substructures of the scaffold of an NP!
  - atom information (elements / atom types, formal charge, chirality) is masked
  - bond information (multiplicity, direction (stereochemistry)) is masked
  - topological information is masked, all bonds in the substructure are removed and instead, 
the atoms become fully connected

\
![Figure 1 C. Details of scaffold-aware contrastive learning. Different colors represent varying weights
defined by scaffold similarity.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig1c.png){width="600" fig-align="center"}
\

- scaffold-aware contrastive learning:
  - For every original input structure, its masked form is set as positive example of a "similar" structure
that should be encoded similarly
  - the other molecules and their masked forms are set as negative examples, so "dissimilar" structures that should be
encoded dissimilarly
  - scaffold similarity is used as a metric for similarity of the complete structures
    - cosine similarity between the MACCS fingerprints of scaffolds is 
      incorporated into the contrastive learning loss function
  - This way, scaffold-subgraph reconstruction and scaffold-aware contrastive learning are combined

\
![Figure 1 A. Overview of NaFM pre-training. After the natural product molecule undergoes scaffold-aware masking, both the masked
and unmasked information are simultaneously input into a multi-layer Graph Isomorphism Network. The masked information is
then processed through a projection head to predict the masked atom attributes, bond attributes, and topological information.
Meanwhile, the masked and unmasked information passes through a pooling function and is used for scaffold-aware contrastive
learning.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig1a.png){width="600" fig-align="center"}
\

- Training data: COCONUT NP (0.6 mio molecules) ([@chandrasekhar_coconut_2024])
- Model architecture: Graph Neural Network (GNN) / Graph Isomorphism Network (GIN) (6 layers, hidden dimensionality of 512)
- Molecule goes in once as original structure and once as scaffold-masked structure
- Average pooling operation as the readout function to extract a global molecular representation of 1024 dimensions
- Projection head (3 Multi-Layer Perceptrons (MLP)) has to reconstruct/predict the masked properties (atom properties, bond properties, topologies)
- a MLP has to embed the original structures and their masked forms (in 256 dimensions)
so that original-masked pairs are seen as similar and other structures as dissimilar, based on scaffold 
similarity (scaffold-aware contrastive learning)

### General Down-stream Fine-tuning Approach
- Original contrastive learning head and reconstruction head are replaced with a randomly initialized MLP
- For all pre-trained baselines (other models that NaFM is compared to below) the same settings as described above are used 
- For baselines trained from scratch, their default configurations are used.

## Performance Evaluation on Downstream Tasks
### NP Taxonomy Classification
- Data: NPClassifier dataset ([@kim_npclassifier_2021])
  - Biosynthetic pathway (7 types), super class (70), and class (563) annotations of 77,000 NP structures
- First experiment on class level
  - Multiple pre-trained (general, not NP-specific!) molecular models were used as baselines
  - Additionally, Extended Connectivity Fingerprints (ECFP) were used as baseline (authors state that they 
  are currently the most popular choice for encoding NP structures)
  - All baseline models, ECFP, and NaFM were used for training a MLP as described above for NP classification
  - Results in Table 1
  - NaFM shows top scores for all dataset sizes, especially for small datasets

Table 1 The AUPRC results for natural product taxonomy classification compared to other baselines.

| Training samples per Class | 4            | 8            | 16           | 24           | 40           | 64           |
|----------------------------|--------------|--------------|--------------|--------------|--------------|--------------|
| N-Gram                     | 44.72 ± 1.91 | 56.61 ± 0.66 | 66.36 ± 1.57 | 71.11 ± 0.77 | 73.34 ± 1.14 | 75.77 ± 0.54 |
| PretrainGNN                | 44.83 ± 0.82 | 61.85 ± 0.78 | 75.76 ± 0.62 | 80.50 ± 0.17 | 85.31 ± 0.36 | 87.82 ± 0.13 |
| D-MPNN                     | 46.63 ± 0.23 | 60.88 ± 0.48 | 75.73 ± 0.12 | 80.96 ± 0.24 | 86.64 ± 0.44 | 89.23 ± 0.79 |
| MolCLR                     | 45.76 ± 1.98 | 65.80 ± 1.51 | 78.14 ± 1.08 | 83.20 ± 0.74 | 85.56 ± 0.11 | 88.22 ± 0.34 |
| Mole-BERT                  | 66.32 ± 1.23 | 73.39 ± 0.66 | 78.25 ± 0.45 | 80.83 ± 0.51 | 83.57 ± 0.59 | 85.69 ± 0.70 |
| ECFP                       | 69.17 ± 0.19 | 78.21 ± 0.79 | 83.82 ± 0.37 | 86.28 ± 0.49 | 88.52 ± 0.48 | 89.75 ± 0.45 |
| GraphMVP                   | 64.50 ± 0.74 | 78.41 ± 0.09 | 85.71 ± 0.43 | 87.88 ± 0.26 | 89.72 ± 0.45 | 91.07 ± 0.43 |
| **NaFM**                   | 70.10 ± 0.92 | 79.89 ± 0.07 | 87.37 ± 1.51 | 89.15 ± 0.22 | 90.77 ± 0.26 | 91.75 ± 0.47 |

- Second experiment comparison to NPClassifier
  - NPClassifier was taken as-is, NaFM was fine-tuned as described above
  - comparison on pathway and super-class levels
  - Results in Figure 2 (PR AUC: Area Under the Curve where x is recall and y is precision)
  - The fine-tuned NaFM outperforms NPClassifier
  - Especially for amino acids and peptides (pathway) and Carotenoids (C45), Tetramate alkaloids, Peptide alkaloids,
    and Anthranilic acid alkaloids (super-classes)
  - The gap in the recognition of γ-lactam-β-lactones can be attributed to the
    limited data for this category, with only 8 instances out of 78,000 data points

\
![Figure 2. The bar plots illustrate the models’ performance comparison of NaFM and NPClassifier on various superclass categories
(a) and biosynthetic pathways (b) including Carbohydrates, Amino Acids and Peptides, Alkaloids, Terpenoids, Shikimates and
Phenylpropanoids, Polyketides, and Fatty Acids in terms of AUPRC. The gray segments indicate the shared performance (minimum
AUPRC) and the light blue/orange segments highlight the additional AURPC achieved by NaFM and NPClassifier beyond each
other, respectively. Higher values indicate better model performance.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig2.png){width="600" fig-align="center"}
\

### Prediction of Source Organism Kingdom
- Based on LOTUS ([@rutz_lotus_2022])
  - 130,000 data pairs of NP structure and source organism (and literature reference)
- NaFM model was fine-tuned to predict kingdom of NP-producing species (i.e. animals, bacteria, 
[chromista](https://ucmp.berkeley.edu/chromista/chromista.html), fungi, and plants)
- Figure 3 shows the resulting embeddings plotted using TMAP [@probst_visualization_2020]
- NaFM appears to be able to separate the NP of the different kingdoms within the representation space
- Kingdom representative structures were obtained by performing a cluster analysis and giving the 
representative of the biggest cluster, respectively
- Without the pre-training weights ("from scratch"), the model is not able to separate the classes (larger figure available in SI)

\
![Figure 3. The obtained representations are projected into two dimensions using TMAP and
visualized with Faerun. For each biological source, a representative example is highlighted in a designated box, showcasing a
characteristic scaffold that typifies compounds derived from that particular source.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig3.png){width="600" fig-align="center"}
\

### Prediction of NP-producing Biosynthetic Gene Cluster (BGC) and protein family
- dataset a combination of MIBiG (Minimum Information about a Biosynthetic Gene cluster database, [@terlouw_mibig_2023]) 
and Pfam (Protein families database, [@mistry_pfam_2021]): 2,000 bacterial and fungal BGCs and their corresponding
  metabolite structures, plus protein families encoded by the respective BGC
- Again, NaFM, ECFP, and baseline molecular pre-trained models were fine-tuned on the task of predicting the producing 
protein family based on an NP structure
- Results in Figure 4 for the top 128 protein families, NaFM again outperforms all others (AUROC: Area Under ROC curve)
  - No idea why they call it NaRel instead of NaFM here...
  - e.g. ID 65 and 115 are protein families are known to be highly conserved

\
![Figure 4. The figure visualizes the AUROC gap between NaFM and various molecular representations. The results show the AUROC
for the 128 most frequent protein families found in bacteria and fungi. The color in each block represents the magnitude of the
AUROC gap, ranging from negative to positive, with colors transitioning from red to blue. Deeper colors indicate a greater absolute
difference.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig4.png){width="600" fig-align="center"}
\

### Predicting bioactivity for specific targets
- data: human protein targets and HIV type-
  1 reverse transcriptase activity data from the NPASS natural product activity database
- As before, NaFM and the baseline models were fine-tuned on this data
- Results in Table 2
- NaFM surpasses all baselines, except for N-Gram on HIV type-1 reverse transcriptase

Table 2 The RMSE results for natural product bioactivity regression compared to other baselines.

| Target Name   | Number of samples | N-Gram              | PretrainGNN     | D-MPNN          | MolCLR          | Mol-BERT        | ECFP            | GraphMVP        | NaFM                |
|---------------|-------------------|---------------------|-----------------|-----------------|-----------------|-----------------|-----------------|-----------------|---------------------|
| PTP1B         | 612               | 0.8493 ± 0.2623     | 0.8652 ± 0.2110 | 0.8577 ± 0.2165 | 0.9912 ± 0.2467 | 0.8624 ± 0.0732 | 1.1409 ± 0.2281 | 1.8865 ± 0.1303 | **0.8243 ± 0.1960** |
| AChE          | 341               | 1.3774 ± 0.3995     | 1.5462 ± 0.2265 | 1.2881 ± 0.1992 | 1.1507 ± 0.1683 | 1.2938 ± 0.3615 | 1.3721 ± 0.0703 | 2.8140 ± 0.3649 | **1.1227 ± 0.1604** |
| COX-2         | 190               | 1.0022 ± 0.3192     | 1.7231 ± 0.3754 | 1.0272 ± 0.2468 | 0.9399 ± 0.1705 | 1.0529 ± 0.1536 | 1.2718 ± 0.1531 | 3.0341 ± 0.6418 | **0.9239 ± 0.1721** |
| HIV type-1 RT | 186               | **1.0606 ± 0.1613** | 1.5210 ± 0.2606 | 1.1826 ± 0.1683 | 1.2649 ± 0.1353 | 1.4348 ± 0.1398 | 1.5446 ± 0.1133 | 2.7829 ± 0.1195 | 1.0802 ± 0.1506     |
| Tyrosinase    | 186               | 0.7822 ± 0.1408     | 1.6957 ± 0.2915 | 0.8206 ± 0.1891 | 0.9165 ± 0.2820 | 1.0440 ± 0.2569 | 1.2553 ± 0.0985 | 3.3965 ± 0.3437 | **0.6927 ± 0.2828** |
| CYP3A4        | 178               | 0.7846 ± 0.2037     | 1.4157 ± 0.2923 | 0.6946 ± 0.1415 | 0.7072 ± 0.1362 | 0.9377 ± 0.1873 | 1.0439 ± 0.1244 | 3.0817 ± 0.2720 | **0.6922 ± 0.1326** |
| MRP4          | 177               | 0.2747 ± 0.0846     | 1.6869 ± 0.2476 | 0.2399 ± 0.0603 | 0.2321 ± 0.0847 | 0.5190 ± 0.0645 | 1.4682 ± 0.2422 | 3.5723 ± 0.4337 | **0.2265 ± 0.0921** |
| COX-1         | 140               | 0.9529 ± 0.1643     | 2.6831 ± 0.2056 | 0.9405 ± 0.1414 | 0.9574 ± 0.1655 | 1.4542 ± 0.3456 | 1.1965 ± 0.1850 | 3.0083 ± 0.2802 | **0.9286 ± 0.1342** |

- Second experiment in this area:
  - acetylcholinesterase(AChE) as the target
  - data: AChE IC50 data from the NPASS database
  - NaFM model is fine-tuned on this data
  - Fine-tuned model is used to predict AChE activity for all COCONUT NP
  - The 600 molecules with the best predicted activity are tested in virtual screening (docking)
  - as control, 600 randomly chosen NP are also tested
  - Figure 5 a: docking scores for the two groups
    - the two distributions are significantly different, the NP with predicted activity are significantly 
    better than the randomly chosen ones
    - Model-selected NP scores have a higher mean and lower variance

\
![Figure 5a. Docking scores of model ranked top 0.1% (approximately 600 molecules) of acetylcholinesterase inhibitors, along with
600 randomly selected molecules. The histogram and distribution of docking scores with acetylcholinesterase are shown in the figure.
Orange represents the model-selected group, while blue represents the random group. The x-axis represents the docking score, and
the y-axis represents frequency. A lower docking score (more left-shifted distribution) indicates higher inhibitory activity. Docking
was performed using Schrödinger Maestro, with detailed settings provided in the Supplementary section 1.3.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig5a.png){width="600" fig-align="center"}
\

- Figure 5c shows some COCONUT NP with high model and docking scores, alongside the most structurally similar NP from the fine-tuning set (NPASS)
  - demonstrated ability of the model to generalise or find novel actives

\
![Figure 5c. The first and second columns display several molecules with high model scores and good
docking activity, along with their most similar positive counterparts encountered during the model fine-tuning process. The third
column shows the maximum common substructure between these two molecules.  source: [@ding_nafm_2025] CC-BY 4.0](NaFM_2025_images/Fig5c.png){width="600" fig-align="center"}
\

## Conclusion
- proposed NaFM, a foundation model for small molecule natural products
- Generated using novel pre-training strategies
- NaFM has been evaluated across a range of tasks, including
  upstream genome mining, biological source identification, structural classification, metabolic pathway prediction,
  and activity prediction and screening. 
- On most of the benchmarks, NaFM outperforms both traditional
  molecular fingerprints and deep-learning based pre-training frameworks
- It is expected that NaFM can accelerate the NP drug discovery process in multiple stages

## Personal comments / questions to discuss
- There is no code available, apparently; sad
- Larger text sections, e.g. the abstract, are quite hard to understand, i.e. there are many "nice" words but no details
- Is the presented model really a "foundation model" and not rather an "encoder"?

## References