02 — Data Processing (Per-Source)¶

📂 Source Code & Notebooks: pipeline/02_data_processing

1. Purpose¶

This is the second step of the EvoAge KG pipeline. Each of the ~50 collected data sources has a different format, schema, and identifier system. This step processes every source individually — cleaning, normalising IDs, resolving gene symbols, and converting raw data into a standardised KG-triple format:

Head | Relation | Tail | Head_type | Tail_type | Source | KG_Source | Head_detail_name | Tail_detail_name | Head_id_is | Tail_id_is | kg_type

Because each data source is different, a tailored processing strategy was developed for each source in its own Jupyter notebook.

2. Input¶

Raw data from Step 01 (Data Collection), located in:

kg_formation/data_collection/<source_name>/

Plus shared reference/mapping dictionaries:

• NCBI gene_info — GeneID ↔ Symbol ↔ Synonyms (per species)
• PubChem CID-Synonym — compound name → PubChem CID
• ChEBI — ChEBI ID mappings
• UniProt — protein ID resolution
• DO / DOID / MONDO — disease ontology mappings
• GO (Quick_GO) — Gene Ontology term resolution
• UBERON — anatomical entity ontology
• HPO — Human Phenotype Ontology
• Reactome — pathway IDs

3. Output¶

Processed KG-triple CSV files saved into:

kg_formation/data_processing/<source_name>/

Each output file follows the standard schema with columns: Head, Relation, Tail, Head_type, Tail_type, Source, KG_Source, Head_detail_name, Tail_detail_name, Head_id_is, Tail_id_is, kg_type

4. Processing Notebooks (Human Data)¶

31 notebooks process human (and sometimes multi-species) data sources:

Aging-Specific Sources¶

General Biomedical KGs¶

Chemical / Drug / Target¶

Other Sources¶

5. Species-Specific Processing¶

Species-specific data sources are processed in a separate subdirectory:

data_processing_02/species_specific_data_source/
├── Celegans/
│   ├── fic.ipynb            ← FICE (Functional Interactome of C. elegans)
│   ├── stitch.ipynb         ← STITCH chemical interactions
│   ├── string.ipynb         ← STRING PPI network
│   ├── wormbase.ipynb       ← WormBase gene annotations
│   └── worminteractomedatabase.ipynb  ← Worm Interactome Database
│
├── Drosophila/
│   ├── flybase.ipynb        ← FlyBase allele/gene data
│   ├── stitch.ipynb         ← STITCH chemical interactions
│   └── string.ipynb         ← STRING PPI network
│
├── Mouse/
│   ├── mgido.ipynb          ← MGI + Disease Ontology mappings
│   ├── stitch.ipynb         ← STITCH chemical interactions
│   └── string_mousenet.ipynb ← STRING + MouseNet v2 combined PPI
│
├── Yeast/
│   ├── esldb.ipynb          ← eSLDB (Essential gene database)
│   ├── sgd.ipynb            ← SGD (Saccharomyces Genome Database)
│   ├── stitch_ch_ch.ipynb   ← STITCH Chemical–Chemical
│   ├── stitch_ch_ge.ipynb   ← STITCH Chemical–Gene
│   └── string_yeastnet_biogrid.ipynb  ← STRING + YeastNet v3 + BioGRID combined
│
└── Zebrafish/
    ├── stitch.ipynb         ← STITCH chemical interactions
    ├── string.ipynb         ← STRING PPI network
    └── zfin.ipynb           ← ZFIN gene/phenotype annotations

6. Common Processing Pattern¶

Although each notebook uses a tailored strategy, they follow a shared general pattern:

1. Load raw data from data_collection/<source>/
2. Load reference dictionaries (NCBI gene_info, PubChem synonyms, ontology mappings)
3. Map identifiers — resolve gene symbols, chemical names, disease IDs to standardised ontology IDs
4. Assign entity types — categorise heads and tails (Gene, ChemicalEntity, Disease, Pathway, etc.)
5. Define relations — map source-specific relationship labels to the EvoAge unified relation vocabulary
6. Build KG triples — create rows with the standard schema columns
7. Save output — write per-relation CSV files to data_processing/<source>/

Standard output schema columns¶

7. Key Design Decisions¶

• One notebook per source: Ensures source-specific edge cases are handled without affecting other sources.
• Reference dictionaries loaded per notebook: Each notebook loads only the dictionaries it needs (e.g., GenAge needs per-species NCBI lookups; DrugAge needs PubChem synonym dictionaries).
• Species separation: Human data and species-specific data are processed in separate notebooks/folders to keep the pipeline clean.
• kg_type assignment: Each triple is tagged as either Aging or Biomedical depending on the source, enabling downstream splitting into KG variants.

8. File Summary¶