AINovice2005/carbon-cpu-enriched-sequences

Carbon CPU-Enriched Sequences

A CPU-enriched, analysis-ready subset of the Carbon genomic pretraining corpus, containing normalized sequence records together with the original Carbon fields and deterministic, row-level derived features.

This dataset is an intermediate data product intended to support downstream quality analysis, sampling, GPU-based enrichment, embedding generation, and genomic foundation-model data preparation.

---

Dataset Summary

Property	Value
Source corpus	Carbon genomic pretraining corpus
Source size	~46.3M rows
Enriched rows	32,410,000
Source coverage	~70%
Processing mode	Streaming
CPU batch size	5,000 rows
Parquet shard size	50,000 rows
Compression	Zstandard (zstd)
Approximate shards	649
CPU hardware	4-core CPU
CPU enrichment wall time	~3h 48m
Final end-to-end throughput	~2,369 rows/s

The dataset deliberately contains both the original Carbon fields and derived enrichment features. Derived features do not replace the source representation.

---

Purpose

The dataset provides a reusable CPU-side canonical representation of the Carbon corpus before computationally expensive GPU enrichment.

The processing pipeline is designed to establish a stable data boundary:

Carbon source corpus
        │
        ▼
CPU normalization / enrichment
        │
        ▼
CPU-enriched Parquet dataset
        │
        ├── quality analysis
        ├── distribution analysis
        ├── representative sampling
        ├── GPU enrichment
        ├── embedding generation
        └── downstream dataset construction

The CPU enrichment stage focuses on deterministic, relatively inexpensive transformations that can be computed once and reused by subsequent stages.

GPU-dependent and corpus-global enrichment is intentionally kept separate.

---

Source Coverage

The original source corpus contains approximately 46.3 million records.

This release contains:

32,410,000 / 46,300,000 ≈ 70%

of the source row population.

The 70% coverage was selected as a practical balance between:

• broad corpus representation,
• CPU processing cost,
• storage requirements,
• downstream GPU compute availability, and
• the need to retain an additional portion of the source corpus for later expansion or validation.

This is therefore not a complete materialization of the source corpus.

---

Processing Methodology

The enrichment pipeline uses a streaming architecture to avoid materializing the complete source dataset in memory.

Hugging Face dataset
        │
        ▼
Streaming reader
        │
        ▼
5,000-row processing batch
        │
        ▼
CPU normalization
        │
        ▼
Deterministic enrichment
        │
        ▼
PyArrow table
        │
        ▼
Parquet writer
        │
        ▼
50,000-row shard

Processing characteristics

• Input is consumed using streaming rather than loading the entire corpus into memory.
• Processing is performed in bounded batches.
• NumPy-based operations are used for computationally intensive sequence-level operations where appropriate.
• PyArrow is used for tabular conversion and Parquet serialization.
• Parquet files use Zstandard compression.
• The output is sharded to keep individual files manageable for downstream processing.

---

Data Preservation

A core design principle of this dataset is source-data preservation.

The output consists of:

Original Carbon columns
        +
CPU-derived enrichment columns

rather than replacing the original representation with derived features.

This allows downstream users to:

• recover the original sequence representation,
• reproduce or audit derived features,
• apply alternative downstream transformations,
• compare enrichment behavior,
• perform additional filtering without returning to the original source.

The current materialized output contains 25 columns, comprising approximately:

• 14 source/raw columns
• 11 derived enrichment columns

The exact schema should be inspected from the Parquet files because future enrichment revisions may add or modify derived fields.

---

Enrichment Semantics

Coding sequences

Coding sequence identification follows the observed representation in the Carbon data.

The coding gene type is represented as:

<cds>

Accordingly, coding-sequence detection uses:

CODING_GENE_TYPES = frozenset({"<cds>"})

This avoids assuming that multiple taxonomy/gene-type representations exist when the observed source representation uses the explicit <cds> tag.

Taxonomy

Taxonomic information is represented using a derived:

taxonomy_depth

rather than generating a fixed collection of positional fields such as:

taxonomy_rank_0
taxonomy_rank_1
...
taxonomy_rank_6

This avoids imposing an arbitrary maximum taxonomy depth on the source data.

Relative gene position

relative_gene_position is intentionally not retained unless a concrete downstream use requires it. This avoids adding derived fields whose semantics do not provide a demonstrated downstream benefit.

---

Sequence Orientation

The source data contains strand/orientation information.

Where applicable, sequence orientation can require strand-aware normalization.

The CPU enrichment pipeline evaluates strand information explicitly rather than assuming that all sequences share the same orientation.

The persisted representation may contain both the original sequence and a strand-normalized representation when required by the downstream pipeline.

Users should therefore treat the original sequence as the source representation and any normalized sequence field as a derived representation.

---

Storage Format

The dataset is stored as Apache Parquet.

Parquet was selected because it provides:

• columnar access,
• efficient compression,
• compatibility with PyArrow,
• compatibility with DuckDB,
• efficient batch processing,
• interoperability with common data-processing frameworks,
• convenient integration with downstream data-lake and lakehouse workflows.

Files use:

compression = zstd

The CPU enrichment run uses approximately:

50,000 rows / Parquet shard

For the 32.41M-row release this corresponds to approximately 649 Parquet shards, with the final shard containing the remaining rows.

---

Recommended Usage

The dataset is intended primarily for downstream processing rather than direct model training without further filtering.

from datasets import load_dataset

dataset = load_dataset(
    "AINovice2005/carbon-cpu-enriched-sequences",
    data_files="data/cpu_enriched/*.parquet",
)

For analytical workloads, the Parquet representation can also be queried using tools such as DuckDB or processed directly with PyArrow.

For large-scale processing, prefer streaming or batch-oriented access rather than materializing the entire dataset in memory.

---

Data Quality

The CPU enrichment pipeline was tested at multi-million-row scale before the final materialization. The resulting artifact is intended to provide a stable, reusable CPU-enriched data boundary for subsequent experiments.

Validation should still be performed by downstream users when:

• changing the enrichment implementation,
• changing schema versions,
• adding new derived features,
• using a different source configuration,
• or constructing a new release.

---

Intended Use

This dataset is suitable for:

• genomic data quality analysis,
• preprocessing research,
• sequence-distribution analysis,
• dataset sampling experiments,
• GPU enrichment experiments,
• embedding generation,
• nearest-neighbor and clustering studies,
• genomic foundation-model data engineering,
• benchmarking data-processing pipelines.

It is not intended to be treated as a clinically validated genomic dataset or as a substitute for curated biological annotations.

---

Data Lineage

Carbon genomic pretraining corpus
                │
                │ ~46.3M source rows
                ▼
       CPU normalization
                │
                ▼
       Deterministic enrichment
                │
                │ 32.41M rows (~70%)
                ▼
   carbon-cpu-enriched-sequences
                │
                ├───────────────┐
                ▼               ▼
        GPU enrichment      Data analysis
                │
                ▼
        Embedding-derived
             features

This separation is intentional: deterministic CPU enrichment establishes the reusable canonical data layer, while computationally expensive GPU and corpus-global operations remain downstream.

---

---

License

The licensing terms of the underlying Carbon source data apply to the corresponding records in this processed dataset.

Users are responsible for verifying that their intended use complies with the licenses and usage conditions of the underlying source data and any associated components.

---