record_id stringlengths 11 14 | start int64 0 1.19B | end int64 62 1.19B | embedding list | embedding_norm float32 24.5 258 |
|---|---|---|---|---|
NW_006267348.1 | 11,820 | 13,147 | [
0.0311279296875,
-0.0220947265625,
0.034912109375,
-11.5625,
-0.150390625,
-0.1181640625,
-0.0556640625,
-0.08203125,
-0.115234375,
-0.228515625,
-0.44140625,
-0.09033203125,
0.10595703125,
0.0198974609375,
0.0712890625,
0.06494140625,
-0.158203125,
0.035400390625,
-0.1904296875,... | 125.5 |
NW_006267348.1 | 17,065 | 17,144 | [
-0.0498046875,
0.000896453857421875,
-0.3828125,
-9,
-0.2314453125,
-0.1005859375,
0.111328125,
-0.3359375,
-0.01263427734375,
0.07666015625,
0.0400390625,
-0.1494140625,
0.02880859375,
0.2236328125,
-0.05517578125,
-0.1884765625,
0.072265625,
0.015380859375,
0.0155029296875,
-... | 98.5 |
NW_006267348.1 | 22,199 | 23,626 | [
0.0859375,
0.05126953125,
-0.1923828125,
-17.25,
-0.0908203125,
-0.1845703125,
0.022705078125,
-0.103515625,
0.0947265625,
0.1591796875,
-0.0291748046875,
-0.1875,
0.0498046875,
-0.06494140625,
0.038818359375,
0.054443359375,
-0.1416015625,
0.01300048828125,
-0.08837890625,
-0.... | 181 |
NW_006267348.1 | 23,988 | 25,516 | [
0.099609375,
-0.00286865234375,
0.09130859375,
-18.125,
-0.126953125,
0.0189208984375,
-0.0164794921875,
-0.134765625,
-0.026611328125,
0.031494140625,
-0.09033203125,
-0.287109375,
0.03369140625,
-0.03955078125,
0.2060546875,
0.154296875,
-0.095703125,
-0.0311279296875,
-0.13085... | 193 |
NW_006267348.1 | 28,410 | 31,305 | [0.014404296875,0.0174560546875,-0.045654296875,-14.0,-0.02294921875,-0.061279296875,0.022705078125,(...TRUNCATED) | 154 |
NW_006267348.1 | 33,171 | 34,881 | [0.057373046875,-0.01422119140625,-0.14453125,-14.5625,-0.0947265625,-0.07421875,-0.0022735595703125(...TRUNCATED) | 156 |
NW_006267348.1 | 35,465 | 37,429 | [0.036865234375,-0.001556396484375,-0.0478515625,-15.0,-0.06103515625,-0.029296875,-0.0245361328125,(...TRUNCATED) | 160 |
NW_006267348.1 | 38,487 | 39,015 | [0.056884765625,0.0830078125,-0.1767578125,-21.125,0.04345703125,-0.09716796875,-0.02001953125,-0.10(...TRUNCATED) | 228 |
NW_006267348.1 | 39,287 | 40,951 | [0.06640625,0.158203125,-0.1943359375,-23.125,-0.0703125,0.115234375,-0.0155029296875,0.016235351562(...TRUNCATED) | 247 |
NW_006267348.1 | 41,141 | 42,854 | [0.072265625,0.0703125,0.134765625,-17.25,-0.123046875,0.0712890625,-0.043212890625,-0.02197265625,-(...TRUNCATED) | 184 |
Dataset Summary
AINovice2005/carbon-embeddings is a derived dataset from the Carbon corpus containing dense vector embeddings of biological sequence records.
Each row corresponds to a source sequence identified by record_id. The dataset retains the sequence's position within the processed corpus through start and end and provides a numerical embedding representing the sequence in the embedding model's learned representation space.
The dataset is intended for downstream embedding-based analysis, including similarity search, clustering, retrieval, visualization, and machine-learning workflows.
Schema
| Column | Type | Description |
|---|---|---|
record_id |
string |
Unique identifier for the source sequence record. |
string_lengths |
int64 |
Length of the corresponding sequence/string. |
start |
int64 |
Starting token position of the record in the processed/tokenized corpus. |
end |
int64 |
Ending token position of the record in the processed/tokenized corpus. |
embedding |
list |
Dense numerical vector representation generated for the sequence by the Carbon embedding model. |
embedding_norm |
float32 |
L2 norm of the embedding vector. |
What the Dataset Represents
The central field is embedding.
For each biological sequence, the embedding model maps the sequence into a fixed-dimensional numerical representation. Sequences that the model represents as similar may occupy nearby regions of the embedding space, depending on the model and similarity metric used.
embedding_norm records the magnitude of each embedding vector. It can be used for diagnostics and for understanding whether embedding magnitudes vary systematically across the corpus.
The dataset therefore functions as a vector representation layer over the Carbon biological sequence corpus.
Embedding Representation
The embedding column is stored as a list containing the numerical components of the sequence embedding.
The embedding should be treated as a model-generated representation rather than as a set of directly interpretable biological measurements. Individual embedding dimensions generally do not correspond to specific biological properties unless that relationship has been established through separate analysis.
For similarity and retrieval applications, users should use a distance or similarity metric appropriate to the embedding model. Cosine similarity and Euclidean distance are common choices, but the appropriate metric depends on how the embeddings were generated and normalized.
Sequence Positioning
start and end identify the corresponding region of the processed/tokenized Carbon corpus.
These are corpus/tokenization positions, not genomic coordinates.
record_id provides the primary record-level association between the embedding and its source sequence.
Intended Uses
The dataset can be used for:
- Biological sequence similarity search
- Nearest-neighbor retrieval
- Sequence clustering
- Embedding-space visualization
- Representation-learning experiments
- Retrieval-augmented biological workflows
- Duplicate and near-duplicate analysis
- Dataset exploration and stratification
- Downstream machine-learning models
Embedding Norm
embedding_norm is the L2 magnitude of the embedding vector.
It is useful for:
- Checking embedding generation consistency
- Detecting anomalous vectors
- Studying embedding magnitude distributions
- Diagnosing numerical or preprocessing issues
Embedding norm should not itself be interpreted as a measure of biological quality or sequence importance.
Relationship to the Carbon Corpus
This dataset does not constitute a separate biological sequence collection. It is a derived computational representation of the Carbon corpus.
The source sequence remains the underlying biological object; the embedding is a model-dependent representation generated from that sequence.
The record_id, start, and end fields provide the linkage between the derived embedding records and the processed source corpus.
Dataset Statistics
The current dataset contains the following principal data types:
- Record-level identifiers:
record_id - Sequence/corpus metadata:
string_lengths,start,end - Vector representation:
embedding - Vector diagnostic:
embedding_norm
- Downloads last month
- 182