Datasets:

LuminScience
/

LuminBench-Nano-ESMC-RAW

Error code:   StreamingRowsError
Exception:    CastError
Message:      Couldn't cast
clustering: struct<cluster_mode: int64, coverage_mode: int64, minimum_coverage: double, minimum_sequence_identit (... 41 chars omitted)
  child 0, cluster_mode: int64
  child 1, coverage_mode: int64
  child 2, minimum_coverage: double
  child 3, minimum_sequence_identity: double
  child 4, method: string
  child 5, scope: string
files: struct<mgnify: struct<cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>,  (... 446 chars omitted)
  child 0, mgnify: struct<cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>, representatives (... 69 chars omitted)
      child 0, cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>
          child 0, bytes: int64
          child 1, path: string
          child 2, rows: int64
          child 3, sha256: string
      child 1, representatives: struct<bytes: int64, path: string, records: int64, sha256: string>
          child 0, bytes: int64
          child 1, path: string
          child 2, records: int64
          child 3, sha256: string
  child 1, omg_img: struct<cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>, representatives (... 69 chars omitted)
      child 0, cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>
          child 0, bytes: int64
          child 1, path: string
          child 2, rows: int64
          child 3, sha256: string
      child 1, representatives: struct<bytes: int64, path: string, records: i
...
ecords: int64, source_ (... 58 chars omitted)
  child 0, cluster_map_bytes: int64
  child 1, representative_bytes: int64
  child 2, representative_records: int64
  child 3, source_specific_member_rows: int64
  child 4, total_published_bytes: int64
reviewed_at: timestamp[s]
release: string
distribution_license: struct<license: string, lumin_science_metadata_license: string, third_party_records_relicensed: bool (... 1 chars omitted)
  child 0, license: string
  child 1, lumin_science_metadata_license: string
  child 2, third_party_records_relicensed: bool
source_arms: struct<mgnify: struct<governing_terms: string, raw_bytes: int64, raw_md5: string, raw_url: string>,  (... 226 chars omitted)
  child 0, mgnify: struct<governing_terms: string, raw_bytes: int64, raw_md5: string, raw_url: string>
      child 0, governing_terms: string
      child 1, raw_bytes: int64
      child 2, raw_md5: string
      child 3, raw_url: string
  child 1, omg_img: struct<governing_terms: string, raw_bytes: int64, raw_manifest_sha256: string, raw_repository: strin (... 21 chars omitted)
      child 0, governing_terms: string
      child 1, raw_bytes: int64
      child 2, raw_manifest_sha256: string
      child 3, raw_repository: string
      child 4, raw_shards: int64
  child 2, uniref90: struct<governing_terms: string, raw_bytes: int64, raw_md5: string, raw_url: string>
      child 0, governing_terms: string
      child 1, raw_bytes: int64
      child 2, raw_md5: string
      child 3, raw_url: string
to
{'distribution_license': {'license': Value('string'), 'lumin_science_metadata_license': Value('string'), 'third_party_records_relicensed': Value('bool')}, 'release': Value('string'), 'reviewed_at': Value('timestamp[s]'), 'schema_version': Value('int64'), 'source_arms': {'mgnify': {'governing_terms': Value('string'), 'raw_bytes': Value('int64'), 'raw_md5': Value('string'), 'raw_url': Value('string')}, 'omg_img': {'governing_terms': Value('string'), 'raw_bytes': Value('int64'), 'raw_manifest_sha256': Value('string'), 'raw_repository': Value('string'), 'raw_shards': Value('int64')}, 'uniref90': {'governing_terms': Value('string'), 'raw_bytes': Value('int64'), 'raw_md5': Value('string'), 'raw_url': Value('string')}}}
because column names don't match
Traceback:    Traceback (most recent call last):
                File "/src/services/worker/src/worker/utils.py", line 147, in get_rows_or_raise
                  return get_rows(
                      dataset=dataset,
                  ...<4 lines>...
                      column_names=column_names,
                  )
                File "/src/libs/libcommon/src/libcommon/utils.py", line 272, in decorator
                  return func(*args, **kwargs)
                File "/src/services/worker/src/worker/utils.py", line 127, in get_rows
                  rows_plus_one = list(itertools.islice(safe_iter(ds, dataset=dataset), rows_max_number + 1))
                File "/src/services/worker/src/worker/utils.py", line 483, in safe_iter
                  yield from ds.decode(False) if ds.features else ds
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2840, in __iter__
                  for key, example in ex_iterable:
                                      ^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2373, in __iter__
                  for key, pa_table in self._iter_arrow():
                                       ~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 2398, in _iter_arrow
                  for key, pa_table in self.ex_iterable._iter_arrow():
                                       ~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 536, in _iter_arrow
                  for key, pa_table in iterator:
                                       ^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/iterable_dataset.py", line 419, in _iter_arrow
                  for key, pa_table in self.generate_tables_fn(**gen_kwags):
                                       ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 343, in _generate_tables
                  self._cast_table(pa_table, json_field_paths=json_field_paths),
                  ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
                File "/usr/local/lib/python3.14/site-packages/datasets/packaged_modules/json/json.py", line 132, in _cast_table
                  pa_table = table_cast(pa_table, self.info.features.arrow_schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2378, in table_cast
                  return cast_table_to_schema(table, schema)
                File "/usr/local/lib/python3.14/site-packages/datasets/table.py", line 2306, in cast_table_to_schema
                  raise CastError(
                  ...<3 lines>...
                  )
              datasets.table.CastError: Couldn't cast
              clustering: struct<cluster_mode: int64, coverage_mode: int64, minimum_coverage: double, minimum_sequence_identit (... 41 chars omitted)
                child 0, cluster_mode: int64
                child 1, coverage_mode: int64
                child 2, minimum_coverage: double
                child 3, minimum_sequence_identity: double
                child 4, method: string
                child 5, scope: string
              files: struct<mgnify: struct<cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>,  (... 446 chars omitted)
                child 0, mgnify: struct<cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>, representatives (... 69 chars omitted)
                    child 0, cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>
                        child 0, bytes: int64
                        child 1, path: string
                        child 2, rows: int64
                        child 3, sha256: string
                    child 1, representatives: struct<bytes: int64, path: string, records: int64, sha256: string>
                        child 0, bytes: int64
                        child 1, path: string
                        child 2, records: int64
                        child 3, sha256: string
                child 1, omg_img: struct<cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>, representatives (... 69 chars omitted)
                    child 0, cluster_map: struct<bytes: int64, path: string, rows: int64, sha256: string>
                        child 0, bytes: int64
                        child 1, path: string
                        child 2, rows: int64
                        child 3, sha256: string
                    child 1, representatives: struct<bytes: int64, path: string, records: i
              ...
              ecords: int64, source_ (... 58 chars omitted)
                child 0, cluster_map_bytes: int64
                child 1, representative_bytes: int64
                child 2, representative_records: int64
                child 3, source_specific_member_rows: int64
                child 4, total_published_bytes: int64
              reviewed_at: timestamp[s]
              release: string
              distribution_license: struct<license: string, lumin_science_metadata_license: string, third_party_records_relicensed: bool (... 1 chars omitted)
                child 0, license: string
                child 1, lumin_science_metadata_license: string
                child 2, third_party_records_relicensed: bool
              source_arms: struct<mgnify: struct<governing_terms: string, raw_bytes: int64, raw_md5: string, raw_url: string>,  (... 226 chars omitted)
                child 0, mgnify: struct<governing_terms: string, raw_bytes: int64, raw_md5: string, raw_url: string>
                    child 0, governing_terms: string
                    child 1, raw_bytes: int64
                    child 2, raw_md5: string
                    child 3, raw_url: string
                child 1, omg_img: struct<governing_terms: string, raw_bytes: int64, raw_manifest_sha256: string, raw_repository: strin (... 21 chars omitted)
                    child 0, governing_terms: string
                    child 1, raw_bytes: int64
                    child 2, raw_manifest_sha256: string
                    child 3, raw_repository: string
                    child 4, raw_shards: int64
                child 2, uniref90: struct<governing_terms: string, raw_bytes: int64, raw_md5: string, raw_url: string>
                    child 0, governing_terms: string
                    child 1, raw_bytes: int64
                    child 2, raw_md5: string
                    child 3, raw_url: string
              to
              {'distribution_license': {'license': Value('string'), 'lumin_science_metadata_license': Value('string'), 'third_party_records_relicensed': Value('bool')}, 'release': Value('string'), 'reviewed_at': Value('timestamp[s]'), 'schema_version': Value('int64'), 'source_arms': {'mgnify': {'governing_terms': Value('string'), 'raw_bytes': Value('int64'), 'raw_md5': Value('string'), 'raw_url': Value('string')}, 'omg_img': {'governing_terms': Value('string'), 'raw_bytes': Value('int64'), 'raw_manifest_sha256': Value('string'), 'raw_repository': Value('string'), 'raw_shards': Value('int64')}, 'uniref90': {'governing_terms': Value('string'), 'raw_bytes': Value('int64'), 'raw_md5': Value('string'), 'raw_url': Value('string')}}}
              because column names don't match

Need help to make the dataset viewer work? Make sure to review how to configure the dataset viewer, and open a discussion for direct support.

LuminBench Nano ESMC Raw 70% Cluster Outputs v1

This repository preserves the source-specific 70%-identity clustering outputs that precede evaluation decontamination and final Parquet packing in LuminScience/LuminBench-Nano-ESMC.

It contains 241,600,826,413 bytes of representative FASTA data covering 765,290,002 source-specific cluster representatives. It also preserves the three cluster-membership maps so the clustering result is not reduced to the representatives alone.

Not decontaminated training data. These FASTAs precede the P@L/P-CORE evaluation exclusion, cross-source ownership assignment, length gate, and train/validation split. Use the final repository above for clean training.

Files

Source arm	Representative FASTA	Representatives	Cluster map	Source-specific member rows
UniRef90 2023_02	38,626,189,542 bytes	92,230,941	23,887,338,192 bytes	165,884,293
MGnify Protein DB 2023_02	93,319,863,106 bytes	348,135,082	88,097,490,576 bytes	611,788,129
OMG/IMG	109,654,773,765 bytes	324,923,979	138,768,938,784 bytes	963,673,186
Total	241,600,826,413 bytes	765,290,002	250,753,767,552 bytes	1,741,345,608 source memberships

Paths are:

representatives/{uniref90,mgnify,omg_img}.fasta
clusters/{uniref90,mgnify,omg_img}.clusters.tsv
receipts/{uniref90,mgnify,omg_img}.verification.json

FASTA identifiers are sha256_<digest>, where the digest is computed from the normalized ASCII amino-acid sequence. Each cluster-map row is:

representative_sha256<TAB>member_sha256

The maps are source-specific. The same exact sequence can therefore occur as a member in more than one source arm.

Does this preserve every clustered protein?

It preserves the cluster assignment relation, but it is not a self-contained copy of every cluster-member sequence:

the representative FASTAs contain the 765.29M representative sequences;
the clusters.tsv files map each representative digest to every member digest;
non-representative member sequences and original upstream accessions are not stored in those TSVs.

To recover all member sequences or original identifiers, redownload the pinned upstream snapshots, rerun normalization and exact deduplication, and join the resulting membership Parquet rows to member_sha256. This repository therefore preserves enough information to reproduce cluster membership when combined with the pinned upstream inputs and build recipe, but the representative FASTAs alone would not be sufficient.

See BUILD_RECIPE.md, manifest.json, and SOURCE_PROVENANCE.json for the exact pipeline, commands, checksums, and limitations.

License and source terms

This is a mixed-terms dataset, so the Hub metadata uses license: other rather than pretending that one license replaces all upstream terms.

Path	Direct source	Governing terms
`representatives/uniref90.fasta`	UniRef90 2023_02	UniProt CC BY 4.0
`representatives/mgnify.fasta`	MGnify Protein DB 2023_02	EMBL-EBI Terms of Use plus applicable original-owner rights; not relicensed by Lumin Science
`representatives/omg_img.fasta`	JGI/IMG records distributed by `tattabio/OMG`	CC BY-SA 4.0 as declared by the direct distribution
`clusters/**`, manifests, receipts, and build documentation	Lumin Science selection, arrangement, and derived clustering metadata	CC BY-SA 4.0

See LICENSE_AND_ATTRIBUTION.md before reuse.

Downloads last month: 53

Total file size:

492 GB