Datasets:

universal-dependencies
/

universal_dependencies

This repository uses Parquet-only HuggingFace dataset branches for Universal Dependencies releases. Current examples use the official dataset location: universal-dependencies/universal_dependencies.

v2.2.x to v2.3.0

Loader/framework v2.3.0 updates the generated Parquet schema for regular CoNLL-U token fields:

upos is now stored as a list of strings instead of HuggingFace ClassLabel indices.
head is now stored as a list of 16-bit integers instead of strings.
Empty-node head values remain strings because CoNLL-U empty nodes may use _ for HEAD.

What Usually Changes

Most users can load and iterate over datasets as before:

from datasets import load_dataset

ds = load_dataset(
    "universal-dependencies/universal_dependencies",
    "en_ewt",
    revision="2.18",
    split="train",
)

Code that expected upos as integer class IDs should now work directly with strings:

upos = ds[0]["upos"]
assert upos[0] == "PROPN"

Code that compared head to strings should compare to integers:

heads = ds[0]["head"]
root_positions = [i for i, head in enumerate(heads) if head == 0]

If you need string output for CoNLL-U serialization, convert at the boundary or use the helper package:

from ud_hf_parquet_tools import write_conllu

write_conllu(ds, "output.conllu")

Compatibility Notes

ud-hf-parquet-tools>=1.3.0 can still reconstruct older Parquet outputs where upos was stored as a ClassLabel. New outputs should use the v2.3.0 schema.

Archived v1.x to v2.0 Notes

The v1 loader used a Python dataset script and required trust_remote_code=True. The v2 line removed the script loader and distributes Parquet files directly, which requires datasets>=4.0.0.

Important v2.0 data changes:

Multi-word token surface forms are no longer included in regular tokens.
Multi-word token information is preserved in the structured mwt field.
Helper functions moved out of the dataset repository and into ud-hf-parquet-tools.

Old v1 code like this:

dataset = load_dataset(
    "universal-dependencies/universal_dependencies",
    "fr_gsd",
    trust_remote_code=True,
    revision="v1.0",
)

should be replaced with current Parquet loading:

dataset = load_dataset(
    "universal-dependencies/universal_dependencies",
    "fr_gsd",
    revision="2.18",
)

For token-count checks, use num_words from metadata.json rather than num_tokens when comparing against regular syntactic-word sequences.

Repository Move

The dataset was formerly maintained at commul/universal_dependencies and now lives under the official Universal Dependencies organization at universal-dependencies/universal_dependencies: https://huggingface.co/datasets/universal-dependencies/universal_dependencies

Support

Changelog: CHANGELOG.md
Dataset repository: https://huggingface.co/datasets/universal-dependencies/universal_dependencies
Helper library: https://github.com/bot-zen/ud-hf-parquet-tools
Discussions: https://huggingface.co/datasets/universal-dependencies/universal_dependencies/discussions