hydra.utils.dataset
Dataset manipulation utilities.
- hydra.utils.dataset.add_adjacency_matrix(dataset: DatasetDict) DatasetDict[source]
Add dataset column used to train the dependency parsing module.
- Returns:
a dataset istance with columns added.
- hydra.utils.dataset.add_graph_matrices(dataset: DatasetDict) DatasetDict[source]
Add dataset columns used to train the graph parsing module.
- Returns:
a dataset istance with columns added.
- hydra.utils.dataset.add_lemma_rules(dataset: DatasetDict) DatasetDict[source]
Add dataset columns used to train the lemmatisation module.
- Returns:
a dataset with columns added.
- hydra.utils.dataset.add_spines_and_attachments(dataset: DatasetDict, compress: bool = False) DatasetDict[source]
Add dataset columns used to train the hybrid syntactic parsing module.
- Returns:
a dataset istance with columns added.
- hydra.utils.dataset.cast_labels(dataset: DatasetDict, columns: list[str]) DatasetDict[source]
Cast the provided columns from datasets.Value into datasets.ClassLabel, updating the features signature.
- Returns:
the dataset with cast columns.
- hydra.utils.dataset.get_dataset_features(dataset: DatasetDict) Features[source]
Get the dataset’s features signature.
- Returns:
the features.
- hydra.utils.dataset.lm_tokenize_and_align(example: DatasetInstance, tokenizer: PreTrainedTokenizerBase, masking_strategy: str = 'first') BatchEncoding[source]
Tokenise the dataset instance using a language model-specific tokeniser.
- Returns:
the tokenised instance, encoded as model input.
- hydra.utils.dataset.masked_word_ids(word_ids: list[int | None], masking_strategy: str = 'first') list[int | None][source]
Mask positions of multiple language model tokens corresponding to single input segment (word).
This is needed to properly align training labels (one per word) with tokens passed to the language model.
- Parameters:
- Returns:
a list of the same length as word_ids, with masked postions repleced with None.
- hydra.utils.dataset.morf_tokenize(text: str, m: Morfeusz) TokenList[source]
Tokenize the text using Morfeusz. The most fine-grained tokenisation is retrieved from the DAG.
- Returns:
the tokenised text.
- hydra.utils.dataset.morfeusz_retokenize(dataset: DatasetDict, masking_strategy: str = 'first', morfeusz_dict_path: str | None = None, morfeusz_dict_name: str | None = None) DatasetDict[source]
Retokenise the dataset using Morfeusz, possibly splitting tokens into smaller parts.
This is to be done before language model tokenisation, which can futrther split the Morfeusz tokens.
- Returns:
the retokenised dataset.