hydra

class hydra.ClassificationTask(name: str, dataset: DatasetDict, pretrained_path: str, segmentation: bool = False, lemmatisation: bool = False, tagging: bool = False, dependency: bool = False, graph: bool = False, hybrid: bool = False, named_entities: bool = False, root_label: str = 'ROOT', force_root_label: bool = False, morfeusz_dict_path: str | None = None, morfeusz_dict_name: str | None = None, nonterminal_features: list[str] | None = None, tree_weight: float = 1.0, batch_size: int = 16)[source]

Bases: object

A Hydra training configuration.

class hydra.Hydra(pretrained_path: str, model_type: str, model: MultiTargetTokenClassification, labels: dict[str, list[str]], segmentation: bool = False, lemmatisation: bool = False, tagging: bool = False, dependency: bool = False, graph: bool = False, hybrid: bool = False, nonterminal_features: list[str] | None = None, named_entities: bool = False, morfeusz_dict_path: str | None = 'polimorf', morfeusz_dict_name: str | None = None, pretrained_tokenizer: PreTrainedTokenizerBase | None = None, tokenizer_kwargs: dict[str, Any] | None = None)[source]

Bases: object

A Hydra parser. Use of train() recommended for new instance creation.

classmethod create(pretrained_path: str, model_type: str, features: Features, segmentation: bool = False, lemmatisation: bool = False, tagging: bool = False, dependency: bool = False, graph: bool = False, hybrid: bool = False, nonterminal_features: list[str] | None = None, named_entities: bool = False, morfeusz_dict_path: str | None = None, morfeusz_dict_name: str | None = 'sgjp', pretrained_tokenizer: PreTrainedTokenizerBase | None = None, tokenizer_kwargs: dict[str, Any] | None = None) → Hydra[source]

Create a new Hydra instance.

Parameters:
  • pretrained_path – pretrained language model to be fine-tuned.

  • model_type – pretrained language model type.

  • features – the features signature of the dataset to be used for training.

  • segmentation – enable segmentation module.

  • lemmatisation – enable lemmatisation module.

  • tagging – enable tagging module.

  • dependency – enable dependency parsing module.

  • graph – enable graph parsing module (extended/additional edges).

  • hybrid – enable hybrid syntactic parsing module.

  • nonterminal_features – nonterminal features to predict (experimental).

  • named_entities – enable NER module.

  • morfeusz_dict_path – path to directory containing Morfeusz dictionary (if not using the default one).

  • morfeusz_dict_name – name of the Morfeusz dictionary (if not using the default one).

  • pretrained_tokenizer – tokenizer for the language model.

  • tokenizer_kwargs – addditonal kwargs for the language model tokenizer.

Returns:

a Hydra instance.

classmethod load(path: str, morfeusz_dict_path: str | None = None) → Hydra[source]

Load a trained Hydra model.

Returns:

a Hydra instance.

parse(sentences: list[str] | str, correct_lemmata: bool = False, return_jsons: bool = False, return_conllu: bool = False, return_trees: bool = False, return_labels: bool = False, return_logits: bool = False, root_label: str | None = None, force_root_label: bool = False, skip_long: bool = False, is_tokenized: bool = False, morph_postprocess: Callable[[str, str, str], tuple[str, str]] | None = None, mst: str = 'mst', return_times: bool = False, mem_limit: int | None = None, verbose: bool = False) → list[HydraAnalysis] | tuple[list[HydraAnalysis], dict[str, int]][source]

Parse a single sentence or a list of sentences.

Parameters:
  • sentences – sentence(s) to process.

  • correct_lemmata – apply lemmata correction using Morfeusz.

  • return_jsons – include full json outputs in the results. Only available for models trained with hybrid=True.

  • return_conllu – include ConLLu outputs in the results. Only available for models trained with dependency=True.

  • return_trees – include tree objects in the results. Only available for models trained with hybrid=True.

  • return_labels – include label lists in the results.

  • return_logits – include raw predicted logits in the results.

  • root_label – used by force_root_label.

  • force_root_label – enforce a specific root label in resulting trees.

  • skip_long – omit sentences with too many tokens (an error message will be logged in the sentences’ metadata).

  • is_tokenized – don’t perform segmentation (treat spaces as segment delimiters).

  • morph_postprocess – custom postprocessing function for tags and lemmata.

  • mst – dependency tree decoding algorithm to use: MST or EAGER.

  • return_times – include parsing time info in the results.

  • mem_limit – memory limit (experimental).

  • verbose – include additional info in the results.

Returns:

the parsing results.

process_labels(labels: LabelDict, tokens: TokenList, correct_lemmata: bool = False, morph_postprocess: Callable[[str, str, str], tuple[str, str]] | None = None, mst: str = 'mst', allow_masked: bool = False) → LabelDict[source]

Process model-predicted labels.

Parameters:
  • labels – model-predicted labels.

  • tokens – the processed sentence.

  • correct_lemmata – apply lemmata correction using Morfeusz.

  • morph_postprocess – custom postprocessing function for tags and lemmata.

  • mst – dependency tree decoding algorithm to use: MST or EAGER.

  • allow_masked – accept masked labels.

Returns:

labels that can be returned as parsing result or used to construct a tree.

save(path: str) → None[source]

Save a trained Hydra model to disk.

class hydra.Trainer(task: ClassificationTask)[source]

Bases: object

Used for creating, training and evaluating instances of Hydra parser.

cross_validate(nfolds: int = 10, epochs: int = 10, patience: int = 4, lr: float = 1e-05, mst: str = 'mst', allow_masked: bool = False) → list[EvalResults][source]

Perform 10-fold cross validation.

Returns:

results of each fold run.

evaluate(parser: Hydra, mst: str = 'mst', allow_masked: bool = False, dataset: DatasetDict | None = None) → EvalResults[source]

Perform evaluation of a trained Hydra model.

Parameters:
  • parser – the parser instance to be evaluated.

  • mst – dependency tree decoding algoriths to be used.

  • allow_masked – accept masked labels in gold data (but exclude them from metric computation).

  • dataset – source of gold instances. Evaluation will be run on ‘validation’ and any ‘test*’ parts. Training dataset will be used as default if this argument is not provided.

Returns:

evaluation results.

train(epochs: int = 10, patience: int = 4, lr: float = 1e-05, log_dir: str | None = None, model_dir: str | None = None, dataset: DatasetDict | None = None) → Hydra[source]

Create and train a parser.

Parameters:
  • epochs – number of epochs.

  • patience – how many epochs are allowed with no training progress. The best model across all epochs is then retrieved.

  • lr – learning rate.

  • log_dir – where to save training logs.

  • model_dir – where to save the final model.

  • dataset – training data. Must contain ‘train’ and ‘validation’ parts (used for training), with any number of ‘test*’ parts for later evaluation.

Returns:

a parser instance.