hydra
- class hydra.ClassificationTask(name: str, dataset: DatasetDict, pretrained_path: str, segmentation: bool = False, lemmatisation: bool = False, tagging: bool = False, dependency: bool = False, graph: bool = False, hybrid: bool = False, named_entities: bool = False, root_label: str = 'ROOT', force_root_label: bool = False, morfeusz_dict_path: str | None = None, morfeusz_dict_name: str | None = None, nonterminal_features: list[str] | None = None, tree_weight: float = 1.0, batch_size: int = 16)[source]
Bases:
objectA Hydra training configuration.
- class hydra.Hydra(pretrained_path: str, model_type: str, model: MultiTargetTokenClassification, labels: dict[str, list[str]], segmentation: bool = False, lemmatisation: bool = False, tagging: bool = False, dependency: bool = False, graph: bool = False, hybrid: bool = False, nonterminal_features: list[str] | None = None, named_entities: bool = False, morfeusz_dict_path: str | None = 'polimorf', morfeusz_dict_name: str | None = None, pretrained_tokenizer: PreTrainedTokenizerBase | None = None, tokenizer_kwargs: dict[str, Any] | None = None)[source]
Bases:
objectA Hydra parser. Use of
train()recommended for new instance creation.- classmethod create(pretrained_path: str, model_type: str, features: Features, segmentation: bool = False, lemmatisation: bool = False, tagging: bool = False, dependency: bool = False, graph: bool = False, hybrid: bool = False, nonterminal_features: list[str] | None = None, named_entities: bool = False, morfeusz_dict_path: str | None = None, morfeusz_dict_name: str | None = 'sgjp', pretrained_tokenizer: PreTrainedTokenizerBase | None = None, tokenizer_kwargs: dict[str, Any] | None = None) Hydra[source]
Create a new Hydra instance.
- Parameters:
pretrained_path – pretrained language model to be fine-tuned.
model_type – pretrained language model type.
features – the features signature of the dataset to be used for training.
segmentation – enable segmentation module.
lemmatisation – enable lemmatisation module.
tagging – enable tagging module.
dependency – enable dependency parsing module.
graph – enable graph parsing module (extended/additional edges).
hybrid – enable hybrid syntactic parsing module.
nonterminal_features – nonterminal features to predict (experimental).
named_entities – enable NER module.
morfeusz_dict_path – path to directory containing Morfeusz dictionary (if not using the default one).
morfeusz_dict_name – name of the Morfeusz dictionary (if not using the default one).
pretrained_tokenizer – tokenizer for the language model.
tokenizer_kwargs – addditonal kwargs for the language model tokenizer.
- Returns:
a Hydra instance.
- classmethod load(path: str, morfeusz_dict_path: str | None = None) Hydra[source]
Load a trained Hydra model.
- Returns:
a Hydra instance.
- parse(sentences: list[str] | str, correct_lemmata: bool = False, return_jsons: bool = False, return_conllu: bool = False, return_trees: bool = False, return_labels: bool = False, return_logits: bool = False, root_label: str | None = None, force_root_label: bool = False, skip_long: bool = False, is_tokenized: bool = False, morph_postprocess: Callable[[str, str, str], tuple[str, str]] | None = None, mst: str = 'mst', return_times: bool = False, mem_limit: int | None = None, verbose: bool = False) list[HydraAnalysis] | tuple[list[HydraAnalysis], dict[str, int]][source]
Parse a single sentence or a list of sentences.
- Parameters:
sentences – sentence(s) to process.
correct_lemmata – apply lemmata correction using Morfeusz.
return_jsons – include full json outputs in the results. Only available for models trained with hybrid=True.
return_conllu – include ConLLu outputs in the results. Only available for models trained with dependency=True.
return_trees – include tree objects in the results. Only available for models trained with hybrid=True.
return_labels – include label lists in the results.
return_logits – include raw predicted logits in the results.
root_label – used by force_root_label.
force_root_label – enforce a specific root label in resulting trees.
skip_long – omit sentences with too many tokens (an error message will be logged in the sentences’ metadata).
is_tokenized – don’t perform segmentation (treat spaces as segment delimiters).
morph_postprocess – custom postprocessing function for tags and lemmata.
mst – dependency tree decoding algorithm to use:
MSTorEAGER.return_times – include parsing time info in the results.
mem_limit – memory limit (experimental).
verbose – include additional info in the results.
- Returns:
the parsing results.
- process_labels(labels: LabelDict, tokens: TokenList, correct_lemmata: bool = False, morph_postprocess: Callable[[str, str, str], tuple[str, str]] | None = None, mst: str = 'mst', allow_masked: bool = False) LabelDict[source]
Process model-predicted labels.
- Parameters:
- Returns:
labels that can be returned as parsing result or used to construct a tree.
- class hydra.Trainer(task: ClassificationTask)[source]
Bases:
objectUsed for creating, training and evaluating instances of Hydra parser.
- cross_validate(nfolds: int = 10, epochs: int = 10, patience: int = 4, lr: float = 1e-05, mst: str = 'mst', allow_masked: bool = False) list[EvalResults][source]
Perform 10-fold cross validation.
- Returns:
results of each fold run.
- evaluate(parser: Hydra, mst: str = 'mst', allow_masked: bool = False, dataset: DatasetDict | None = None) EvalResults[source]
Perform evaluation of a trained Hydra model.
- Parameters:
parser – the parser instance to be evaluated.
mst – dependency tree decoding algoriths to be used.
allow_masked – accept masked labels in gold data (but exclude them from metric computation).
dataset – source of gold instances. Evaluation will be run on ‘validation’ and any ‘test*’ parts. Training dataset will be used as default if this argument is not provided.
- Returns:
evaluation results.
- train(epochs: int = 10, patience: int = 4, lr: float = 1e-05, log_dir: str | None = None, model_dir: str | None = None, dataset: DatasetDict | None = None) Hydra[source]
Create and train a parser.
- Parameters:
epochs – number of epochs.
patience – how many epochs are allowed with no training progress. The best model across all epochs is then retrieved.
lr – learning rate.
log_dir – where to save training logs.
model_dir – where to save the final model.
dataset – training data. Must contain ‘train’ and ‘validation’ parts (used for training), with any number of ‘test*’ parts for later evaluation.
- Returns:
a parser instance.