hydra.utils.evaluation

Model evaluation utilities.

class hydra.utils.evaluation.AccuracyMetric(name: str)[source]

Bases: TokenMetric

Base class for any evaluation metric calculating token-level accuracy.

calculate_token_metric(true: list[Any], pred: list[Any]) → Any[source]

Calculate the accuracy score.

Returns:

the numeric score or ‘N/A’ if no unmasked data was provided.

class hydra.utils.evaluation.ConfusionMatrix(name: str)[source]

Bases: TokenMetric

Confusion matrix calculation.

calculate_token_metric(true: list[Any], pred: list[Any]) → Any[source]
class hydra.utils.evaluation.DepArcs[source]

Bases: PrecisionRecallFscore

Precision/recall/F1 metrics for dependency arcs.

get_unit_key(unit: tuple[int, tuple[str, int]]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[int, tuple[str, int | None]]] | None[source]
class hydra.utils.evaluation.DepArcsConfusionMatrix[source]

Bases: ConfusionMatrix

Confusion matrix for dependency arcs (heads and relation labels).

calculate_token_metric(true: list[Any], pred: list[Any]) → Any[source]
get_labels(tags: LabelDict) → list[tuple[str, int]] | None[source]
class hydra.utils.evaluation.Deprels[source]

Bases: PrecisionRecallFscore

Precision/recall/F1 metrics for dependency relation labels.

get_unit_key(unit: tuple[int, str]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[int, str]] | None[source]
class hydra.utils.evaluation.DeprelsConfusionMatrix[source]

Bases: ConfusionMatrix

Confusion matrix for dependency relation labels.

get_labels(tags: LabelDict) → list[str] | None[source]
class hydra.utils.evaluation.ELAS[source]

Bases: PrecisionRecallFscore

Enhanced Labeled Attachment Score (standard and enhanced edges).

get_unit_key(unit: tuple[int, tuple[str, int]]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[int, tuple[str, int | None]]] | None[source]
class hydra.utils.evaluation.EUAS[source]

Bases: PrecisionRecallFscore

Enhanced Unlabeled Attachment Score (standard and enhanced edges).

get_unit_key(unit: tuple[int, int]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[int, int | None]] | None[source]
class hydra.utils.evaluation.Edges[source]

Bases: PrecisionRecallFscore

Precision/recall/F1 metrics operating on hybrid tree edges.

get_unit_key(unit: tuple[str, str, int, int, str]) → tuple[str, str, str][source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[str, str, int, int, str]] | None[source]
class hydra.utils.evaluation.EvaluationMetric(name: str)[source]

Bases: object

Base class for any evaluation metric.

calculate() → dict[str, Any][source]

Calculate the overall metric result after being fed all the data.

Returns:

total seen and excluded because of masking.

Return type:

any scores and statistics assoaciated with the metric, number of instances

do_calculate() → Any[source]

Perform metric-specific calculations of the overall metric result after being fed all the data.

Subclasses must implement this method.

Raises:

NotImplementedError – Always.

update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) → None[source]

Gather any statistics needed for calculating this metric from a true/predicted sentence pair.

Subclasses must implement this method.

Raises:

NotImplementedError – Always.

class hydra.utils.evaluation.GLAS[source]

Bases: PrecisionRecallFscore

Graph Labeled Attachment Score (enhanced edges only).

get_unit_key(unit: tuple[int, tuple[str, int]]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[int, tuple[str, int | None]]] | None[source]
class hydra.utils.evaluation.GUAS[source]

Bases: PrecisionRecallFscore

Graph Unlabeled Attachment Score (enhanced edges only).

get_unit_key(unit: tuple[int, int]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[int, int | None]] | None[source]
class hydra.utils.evaluation.HeadedSpans(**kwargs: Any)[source]

Bases: Spans

Precision/recall/F1 metrics operating on tree constituents.

Takes token spans, syntactic categories and head marking into account.

class hydra.utils.evaluation.HeadedSpansDiscontinuous(**kwargs: Any)[source]

Bases: Spans

Like HeadedSpans, but considers only discontinuous constituents.

class hydra.utils.evaluation.LAS[source]

Bases: AccuracyMetric

Labeled Attachment Score.

get_labels(tags: LabelDict) → list[str] | None[source]

Gather dependency head indices and dependency relation labels.

Strips UD-style language-specific dependency relation sublabels (anything after a semicolon).

Returns:

indices with deprels or None if masked.

class hydra.utils.evaluation.Lemmatisation[source]

Bases: AccuracyMetric

Lemmatisation accuracy.

get_labels(tags: LabelDict) → list[str] | None[source]

Gather lemmata.

Returns:

lemmata or None if masked.

class hydra.utils.evaluation.LemmatisationCorr(morfeusz: Morfeusz)[source]

Bases: AccuracyMetric

Lemmatisation accuracy when Morfeusz correction is applied.

update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) → None[source]

Override update() to inject Morfeusz correction.

class hydra.utils.evaluation.NER[source]

Bases: PrecisionRecallFscore

Precision/recall/F1 metrics for named entities.

get_unit_key(unit: tuple[tuple[int, ...], str]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[int, tuple[str, int | None]]] | None[source]
class hydra.utils.evaluation.NonHeadedSpans(**kwargs: Any)[source]

Bases: Spans

Precision/recall/F1 metrics operating on tree constituents.

Takes token spans and syntactic categories into account.

class hydra.utils.evaluation.NonHeadedSpansDiscontinuous(**kwargs: Any)[source]

Bases: Spans

Like NonHeadedSpans, but considers only discontinuous constituents.

class hydra.utils.evaluation.PrecisionRecallFscore(name: str)[source]

Bases: EvaluationMetric

Precision, recall and F1-score reported for entire data and specific types of predicted units.

do_calculate() → Any[source]

Perform calculations of the overall metric results after being fed all the data.

Returns:

calculated results.

get_unit_key(unit: Any) → Any[source]

Get unit type for calculating partial scores (e.g. a constituent type for tree spans).

Subclasses must implement this method.

Raises:

NotImplementedError – Always.

get_units(tree: TreeNode | None, tags: LabelDict) → set[Any] | None[source]

Gather predicted units (spans, constituency edges…) to be compared between gold and predicted sequences.

Subclasses must implement this method.

Raises:

NotImplementedError – Always.

update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) → None[source]

Gather units needed for calculating this metric from a true/predicted sentence pair.

Metric-specific unit gathering logic must be implemented by overriding get_units() and get_unit_key().

class hydra.utils.evaluation.Spans(name: str, labeled: bool, headed: bool, discontinuous_only: bool = False, skip_categories: list[str] | None = None)[source]

Bases: PrecisionRecallFscore

Base class for precision/recall/F1 metrics operating on tree constituents.

Implements the logic shared between specific metrics.

get_unit_key(unit: tuple[tuple[int, ...], str, bool]) → str[source]
get_units(tree: TreeNode | None, tags: LabelDict) → set[tuple[tuple[int, ...], str, bool]] | None[source]
class hydra.utils.evaluation.Tagging[source]

Bases: AccuracyMetric

Tagging accuracy.

get_labels(tags: LabelDict) → list[str] | None[source]

Gather inflectional tags.

Returns:

tags or None if masked.

class hydra.utils.evaluation.TaggingCorr(morfeusz: Morfeusz)[source]

Bases: AccuracyMetric

Tagging accuracy when Morfeusz correction is applied.

update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) → None[source]

Override update() to inject Morfeusz correction.

class hydra.utils.evaluation.TokenMetric(name: str)[source]

Bases: EvaluationMetric

Base class for any evaluation metric operating on token level.

calculate_token_metric(true: list[Any], pred: list[Any]) → Any[source]

Calculate a specific metric operating on token level.

Subclasses must implement this method.

Raises:

NotImplementedError – Always.

do_calculate() → Any[source]

Perform metric-specific calculations of the overall metric result after being fed all the data.

Metric-specific calculation logic must be implemented by overriding calculate_token_metric().

Returns:

calculated result or ‘N/A’ if not possible.

get_labels(tags: LabelDict) → list[Any] | None[source]

Gather token labels to be compared between gold and predicted sequences.

Subclasses must implement this method.

Raises:

NotImplementedError – Always.

update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) → None[source]

Gather token-level labels needed for calculating this metric from a true/predicted sentence pair.

Metric-specific label gathering logic must be implemented by overriding get_labels().

class hydra.utils.evaluation.UAS[source]

Bases: AccuracyMetric

Unlabeled Attachment Score.

get_labels(tags: LabelDict) → list[int] | None[source]

Gather dependency head indices.

Returns:

indices or None if masked.

class hydra.utils.evaluation.UnlabeledSpans(**kwargs: Any)[source]

Bases: Spans

Precision/recall/F1 metrics operating on tree constituents.

Only takes token spans into account.

class hydra.utils.evaluation.UnlabeledSpansDiscontinuous(**kwargs: Any)[source]

Bases: Spans

Like UnlabeledSpans, but considers only discontinuous constituents.