hydra.utils.evaluation
Model evaluation utilities.
- class hydra.utils.evaluation.AccuracyMetric(name: str)[source]
Bases:
TokenMetricBase class for any evaluation metric calculating token-level accuracy.
- class hydra.utils.evaluation.ConfusionMatrix(name: str)[source]
Bases:
TokenMetricConfusion matrix calculation.
- class hydra.utils.evaluation.DepArcs[source]
Bases:
PrecisionRecallFscorePrecision/recall/F1 metrics for dependency arcs.
- class hydra.utils.evaluation.DepArcsConfusionMatrix[source]
Bases:
ConfusionMatrixConfusion matrix for dependency arcs (heads and relation labels).
- class hydra.utils.evaluation.Deprels[source]
Bases:
PrecisionRecallFscorePrecision/recall/F1 metrics for dependency relation labels.
- class hydra.utils.evaluation.DeprelsConfusionMatrix[source]
Bases:
ConfusionMatrixConfusion matrix for dependency relation labels.
- class hydra.utils.evaluation.ELAS[source]
Bases:
PrecisionRecallFscoreEnhanced Labeled Attachment Score (standard and enhanced edges).
- class hydra.utils.evaluation.EUAS[source]
Bases:
PrecisionRecallFscoreEnhanced Unlabeled Attachment Score (standard and enhanced edges).
- class hydra.utils.evaluation.Edges[source]
Bases:
PrecisionRecallFscorePrecision/recall/F1 metrics operating on hybrid tree edges.
- class hydra.utils.evaluation.EvaluationMetric(name: str)[source]
Bases:
objectBase class for any evaluation metric.
- calculate() dict[str, Any][source]
Calculate the overall metric result after being fed all the data.
- Returns:
total seen and excluded because of masking.
- Return type:
any scores and statistics assoaciated with the metric, number of instances
- do_calculate() Any[source]
Perform metric-specific calculations of the overall metric result after being fed all the data.
Subclasses must implement this method.
- Raises:
NotImplementedError – Always.
- update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) None[source]
Gather any statistics needed for calculating this metric from a true/predicted sentence pair.
Subclasses must implement this method.
- Raises:
NotImplementedError – Always.
- class hydra.utils.evaluation.GLAS[source]
Bases:
PrecisionRecallFscoreGraph Labeled Attachment Score (enhanced edges only).
- class hydra.utils.evaluation.GUAS[source]
Bases:
PrecisionRecallFscoreGraph Unlabeled Attachment Score (enhanced edges only).
- class hydra.utils.evaluation.HeadedSpans(**kwargs: Any)[source]
Bases:
SpansPrecision/recall/F1 metrics operating on tree constituents.
Takes token spans, syntactic categories and head marking into account.
- class hydra.utils.evaluation.HeadedSpansDiscontinuous(**kwargs: Any)[source]
Bases:
SpansLike
HeadedSpans, but considers only discontinuous constituents.
- class hydra.utils.evaluation.LAS[source]
Bases:
AccuracyMetricLabeled Attachment Score.
- class hydra.utils.evaluation.Lemmatisation[source]
Bases:
AccuracyMetricLemmatisation accuracy.
- class hydra.utils.evaluation.LemmatisationCorr(morfeusz: Morfeusz)[source]
Bases:
AccuracyMetricLemmatisation accuracy when Morfeusz correction is applied.
- class hydra.utils.evaluation.NER[source]
Bases:
PrecisionRecallFscorePrecision/recall/F1 metrics for named entities.
- class hydra.utils.evaluation.NonHeadedSpans(**kwargs: Any)[source]
Bases:
SpansPrecision/recall/F1 metrics operating on tree constituents.
Takes token spans and syntactic categories into account.
- class hydra.utils.evaluation.NonHeadedSpansDiscontinuous(**kwargs: Any)[source]
Bases:
SpansLike
NonHeadedSpans, but considers only discontinuous constituents.
- class hydra.utils.evaluation.PrecisionRecallFscore(name: str)[source]
Bases:
EvaluationMetricPrecision, recall and F1-score reported for entire data and specific types of predicted units.
- do_calculate() Any[source]
Perform calculations of the overall metric results after being fed all the data.
- Returns:
calculated results.
- get_unit_key(unit: Any) Any[source]
Get unit type for calculating partial scores (e.g. a constituent type for tree spans).
Subclasses must implement this method.
- Raises:
NotImplementedError – Always.
- get_units(tree: TreeNode | None, tags: LabelDict) set[Any] | None[source]
Gather predicted units (spans, constituency edges…) to be compared between gold and predicted sequences.
Subclasses must implement this method.
- Raises:
NotImplementedError – Always.
- update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) None[source]
Gather units needed for calculating this metric from a true/predicted sentence pair.
Metric-specific unit gathering logic must be implemented by overriding
get_units()andget_unit_key().
- class hydra.utils.evaluation.Spans(name: str, labeled: bool, headed: bool, discontinuous_only: bool = False, skip_categories: list[str] | None = None)[source]
Bases:
PrecisionRecallFscoreBase class for precision/recall/F1 metrics operating on tree constituents.
Implements the logic shared between specific metrics.
- class hydra.utils.evaluation.Tagging[source]
Bases:
AccuracyMetricTagging accuracy.
- class hydra.utils.evaluation.TaggingCorr(morfeusz: Morfeusz)[source]
Bases:
AccuracyMetricTagging accuracy when Morfeusz correction is applied.
- class hydra.utils.evaluation.TokenMetric(name: str)[source]
Bases:
EvaluationMetricBase class for any evaluation metric operating on token level.
- calculate_token_metric(true: list[Any], pred: list[Any]) Any[source]
Calculate a specific metric operating on token level.
Subclasses must implement this method.
- Raises:
NotImplementedError – Always.
- do_calculate() Any[source]
Perform metric-specific calculations of the overall metric result after being fed all the data.
Metric-specific calculation logic must be implemented by overriding
calculate_token_metric().- Returns:
calculated result or ‘N/A’ if not possible.
- get_labels(tags: LabelDict) list[Any] | None[source]
Gather token labels to be compared between gold and predicted sequences.
Subclasses must implement this method.
- Raises:
NotImplementedError – Always.
- update(tokens: TokenList, tags_true: LabelDict, tags_pred: LabelDict, tree_true: TreeNode | None, tree_pred: TreeNode | None) None[source]
Gather token-level labels needed for calculating this metric from a true/predicted sentence pair.
Metric-specific label gathering logic must be implemented by overriding
get_labels().
- class hydra.utils.evaluation.UAS[source]
Bases:
AccuracyMetricUnlabeled Attachment Score.
- class hydra.utils.evaluation.UnlabeledSpans(**kwargs: Any)[source]
Bases:
SpansPrecision/recall/F1 metrics operating on tree constituents.
Only takes token spans into account.
- class hydra.utils.evaluation.UnlabeledSpansDiscontinuous(**kwargs: Any)[source]
Bases:
SpansLike
UnlabeledSpans, but considers only discontinuous constituents.