Add Evaluation Record

The previous section covered how to register an evaluation recorder. This guide covers how you can use this recorder to add individual records to a MarkovML evaluation recording. Model evaluation can be performed either at the end of the training process or offline at a later time. MarkovML supports both modes, allowing you to conveniently record and track evaluation metrics for your models.

📘

Note

The records are sent to the MarkovML backend asynchronously using multiple worker threads.

Adding Evaluation Record to Recorder

MarkovML supports users in evaluating models that provide outcomes with single labels or multiple labels. For instance, it can handle binary classification tasks (assigning one label to each outcome) or tasks like identifying multiple objects in an image (assigning multiple labels).

SingleTagInferenceRecord

This record is used to store and send evaluation data for models that assign only one label to each outcome in an evaluation dataset.

MultiTagInferenceRecord

This record is used to store and send evaluation data for models that assign multiple labels to a single outcome in an evaluation dataset.

How it works

The process involves creating either a SingleTagInferenceRecord or a MultiTagInferenceRecord and then adding it to the evaluation recorder. This simple two-step process allows users to efficiently evaluate their models and record the results for analysis and tracking.

Psuedo-Code Sample

The pseudo-code will look like this,

1. Create a SingleTagInferenceRecord or MultiTagInferenceRecord
2. Add SingleTagInferenceRecord or MultiTagInferenceRecord to the recorder

Sample Code to Register EvaluationRecorder to MarkovML

from markov import EvaluationRecorder

evaluation_recorder = EvaluationRecorder(
    name=f"Evaluating model YOUR_MODEL_NAME",
    notes=f"Testing evaluation with MarkovML",
    model_id="YOUR_MODEL_ID"  # or my_model.model_id
)
# This method registers a recorder with Markov for the backend to process the records
# sent by this recorder. 
evaluation_recorder.register()