As you already know, MarkovML uses Recorder objects to record data into the MarkovML backend. In addition to the Experiment Recorder, MarkovML features an Evaluation Recorder dedicated to capturing all evaluation-related data within the MarkovML environment.

Create Model Object

You can run a model evaluation against a registered model object. There are two ways to create a model object on MarkovML.

1. By running an experiment using an experiment recorder
2. Creating a placeholder model object in a project

📘

Note

The model object is a harness for the model artifact generated from a model-training process.

1. Create a Model Object using an Experiment Recorder

When an experiment is run, the model object is automatically created and registered with MarkovML. You can get the model_id of the associated model object by the model_id property of the ExperimentRecorder instance.

2. Create a Model Object when using Integrations with ML frameworks

As you are aware, experiments can be easily created using integrations with supported ML frameworks. You can utilize the auto_record() method to streamline this process. This section elaborates on how you can retrieve the model_id from an experiment created using auto_record().

When you initiate an experiment using auto_record(), the model object is automatically generated and registered with MarkovML. For instance, if you have created an experiment using a Keras model with markov.keras.auto_recordonce you have called the model.fit() method, you can access the model_id by referencing model.markov_model_id on the model on which fit() was called.

3. Creating a New Model Object

You can also create the Model placeholder explicitly using Markov SDK. Use the create_model() to create model object placeholder for the Project and register it with MarkovML using register(). Provide the following details to use the create_model()

1. model_name: Name of the model placeholder for the Project.
2. model_description: Add some description for future reference.
3. model_class: State the model class, for example, classification model.

Complete Sample code

from markov import Project, ModelClass

# get existing project by name
my_project = Project.get_by_name(project_name="MarkovMLTestProject")
# or get existing project by id
my_project = Project.get_by_id(project_id="project_id")
# If the model object does not exist, create a placeholder using this code
my_model = my_project.create_model(
    model_name="MyModel",
    model_description="Model created from project object",
    model_class=ModelClass.CLASSIFICATION
)

# Call this method to register the Model with MarkovML backend. 
my_model.register()

To run an evaluation against an existing model, you can get the model object as follows.

📘

Note

The evaluation dataset should be registered with MarkovML. This is required to create a lineage between the dataset and evaluations. This also helps in root cause analysis to plan future iterations.

Getting Existing Model for Evaluation

You can easily retrieve a project in MarkovML using the Project.get_by_name() method by specifying the project name or using the Project.get_by_id() method by providing the project ID. Similarly, you can obtain an existing model within a project using the get_model_by_name() function, where you specify the model name, or by utilizing the get_model_by_id() function, providing the model ID. These methods allow you to efficiently access projects and models within your MarkovML workspace.

from markov import Project

my_project = Project.get_by_name(project_name="MarkovMLTestProject")
# or get existing project by id
my_project = Project.get_by_id(project_id="project_id")

# get an existing Model of this project by name
my_model = my_project.get_model_by_name(model_name="MyFavoriteModel")
# or get Model by model_id
my_model = my_project.get_model_by_id(model_id="model_id")

Create an Evaluation Recorder

To create and register an EvaluationRecorderin MarkovML, you can utilize the EvaluationRecorder(). First, ensure you have the necessary information such as the model_id obtained from a previously registered model and the dataset_id of the dataset used for evaluation.

With these details, instantiate an EvaluationRecorder object, providing a name and any relevant notes. Then, register the recorder with MarkovML using the register() method. This process enables the recording of evaluation-related data for further analysis and tracking within the MarkovML backend.

Provide the following information to use the EvaluationRecorder:

1. name: Add the evaluating model name.
2. notes: Notes for future reference.
3. model_id: Model ID of a registered model you wish to perform evaluation tasks on.
4. dataset_id: Dataset ID of the registered dataset used for the model training.
5. pos_label: If you are evaluating a binary classifier, set the positive label to compute the AUC/PR curve.

Sample code

from markov import EvaluationRecorder
import markov

# You can use the model_id from my_model.model_id to create
# and register EvaluationRecorder
model_id = my_model.model_id
dataset_id = markov.dataset.get_by_name("my_dataset").ds_id

evaluation_recorder = EvaluationRecorder(
    name=f"Evaluating model YOUR_MODEL_NAME",
    notes=f"Testing evaluation with MarkovML",
    model_id= model_id,
    # dataset_id these evaluation records belong to. 
    # You should register the evaluation dataset with MarkovML 
    # for lineage and advanced analytics. 
    dataset_id=dataset_id,
    # If you are evaluating a binary classifier, set the positive label
    # to compute the AUC/PR curve. POS_LABEL provided.
    pos_label = "POS_LABEL"
)

# This method should be called to register a recorder with MarkovML to accept
# the incoming evaluation records.
evaluation_recorder.register()

After creating an evaluation recorder, you can proceed to input evaluation records.

You can evaluate a model multiple times by supplying the same model_id to your EvaluationRecorder.