Send Custom Metric with Each Record
You can also add any custom_metric to your record for bookkeeping. Custom metrics are business metrics that you can include for further analysis. For example, if your business has low latency requirements, you might want to capture inference time as a custom metric to evaluate performance.
Note
Custom metrics should be numeric.
The below sample code outlines the process of creating and registering a recorder to capture evaluation data in MarkovML.
How does it work?
Within the validation loop, predictions and corresponding scores are generated for each input value by the model. These predictions are then encapsulated within SingleTagInferenceRecord objects along with actual labels and scores.
Additionally, custom metrics can be computed and added to the records using the add_custom_metric method. Once all records have been processed, the recording is closed by calling finish, triggering the computation of the evaluation report on the MarkovML backend. This streamlined approach ensures comprehensive tracking and analysis of model predictions, including custom metrics tailored to specific business needs.
Sample Code
# create recorder
recorder = ...
recorder.register()
def your_custom_metric():
# optional: method to compute custom metric if required
....
# Your validation loop that goes over each record to get a score from the model
while ...
predicted_label, score = model.predict(input_value)
urid= recorder.gen_urid(input_value)
record = SingleTagInferenceRecord(urid=urid,
inferred=predicted_label,
actual="YOUR_ACTUAL_LABEL",
score=score)
cust_metric = your_custom_metric()
record.add_custom_metric(label='custom_metric', value=cust_metric)
# Finish New LiIn addition, you can include any custom metrics
# in your record for bookkeeping purposes. These metrics are specific to your business and can be utilized for more in-depth analysis. For instance, if your business demands quick response times, you may want to track inference time as a custom metric for evaluating overall performance.nes to add custom metrics
recorder.add_record(record)
# Call finish to close the recording on MarkovML backend to
# start computation of evaluation report.
evaluation_recorder.finish()
Updated 2 months ago