Send Additional Metadata with Each Record
You can also send additional metadata with each record for future analysis. For example, you might want to send all the probabilities generated by the soft-max classifier for different classes. The supported meta_types
are here.
How does it work?
The below sample code demonstrates the process of creating and registering a recorder to capture evaluation data in MarkovML. Within the validation loop, predictions and corresponding scores are generated for each input value by the model. These predictions, along with actual labels and scores, are encapsulated within SingleTagInferenceRecord
objects.
Additionally, metadata in JSON format is included for each record, providing additional context or information related to the predictions. This metadata, such as class probabilities, is added to the records using the add_meta_data_instance
method.
Once all records have been processed, the recording is closed by calling finish, signaling the completion of the evaluation data collection process in MarkovML. This enables comprehensive tracking and analysis of model predictions, including relevant metadata, to facilitate informed decision-making and performance evaluation.
Sample Code
import json
# create recorder code
recorder = ...
recorder.register()
while ...# Your validation loop
# This is the prediction from your model and score
predicted_label, score = model.predict(input_value)
urid= recorder.gen_urid(input_value)
record = SingleTagInferenceRecord(urid=urid,
inferred=predicted_label,
actual="YOUR_ACTUAL_LABEL",
score=score)
## New Lines to send additional metadata to MarkovML for each record
## This will store the JSON as a string
prob_json = json.dumps({"class_a":0.75,"class_b":0.20,"class_c":0.05})
record.add_meta_data_instance(key="probabilities",
value=prob_json,
meta_type=RecordMetaType.Text)
## To visualize the prob_json as a histogram, save it as RecordMetaType.HISTOGRAM
## This is helpful when you are interested in visualizing relative value
prob_json = {"class_a":0.75,"class_b":0.20,"class_c":0.05}
record.add_meta_data_instance(key="probabilities",
value=prob_json,
meta_type=RecordMetaType.Histogram)
##New Lines to send additional metadata to MarkovML for each record
recorder.add_record(record)
evaluation_recorder.finish()
Updated about 2 months ago