[Python] Answer evaluation with Azure AI

I would like to create an answer evaluation app for test practice. In this concept, the question is What is the capital of France?

For the test, I put a static possible answer in the code and I would like to compare the input to this: The capital of France is Paris. It's a famous city known for its art, fashion, and landmarks like the Eiffel Tower and the Louvre Museum. The idea is that in the final version, I either use ChatGPT model to get the answer, or I will have pre-defined question/answer pairs and compare that to the kid's input. A nice-to-have feature would be to have a percentage, like how detailed is the answer compared to the pre-defined one.

The evaluation results are not really good: |example answer|result| |:-----|:----:| |Paris | True| |France | True| |Amsterdam | False |Amsterdam is not the capital of France | True |Paris is not the capital of France | True |Paris is the capital of France | True| |Budapest is the capital of Hungary|True|

What is the correct approach to have a reasonable good evaluation? I would like to first check if the answer itself is factually correct, then if is similar to the correct answer. The code I have here is below, so far (maybe a bit chaotic, there were many iterations). Copilot recommends sentence_transformers, but I would like to use Azure built-in features.

from flask import Flask, request, jsonify
from azure.ai.textanalytics import TextAnalyticsClient
from azure.core.credentials import AzureKeyCredential
import json
import os
from dotenv import load_dotenv

app = Flask(__name__)

load_dotenv()

# Azure Cognitive Services configuration
AZURE_COGNITIVE_SERVICES_ENDPOINT = os.getenv("AZURE_COGNITIVE_SERVICES_ENDPOINT")
AZURE_COGNITIVE_SERVICES_KEY = os.getenv("AZURE_COGNITIVE_SERVICES_KEY")

text_analytics_client = TextAnalyticsClient(
endpoint=AZURE_COGNITIVE_SERVICES_ENDPOINT,
credential=AzureKeyCredential(AZURE_COGNITIVE_SERVICES_KEY)
)

with open('questions.json') as f:
questions = json.load(f)

# Static correct answer for testing
correct_answer = "The capital of France is Paris. It's a famous city known for its art, fashion, and landmarks like the Eiffel Tower and the Louvre Museum."

def evaluate_answer(user_answer, correct_answer):
# Extract key phrases
documents = [user_answer, correct_answer]
key_phrases_response = text_analytics_client.extract_key_phrases(documents=documents)
user_key_phrases = set(key_phrases_response[0].key_phrases)
correct_key_phrases = set(key_phrases_response[1].key_phrases)
key_phrase_match = user_key_phrases.intersection(correct_key_phrases)

# Perform sentiment analysis
sentiment_response = text_analytics_client.analyze_sentiment(documents=documents)
user_sentiment = sentiment_response[0].sentiment
correct_sentiment = sentiment_response[1].sentiment

# Check if key phrases match and sentiments align
return len(key_phrase_match) > 0 and user_sentiment == correct_sentiment

@app.route('/evaluate', methods=['POST'])
def evaluate():
user_answer = request.json.get('answer')
question = questions[0]['question']

# Evaluate the user's answer using Azure Cognitive Services
match = evaluate_answer(user_answer, correct_answer)

evaluation = {'question': question, 'correct_answer': correct_answer, 'match': match}

return jsonify({'evaluation': evaluation})

if __name__ == '__main__':
app.run(port=3000)

Continue reading...