[Python] Langsmith TypeError: DynamicRunEvaluator.evaluate_run() got an unexpected keyword...

Error on running evaluation on dataset with ground truth to check generated SQL queries.

Getting Error for missing evaluator_run_id, not able to pass in evaluate function.

from langsmith import wrappers, Client
from pydantic import BaseModel, Field
from openai import OpenAI

client = Client()
openai_client = wrappers.wrap_openai(OpenAI())

def target(inputs: dict) -> dict:
payload = {
"flowId": "xxx",
"input": {
"input_query": inputs['question']
},
"operateCmd": "testFlow",
"userName": 'xxx'
}

response = requests.post(url, headers=headers, json=payload)
print(response.json()['data']['result'])
return response.json()['data']['result']

experiment_results = client.evaluate(
target,
data="Sample dataset",
evaluators=[
accuracy,
],
max_concurrency=2,
)


Error running evaluator <DynamicRunEvaluator accuracy> on run 2e488952-ce0f-47a7-a112-adc49f49dfc3: TypeError("DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id)
Traceback (most recent call last):
File c:\Python313\Lib\site-packages\langsmith\evaluation\_runner.py, line 1634, in _run_evaluators
evaluator_response = evaluator.evaluate_run( # type: ignore[call-arg]
run=run,
example=example,
evaluator_run_id=evaluator_run_id,
)
TypeError: DynamicRunEvaluator.evaluate_run() got an unexpected keyword argument evaluator_run_id

Continue reading...