[Python] Langchain workaround for with_structured_output using ChatBedrock

I'm working with the langchain library to implement a document analysis application. Especifically I want to use the routing technique described in this documentation. i wanted to follow along the example, but my environment is restricted to AWS, and I am using ChatBedrock instead of ChatOpenAI due to limitations with my deployment.

According to this overview the with_structured_output method, which I need, is not (yet) implemented for models on AWS Bedrock, which is why I am looking for a workaround or any method to replicate this functionality.

The key functionality I am looking for is shown in this example:

from typing import List
from typing import Literal

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.pydantic_v1 import BaseModel, Field
from langchain_openai import ChatOpenAI



class RouteQuery(BaseModel):
"""Route a user query to the most relevant datasource."""

datasources: List[Literal["python_docs", "js_docs", "golang_docs"]] = Field(
...,
description="Given a user question choose which datasources would be most relevant for answering their question",
)

system = """You are an expert at routing a user question to the appropriate data source.

Based on the programming language the question is referring to, route it to the relevant data source."""
prompt = ChatPromptTemplate.from_messages(
[
("system", system),
("human", "{question}"),
]
)

llm = ChatOpenAI(model="gpt-3.5-turbo-0125", temperature=0)
structured_llm = llm.with_structured_output(RouteQuery)
router = prompt | structured_llm
router.invoke(
{
"question": "is there feature parity between the Python and JS implementations of OpenAI chat models"
}
)


The output would be:

RouteQuery(datasources=['python_docs', 'js_docs'])


The most important fact for me is that it just selects items from the list without any additional overhead, which makes it possible to setup the right follow up questions.

Did anyone find a workaround how to resolve this issue?

Continue reading...