[Python] Langchain with_structured_output() does not work with HuggingFace

When I create the LLM instance by calling the with_structured_output() I get an error of

HfHubHTTPError: 422 Client Error: Unprocessable Entity for url: https://api-inference.huggingface.co/models/meta-llama/Llama-3.1-70B-Instruct/v1/chat/completions

Failed to deserialize the JSON body into the target type: tool_choice: data did not match any variant of untagged enum ToolTypeDeserializer at line 1 column 1377


I am using the model Llama-3.1-70B-Instruct. I am relatively new to this and I am not sure if it is an issue with HF or Langchain. Tried the exact same thing with ChatGroq and yet it works. Is there any work around with HF except manual parsing?

llm = HuggingFaceEndpoint(
repo_id="meta-llama/Llama-3.1-70B-Instruct",
task="text-generation",
)

chat_model = ChatHuggingFace(llm=model, verbose=True)
structured_chat_model = llm.with_structured_output(ContextPreparerOutput)


According to LangChain's documentation, the method should also work with the HuggingFaceEndpoint. But when i called the method, what i get was a NotImplementedError.

Continue reading...