[Python] Chatbot Not Responding: Colab FastAPI + Flask + Ngrok Setup

I'm setting up a chatbot where the language model runs on Google Colab (FastAPI exposed via Ngrok), and a local Flask app serves the HTML frontend, acting as a proxy. The Ngrok tunnel for the Colab FastAPI appears to be running, but the chatbot isn't providing responses.

I suspect there's a disconnect in how the components are communicating.

Current Setup Details

1. Colab Code (FastAPI + Ngrok): This code runs in Google Colab to host the language model (Mistral) via a FastAPI endpoint and expose it publicly using pyngrok.

!pip install fastapi uvicorn nest_asyncio pyngrok transformers torch
from fastapi import FastAPI
from pydantic import BaseModel
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
import uvicorn
import nest_asyncio
import threading
from pyngrok import ngrok
import time # Added for keeping Colab alive

# Apply the fix for Colab event loop
nest_asyncio.apply()

model_path = "/content/drive/MyDrive/mistral-finetuned-alpaca"
tokenizer = AutoTokenizer.from_pretrained(model_path)
model = AutoModelForCausalLM.from_pretrained(model_path, device_map="auto")

# Define FastAPI app
app = FastAPI()

class RequestModel(BaseModel):
message: str

@app.post("/predict")
def predict(data: RequestModel):
input_text = data.message
# Ensure inputs are moved to the correct device (e.g., "cuda" if available)
inputs = tokenizer(input_text, return_tensors="pt").to(model.device)
output = model.generate(**inputs, max_length=100)
return {"response": tokenizer.decode(output[0], skip_special_tokens=True)}

# Function to run FastAPI
def run_fastapi():
uvicorn.run(app, host="0.0.0.0", port=8000)

# Run FastAPI in a separate thread
fastapi_thread = threading.Thread(target=run_fastapi)
fastapi_thread.start()

ngrok.set_auth_token("###################") # (Placeholder, actual token used)


colab_public_url = ngrok.connect(addr="8000", proto="http")
print("Colab FastAPI Public URL:", colab_public_url)

# Keep the Colab environment alive
while True:
time.sleep(60)

1. Flask Code (Local Proxy) This Flask app runs on my local machine, serves chat.html, and proxies requests to the Colab FastAPI.

from flask import Flask, render_template, request, jsonify
import requests

app = Flask(__name__)


COLAB_API_BASE_URL = "https://2b19-24-105-72-44.ngrok-free.app" # Example URL from Colab

@app.route("/", methods=["GET"])
def home():
return render_template("chat.html")

@app.route("/chat", methods=["POST"])
def chat():
user_input = request.json.get("message")

try:
response = requests.post(f"{COLAB_API_BASE_URL}/predict", json={"message": user_input})

if response.status_code == 200:
bot_reply = response.json()["response"]
return jsonify({"answer": bot_reply})
else:
print(f"Error from Colab API: {response.status_code} - {response.text}")
return jsonify({"answer": "Sorry, the bot is offline or unavailable. (Colab API error)"}), 500
except requests.exceptions.RequestException as e:
print(f"Request to Colab API failed: {e}")
return jsonify({"answer": "Sorry, the bot is offline or unavailable. (Network error)"}), 500

if __name__ == "__main__":
app.run(debug=True)

1. HTML Frontend (chat.html)

<!DOCTYPE html>
<html>
<head lang="en">
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Mistral Chatbot</title>
<script src="https://ajax.googleapis.com/ajax/libs/jquery/3.2.1/jquery.min.js"></script>
<link href="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/css/bootstrap.min.css" rel="stylesheet">
<link href="https://maxcdn.bootstrapcdn.com/font-awesome/4.6.3/css/font-awesome.min.css" rel="stylesheet">
<style type="text/css">
/* (CSS styles omitted for brevity, identical to original) */
</style>
</head>
<body>
<div class="container background-color: rgb(255,0,255);">
<br /><br /><br />
<div class="row">
<div class="col-md-8 col-md-offset-2">
<div id="chatPanel" class="panel panel-info">
<div class="panel-heading">
<strong><span class="glyphicon glyphicon-globe"></span> Welcome To Mistral Chatbot !!!
(You: Green / Bot: White) </strong>
</div>
<div class="panel-body fixed-panel">
<ul class="media-list"></ul>
</div>
<div class="panel-footer">
<form method="post" id="chatbot-form">
<div class="input-group">
<input type="text" class="form-control" placeholder="Enter Message" name="messageText"
id="messageText" autofocus />
<span class="input-group-btn">
<button class="btn btn-info" type="button" id="chatbot-form-btn">Send</button>
</span>
</div>
</form>
</div>
</div>
</div>
</div>
</div>
<script src="https://code.jquery.com/jquery-1.12.4.min.js"></script>
<script src="https://maxcdn.bootstrapcdn.com/bootstrap/3.3.6/js/bootstrap.min.js"></script>
<script>
var exports = {};
</script>
<script>
$(function () {
$('#chatbot-form-btn').click(function (e) {
e.preventDefault();
$('#chatbot-form').submit();
});

$('#chatbot-form').submit(function (e) {
e.preventDefault();
var message = $('#messageText').val();
if (message.trim() === "") {
return;
}
$(".media-list").append(
'<li class="media"><div class="media-body"><div class="media"><div style = "text-align:right; color : #2EFE2E" class="media-body">' +
message + '<hr/></div></div></div></li>');

$.ajax({
type: "POST",
url: "/chat", // Correctly points to local Flask app's /chat endpoint
data: JSON.stringify({ message: message }),
contentType: "application/json",
success: function (response) {
$('#messageText').val('');
var answer = response.answer;
$(".media-list").append(
'<li class="media"><div class="media-body"><div class="media"><div style = "color : white" class="media-body">' +
answer + '<hr/></div></div></div></li>');
$(".fixed-panel").stop().animate({
scrollTop: $(".fixed-panel")[0].scrollHeight
}, 1000);
},
error: function (error) {
console.error("Error sending message to Flask:", error);
$(".media-list").append(
'<li class="media"><div class="media-body"><div class="media"><div style = "color : red" class="media-body">Error: Could not get a response from the bot.</div></div></div></li>');
}
});
});
});
</script>
</body>
</html>


Problem and Troubleshooting Performed

The chatbot interface loads correctly when I navigate to http://127.0.0.1:5000/. When I type a message and hit "Send", nothing appears in the chat history from the bot.

Here's what I've tried so far for debugging:

1. 
   Colab Ngrok Tunnel Confirmation: The Colab notebook outputs the Ngrok URL (e.g., https://2b19-34-125-72-30.ngrok-free.app), indicating the tunnel is established.
   
   
2. Direct FastAPI Test (from PowerShell):

I performed a direct test of the FastAPI endpoint using PowerShell's native Invoke-WebRequest cmdlet to confirm the backend's accessibility:

# Define the JSON payload as a PowerShell Hashtable, then convert it to JSON
$body = @{
"message" = "Hello, what is the capital of France?"
} | ConvertTo-Json

# Define the headers
$headers = @{
"Content-Type" = "application/json"
}

# Ngrok URL
$ngrokUrl = "COLAB_NGROK_URL_EXAMPLE/predict"

# Execute the web request
try {
$response = Invoke-WebRequest -Uri $ngrokUrl -Method POST -Headers $headers -Body $body -UseBasicParsing
$content = $response.Content | ConvertFrom-Json
Write-Host "Response from Colab FastAPI: $($content.response)"
}
catch {
Write-Host "Error during Invoke-WebRequest:"
Write-Host $_.Exception.Message
}


Output from this command:

Response from Colab FastAPI: Hello, what is the capital of France?
Answer: Paris


This direct test succeeded, confirming that the Colab FastAPI backend and its Ngrok tunnel are functioning correctly and are directly reachable from my local machine.

What I'm Looking For Since the direct call to the Colab FastAPI works, the issue likely lies in the communication between my local Flask app and the Colab FastAPI, or how the Flask app handles the response.

My questions are:

What specific steps should I take to debug the communication flow from the Flask app to the Colab FastAPI? What logs or console outputs should I be checking most carefully?

Are there any common Flask or requests library pitfalls when calling an external API like this that could lead to the bot not responding even if the direct curl test works?

Given the provided code, are there any subtle errors I might have overlooked in the Flask app or chat.html that could cause the response not to be displayed, even if the data is being sent correctly?

Continue reading...