[Python] 404 on certain files on select devices after repeated visits

I run a webserver on my local network to host my movie collection. I also have a custom android app running on an amazon fire stick. Recently I have been working to enable video playback from external sources.

The app sends requests to the webserver, like /externalsearch.php?title=(movie). The site then calls a selenium python web scraper and returns various information in the form of json. All of this works perfectly, about 4 or 5 times. After that, only requests to pure html pages seem to work from the app. The php pages all return 404. Other static files, like /catalog.json also return a 404. Restarting the system gets you another 4 or 5 queries before breaking again.

When visiting the same pages on my computer, they all function perfectly no matter what. However, visiting the pages on my iPhone causes the same result. The only other things running on the

Here is some addition information and things I have already checked. Memory usage is not abnormally high. The app is properly disconnecting. The issue persisted through a completely fresh Debian install on the server. The only other services running on the server are ssh to control it and an smb file share into the web page directory.

Below is one of the kotlin functions that calls the page, the webpage externalsearch.php and the associated python file, with some of the specifics obscurated.

android function

fun external(view : View)
{
val searchStringLower = searchString.lowercase()

CoroutineScope(Dispatchers.IO).launch {
var connection: HttpURLConnection? = null
try {
val urlString = "${MainActivity.settings.ip}/externalsearch.php?title=${URLEncoder.encode(searchStringLower, "UTF-8")}"
val url = URL(urlString)
connection = url.openConnection() as HttpURLConnection

connection.requestMethod = "GET"
connection.connectTimeout = 5000
connection.readTimeout = 15000

val responseCode = connection.responseCode
if (responseCode == HttpURLConnection.HTTP_OK) {
val response = connection.inputStream.bufferedReader().use { it.readText() }

val json = Json { ignoreUnknownKeys = true }
val results: List<ExternalResult> = json.decodeFromString(response)

lifecycleScope.launch(Dispatchers.Main) {
val adapter = ExternalItemAdapter(this@SearchActivity, results, 0)
searchGrid.adapter = adapter
}
} else {
println("Error: $responseCode")
}
} catch (e: Exception) {
e.printStackTrace()
} finally {
connection?.disconnect()
}
}
}


externalsearch.php

<?php
if (!isset($_GET['title'])) {
echo json_encode(["error" => "Missing title parameter"]);
exit;
}

// Escape argument to prevent command injection
$title = escapeshellarg($_GET['title']);

// Local script reference
$script = __DIR__ . "/Scripts/externalsearch.py";

// Build command and execute
$cmd = "xvfb-run -a python3 $script $title 2>&1";
$output = shell_exec($cmd);

header("Content-Type: application/json");
echo $output;
?>


/Scripts/externalsearch.py

import sys
import json
import re

from selenium import webdriver
from selenium.webdriver.common.by import By
from selenium.webdriver.support.ui import WebDriverWait
from selenium.webdriver.support import expected_conditions as EC
from selenium.webdriver.firefox.service import Service


from bs4 import BeautifulSoup
from threading import Lock

if len(sys.argv) < 2:
print(json.dumps({"error": "Missing title parameter"}))
sys.exit(1)

title = sys.argv[1]

url = f"(url)"

options = webdriver.FirefoxOptions()
options.add_argument("--headless")
service = Service('/usr/local/bin/geckodriver')
driver = webdriver.Firefox(service=service, options=options)
results = []

try:
driver.get(url)

html = driver.page_source
soup = BeautifulSoup(html, "html.parser")

...
Scraping code
...

except Exception as e:
print(json.dumps({"error": str(e)}))
finally:
driver.quit()

print(json.dumps(results))


I'm not entirely sure which part of the stack is at fault for this issue, so I have not included any logs. If you think something additional would be helpful please let me know.

Continue reading...