[Python] A website I am trying to scrape yields much faster response for a python script...

I have been trying to scrape this site with node.js but the response is too slow to complete, taking over a minute on average.

Requesting the same site with python fetches the results in under a couple seconds

import fetch from "node-fetch";

const url = "https://ktu.edu.in/eu/core/announcements.htm";

console.log("Fetching the website");
const res = await fetch(url);

console.log("Request completed");

const data = await res.text();
console.log("Website fetched: Node");

console.log(data);


Here is the repl to test the code. I think the server is sending the response in too many "chunks"(you can see it if you run the repl)? Can this be solved from my side? Also why doesn't python have any issues with this?

I have tried setting the same request headers for node.js and python(requests library).

I used the https, axios and node-fetch modules with node.js, same result.

Continue reading...