[Python] Parallelizing GET requests using PyO3

I'm trying to optimize some Python code using the PyO3 library which allows me to write custom Rust code that I can import into my Python code. In particular, I'm trying to parallize requesting multiple urls in Rust, which I would then import into the Python code to get multiple URLs in parallel which I would then parse using BeautifulSoup. This is the Rust code which is giving me some issues.

use pyo3:relude::*;
use tokio::runtime::Runtime;
use reqwest;
use futures::future;

// Function to fetch URLs in parallel
#[pyfunction]
fn fetch_urls(urls: Vec<String>) -> PyResult<Vec<String>> {
let rt = Runtime::new().unwrap(); // Create a Tokio runtime
let responses = rt.block_on(async {
let futures = urls.into_iter().map(|url| {
async {
reqwest::get(&url).await.ok().and_then(|resp| Some(resp.text()))
}
});
future::join_all(futures).await
});
Ok(responses.into_iter().filter_map(|x| x).collect())
}

// PyO3 module
#[pymodule]
fn rust_requests(_py: Python, m: &PyModule) -> PyResult<()> {
m.add_function(wrap_pyfunction!(fetch_urls, m)?)?;
Ok(())
}


These are the errors I'm getting when I run maturin develop

error[E0308]: mismatched types
--> src/lib.rs:13:63
|
13 | reqwest::get(&url).await.ok().and_then(|resp| resp.text())
| ^^^^^^^^^^^ expected `Option<_>`, found future
|
help: try wrapping the expression in `Some`
|
13 | reqwest::get(&url).await.ok().and_then(|resp| Some(resp.text()))
| +++++ +

error[E0599]: no method named `add_function` found for reference `&pyo3::types:yModule` in the current scope
--> src/lib.rs:24:7
|
24 | m.add_function(wrap_pyfunction!(fetch_urls, m)?)?;
| ^^^^^^^^^^^^ method not found in `&PyModule`

error[E0277]: the trait bound `&pyo3::types:yModule: WrapPyFunctionArg<'_, _>` is not satisfied
--> src/lib.rs:24:20
|
24 | m.add_function(wrap_pyfunction!(fetch_urls, m)?)?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
| |
| the trait `WrapPyFunctionArg<'_, _>` is not implemented for `&pyo3::types:yModule`
| required by a bound introduced by this call
|
= help: the following other types implement trait `WrapPyFunctionArg<'py, T>`:
OnlyBound<T>
pyo3::Bound<'py, pyo3::types:yModule>
pyo3::Borrowed<'_, 'py, pyo3::types:yModule>
pyo3:ython<'py>
&pyo3::Bound<'py, pyo3::types:yModule>
&pyo3::Borrowed<'_, 'py, pyo3::types:yModule>
= note: this error originates in the macro `wrap_pyfunction` (in Nightly builds, run with -Z macro-backtrace for more info)

error[E0277]: the trait bound `&pyo3::types:yModule: From<BoundRef<'_, '_, pyo3::types:yModule>>` is not satisfied
--> src/lib.rs:22:1
|
22 | #[pymodule]
| ^^^^^^^^^^^ the trait `From<BoundRef<'_, '_, pyo3::types:yModule>>` is not implemented for `&pyo3::types:yModule`, which is required by `BoundRef<'_, '_, pyo3::types:yModule>: Into<_>`
|
= note: required for `BoundRef<'_, '_, pyo3::types:yModule>` to implement `Into<&pyo3::types:yModule>`
= note: this error originates in the attribute macro `pymodule` (in Nightly builds, run with -Z macro-backtrace for more info)

Some errors have detailed explanations: E0277, E0308, E0599.


I'm not sure how to fix these errors as my knowledge of Rust is limited so any help would be appreciated. Here's the accompanying Cargo.toml and Python code

[package]
name = "python-rust-interop"
version = "0.1.0"
edition = "2021"

[lib]
name = "rust_requests"
crate-type = ["cdylib"]

[dependencies]
pyo3 = { version = "0.22", features = ["extension-module"] }
tokio = { version = "1.41.0", features = ["full"] } # Add tokio with all features
reqwest = { version = "0.11", features = ["json"] } # Add reqwest with JSON support
futures = "0.3"


from rust_requests import fetch_urls
from bs4 import BeautifulSoup

# List of URLs to scrape
urls = ["http://example.com", "http://example.org"]

# Fetch data using Rust
html_responses = fetch_urls(urls)

# Parse each response with BeautifulSoup
for html in html_responses:
soup = BeautifulSoup(html, "html.parser")
print(soup.title.text)

Continue reading...