[Python] Using the free-proxy library with requests to access general https websites

When basically requesting a proxy, what happens is that it delivers an http that currently seems to me to be unusable because the vast majority of sites use https and this causes the request to be made using my own IP instead of the collected proxy, as we can see here:

from fake_useragent import UserAgent
from fp.fp import FreeProxy
import requests


def pack_requests(the_url, timeout=5):
ua = UserAgent()
proxy = FreeProxy(rand=True, timeout=timeout).get()
protocol = "https" if proxy.startswith("https") else "http"
pack_header = {
'User-Agent': ua.random
}
pack_response = requests.get(the_url, headers=pack_header, proxies={protocol: proxy}, timeout=timeout)
return pack_response

print(pack_requests('https://geolocation-db.com/json').text)


In the most current version of free-proxy (1.1.3) the https=True option is generating an error, so I installed version 1.1.2 which works perfectly with the option mentioned above, the only thing I need to do is define the protocol manually because the proxy it comes with the http protocol when I print the return of FreeProxy(...).get():

from fake_useragent import UserAgent
from fp.fp import FreeProxy
import requests


def pack_requests(the_url, timeout=5):
ua = UserAgent()
proxy = FreeProxy(rand=True, https=True, timeout=timeout).get()
pack_header = {
'User-Agent': ua.random
}
pack_response = requests.get(the_url, headers=pack_header, proxies={"http": proxy, "https": proxy}, timeout=timeout)
return pack_response

print(pack_requests('https://geolocation-db.com/json').text)


The problem is that this way for websites in general, such as search sites in general, news websites and the vast majority of them (including my own website which I also tested using just to see the result) a connection problem with the Proxy is generated at the time of the requests or a problem with the SSL certificate at the time of the requests.

Is there a way to generate a good result using these two libraries (free-proxy and requests) together to achieve at least the basic needs or has trying to use these proxy bases become obsolete and let's say "impossible" to generate good results?

Continue reading...