[Python] How to handle HTTPS request with (Urllib2 + SSL) though a HTTP proxy

I am trying to test a proxy connection by using urllib2.ProxyHandler. However, there probably some situation that I am going to request a HTTPS website (eg: https://www.whatismyip.com/)

Urllib2.urlopen() will throw ERROR if request a HTTPS site. So I tried to use a helper function to rewrite the URLOPEN method.

Here is the helper function:

def urlopen(url, timeout):
if hasattr(ssl, 'SSLContext'):
SslContext = ssl.create_default_context()
SslContext.check_hostname = False
SslContext.verify_mode = ssl.CERT_NONE
return urllib2.urlopen(url, timeout=timeout, context=SslContext)
else:
return urllib2.urlopen(url, timeout=timeout)


This helper function based on answer

Then I use:

urllib2.install_opener(
urllib2.build_opener(
urllib2.ProxyHandler({'http': '127.0.0.1:8080'})
)
)


to setup HTTP proxy for urllib.opener.

Ideally, it should working when I request a website by using urlopen('http://whatismyip.com', 30) and it should pass all traffic through HTTP proxy.

However, the urlopen() will fall into if hasattr(ssl, 'SSLContext') all the time even if it is a HTTP site. In addition, HTTPS site is not using HTTP proxy either. This cause the HTTP proxy become invalid and all traffic going through unproxied network

I also tried this answer to change HTTP into HTTPS urllib2.ProxyHandler({'https': '127.0.0.1:8080'}) but it still not working.

My proxy is working. If I am using urllib2.urlopen() instead of the rewrite version urlopen(), it works for HTTP site.

But, I do need consider the suitation if the urlopen gonna need to be used on a HTTPS ONLY site.

How to do that?

UPDATE1: I cannot get this work with Python 2.7.11 and some of server working properly with Python 2.7.5. I assume it is python version issue.

Urllib2 will not go through HTTPS Proxy so all HTTPS web address will failed to use proxy.

Continue reading...