[Python] Blacklists in Lists Python, while grabbing data from webpages

Basically, I have created a very messy code to grab links from bing search queries. The problem I am facing, is that I am receiving too many bing related links.

I have tried this current code to remove these, but I would much rather prefer a blacklist.

This is my code:

import re, urllib
class MyOpener(urllib.FancyURLopener):
version = 'Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.15) Gecko/20110303 Firefox/3.6.15'
myopener = MyOpener()
dork = raw_input("Dork:")
pagevar = ['1','11','23','34','45','46','47','58','69']
for page in pagevar:
bingdork = "http://www.bing.com/search?q=" + str(dork) + "&first=" + str(page)
bingdork.replace(" ", "+")
links = re.findall('''href=["'](.[^"']+)["']''', myopener.open(bingdork).read(), re.I)
toremove = []
for i in links:
if "bing.com" in i:
toremove.append(i)
elif "wlflag.ico" in i:
toremove.append(i)
elif "/account/web?sh=" in i:
toremove.append(i)
elif "/?FORM" in i:
toremove.append(i)
elif "javascript:void(0);" in i:
toremove.append(i)
elif "javascript:" in i:
toremove.append(i)
elif "go.microsoft.com/fwlink" in i:
toremove.append(i)
elif "g.msn.com" in i:
toremove.append(i)
elif "onlinehelp.microsoft.com" in i:
toremove.append(i)
elif "feedback.discoverbing.com" in i:
toremove.append(i)
elif "/account/web?sh=" in i:
toremove.append(i)
elif "/?scope=web" in i:
toremove.append(i)
elif "/explore?q=" in i:
toremove.append(i)
elif "https://feedback.discoverbing.com" in i:
toremove.append(i)
elif "/images/" in i:
toremove.append(i)
elif "/videos/" in i:
toremove.append(i)
elif "/maps/" in i:
toremove.append(i)
elif "/news/" in i:
toremove.append(i)
for i in toremove:
links.remove(i)
for i in links:
print i


Suppose I inputted: Dork: cfm id

The results I would get would be:

http://pastebin.com/eGgUKYwV

Where as the results I would like would be:

http://pastebin.com/Xi28BzXs

I would like to remove stuff like:

/search?q=cfm+id&lf=1&qpvt=cfm+id
/account/web?sh=5&ru=%2fsearch%3fq%3dcfm%2520id%26first%3d69&qpvt=cfm+id
/search?q=cfm+id&rf=1&qpvt=cfm+id
/search?q=cfm+id&first=69&format=rss
/search?q=cfm+id&first=69&format=rss
/?FORM=Z9FD1
javascript:void(0);
/account/general?ru=http%3a%2f%2fwww.bing.com%2fsearch%3fq%3dcfm+id%26first%3d69&FORM=SEFD
/?scope=web&FORM=HDRSC1
/images/search?q=cfm+id&FORM=HDRSC2
/videos/search?q=cfm+id&FORM=HDRSC3


Basically, I need a filter that allows me to grab only VALID links from bing, and removes all the crap from bings side.

Thanks so much, BK P.S Sorry if my explanation is bad.

Continue reading...