[Python] Can't scrape all the titles from the map on the webpage using the requests module

Discussão em 'Python' iniciado por Stack, Julho 20, 2025.

  1. Stack

    Stack Membro Participativo

    I'm trying to create a script in Python to scrape all available titles that show up when clicking on the black-colored area on the map in this website.

    For example, when I click on a certain area on the map, I see specific geometry details in the developer tools for that property, and I can retrieve the corresponding title using the parameters in the script. How can I get all the titles from the map using the requests module?

    import re
    import json
    import urllib3
    import requests
    from pprint import pprint
    from bs4 import BeautifulSoup

    urllib3.disable_warnings(urllib3.exceptions.InsecureRequestWarning)


    link = 'https://www.bcassessment.ca/Property/Info/SjAwMDAwQzRZMQ=='

    params = {
    'f': 'json',
    'where': '',
    'geometry': {'x':-13679057.376515405,'y':6319660.727004874,'spatialReference':{'wkid':102100,'latestWkid':3857}},
    'returnGeometry': 'true',
    'geometryType': 'esriGeometryPoint',
    'inSR': '102100',
    'outFields': 'AFP_OID,UNIT_NUMBER,TOTAL_ASSESSED,ROLL,AREA_EVBC,JUR,TOTAL_LAND,TOTAL_BUILDING,ADDRESS,DESCRIPTION,STREET_NUMBER,STREET_NAME,OID_EVBC,SHORT_ADDRESS,IS_STRATA,FARM_FLAG,UTILITY_FLAG,MAJ_INDUSTRY_FLAG,MANAGED_FOREST_FLAG',
    'orderByFields': 'STREET_NAME,STREET_NUMBER,UNIT_NUMBER,SHORT_ADDRESS',
    'outSR': '102100'
    }
    headers = {
    'accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.7',
    'accept-language': 'en-US,en;q=0.9',
    'host': 'www.bcassessment.ca',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36'
    }

    headers_ano = {
    'accept': '*/*',
    'accept-language': 'en-US,en;q=0.9',
    'host': 'arcgis.bcassessment.ca',
    'origin': 'https://www.bcassessment.ca',
    'referer': 'https://www.bcassessment.ca/',
    'user-agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/138.0.0.0 Safari/537.36'
    }

    with requests.Session() as session:
    resp = session.get(link,headers=headers,verify=False)
    print(resp.url)

    params['token'] = re.findall(r"gistoken[^']+\'(.*?)\';",resp.text)[0]
    query_url = re.findall(r"mapserverUrl[^']+\'(.*?)\';",resp.text)[0] + "/0/query"

    res = session.get(query_url,params=params,headers=headers_ano,verify=False)
    print(res.status_code)
    for item in res.json()['features']:
    print(item['attributes']['ADDRESS'])

    Continue reading...

Compartilhe esta Página