[Python] Marketplace scraper suddenly stopped, they updated the hidden json?

Discussão em 'Python' iniciado por Stack, Outubro 24, 2024.

  1. Stack

    Stack Membro Participativo

    Since yesterday, I encountered an issue where my facebook marketplace scraper ceased to fetch data, I'm currently using scrapy due to his features, am I doing any mistakes?

    from scrapy import Spider
    import logging
    from json import loads


    class Facebook(Spider):
    name = 'facebook'
    start_urls = ["https://www.facebook.com/marketplace/112047398814697/search?query=funko&sortBy=creation_time_descend&radius=500"]

    def parse(self, response):

    flex_selector = response.xpath('//*[@id="facebook"]/body/script/text()')

    ads = []

    for script in flex_selector.getall():
    try:
    ads = loads(script)
    ads = ads['require'][0][3][0]['__bbox']['require'][0][3][1]['__bbox']['result']['data']['marketplace_search']['feed_units']['edges']
    break
    except:
    pass
    # if(len(ads) == 0):

    for item in ads:
    try:
    title = adJson['node']['listing']['marketplace_listing_title']
    from pdb import set_trace; set_trace()


    To run the code below, runs: scrapy runspider main.py

    I'm currently using py 3.12.2 and scrapy 2.11.2

    Continue reading...

Compartilhe esta Página