[Python] Response Python request

I’m trying to scrape that web page with Python. Here is the website: https://portal.cfm.org.br/busca-medicos/

Once the user fills any of the options doctor’s name (Nome do médico), state (UF), doctor identification (CRM), and the other options or even click on the button search (Buscar) and make the request, here is the response and information I’m trying to grab:

Aarao Andrade Napoleao Lima
CRM: 194528-SP
Data de Inscrição: 06/03/2018
Primeira inscrição na UF: 06/03/2018
Inscrição: Principal
Situação: Regular
Inscrições em outro estado: PI/6943 (Transferido)
Especialidades/Áreas de Atuação:
DERMATOLOGIA - RQE Nº: 91483
Endereço: Exibição não autorizada pelo médico.
Telefone: Exibição não autorizada pelo médico.


For example, the URL below return some doctor's information expect the specialization:

https://portal.cfm.org.br/api_rest_php/api/v1/medicos/buscar_foto/100/AL


The number 100 is the unique doctor's identification, this is a sequential number. The "AL" is the state where the doctors are registered.

So, the doctors specialization came from that URL:

https://portal.cfm.org.br/api_rest_php/api/v1/medicos/listar_especialidades


The return of the specialization URL is all specialization. In my understanding, the JavaScpric does the relationship between the doctor and his specialization in the HTML page. Or I may be wrong.

However, after a while, I wrote the code below to grab that information:

import simplejson as json
import requests

data ={"captcha":"03AGdBq26w_DFk3EDWKK7UeflMOfjABfIijncuV3lKsyPuE3uCObVS3Dh0t9kgUSW1xx48PPoXuonGdwZEon9HnaBAWaqft7ZnIX6dCVhI0HP_z9F6rlAw5JHoBzveqqInngxE7lEM4sSYKgKOsj3k57mzNME7dLeYDzMr18-CIr23RIiVx32QaGiC7Ti2KwccC-NlM6yejbrSTM49_oQ4UjQvUY3ylu00canEORjWQLFJRXScgSVPTwMIL3agpYhAZ45EbdVGngMRg8tfTUqmptJtFXgazefYkf_TMJi9Sjcxtm-TK3I-pBTJxsYyuI-pKK0DozwDhweJpMBTfqjcLwnx0BrdGL7DWaTJQP-6PR8Wc1kybg8HUoUcN73mG2vfhL7Xd731lK74H13NTQbZ7xW6Hj5NfsUHt90hTjhsmhuc4-qdfFHsdtRNdxrn4AGJ75-5UMJZAeaJ","medico":{"nome":"","ufMedico":"SP","crmMedico":"194528","municipioMedico":"","tipoInscricaoMedico":"","situacaoMedico":"","detalheSituacaoMedico":"","especialidadeMedico":"","areaAtuacaoMedico":""},"page":1,"pageNumber":1,"pageSize":10}
json_string = json.dumps(data, iterable_as_array=True)

url= "https://portal.cfm.org.br/api_rest_php/api/v1/medicos/buscar_medicos"
headers = {}
r = requests.post(url, data=json_string, headers=headers)

r.text


As result, I got this, "call_user_func_array() expects parameter 2 to be array, object given in":

'<br />\n<b>Warning</b>: call_user_func_array() expects parameter 2 to be
array, object given in
<b>/var/www/html/portalmedico/api_rest_php/api/v1/index.php</b> on line
<b>37</b><br />\n{"status":"sucesso","dados":null}'


I believe I'm doing something wrong. I'm not that familiar with the request.

Can anybody help me?

Or trying to make this web scraping with Selenium, Scrapy would be better?

My sincere apologies for the long question.

Continue reading...