[Python] How to pass correct cookies when trying to obtain data from a website?

I'm trying to scrape information from following website https://pretrazivac-obrta.gov.hr/. When you click "Traži", you'll get the following response: https://pretrazivac-obrta.gov.hr/pretraga?izvrsiDohvat. In this response (POST), you'll need to pass some data in order to get the data. None of this is a problem, the problem begins with cookies. What I would do is copy the curl into curl converter and this code would work until the cookies expire.

Cookies should look something like this:

{
'__utmz': '150669943.1685031401.1.1.utmcsr=(direct)|utmccn=(direct)|utmcmd=(none)',
'__utmc': '150669943',
'TS00000000076': '08cc731c44ab280009cbed137f8d48311c7fd5bc05553b3acd6b9d0d9449bd40409cf070fa5896acd7bdc232e57afeac086dc7f28109d0004282940bbf620471f904e4ee720e9e52120116681fa2dde9d3ccc250787649e7e86b7e5d50be6800de490c2492f2911f1cb8b9b82c70dcd95a83f589b4250a37bc0579e8061d009baf28546b163c2119e64150c3e00efda065e0ad17693fe9eecaff9c4b8d5c3d30d46863b939b59dbbb1b3d1eceebef41884eb2f3ac26595ebfb5a50745e2507d241697774cde6c81391b3915f901ecd3135765c89914b283e23c41da760e249a9a0bb1dd32cca209d4c88b5de3e0a421f9ebba95d421b56e461da2bb92925fd2a5e0def302e2921e8',
'TSPD_101_DID': '08cc731c44ab280009cbed137f8d48311c7fd5bc05553b3acd6b9d0d9449bd40409cf070fa5896acd7bdc232e57afeac086dc7f28106380041b623f71368ac6b70682da512c2abc24018a2c05e8fb962f71b606f0a158f0bbb9765499fad2aa909a56a1d77a2df7e5b856d0b5e9cc275',
'JSESSIONID': 'srv01~4EBB883F48EE71B2DD61BDE78CD46B45',
'TS01a2590c': '01402d66b96cd92a73d364af64837992731a9dffaa0320858aaf7d01ff45906ed34e34be58bb8cbc0721f315ee640c0292537d402ca30df3621f172ec8226a6d728ba91074',
'TSPD_101': '08cc731c44ab2800a5ab5326d7601cfa453f25bb4696ef0cd9a28538f30faca54237a1836b24ec1bc41da38bbff56a65087190c3fb0518000d84d6b7cd598b945ccda15be985d454811e1204a60a8de4',
'__utma': '150669943.1828940763.1685031401.1685040031.1685049060.3',
'__utmt': '1',
'__utmb': '150669943.1.10.1685049060',
'TS9a3ff16f029': '08cc731c44ab2800f487994d79d6b30a26eac66b279ff7e77a109eae31e22261b23ee80416eee3308ca52726cb4e2872',
'TS74048527027': '08cc731c44ab20008fad2e02e7a4354541f21179ddc82d2d7bce1475deed792eb6158f3ffa00ff0f087eaa5f81113000238281e97b3866df007962363c348138148ed4bfa679013825ce225893f0bc5830b5bd0d1cd239b11c4192faf8209bb9',
'TS9a3ff16f077': '08cc731c44ab2800ee499112ef23f263d4ec8d03a2f452ddb1a1deae2db4b054ceb5acd452b867cf961d7ab66c3f5ccb0819b09526172000c3b1c62a1f5c2f3394c546454ec0b4721bf6c3059cffc133e7fb811bee527d4b',
}


Problem is, I can't obtain the cookies. I tried the following:

import requests


headers = {
'Content-Type': 'application/json',
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_7) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/113.0.0.0 Safari/537.36',
}

session = requests.Session()
response = session.get("https://pretrazivac-obrta.gov.hr/pretraga?izvrsiDohvat", headers=headers)

print(response.cookies.get_dict())


This returns following:

{'TS74048527027': '08cc731c44ab2000eff496fb99f369e0643884fa0f6804056b1bb93d0d244c5864808e6247cd662208d24b50c01130003ecb0ddc3ea6fdacc9b63cbd5be96a0fce282beb2ef736208faf7ae21372e434564afe0e2d22b68ba75bad8aee8080f2'}


Unfortunately, this does not work as when I try to pass these cookies, I get following:

b"<html><head><title>Request Rejected</title></head><body>The requested URL was rejected. Please consult with your administrator.<br><br>Your support ID is: < 6060970845093779227><br><br><a href='javascript:history.back();'>[Go Back]</body></html>"


I also tried with Playwright and no luck, although I managed to grab more cookies like this and the error was the same as above:

from playwright.sync_api import sync_playwright


with sync_playwright() as p:
browser = p.firefox.launch(headless=True)
context = browser.new_context()
page = context.new_page()
page.goto("https://pretrazivac-obrta.gov.hr/pretraga")
page.get_by_role("button", name="Traži").click()
cookies = context.cookies()
cj = {}
for cookie in cookies:
name = cookie["name"]
value = cookie["value"]
cj[name] = value


I want to know, is there any way how can I grab all of the cookies? Thank you!

Continue reading...