[Python] I'm trying to scrape a web page but keep getting back encrypted HTML and the JS file...

So I am learning how to scrape web pages right now in python. First I tried beautiful soup and now selenium webdriver.

I am trying to scrape some crossword puzzle clues from some crossword website but whenever I scrape the page, the puzzle data part of the html code is encrypted. It looks like this:

boatload_puzzles_puz_num=17573;
boatload_puzzles_puz_data='F5955C88A2E68B4CF2143A30085A02BC0CD93EBC23363509A09749D79C4DE80C'
+ '88F9828CB7570B90F73D72AC773FAFA36B6EB87BDDC7EF12669BF8C4BEFA0C98'
+ '18DE48510139817914FD14EF5793DC6E00488A4CB5FF22A2A542D5DF0FF35D07'
+ 'A93FD6A2E3CFEF30CACAB964A27B08573DB7A6D430F233DD070CA17AA3715356'
+ '6A3D8EA8E8C3BEAADAEADA6ED1CF156CA0083B55780ED57406F391F443D80AFB'
+ '7B38F201495C80713D07DD04D453C1CDA26BCF3EEB54B8CDBBAF40AB15E38E9E'
+ '6E659CB7D50245E53880DD6B23B25DBA04339769C824FF26B705297DFD31139A'
+ '516662DF70A133F0A996EC86AFC59C4848A3FF32D17AD1344F6388DC0D952A79'
+ '100473B0DCF66A585EC0D9BC963F263B3F01496B96D93CCAE3BCF220E857BEA6'
+ '36320A36D6ED7DC710572411E7B6A5B3493C409270B6D26B50D3E4A8E95B71AD'
+ '7362C75FEB1238857512FED16758548ABDD1AD8A42AFF8A841A34FD9A88FEE56'
+ '84AE28E3D5B9291224425D28B6564792E0A908310CFD556539144E5E9FF09BDD'
+ '36B237B4E7BE5DBDD6CD341E3F876242DCBD2B4313459CF77D961284ABEDAE6D'
+ '25C12264FF1D864218DA2512FE7C42A6AF262A64C044C3D2BB960137C4F42EEC'
+ 'D42B539DCEA7721EB0ED39CBF52F4AA0D1587D4561077D16485EF90CB700ED06'
+ '1B2030C37455031CFEE229A805411FCE86302C94454E4270AAFBED4D5F309865'
+ '7218CDE64132FA5E1169E32A3123E9140D089DA66C40D9958C44C0C573B464C6'
+ '20263ACAFD614F43B269779983BA744BE1B05E863935E1389B72A9C043999267'
+ 'FEC371761534DD67C41E3835C28CB9E651ADCD9E5EE4C790F84295A21E919AD1'
+ '03470BD11502C51DAC506999CFEA9B53B1ABDBA319F38C8B4A3D06E53BC96AC9'
+ 'CBD3664A11B0216A1F1B6C3041B2877BCCA4216CB1FC823BABABD8D35D455794'
+ 'C1D7494FEBC9B3192E76A8A0D6C41417F9E2D72F462AD41EF262829E29B3B594'
+ '153B8505A8006468A0CEE7870DF89C4C1601370770915AC9061DF24C4711C806'
+ '20DCFE17F482047EE7EA1BB71524B804C8AEB341D5F66FCE54E504BA7E960DB7'
+ '1FE5B219677411C44734374C805B945F69E7AE98DB2AEA4CF67D9B1A4C36FB36'
+ '37B42A1BC3DB1A8E66A453A4374EF83313EFEDB0168CA29AD9D434BDF4B4B940'
+ '10BBF3365D0E1754B7DCBDC6520E60A6B21F1EB9EF67C33C1FB8E1A1CA9060FE'
+ 'D2E5E16C39F97E791C99719589B573D7AFADA2AD0A0533A19D06BBD3CDAD7CDF'
+ '52897281CED8A1764B748BE7785B3859B65182D056E651C739A1028B7AE66C3D'
+ '004CC274DE6F93AFA7D979249742C95A7BA04092';
boatload_puzzles_key='2401B369A829B52C8F556227236EDAEC';


The weird thing is that when I inspect the elements on the actual web page I can see the puzzle clues and data just fine.

Lastly, I tried looking into the pages js file to find some decryption logic but it is super obfuscated and looks crazy. I don't even know javascript so this is a bit over my head.

I appreciate any help I can get!

Continue reading...