Does the Google bot crawl configuration URLs that are within script tags? We are seeing numerous 404 reports related to our endpoints for various functions. To access these endpoints, we construct URLs based on them. Somehow, these endpoints are being discovered by the Google bot. We’ve spent considerable time searching for exposures in HTML, sitemap.xml, robots.txt, etc. The only place we can find them is in a script tag outside of the head and body sections, as we use Angular SSR to transfer configurations. This issue began when we updated Angular to its latest version, v18.2.7. Previously, we used v15.2.10, where this script tag for transferring configurations was encoded. Before: <script id="serverApp-state" type="application/json"> {&q;context&q;:{&q;isBrowser&q;:false,&q;isServer&q;:true,&q;platform&q;:&q;mobile&q;,&q;isBootstrapped&q;:false,&q;slug&q;:&q;/&q;,&q;lang&q;:&q;local&q;,&q;navigationHistory&q;:[],&q;navigationDatabase&q;:[],&q;configs&q;:{&q;translation&q;:{&q;commons&q;:{&q;Subscribe&q; </script> After: <script id="serverApp-state" type="application/json"> {"context":{"isBrowser":false,"isServer":true,"platform":"mobile","isBootstrapped":false,"slug":"/","lang":"local","navigationHistory":[],"navigationDatabase":[],"configs":{"translation":{"commons":{ </script> This is why we believe that the Google crawler reads these types of script tags and even generates possible URLs for crawling. Continue reading...
