[Python] Extracting data between two h1 tags using BeautifulSoup

Beautiful soup: Extract everything between two tags

I am working on extracting content between two specific HTML tags using BeautifulSoup. The tags do not have any specific attributes or IDs, and I want to extract everything between the first and second occurrence of the tags. For example, given the following HTML:

<h1></h1>
Text <i>here</i> has no tag
<div>This is in a div</div>
<h1></h1>


I want the output to be:

Text <i>here</i> has no tag
<div>This is in a div</div>


However, if the tags had distinct id attributes, for example:

<h1 id="start"></h1>
Text <i>here</i> has no tag
<div>This is in a div</div>
<h1 id="end"></h1>



I would still like to extract the same content, but using the specific id attributes as boundaries.

Code:

Here’s the Python code I’ve written using BeautifulSoup to achieve this:

from bs4 import BeautifulSoup

html_doc = '''
This I <b>don't</b> want
<h1></h1>
Text <i>here</i> has no tag
<div>This is in a div</div>
<h1></h1>
This I <b>don't</b> want too
'''

# Parse the HTML
soup = BeautifulSoup(html_doc, 'html.parser')

# Find the first and second <h1> tags
h1_tags = soup.find_all('h1')

if len(h1_tags) >= 2:
# Extract content between the two <h1> tags
between_tags = []
for element in h1_tags[0].next_siblings:
if element == h1_tags[1]:
break
between_tags.append(str(element))

# Join and print the result
print(''.join(between_tags).strip())


Prints:

Text <i>here</i> has no tag
<div>This is in a div</div>


This is working without ids.

What if the ht tags had distinct idsd like

"""
This I <b>don't</b> want
<h1 id="start"></h1>
Text <i>here</i> has no tag
<div>This is in a div</div>
<h1 id="end"></h1>
This I <b>don't</b> want too
'''
"""

Continue reading...