[Python] Problem scraping table row data into an array

Background

I am looking to measure statistics from a website wotstars for a XBOX and Playstation game, World of Tanks Console.

Initially I tried just using Excel to scrape the site directly for me into Power Query, the immediate issue was that only 5 rows of recent match data (5 games) is available from the website as loaded, a button needs to be click to view the last 100 matches.



I thought this was a good idea to learn more about web-scraping with python.

I successfully have managed to use Selenium to activate the "View More" button to shows all 100 rows and I can return the table of interest.

Problem

I am currently stuck through writing the td data to a 2D array that I can then dump to a csv file, I seem to be writing 100 columns rather than 100 rows?

I have looked at sites such as A Guide to Scraping HTML Tables with Pandas and BeautifulSoup for approaches (but this particular example doesn't show the source table).

Other

If I look at Excel the query connection that attaches to this table is shown below



Code

from selenium import webdriver
from selenium.webdriver.common.by import By
import time
from bs4 import BeautifulSoup
import pandas as pd

# Set up the WebDriver
driver = webdriver.Chrome()
# Famous player
url = 'https://www.wotstars.com/xbox/6757320'

# Open the login page
driver.get(url)
time.sleep(5)

login_button = driver.find_elements(By.CLASS_NAME, "_button_1gcqp_2")
for login in login_button:
print (login.text)

if login_button[3].text == 'Start tracking':
login_button[4].click()

print("button activated")

time.sleep(2)
#login_button[3].click()

soup = BeautifulSoup(driver.page_source, 'html.parser')

rows = []
for child in soup.find_all('table')[3].children:
row = []
for td in child:
#print (td.text)
try:
row.append(td.text.replace('\n\r', ''))
except:
continue
if len(row) > 0:
#print (row)
rows.append(row)
print (len(rows[0]))

#not working
#df = pd.DataFrame(rows[1:], columns=rows[0])
#display(HTML(df.to_html()))

print("done")

# Close the browser
driver.quit


Object Inspector

Continue reading...