[Python] "AssertionError: Q2: Your answer is not close enough to ours to be...

I need some assistance with my Assignment 4 for Introduction to Data Science in Python.

Question 2 of assignment 4.

For this question, calculate the win/loss ratio's correlation with the population of the city it is in for the NBA using 2018 data.
​

When I submit the assignment this is the error i get

You have failed this test due to an error. The traceback has been removed because it may contain hidden tests. This is the exception that was thrown:

AssertionError: Q2: Your answer is not close enough to ours to be correct.
​

The answer I get is

-0.16087318003059875
​

I have a feeling the answer must be around

-0.1763635064218294
​

since that is what the other online resources show me for people who have taken part in the test,

Here is my Dataframe output:

NBA DataFrame:
W/L% Population
Metropolitan area
Atlanta 0.293 5789700.0
Boston 0.671 4794447.0
Charlotte 0.439 2474314.0
Chicago 0.329 9512999.0
Cleveland 0.610 2055612.0
Dallas–Fort Worth 0.293 7233323.0
Denver 0.561 2853077.0
Detroit 0.476 4297617.0
Houston 0.793 6772470.0
Indianapolis 0.585 2004230.0
Los Angeles 0.512 13310447.0
Memphis 0.268 1342842.0
Miami–Fort Lauderdale 0.537 6066387.0
Milwaukee 0.537 1572482.0
Minneapolis–Saint Paul 0.573 3551036.0
New Orleans 0.585 1268883.0
New York City 0.341 20153634.0
Oklahoma City 0.585 1373211.0
Orlando 0.305 2441257.0
Philadelphia 0.634 6070500.0
Phoenix 0.256 4661537.0
Portland 0.598 2424955.0
Sacramento 0.329 2296418.0
Salt Lake City 0.585 1186187.0
San Antonio 0.573 2429609.0
San Francisco Bay Area 0.707 6657982.0
Toronto 0.720 5928040.0
Washington, D.C. 0.524 6131977.0
Correlation Coefficient: -0.16087318003059875
-0.16087318003059875




This is my solution for question 2 run in Jupyter notebook

import pandas as pd
import numpy as np
import scipy.stats as stats
import re

cities = pd.read_html("assets/wikipedia_data.html")[1]
cities = cities.iloc[:-1, [0, 3, 5, 6, 7, 8]]
cities.rename(columns={"Population (2016 est.)[8]": "Population"},
inplace=True)
cities['NFL'] = cities['NFL'].str.replace(r"\[.*\]", "")
cities['MLB'] = cities['MLB'].str.replace(r"\[.*\]", "")
cities['NBA'] = cities['NBA'].str.replace(r"\[.*\]", "")
cities['NHL'] = cities['NHL'].str.replace(r"\[.*\]", "")

Big4='NBA'

def nba_correlation():

team = cities[Big4].str.extract('([A-Z]{0,2}[a-z0-9]*\ [A-Z]{0,2}[a-z0-9]*|[A-Z]{0,2}[a-z0-9]*)([A-Z]{0,2}[a-z0-9]*\ [A-Z]{0,2}[a-z0-9]*|[A-Z]{0,2}[a-z0-9]*)([A-Z]{0,2}[a-z0-9]*\ [A-Z]{0,2}[a-z0-9]*|[A-Z]{0,2}[a-z0-9]*)')
team['Metropolitan area']=cities['Metropolitan area']
team = pd.melt(team, id_vars=['Metropolitan area']).drop(columns=['variable']).replace("",np.nan).replace("—",np.nan).dropna().reset_index().rename(columns = {"value":"team"})
team=pd.merge(team,cities,how='left',on = 'Metropolitan area').iloc[:,1:4]
team = team.astype({'Metropolitan area': str, 'team': str, 'Population': int})
team['team']=team['team'].str.replace('[\w.]*\ ','')

_df=pd.read_csv("assets/"+str.lower(Big4)+".csv")
_df = _df[_df['year'] == 2018]
_df['team'] = _df['team'].str.replace(r'[\*]',"")
_df['team'] = _df['team'].str.replace(r'\(\d*\)',"")
_df['team'] = _df['team'].str.replace(r'[\xa0]',"")
_df = _df[['team','W/L%']]
_df['team'] = _df['team'].str.replace('[\w.]* ','')
_df = _df.astype({'team': str,'W/L%': float})

merge=pd.merge(team,_df,'outer', on = 'team')
merge=merge.groupby('Metropolitan area').agg({'W/L%': np.nanmean, 'Population': np.nanmean})

population_by_region = merge['Population'] # pass in metropolitan area population from cities
win_loss_by_region = merge['W/L%'] # pass in win/loss ratio from _df in the same order as cities["Metropolitan area"]

assert len(population_by_region) == len(win_loss_by_region), "Q2: Your lists must be the same length"
assert len(population_by_region) == 28, "Q2: There should be 28 teams being analysed for NBA"

return stats.pearsonr(population_by_region, win_loss_by_region)[0]
nba_correlation()


I am not so sure how to go with this since am sure i have worked out the assignment to my full capabilities, also below is my question 1 code just in case it cuts across.

import pandas as pd
import numpy as np
import scipy.stats as stats
import re

def clear_data(string1):
if re.search(r'\[[a-z]* [0-9]+\]', string1) is None:
return string1
else:
return string1.replace(re.search(r'\[[a-z]* [0-9]+\]', string1).group(), '')

def get_area(team):
for each in list(nhl_cities.index.values):
if team in each:
return nhl_cities.at[each, 'Metropolitan area']

nhl_df=pd.read_csv("assets/nhl.csv")
cities=pd.read_html("assets/wikipedia_data.html")[1]
cities=cities.iloc[:-1,[0,3,5,6,7,8]]
cities['NHL'] = cities['NHL'].apply(lambda x: clear_data(x))
nhl_cities = cities[['Metropolitan area', 'NHL']].set_index('NHL')
nhl_cities = nhl_cities.drop(['—', ''], axis=0)
nhl_df = nhl_df[nhl_df['year'] == 2018].drop([0, 9, 18, 26], axis=0) # get only 2018 stats
population = cities[['Metropolitan area', 'Population (2016 est.)[8]']]
population = population.set_index('Metropolitan area')
nhl_df['team'] = nhl_df['team'].apply(lambda x: x[:-1].strip() if x.endswith("*") else x.strip())
nhl_df['area'] = nhl_df['team'].apply(lambda x: x.split(" ")[-1])
nhl_df['area'] = nhl_df['area'].apply(lambda x: get_area(x))
out = []
for group, frame in nhl_df.groupby('area'):
total_wins = np.sum(pd.to_numeric(frame['W']))
total_losses = np.sum(pd.to_numeric(frame['L']))
total_matches = total_wins + total_losses
ratio = total_wins / total_matches
out_dict = {
'Area': group,
'Ratio': ratio
}
out.append(out_dict)
new_df = pd.DataFrame(out)
new_df = new_df.set_index('Area')
out_df = pd.merge(new_df, population, how="inner", left_index=True, right_index=True)
out_df['Population (2016 est.)[8]'] = pd.to_numeric(out_df['Population (2016 est.)[8]'])

def nhl_correlation():
population_by_region = out_df['Population (2016 est.)[8]'].to_list()
win_loss_by_region = out_df['Ratio'].to_list()

assert len(population_by_region) == len(win_loss_by_region), "Q1: Your lists must be the same length"
assert len(population_by_region) == 28, "Q1: There should be 28 teams being analysed for NHL"

return np.float64(stats.pearsonr(population_by_region, win_loss_by_region)[0])

def get_nhl_data():
return out_df


And here is the question to assignment 4.

In this assignment you must read in a file of metropolitan regions and associated sports teams from assets/wikipedia_data.html and answer some questions about each metropolitan region. Each of these regions may have one or more teams from the "Big 4": NFL (football, in assets/nfl.csv), MLB (baseball, in assets/mlb.csv), NBA (basketball, in assets/nba.csv or NHL (hockey, in assets/nhl.csv). Please keep in mind that all questions are from the perspective of the metropolitan region, and that this file is the "source of authority" for the location of a given sports team. Thus teams which are commonly known by a different area (e.g. "Oakland Raiders") need to be mapped into the metropolitan region given (e.g. San Francisco Bay Area). This will require some human data understanding outside of the data you've been given (e.g. you will have to hand-code some names, and might need to google to find out where teams are)!

For each sport I would like you to answer the question: what is the win/loss ratio's correlation with the population of the city it is in? Win/Loss ratio refers to the number of wins over the number of wins plus the number of losses. Remember that to calculate the correlation with pearsonr, so you are going to send in two ordered lists of values, the populations from the wikipedia_data.html file and the win/loss ratio for a given sport in the same order. Average the win/loss ratios for those cities which have multiple teams of a single sport. Each sport is worth an equal amount in this assignment (20%*4=80%) of the grade for this assignment. You should only use data from year 2018 for your analysis -- this is important!

Notes Do not include data about the MLS or CFL in any of the work you are doing, we're only interested in the Big 4 in this assignment. I highly suggest that you first tackle the four correlation questions in order, as they are all similar and worth the majority of grades for this assignment. This is by design! It's fair game to talk with peers about high level strategy as well as the relationship between metropolitan areas and sports teams. However, do not post code solving aspects of the assignment (including such as dictionaries mapping areas to teams, or regexes which will clean up names). There may be more teams than the assert statements test, remember to collapse multiple teams in one city into a single value! As this assignment utilizes global variables in the skeleton code, to avoid having errors in your code you can either:

You can place all of your code within the function definitions for all of the questions (other than import statements). You can create copies of all the global variables with the copy() method and proceed as usual.
​

Here is some sample data so one can understand how the data in the CSV files looks like in Jupyter Notebook:

mlb.csv

nfl.csv

nba.csv

nhl.csv

wikipedia_data.html

Continue reading...