[Python] Django Scraper Matching Issue: match_maker Only Returns 4 Members Instead of 150

Question:

I'm working on a Django project that uses Scrapy to scrape member profiles from a website. The scraped data is processed by a method called match_maker. However, I'm encountering an issue where match_maker only returns 4 members, despite having 150 members in the database (excluding 3 staff members).

Details:

Database: Contains 153 members; 3 are staff members, leaving 150 regular members. Profile Types: Each member has a profile_type of either 'Man', 'Woman', 'Trans', or 'Couple'.

Issue:

In the match_maker method, there's a loop that processes rooms and assigns them to members. A set named used_rooms is used to track assigned rooms to ensure each room is only assigned once. The relevant code snippet is:

if room["username"] in used_rooms:
continue


When this condition is active, only 4 members are returned. If I comment out this check, the method processes all 150 members, but the number of available rooms exceeds one million, which is incorrect.

Objective:

I need each room to be assigned to only one member, ensuring no more than one member owns a particular room. I'm looking for guidance on how to resolve this issue so that match_maker correctly processes all 150 members without assigning multiple members to the same room.

What I've Tried:

Ensured Uniqueness: Verified that room["username"] is unique for each room. Debugged used_rooms: Printed the contents of used_rooms before and after the check to ensure it's being populated correctly. Checked Room Data Structure: Confirmed that room["username"] is unique across all rooms. Despite these efforts, the issue persists. Any insights or suggestions would be greatly appreciated.

Code snippet:

def match_maker(self, members, room_data: list):
matched_items = []
used_rooms = set() # Keep track of assigned rooms
room_data_copy = deepcopy(room_data)

# Group rooms by profile_type for quick lookup
rooms_by_type = defaultdict(list)

for room in room_data_copy:
if len(room["body"]) >= 5: # Only store rooms with a valid description
rooms_by_type[room["profile_type"]].append(room)

users = set()

for member in members:
if member.is_staff:
continue

# Get matching rooms for this profile_type
available_rooms = rooms_by_type.get(member.profile_type, [])

for room in available_rooms:
if room["id"] in used_rooms:
continue

if room["profile_type"] != member.profile_type:
continue

users.add(member.username)
matched_items.append((member, room))
used_rooms.add(room["id"])

print(f"The users are: {users}")
print(f"The number of users are: {len(users)}")


random.shuffle(matched_items)

return matched_items

Continue reading...