[Python] How do I write this discord.py command so it works on a large array of server members?

Background

Hi everyone. I've used the discord.py package to write a Discord bot for a server that I own and manage. The server is an unofficially official community for a retail brand in the United States and is dominated by employees. We have a role (@U.S. Employee) to distinguish employees and grant access to employee-only channels. Once a year, we require employees to verify that they're still an employee. Once verified, they're given the @Verified - 2022 role.

I've written a command called prune_unverified where it will iterate over all server members and checks whether or not they have either of these roles.

Here's how the command works: If the server member doesn't have the @U.S. Employee role (or the @Canada Employee role), then don't do anything. If the employee has either of these employee roles and also has the @Verified - 2022 role, then also don't do anything. However, if the employee has either employee role and does not have the @Verified - 2022 role, then all of their roles are removed and replaced with the @Former Employee role.

The Problem

The code that I've written runs and executes flawlessly with zero errors during my small scale tests. It works exactly as expected. The issue is that every year when running the command for real, the number of server members who have either employee role but does not have the verified role (and thus will be given the former employee role) is well over 1,000 server members.

I'll run the command, it will work perfectly for less than 200 server members and then it will just stop. The bot is supposed to reply with a message once it completes, but it never does. The bot doesn't crash though - it's literally just this one command will stop working. Even trying it again won't work anymore. Even rebooting the bot won't work. All other functions of the bot continues to work perfectly fine.

The source code of the bot is hosted in a private GitHub repository which is synced with a Heroku app on the heroku-20 stack with automated deployments enabled. It's slug size right now is 62.9 MiB of 500 MiB. It's also using hobby dynos - the one that only costs $7/dyno/month.

The code for the command I'm talking about is below. So how can I write this code better so that it works actioning upon over 1,000 server members at once? Am I being limited by the Discord API? Or the discord.py framework? Do I not have enough memory or performance on the Heroku hobby dyno? Any help would be greatly appreciated. I really would like to get this code to work but I'm not the greatest programmer and I'm not sure how else I can write this code.

@bot.command(name = "prune_unverified", help = "prune the unverified employees", enabled = True)
@commands.has_role("Owner")
async def prune_unverified(context):
await context.message.delete()
guild = discord.utils.get(bot.guilds, name = GUILD)

ServerMembers = [i for i in guild.members]
VerifiedEmployees = []
PrunedUsers = []

EmployeeRoleUS = guild.get_role(EMPLOYEE_ROLE)
EmployeeRoleCA = guild.get_role(EMPLOYEE_CA_ROLE)
VerifiedRole = guild.get_role(EMPLOYEE_VERIFIED_ROLE)

for user in ServerMembers:
if (EmployeeRoleUS or EmployeeRoleCA) in user.roles:
if VerifiedRole in user.roles:
VerifiedEmployees.append(user)
else:
PrunedUsers.append(user)
else:
continue

for user in PrunedUsers:
await guild.get_member(user.id).edit(roles = []) # Remove all roles
await guild.get_member(user.id).add_roles(guild.get_role(EMPLOYEE_FORMER_ROLE)) # Award the former role
await asyncio.sleep(15)
# pass # Adding a pass here because we are just testing

# create a .csv file of pruned users
with open("pruned_users.csv", mode = "w") as pu_file:
pu_writer = csv.writer(pu_file, delimiter = ",")
pu_writer.writerow(["Server Nickname", "Username", "ID"])
for user in PrunedUsers:
pu_writer.writerow([f"{user.nick}", f"{user.name}#{user.discriminator}", f"{user.id}"])

# create a .csv file of verified users
with open("verified_users.csv", mode = "w") as vu_file:
vu_writer = csv.writer(vu_file, delimiter = ",")
vu_writer.writerow(["Server Nickname", "Username", "ID"])
for user in VerifiedEmployees:
vu_writer.writerow([f"{user.nick}", f"{user.name}#{user.discriminator}", f"{user.id}"])

embed = discord.Embed(
description = f":crossed_swords: **`{len(PrunedUsers)}` users were pruned by <@{context.author.id}>.**"
+ f"\n:shield: **There are `{len(VerifiedEmployees)}` users who completed re-verification.**"
+ f"\n\n**Here is a `.csv` file of all the users that were pruned:**",
color = discord.Color.blue()
)
await context.send(embed = embed)
await context.send(file = discord.File(r"pruned_users.csv")) # upload the .csv file we created
await context.send(file = discord.File(r"verified_users.csv"))
await log_command(context, "prune_unverified")


UPDATE: Sunday June 26, 2022 @ ~4:13 PM EST

Thanks to your help, I've rewritten the logic and also changed how I retrieve all server members. After running this new code, it took the command approximately 24 minutes to execute and it worked flawlessly. Thank you everybody!

@bot.command(name = "prune_unverified", help = "prune the unverified employees", enabled = True)
@commands.has_role("Owner")
async def prune_unverified(context):
await context.message.delete()

VerifiedEmployees = []
PrunedUsers = []

EmployeeRoleUS = context.guild.get_role(EMPLOYEE_ROLE)
EmployeeRoleCA = context.guild.get_role(EMPLOYEE_CA_ROLE)
VerifiedRole = context.guild.get_role(EMPLOYEE_VERIFIED_ROLE)

# Begin new code here
async for user in context.guild.fetch_members(limit = None):
if (EmployeeRoleUS in user.roles) or (EmployeeRoleCA in user.roles):
if VerifiedRole in user.roles:
VerifiedEmployees.append(user)
else:
PrunedUsers.append(user)
else:
continue
# End new code here

for user in PrunedUsers:
#await user.edit(roles = []) # Remove all roles (Commented out for testing)
await user.add_roles(context.guild.get_role(990703166041501806)) # Award the former role

# create a .csv file of pruned users
with open("pruned_users.csv", mode = "w") as pu_file:
pu_writer = csv.writer(pu_file, delimiter = ",")
pu_writer.writerow(["Server Nickname", "Username", "ID"])
for user in PrunedUsers:
pu_writer.writerow([f"{user.nick}", f"{user.name}#{user.discriminator}", f"{user.id}"])

# create a .csv file of verified users
with open("verified_users.csv", mode = "w") as vu_file:
vu_writer = csv.writer(vu_file, delimiter = ",")
vu_writer.writerow(["Server Nickname", "Username", "ID"])
for user in VerifiedEmployees:
vu_writer.writerow([f"{user.nick}", f"{user.name}#{user.discriminator}", f"{user.id}"])

embed = discord.Embed(
description = f":crossed_swords: **`{len(PrunedUsers)}` users were pruned by <@{context.author.id}>.**"
+ f"\n:shield: **There are `{len(VerifiedEmployees)}` users who completed re-verification.**"
+ f"\n\n**Here is a `.csv` file of all the users that were pruned:**",
color = discord.Color.blue()
)
await context.send(embed = embed)
await context.send(file = discord.File(r"pruned_users.csv")) # upload the .csv file we created
await context.send(file = discord.File(r"verified_users.csv"))
await log_command(context, "prune_unverified")

Continue reading...