[Python] Xwayland gradually consumes all RAM when running tkinter + matplotlib + RPI Connect

I have a Raspberry Pi 4 (4 GB RAM) running a Python program 24/7. This program uses tkinter for the GUI along matplotlib’s FuncAnimation to update graphs. After launching the Python program, the Xwayland service starts consuming about 160 MB of RAM, as it can be checked in this screenshoot of the task manager:



The problem is that this usage keeps growing gradually each day until Xwayland eventually takes up all the available RAM. The RPI uses Raspberry Pi OS Bookworm (fully updated). I also have RPI Connect service activated in Desktop mode, as I need to check the graphs in real time whenever I need remotely.

I was able to work around this issue by restricting Xwayland’s resource usage. Specifically, I did two things:

First action: Creating an override.conf file

[Service]

Environment="XWAYLAND_NO_GLAMOR=1"
Environment="MESA_LOADER_DRIVER_OVERRIDE=software"
Environment="MALLOC_TRIM_THRESHOLD_=100000"

LimitAS=1073741824
LimitRSS=536870912
LimitDATA=536870912
LimitVMEM=1073741824

IPCNamespacingSupported=yes


Then, I have executed: sudo systemctl daemon-reload

Second action: Configuring environment variables globally:

echo 'XWAYLAND_NO_GLAMOR=1' | sudo tee -a /etc/environment
echo 'MESA_LOADER_DRIVER_OVERRIDE=software' | sudo tee -a /etc/environment


With these two actions, Xwayland’s RAM consumption now stays constant. However, I am not sure if this workaround is ok in the long term.

My question is: Why did Xwayland showed this progressive RAM increase?.

These are the important functions of the python code:

def inicializarXeY(self):
# We create a list for both x (time) and y (numeric values)
self.x_vals = []
self.y_vals = []
# Initialize a list of lists, as there will be 20 lists, each storing the values of each of the 20 data
for i in range(self.NUMEROMAXIMODEDATOS):
self.y_vals.append([])

def configurarGrafica(self):

# Ensure matplotlib uses the correct backend
plt.switch_backend('TkAgg')

# We create a figure with its axes. figsize determines the graph size (considering it occupies columnspan as below)
# The dpi establishes the text size of axes and title
self.fig, self.ax = plt.subplots(figsize=(7, 3), dpi=85, label = 'Start')

# Configure to optimize memory
self.fig.patch.set_facecolor('white')
self.ax.set_facecolor('white')

def dibujarGrafica(self):

# Create the tkinter canvas
self.canvas = FigureCanvasTkAgg(plt.gcf(), master=self.window)
self.canvas.get_tk_widget().grid(column=0, row=6, padx=1, pady =1, columnspan = 8, rowspan=11, sticky=W)

# Execute animation every 2 seconds
# Must pass cache_frame_data=False, since this animation is real-time and has no predictable end, and it's safe
# because we don't need to go back in frames or save them in memory.
self.ani = FuncAnimation(self.fig, self.animate, interval=2000, blit=False, cache_frame_data=False)

# list of possible colors to use
colores = [
"red","blue","green","purple","grey","black","pink","orange","violet","yellow",
"red","blue","green","purple","grey","black","pink","orange","violet","yellow"
]
# List of all possible lines to draw to use later in the legend
nombres = [
self.configdata["dato1"], self.configdata["dato2"], self.configdata["dato3"], self.configdata["dato4"],
self.configdata["dato5"], self.configdata["dato6"], self.configdata["dato7"], self.configdata["dato8"],
self.configdata["dato9"], self.configdata["dato10"], self.configdata["dato11"], self.configdata["dato12"],
self.configdata["dato13"], self.configdata["dato14"], self.configdata["dato15"], self.configdata["dato16"],
self.configdata["dato17"], self.configdata["dato18"], self.configdata["dato19"], self.configdata["dato20"]
]
# Create drawing lines
self.lineas = [
self.ax.plot([], [], color=colores, label=nombres)[0] for i in range(self.NUMEROMAXIMODEDATOS)
]

def animate(self, i):

# Detect if strange character is received through serial port
tipocadenarecibida = self.verificar_y_asignar_cadena(self.tiempo_texto.get())
#print("Type of received string = ", tipocadenarecibida)

artistas = []

# Only plot if serial data is being received and no strange character is received
if (self.FlagdibujarGrafica == True and tipocadenarecibida == 0):

# We assign x and y values. The x will always be time
self.xactual = self.tiempo_valor

# We put the first time value in a list to then count cycle times correctly
if (self.flagprimervalorrecibido == True):
self.rangosdexparalosciclos.append(self.xactual)
self.flagprimervalorrecibido = False

# When there's a cycle change, do several things
if (self.xprevio > self.xactual):
# we add the last value so it adds the necessary offset so that values don't overlap in the temporal vector
self.xoffsetacumulado = self.xoffsetacumulado + self.xprevio

# We build a vector to know which time ranges are each cycle and thus be able to draw them correctly when desired
self.rangosdexparalosciclos.append(self.xoffsetacumulado)

# The time value with accumulated offset is added to the list
self.x_vals.append(self.xactual + self.xoffsetacumulado)

# We save the last value to store it for the next serial record
self.xprevio = self.xactual

# The y will depend on what's selected.
for i in range(self.NUMEROMAXIMODEDATOS):
y = 0.0
if (self.listadetextosdelstring.get() == ""):
y = 0.0
else:
# Because sometimes the character \x00 arrives at some position in the string, we have to remove them so that the value can be plotted correctly later
rawtext = self.listadetextosdelstring.get()
rawtext = rawtext.replace('\x00','').strip().replace(',','.')
# use try-except to avoid exceptions in conversions, which make the graph crash
try:
y = float(rawtext)
except ValueError:
y = -77.0 # default value if not numeric
# Show exception in terminal and log for debugging purposes
print("Problem converting a value from the string text list: ")
print(repr(rawtext))
date_str = datetime.datetime.now().strftime("\n%d_%m_%Y %H:%M:%S")
self.logfile.append(date_str + " Problem converting a value from the string text list: ")
self.logfile.append(repr(rawtext))
# Add to list, necessary for real-time drawing
self.y_vals.append(y)

# See which ones have been selected as visible
visibles = set(int(k) - 1 for k in self.listadatosadibujar if k.isdigit())
for j in range(self.NUMEROMAXIMODEDATOS):
self.lineas[j].set_visible(j in visibles)

# Set data for each visible line
for i in visibles:
self.lineas.set_data(self.x_vals, self.y_vals)
# accumulate Artist objects to return
artistas.append(self.lineas)

# Limit X axis
self.ax.set_xlim(min(self.x_vals), max(self.x_vals))

# Limit Y axis
y_min, y_max = None, None
for j in visibles:
datos = self.y_vals[j]
if len(datos) > 0:
min_d = min(datos)
max_d = max(datos)
if y_min is None or min_d < y_min:
y_min = min_d
if y_max is None or max_d > y_max:
y_max = max_d
self.ax.set_ylim(y_min, y_max)

# Only draw active legends
if visibles != self.ultima_lista_visible:
# Safer handling of existing legend
if self.legend_obj:
try:
self.legend_obj.remove()
except (ValueError, AttributeError):
# Legend no longer exists or is not associated with current axes
pass

# Create new legend
handles = [self.lineas[j] for j in visibles]
labels = [self.lineas[j].get_label() for j in visibles]
self.legend_obj = self.ax.legend(handles, labels)
self.ultima_lista_visible = visibles.copy()
artistas.append(self.legend_obj)

# Set data type, graph and limit
self.tipoDatoytipoGrafica()

# Always return a list (even if empty) to avoid errors with blit=True
return artistas

# Key function of the class, runs every two seconds to check if something has been received through serial port to read it
def periodicCall(self):

# Update port status text continuously
if (self.serie.getEstadoPuerto() == True):
self.lbl_estadopuerto.configure(text = "Open",bg="green", font = 'Helvetica 10 bold')
else:
self.lbl_estadopuerto.configure(text = "Closed", bg="red", font = 'Helvetica 10 bold')

# While loop to check if queue has something or not
while self.queue.qsize( ):
try:
# Data file is only created if port is open and hasn't been created yet, since it's only created once
if (self.serie.getEstadoPuerto() == True and self.datafileflag == False):

# Call function to create data file
self.createDataFile()

# Reset flag so it doesn't enter again
self.datafileflag = True

# Show only once that data is being received through serial port
if (self.flagmostrarenlogfechayhoraprimerdatorecibido == False):

# Call queue and save it in msg variable
msg = self.queue.get(0)

# First write strings in local file. Designed to read only numeric values, not strings
if msg == None:
print("Empty string...")
return
#line = re.sub(b'\r\n', b'', msg).decode('utf-8')
#print("line: " + line)

try:
# Assume msg is the variable containing the original message
line = re.sub(b'\n', b'', msg).decode('utf-8')
print(line)
except (re.error, UnicodeDecodeError) as e:
# Capture regular expression and decoding exceptions
print(f"An error occurred: {e}")
date_str = datetime.datetime.now().strftime("\n%d-%m-%Y %H:%M:%S")
self.logfile.append(date_str + " SERIAL PORT Error receiving data through serial port. Check that source and destination baudrate match: " + str(e))
return

# If port is open, record received string in text file (data + parameters, everything)
if (self.serie.getEstadoPuerto() == True):

# Record string in local txt file
self.datafile.append(line)
# Add carriage return
self.datafile.append("\n")

# There are several strings sent from Mduino that have specific actions. This order must be respected for NEWFILE to work well and graph to draw correctly
# Then there's "NEWFILE", without commas, as various special actions must be performed
# Has to go after header, so config file is read correctly and legends are updated correctly
if "NEWFILE" in line:

# Call dedicated function
self.newfilefunction()

# Extract information from string to show in HMI. This has to go before len(line_split) afterwards
# Depending on whether we receive value string or parameters, save independent split string
# Initialize split strings
lineparam_split = ''

# Make periodic call to this function every 2000 ms, that's how it's done in tkinter
self.window.after(2000, self.periodicCall)

# Main tkinter window closing
def ExitApplication(self):

MsgBox = messagebox.askquestion ('Exit','Confirm to close program',icon = 'warning')
if MsgBox == 'yes':

# CLEANUP BEFORE CLOSING
print("Performing cleanup before closing...")

# Stop animation
if hasattr(self, 'ani') and self.ani:
self.ani.event_source.stop()

# Close serial port
if hasattr(self, 'serie'):
self.serie.serieClose()

# Close all matplotlib figures
plt.close('all')

# Destroy window
self.window.destroy()

# Force final garbage collection
import gc
gc.collect()

# Close program
sys.exit()

# Clear graph button
def BotonborrarGrafica(self):

print("Graph cleared by user")

self.borrarGrafica()

# Create action log
date_str = datetime.datetime.now().strftime("\n%d-%m-%Y %H:%M:%S")
self.logfile.append(date_str + " Graph cleared by user")

# Clear graph function, used in various places
def borrarGrafica(self):

# 1. STOP ANIMATION FIRST
if hasattr(self, 'ani') and self.ani:
try:
self.ani.event_source.stop()
self.ani = None
except:
pass

# 2. COMPLETELY CLEAR THE FIGURE
if hasattr(self, 'fig') and self.fig:
# Clear all axes and objects
self.fig.clear()
plt.close(self.fig) # CRITICAL! Close figure to free memory
self.fig = None

# 3. DESTROY TKINTER CANVAS
if hasattr(self, 'canvas') and self.canvas:
try:
self.canvas.get_tk_widget().destroy() # Destroy widget
self.canvas = None
except:
pass

# 4. CLEAR REFERENCES TO MATPLOTLIB OBJECTS
self.lineas = []
self.ax = None
if hasattr(self, 'legend_obj'):
self.legend_obj = None

# 5. CLEAR ACCUMULATED DATA (LIMIT SIZE)
self.inicializarXeY()

# 6. RECREATE GRAPH COMPLETELY
self.configurarGrafica()
self.dibujarGrafica()

# 7. RESET VARIABLES
self.valorinferiordeciclo = 0
self.valorsuperiordeciclo = 0
self.tiposeleccionciclo = 1
self.cajaciclosnormal = -1
self.ultima_lista_visible = set()

# 8. FORCE GARBAGE COLLECTION
gc.collect()

print("Graph cleared and memory freed correctly")


Sorry that variable names are in Spanish. The program works this way: it receives serial data every 2 seconds, that is used by matplotlib funcanim class to draw the data also every 2 seconds (periodic Call function). Once every several hours (more than 10000 points on each figure), the program erases the real time graph and starts a new one (borrargrafica function).

Continue reading...