[Python] (Python - Corrupted files with TCP process) How to ensure the transfer? - Edited

I'm facing an issue with MP4 file transfers over TCP. I’m trying to send MP4 files from a Raspberry Pi with a camera to another Raspberry Pi with an SSD connected for storage. According to online sources, this process should result in a very low percentage of corrupted transfers, but after transferring a large amount (6445 transfers, 1 FPS, 10-second recordings), I have 2.93% corrupted files.

I wrote a code to verify file formats : MP4 : 6256 | None : 189 2.9325058184639256 % corrupted files

• 
  How does my process work? I set up a TCP connection on the storage Raspberry Pi. Every X time, the camera Raspberry Pi connects to the storage Pi and sends the video file. Once the file is sent and written, the connection is closed. The Raspberry aren't connected to internet and they are linked with Ethernet cables
  
  
• 
  What did I try? I implemented an end key sent by the camera Pi at the end of each file transfer, but I rarely detect it on the storage Raspberry Pi (only around 10% success).
  

I compared the file sizes after transfer with the original on the camera Pi, but this method resulted in 100% corrupted files (I suspect the problem lies in my code, but I’m not sure where it went wrong).

I also experimented with adjusting the FPS and recording time to see if it was a writing time issue or a RAM problem, but it didn’t lead to any significant change.

• What am I aiming for? I really want to achieve 0% corrupted files. I’m thinking of implementing a verification step after writing by checking the file format, and if it's a NoneType, I would request a resend. However, I want to optimize my program to minimize unnecessary resend iterations.

Code on storage Raspberry

def receive_file(self, client_socket, file_name):
with open(file_name+"/"+datetime.now().strftime("%y_%m_%d_%H_%M_%S"), 'wb') as f:
while True:
chunk = client_socket.recv(1500)
if not chunk:
break
f.write(chunk)
if chunk == b'END':
print("End Key, conenction will be closed")
break

def start_serv(self):
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((self.server_host, self.server_port))
server_socket.listen(3)
print("[TCP] Server Started")
while True:
read, _, _ = select.select(self.connected + [server_socket], [], [])

for sock in read:
if sock == server_socket:
client_socket, addr = server_socket.accept()
self.connected.append(client_socket)
print(f"[TCP] New connection : {client_socket}")
print(f"[TCP] Currently in queue : {self.connected} ")
else:
try:
self.receive_file(sock, self.video_path+"/"+addr[0])
sock.close()
self.connected.remove(sock)

print(f"[TCP] Video stored in {self.video_path +'/'+ addr[0]}")
except Exception as e:
print(f"[TCP] Erreur lors de la réception : {e}")
self.connected.remove(sock)
sock.close()


Code on Camera Pi

def send_file(self, file_path: str):
"""Envoie un fichier au serveur via TCP."""
if not self.client_socket:
print("[ERROR] TCP not linked anymore.")
return

if not os.path.exists(file_path):
print(f"[ERROR] wrong file path : {file_path}")
return

try:
with open(file_path, "rb") as file:
while chunk := file.read(1500):
self.client_socket.sendall(chunk)
self.client_socket.sendall(b'END')
print(f"[TCP] file '{file_path}' sent.")
os.remove(file_path)
except Exception as e:
print(f"[ERROR] Hu, something wrong happend : {e}")


(I close it in another function)

[EDIT]

Thank you for your feedback. I couldn't find how to reply to your comments, so I’m editing the question.

I removed the END key as Steffen Ullrich recommended, indeed it didn’t serve much purpose.

Regarding what ewokx said, I found the error with the size sending. The method of adding chunks using "+=" is not viable and causes many errors. So, I now append the chunks to a list and use b' '.join(chunk) before writing it.

I also downloaded a video online to focus solely on the send/receive part and see if the problem was on this side or on the camera's storage side. After sending the file 1000 times, I initially obtained files with sizes very close to, or equal to, the reference file, but without an extension.

I tried manually adding the .mp4 extension to the file, but it was not openable. (I suspected it wouldn’t be that easy....)

I then tried to see if the file extension could be identified before writing it, and for that, I used the magic library with magic.from_buffer(file in bytes, mime=True). Unfortunately, the extension couldn’t recognize the file format. Then, I had the idea of comparing the first 10 lines between the received file and the sent file, Bingo! I found this :

File with no extension :

raspberry@raspberrypi:~ $ hexdump -C /home/raspberry/Tcp/25_04_11_13_22_54 | head -n 10
00000000 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 00 |............@...|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 02 00 00 02 32 |...............2|
00000030 74 72 61 6b 00 00 00 5c 74 6b 68 64 00 00 00 03 |trak...\tkhd....|
00000040 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 |................|
*
00000080 00 00 00 00 40 00 00 00 0c a8 00 00 08 70 00 00 |....@........p..|
00000090 00 00 00 24 65 64 74 73 00 00 00 1c 65 6c 73 74 |...$edts....elst|

File with manual extension :

raspberry@raspberrypi:~ $ hexdump -C /home/raspberry/Tcp/25_04_11_13_22_54.mp4 | head -n 10
00000000 00 00 00 00 00 00 00 00 00 00 00 00 40 00 00 00 |............@...|
00000010 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000020 00 00 00 00 00 00 00 00 00 00 00 02 00 00 02 32 |...............2|
00000030 74 72 61 6b 00 00 00 5c 74 6b 68 64 00 00 00 03 |trak...\tkhd....|
00000040 00 00 00 00 00 00 00 00 00 00 00 01 00 00 00 00 |................|
00000050 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
00000060 00 00 00 00 00 01 00 00 00 00 00 00 00 00 00 00 |................|
*
00000080 00 00 00 00 40 00 00 00 0c a8 00 00 08 70 00 00 |....@........p..|
00000090 00 00 00 24 65 64 74 73 00 00 00 1c 65 6c 73 74 |...$edts....elst|

Original file :

raspberry@raspberrypi:~ $ hexdump -C /home/raspberry/Tcp/reference.mp4 | head -n 10
00000000 00 00 00 1c 66 74 79 70 69 73 6f 35 00 00 02 00 |....ftypiso5....|
00000010 69 73 6f 35 69 73 6f 36 6d 70 34 31 00 00 03 30 |iso5iso6mp41...0|
00000020 6d 6f 6f 76 00 00 00 6c 6d 76 68 64 00 00 00 00 |moov...lmvhd....|
00000030 00 00 00 00 00 00 00 00 00 00 03 e8 00 00 00 00 |................|
00000040 00 01 00 00 01 00 00 00 00 00 00 00 00 00 00 00 |................|
00000050 00 01 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |................|
*
00000070 40 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 |@...............|
00000080 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 02 |................|
00000090 00 00 02 32 74 72 61 6b 00 00 00 5c 74 6b 68 64 |...2trak...\tkhd|


The problem is that the first 5 lines are missing, and the 6th is partially there. If I understood it well that causes everything, including the file extension, to shift.

Now I need to find how to solve this ! New Code for Storage Raspberry.(I just check the file's size but don't compare it)

import socket
import select
from datetime import datetime
import magic
server_host="192.168.1.100"
server_port=8000
video_path = "/home/raspberry/Tcp"

connected = []

def receive_file(client_socket, file_name):
Transfer_size = client_socket.recv(100)
Transfer_size = int.from_bytes(Transfer_size,'big')
print(f"[TCP] Fichier video de {Transfer_size}")
sending=True
vid = []

while sending:
chunk = client_socket.recv(1500)
vid.append(chunk)
if not chunk:
break
contenu = b''.join(vid)
mime = magic.from_buffer(contenu,mime=True)
print(f"Extension detected : {mime}")

with open(file_name+"/"+datetime.now().strftime("%y_%m_%d_%H_%M_%S"), 'wb') as f:

print(f"Transfert end : {len(chunk)/Transfer_size}")
f.write(b''.join(vid))




def start_serv():
server_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
server_socket.setsockopt(socket.SOL_SOCKET, socket.SO_REUSEADDR, 1)
server_socket.bind((server_host, server_port))
server_socket.listen(3)
print("[TCP] Server Started")
while True:
read, _, _ = select.select(connected + [server_socket], [], [])

for sock in read:
if sock == server_socket:
client_socket, addr = server_socket.accept()
client_socket.recv(100)
connected.append(client_socket)
print(f"[TCP] New connection : {client_socket}")
print(f"[TCP] Currently in queue : {connected} ")
else:
try:
receive_file(sock, video_path)
sock.close()
connected.remove(sock)

print(f"[TCP] Video stored in {video_path}")
except Exception as e:
print(f"[TCP] Erreur lors de la réception : {e}")
connected.remove(sock)
sock.close()

start_serv()


New Camera Raspberry :

import socket
import os
server_host="192.168.1.100"
server_port=8000
server_host = server_host
server_port = server_port
client_socket = None
ip = "192.168.1.102"
path = "/home/raspberry/recorded_videos/13434423_3240_2160_25fps.mp4"
taille = os.path.getsize(path)
def connect():
try:
client_socket = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
client_socket.connect((server_host, server_port))
print(f"[TCP] connected {server_host}:{server_port}")
except Exception as e:
print(f"[ERROR TCP] : {e}")
client_socket = None
return client_socket

def send_file( file_path: str,client_socket,taille = taille):
compteur_envoi = 0
client_socket.send(taille.to_bytes(100,'big'))
if not client_socket:
print("[ERROR] TCP not linked anymore.")
return
if not os.path.exists(file_path):
print(f"[ERROR] wrong file path : {file_path}")
return
try:
with open(file_path, "rb") as file:
while chunk := file.read(1500):
client_socket.sendall(chunk)
compteur_envoi += len(chunk)
print(f"[TCP] file '{file_path}' sent.")
except Exception as e:
print(f"[ERROR] Hu, something wrong happend : {e}")

def close(client_socket):
if client_socket:
client_socket.close()
print("[TCP] Link closed.")
i =0
while i < 1000:
client_socket = connect()
send_file(path,client_socket )
close(client_socket)
i+=1
print(i)

Continue reading...