[Python] Why are images from my custom RTP-to-JPEG decoder distorted, while OpenCV retrieves...

I am working with an Ethernet camera that streams images via an RTSP stream. The camera supports PTP (Precision Time Protocol) and writes the PTP timestamp in the header extension of the RTP header. The extension length of the header is set to 2, followed by the PTP Unix timestamp and milliseconds (as visible in this Wireshark screenshot).

I wrote a custom function to retrieve single frames from the RTP stream, assemble the JPEG payload, and create a JFIF header so I can save the image and include the PTP timestamp when the image was taken. Here's the relevant part of my code:

def process_rtp_packets(self):
buffer = bytearray()
last_sequence_number = None
quantization_table = None
height_blocks = None
width_blocks = None
while True:
try:
data, addr = self.udp_sock.recvfrom(4096)
package = self.DecodeRTPpacket(data)

sequence_number = package['rtsp']['sequence_number']
if last_sequence_number is not None and sequence_number != last_sequence_number + 1:
print(f"Warning: Packet loss or out-of-order packets detected. Expected {last_sequence_number + 1}, got {sequence_number}")
buffer = bytearray()
last_sequence_number = sequence_number

if package['jpeg']['fragment_offset'] == 0:
buffer = bytearray() # Reset buffer for a new frame
quantization_table = package['jpeg'].get('quantization_table', None) # get the quantization table from first packet
height_blocks = package['jpeg'].get('height', None) # get the height from first packet
width_blocks = package['jpeg'].get('width', None) # get the width from first packet
if not quantization_table:
print("Error: Missing quantization table in the first packet.")
continue

# Extract the payload and fragment offset
fragment_offset = package['jpeg']['fragment_offset']
payload = package['jpeg']['payload']

# Ensure the buffer is large enough to accommodate the payload at the specified offset
if len(buffer) < fragment_offset + len(payload):
buffer.extend(b'\x00' * (fragment_offset + len(payload) - len(buffer)))

# Place the payload at the correct offset
buffer[fragment_offset:fragment_offset + len(payload)] = payload

if package['rtsp']['marker'] == 1:
lqt = quantization_table['quantization_table_data'][:64]
cqt = quantization_table['quantization_table_data'][64:128]
jfif_image = self.make_headers(0, width_blocks, height_blocks, lqt, cqt, 0) + buffer

with open("output.jpg", "wb") as f:
f.write(jfif_image)

# convert to cv2 image
# nparr = np.frombuffer(jfif_image, np.uint8)
# image = cv2.imdecode(nparr, cv2.IMREAD_COLOR)
# cv2.imwrite('output_cv2.jpg', image)

except KeyboardInterrupt:
break


I retrieve the quantization tables from the first packet of the RTP stream to generate an appropriate JFIF header. I have reduced the camera's FPS to 1 to minimize network load and system resource usage. The system resource monitor confirms that there are no performance issues.

Despite my efforts, the images generated by my code are distorted. The top part of the image looks correct, but as you go further down, there are pixel offsets to the left, and the colors are completely off. Here's an example of the distorted image: image generated by my code.

When I use OpenCV to retrieve frames from the same RTSP stream, the images look normal, so the issue is not with the camera. Here's an example of the correct image: correct image.

What I Have Tried:

• Verified that the quantization tables are correctly extracted from the first RTP packet.
• Ensured that the payloads are placed at the correct offsets in the buffer.
• Reduced the camera's FPS to 1 to avoid network congestion or packet loss.
• Checked for packet loss or out-of-order packets using debug logs.

My Questions:

• What could be causing the pixel offsets and color distortion in the generated images?
• How can I ensure that the payloads from the RTP packets are correctly assembled into a valid JPEG image?
• Could the issue be related to the PTP timestamp in the RTP header extension, and if so, how should I handle it?
• Is my whole attempt flawed and there is a better way to retrieve the PTP-timestamp?

Thats how i generate the headers:

def make_headers(self, image_type, width, height, lqt, cqt, dri)-> bytes:
"""
Constructs the JPEG headers for the image.

Returns:
bytes: The constructed JPEG headers.
"""
if not isinstance(lqt, (bytes, bytearray)) or len(lqt) != 64:
raise ValueError("Luminance quantization table must be a 64-byte array.")
if not isinstance(cqt, (bytes, bytearray)) or len(cqt) != 64:
raise ValueError("Chrominance quantization table must be a 64-byte array.")
if not (image_type == 0 or image_type == 1):
raise ValueError("Image type must be 0 (4:2:2) or 1 (4:2:0).")

# Start constructing the headers
headers = bytearray()

# Start of Image (SOI) marker
headers.append(0xff)
headers.append(0xd8)

# JFIF header
headers.extend(self.jfif_header)

# Add Quantization Tables (DQT)
headers.extend(self.quant_flag)
headers.extend((132).to_bytes(2, byteorder='big')) # Length LSB (64 bytes + 3 bytes for the header)
headers.append(0) # Table number
headers.extend(self.make_quant_header(lqt, 0)) # Luminance
headers.append(1) # Table number
headers.extend(self.make_quant_header(cqt, 1)) # Chrominance

# Add Restart Interval (DRI) if applicable
if dri != 0:
headers.extend(self.make_dri_header(dri))

# Add Huffman Tables (DHT)
headers.extend(self.huf_flag)
headers.extend((418).to_bytes(2, byteorder='big')) # Length LSB (2 bytes + 16 + 16 + 16 + 16)
headers.append(0) # Table class and table number for DC Luminance
headers.extend(self.make_huffman_header(self.lum_dc_codelens, self.lum_dc_codelens, 0, 0)) # Luminance DC
headers.extend(self.make_huffman_header(self.lum_ac_codelens, self.lum_dc_symbols, 0, 1)) # Luminance AC
headers.extend(self.make_huffman_header(self.chm_dc_codelens, self.lum_ac_symbols, 1, 0)) # Chrominance DC
headers.extend(self.make_huffman_header(self.chm_ac_codelens, self.chm_dc_symbols, 1, 1)) # Chrominance AC
headers.extend(self.make_huffman_header(self.lum_dc_codelens, self.chm_ac_symbols, 2, 0)) # Luminance DC

# start of frame header
headers.append(0xff)
headers.append(0xc0) # SOF marker
headers.extend(self.sof_header)

# header
headers.append(0xff)
headers.append(0xdd)
headers.extend([0x00, 0x04, 0x00, 0x00])

# start of scan header
headers.append(0xff)
headers.append(0xda) # SOS marker
headers.extend(self.start_of_scan_header)

return bytes(headers)
def make_huffman_header(self, codelens, symbols, table_no, table_class):
"""
Constructs a Huffman header for the JPEG image.

Args:
codelens (list): The code lengths for the Huffman table.
symbols (list): The symbols for the Huffman table.
table_no (int): The table number (0-3).
table_class (int): The table class (0 for DC, 1 for AC).

Returns:
bytes: The constructed Huffman header.
"""
# Validate inputs
if not isinstance(codelens, (list, bytes)) or not isinstance(symbols, (list, bytes)):
raise ValueError("codelens and symbols must be lists or bytes.")
if not (0 <= table_no <= 3):
raise ValueError("table_no must be between 0 and 3.")
if not (0 <= table_class <= 1):
raise ValueError("table_class must be 0 (DC) or 1 (AC).")

header = bytearray()

# Calculate lengths
ncodes = len(codelens)
nsymbols = len(symbols)

# header.append(0x00)
# header.append(3 + ncodes + nsymbols) # Length LSB
#header.append((table_class << 4) | table_no) # Table class and table number

# # Append code lengths and symbols
# header.extend(codelens)
header.extend(symbols)

return bytes(header)

def make_quant_header(self, qt, table_no):
"""
Constructs a Quantization Table (DQT) header for the JPEG image.

Args:
qt (bytes): The quantization table (64 bytes).
table_no (int): The table number (0-3).

Returns:
bytes: The constructed Quantization Table header.
"""
if not isinstance(qt, (bytes, bytearray)) or len(qt) != 64:
raise ValueError("Quantization table must be a 64-byte array.")
if not (0 <= table_no <= 3):
raise ValueError("Table number must be between 0 and 3.")

# Construct the DQT header
header = bytearray()

header.extend(qt) # Append the quantization table data

return bytes(header)

Continue reading...