[Python] zstd: Unknown frame descriptor

I have a binary file that appears to be originally generated by a Java hashmap serialization. The majority of the file seems to be a zstandard-compressed archive:

00000000 28 B5 2F FD:00 58 9D 21|00 16 BC D4:4B B0 56 55
00000010 3A 52 74 EB5 B9 A7 C3|FD 38 04 39:ED EC 86 F3
00000020 C7 08 8F 4C:77 82 FA D7|10 47 EC 77:73 23 B7 D0
00000030 E6 D0 27 019 08 D4 F5|37 52 38 66:1C 26 B1 71
00000040 0A 24 95 8B:A4 26 58 B6|C5 8D 0B 92:36 A5 F0 D2
00000050 D1 15 BC F77 FF 4F 01|BE 00 BC 00:C2 00 17 F0
00000060 DF C5 04 F3:FF BF 60 41|58 EC A1 2A:70 BD 18 98
00000070 C9 9B 58 CD:3C 7B EB 59|15 7D FD A6:FA 6A 09 0A
00000080 03 46 8A 3C:01 C0 12 DA|1C 70 B5 1C:4A F4 CC C4


Here, we can clearly see the magic bytes for zstd - 28 BF 2F FD. I inspected the entire file in hexedit, there appear to be multiple instances of zstd's magic bytes, suggesting that maybe there are multiple concatenated archives in this file.

Regardless, after apt install zstd and zstd -d zstandardblock.zst, I get the error: zstandardblock.zst : 0 B... zstd: zstandardblock.zst: unsupported format

I then used this Python program to try decoding it by chunks:

import zstandard as zstd

def decompress_until_error(input_file, output_prefix, chunk_size=65536):
"""Decompress a Zstd file in large chunks until an error occurs, writing each chunk to separate files."""
with open(input_file, "rb") as compressed:
dctx = zstd.ZstdDecompressor()
reader = dctx.stream_reader(compressed)
chunk_index = 0
try:
while True:
chunk = reader.read(chunk_size)
if not chunk: # Stop if no more data
break
# Check for skippable frame marker (0x184D2A50)
if chunk[:4] == b'\x50\x2A\x4D\x18':
print(f"Skippable frame detected at chunk {chunk_index}, skipping...")
continue
output_file = f"{output_prefix}_{chunk_index}.bin"
with open(output_file, "wb") as f:
f.write(chunk)
print(f"Chunk {chunk_index}: {len(chunk)} bytes written to {output_file}")
chunk_index += 1
except zstd.ZstdError as e:
print(f"Zstandard error encountered after {chunk_index} chunks: {e}")
except Exception as e:
print(f"Unexpected error encountered after {chunk_index} chunks: {e}")
print("Decompression halted due to an error.")

# Example usage
decompress_until_error("zstandardblock.zst", "final_output")


Just a basic program to read the file. Curiously enough, I was able to extract a corrupted Java hashmap, so this zstd block does contain some data. However, the script terminates with this error:

Chunk 0: 2098 bytes written to final_output_0.bin
Zstandard error encountered after 1 chunks: zstd decompress error: Unknown frame descriptor
Decompression halted due to an error.


I'm confident this archive contains usable information. Is there a flag or setting for zstd, or some way for me to truncate or modify the binary data, to decompress it?

I'm not sure of the etiquette in providing files, but I've trimmed my workspace and made it into a codesandbox project: https://codesandbox.io/p/devbox/3l96hf
The original binary and output files are all there.

Continue reading...