[Python] Byte Shifting Error White Reading Safetensor File in C

I'm trying to read safetensors file in C. As a read on Huggingface's documentation, I'm taking first 8 byte of file for header size;

...
uint64_t header_size = 0;
int read_header_size = fread(&header_size, sizeof(uint64_t), 1, fp);
...


And after that I'm reading the header of data;

...
char *header_str = MALLOC(header_size + 1);
int read_header_str = fread(header_str, 1, header_size, fp);
header_str[header_size] = '\0';
...


I'am parsing the header with cJSON safe_tensor->header = cJSON_Parse(header_str); and there is no any error. Everything works great in header file.

{
"conv2d/bias": {
"dtype": "F32",
"shape": [32],
"data_offsets": [0, 128]
},
"conv2d/kernel": {
"dtype": "F32",
...
...



Then I'm taking each header item and reading as byte as offset;

uint64_t tensor_idx = 0;
uint64_t tensor_offset = 8 + header_size;
cJSON_ArrayForEach(tensor_item, safe_tensor->header) {
if(strcmp(tensor_item->string, "__metadata__") == 0) continue;

cJSON* dtype_json = cJSON_GetObjectItem(tensor_item, "dtype");
cJSON* shape_json = cJSON_GetObjectItem(tensor_item, "shape");

...
...

fseek(fp, tensor_offset, SEEK_SET);
int read_tensor_data = fread(safe_tensor->data[tensor_idx]->data, data_size, num_elements, fp);

tensor_offset += data_size * num_elements;
tensor_idx++;



data_size is 4 because of f32 and I'm calculating num_elements from shape (Multiplying all array elements in shape_json).

It was working with dataset that I found from huggingface models and dataset I generated. But when I try to export Keras's model as safetensors It's not working;

def get_layer_parameter_names(layer):
"""Returns parameters based on layer name."""
layer_type = layer.__class__.__name__
param_names = []

# Dense & Conv
if layer_type in ["Dense", "Conv1D", "Conv2D", "Conv3D"]:
param_names = ["kernel", "bias"]

# Normalization
elif layer_type == "BatchNormalization":
param_names = ["gamma", "beta", "moving_mean", "moving_variance"]
elif layer_type == "LayerNormalization":
param_names = ["gamma", "beta"]
...
...
else:
weights = layer.get_weights()
param_names = [f"weight_{i}" for i in range(len(weights))] if weights else []

return param_names

def save_keras_model_to_safetensors(model, output_path):
tensors = {}
all_layer_names = []

for layer in model.layers:
all_layer_names.append(layer.name)
layer_weights = layer.get_weights()
param_names = get_layer_parameter_names(layer)

for name, weight in zip(param_names, layer_weights):
key = f"{layer.name}/{name}"
tensors[key] = np.array(weight, dtype=np.float32)


save_file(tensors, output_path, metadata=None)


safetensors library can handle export and import very well with this code and it's actually exporting all the parameters of class. But when I try to import safetensors file that I exported with this function I got a problem.

It reads the header file exactly correctly. Then it reads the first tensor file exactly correctly. But when it comes to the second tensor, every variable is shifted to the right by 1 zero. Likewise, in the 3rd tensor, 2 zeros are shifted to the right (1 zero to the right compared to the previous one). For example;

Tensor[0][0] = 0.16358605 // expected: 0.16358605
Tensor[0][1] = -0.087356716 // expected: -0.087356716
Tensor[1][0] = -0.40413815 // expected: -4.04138158
Tensor[1][1] = 0.13498262 // expected: 1.34982622
Tensor[2][0] = -0.00154723 // expected: -1.54723778
Tensor[2][1] = 0.00372836 // expected: 3.72836853


What could my error be? Do you have and idea?

Continue reading...