[Python] How to convert from Python pandas Timestamp to repeated google.protobuf.Timestamp?...

I am trying to write some code which converts the contents of a pandas.DataFrame to a protobuf object which can be serialized and written to a file.

Here is my protobuf definition.

syntax = "proto3";

import "google/protobuf/timestamp.proto"

message BidAskTimeseries {
repeated double bid = 1;
repeated double ask = 2;
repeated google.protobuf.Timestamp timestamp = 3;
}


It can be compiled using

protoc --proto_path=. --python_out=. bid_ask_timeseries.proto


The pandas.DataFrame has been loaded from a CSV file.

import pandas

df = pandas.read_csv('df.csv')


The datatypes are

df_data.dtypes
ask float64
bid float64
ts object


however ts is actually a String.

The timestamp column can be converted to a pandas.Timestamp in the usual way

df['ts'] = pandas.to_datetime(df['ts'])


however, this step should be considered optional.

I could not figure out how to create a Python protobuf object from this data.

import bid_ask_timeseries_pb2
from google.protobuf.timestamp import Timestamp

bid_ask_timeseries = bid_ask_timeseries_pb2.BidAskTimeseries(
bid=df['bid'],
ask=df['ask'],
timestamp=df['ts'],
)


It seemed like keeping the 'ts' column as a String would be the most straightforward approach, since dir(Timestamp) shows that Timestamp has a FromString method. However, this produced the following error.

Traceback (most recent call last):
File "<stdin>", line 1, in <module>
TypeError: expected bytes, str found


Converting to pandas.Timestamp is preferable, since other parts of my code expect a DataFrame with datetime values stored as a pandas.Timestamp type, rather than a String type.

However (and I may be wrong here) the pandas.Timestamp appears to be a two-element struct containing a count of number of seconds, and count of number of nanoseconds, exactly like the Linux struct timespec type. (If I'm wrong about this please let me know, I wasn't too sure from looking at the documentation.)

• https://www.man7.org/linux/man-pages/man3/timespec.3type.html

Assuming this is correct however, this format does seem as if it would line up with the protobuf format for a Timestamp which is an int64 count of seconds and a int32 count of nanos.

• https://github.com/protocolbuffers/protobuf/blob/main/src/google/protobuf/timestamp.proto

So it seems like there should be a relatively straightforward way to convert from the dataframe I am reading to the protobuf serialized format. But I can't quite figure out how to do it in a sensible way. (Meaning not element-by-element at the Python level, which is likely to be extremely slow.)

Continue reading...