[Python] Pyspark, PandasUDF; How to return a matrix using Pyspark.PandasUDF?

I would like to implement a pandasUDF function in PySpark that returns a matrix of float numbers. I have tried, but the below appeared.

RuntimeError: ('Exception thrown when converting pandas.Series (float64) to
Arrow Array (list<item: float>). It can be caused by overflows or other
unsafe conversions warned by Arrow. Arrow safe type check can be disabled by
using SQL config `spark.sql.execution.pandas.convertToArrowArraySafely`.',
ArrowNotImplementedError("NumPyConverter doesn't implement <list<item: float>>
conversion. ",))


I wonder if you have an idea to get a Spark DataFrame as the table below.

Desired output:

+--------------------+--------------------+--------------------+
| P0| P1| P2|
+--------------------+--------------------+--------------------+
|-6.384 |-2.950 |-2.095 |
|-4.102 |-7.101 |-8.184 |
|-1.614 |4.4842 |-10.00 |
|-6.166 |-3.186 |-2.076 |
.
.
.
+--------------------+--------------------+--------------------+

Continue reading...