[Python] How to deploy deep learning programs on standalone Spark cluster efficiently? Or any...

I have live streaming data (big data) and a standalone spark cluster of CPU systems. I need to deploy a deeplearning program to process this data and predict result.

My DL model(Keras) is a functional model with multi-inputs and consists of LSTM-CNN layers.

<class ‘keras.src.engine.functional.Functional’> TensorFlow version: 2.15.0 Keras version: 2.15.0 Keras backend: tensorflow

I tried different methods. However, nothing gives me an efficient way of execution.

1. Initially, I trained and developed a DL model in python and saved it as model.h5. Then I created a Pyspark application where I preprocessed the streaming data as batches, imported the model.h5 and predicted the result. But, I couldn't deploy Pyspark application in cluster mode as Spark doesn't support Python in cluster mode. In client mode, it runs for 20-30minutes and then it terminates.

From Spark Docs:

Currently, the standalone mode does not support cluster mode for Python applications.

1. Then I tried to develop deeplearning application in Scala language. For this I used DL4J library. But, when I try to load the model, I got error like

A fatal error has been detected by the Java Runtime Environment:
#
# SIGSEGV (0xb) at pc=0x000072104b173475, pid=76220, tid=76222
#
# JRE version: OpenJDK Runtime Environment (11.0.24+8) (build 11.0.24+8-post-Ubuntu-1ubuntu322.04)
# Java VM: OpenJDK 64-Bit Server VM (11.0.24+8-post-Ubuntu-1ubuntu322.04, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C [libnd4jcpu.so+0x1573475] sd::NDArray::getOffset(long long) const+0x25
#
# Core dump will be written.


However, models containing layers other than LSTM/RNN doesn't raise any errors.

Can someone guide me on the other possibilities that I can try?
Is there any way to deploy python in a distributed manner? Is it challenging to deploy AI programs on Spark? If so, can someone suggest any other platforms or tools or package or language that can help me do my task of applying AI model (DL) on large streaming data?

I'm new to this area and have been stuck with this for a long time. Any suggestion would be appreciated!

Continue reading...