I have a text file with fixed length format like below - K57 details 1234 K58 abcdefghijklmnopqrstuvwxyz 0123456789 K58 abcdefghijklmnopqrstuvwxyz 0123456789 K58 abcdefghijklmnopqrstuvwxyz 0123456789 K57 details 5678 K58 abcdefghijklmnopqrstuvwxyz 9876543210 K58 abcdefghijklmnopqrstuvwxyz 9876543210 K58 abcdefghijklmnopqrstuvwxyz 9876543210 ..... ..... Here K57 is header record of each batch and K58 records are detail records of the batch. There can be multiple batches each having its header(K57) record followed by multiple detail(K58) records. I need to extract values from each K58 record along with its header record detail in following format - +----------+------------+------------+ |K57_detail|K58_detail_1|K58_detail_2| +----------+------------+------------+ |1234 |abcdef |01234 | |1234 |abcdef |01234 | |1234 |abcdef |01234 | |5678 |abcdef |98765 | |5678 |abcdef |98765 | |5678 |abcdef |98765 | +----------+------------+------------+ I know Pyspark substring() function which can help to extract required value from each row. But my real challenge is to bring info of header K57 record to each of its detail records. Any help to achieve the above result will be appreciated. Thanks in advance. Continue reading...
