[Python] How to flatten nested JSON in pyspark

I have a JSON file that looks like this:

[
{ "student_id": 1234,
"room_id": "abc",
"enrolled": false
},
{ "student_id": 4321,
"room_id": "def",
"enrolled": true,
"enrollment": {
"type": "home",
"date": "01-01-2020"
}
},
{ "student_id": 678,
"room_id": "htf",
"sports": {
"team": "hockey",
"position": "forward"
}
]


I able to flatten it partially by doing:

df = sc.parallelize(data).map(lambda x: json.dumps(x))


This gives me:

student_id room_id enrolled enrollment sports
1234 abc false NULL NULL
4321 def true {home, 01-01-2020} NULL
678 htf NULL NULL {hockey, forward}

How can I flatten this further to get:

student_id room_id enrolled type date team position
1234 abc false NULL NULL NULL NULL
4321 def true home 01-01-2020 NULL NULL
678 htf NULL NULL NULL. hockey forward

Continue reading...