[Python] Fabric PySpark Notebook Parsing JSON string with double quote escape chars

I am working in a PySpark notebook that gets its input parameter as a JSON String from a Pipeline and the notebook need to process the string further. Below is the example string that the notebook gets as input and example code that try to process the string and fails because the Details section has text enclosed with " ("Internal Server Error") and I am not sure how do I parse these unescaped " chars inside the message.

I am able to parse \n \t and \r but stuck with \". I can't even directly do replace on the entire file as the element names has to be enclosed with ".

Please suggest how can I parse these types of messages that are having text enclosed with ".

Here is the example JSON string that the notebook received and plan to perse.

#Input String Received from Pipeline
var_str_log_data = "[{\"ApplicationName\": \"APP\",\"WorkspaceId\": \"0addb382-fa0d-4ce1-9c3c-95e32b957955\",\"Environment\": \"DEV\",\"Level\": \"ERROR\",\"Severity\": \"SEV 4\",\"Component\": \"PL_APP_Ingest\",\"Operation\": \"ETL\",\"Run_Id\": \"88276e4f-4e60-47ec-a6d6-dd9e475d2c3a\",\"SessionId\": \"\",\"Message\": \"[DEV] - APP - Pipeline Execution Failed - 20250329\",\"Status\": \"Success\",\"Details\": \"Pipeline Name: PL_APP_Ingest\\nRun Id: 88257e4f-4e60-34ec-a6d6-dd9e475d2c3a\\nError Message: Notebook execution failed at Notebook service with http status code - '200', please check the Run logs on Notebook, additional details - 'Error name - Py4JJavaError, Error value - An error occurred while calling o4628.load.\n: java.util.concurrent.ExecutionException: Operation failed: \"Internal Server Error\", 500, HEAD, http://msit-onelake.dfs.fabric.microsoft.com/MYWORKSPACE/dataverse_name_cds2_workspace_unq60feaf930cbf236eb2519b27f2d6b.Lakehouse/Tables/app_riskparcel/_delta_log?upn=false&action=getStatus&timeout=90\n at com.google.common.util.concurrent.AbstractFuture$Sync.getValue(AbstractFuture.java:306)\n at com.google.common.util.concurrent.AbstractFuture$Sync.get(AbstractFuture.java:293)\n at com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)\n at com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)\n ... 37 more\n' : \\nExecution URL: https://msit.powerbi.com/workloads/data-pipeline/artifacts/workspaces/0addb122-fa0d-4ce1-9c3c-95e25b957955/pipelines/PL_APP_Ingest/88223e4f-4e60-47ec-a6d6-dd9e475d2c3a?experience=power-bi\\nApp Name: APP\",\"CorrelationId\": \"\",\"User\": \"System\"}]"

#Example Code Try to Process the JSON provided a String

var_str_log_data = re.sub(r'(?<!\\)\n', r'\\n', var_str_log_data)
var_str_log_data = re.sub(r'(?<!\\)\t', r'\\t', var_str_log_data)
var_str_log_data = re.sub(r'(?<!\\)\r', r'\\r', var_str_log_data)

print(f"Log Data Format and Content: {type(var_str_log_data)} - {var_str_log_data}.")

try:
data = json.loads(var_str_log_data)
print("JSON loaded successfully")
except json.JSONDecodeError as e:
error_pos = e.pos
print("Invalid JSON:", e)
print(f"Character at position {error_pos}: {repr(var_str_log_data[error_pos-1])}")
start = max(0, error_pos - 20)
end = min(len(var_str_log_data), error_pos + 20)
print("Surrounding text:", repr(var_str_log_data[start:end]))


Thanks, Prabhat

Continue reading...