Parsing json string with escaped quotes in spark scala [duplicate] - json

Question

This question already has answers here:
Parse a String in Scala [closed]
(2 answers)
Closed 1 year ago.
I am trying to parse the string given below as a JSON using Scala, but haven't been able to do so because of the escaped quotes \" occurring in fields.
{\"TimeCreated\":\"2021-01-09T04:29:21.413Z\",\"Computer\":\"WIN-10.atfdetonate.local\",\"Channel\":\"Security\",\"Provider\":\"Microsoft-Windows-Security-Auditing\",\"Category\":\"AUDIT_SUCCESS\",\"Version\":\"1\",\"EventID\":\"4698\",\"EventRecordID\":\"12956650\",\"SubjectUserSid\":\"S-1-5-18\",\"SubjectUserName\":\"WIN-10$\",\"SubjectDomainName\":\"ATFDETONATE\",\"SubjectLogonId\":\"0x3e7\",\"TaskName\":\"\\Microsoft\\Windows\\UpdateOrchestrator\\Universal Orchestrator Start\",\"TaskContent\":\"<?xml version=\"1.0\" encoding=\"UTF-16\"?>\r <Task version=\"1.2\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\r <RegistrationInfo>\r <URI>\\Microsoft\\Windows\\UpdateOrchestrator\\Universal Orchestrator Start</URI>\r <SecurityDescriptor>D:P(A;;FA;;;SY)(A;;FRFX;;;LS)(A;;FRFX;;;BA)</SecurityDescriptor>\r </RegistrationInfo>\r <Triggers>\r <TimeTrigger>\r <StartBoundary>2021-01-09T11:42:00.000Z</StartBoundary>\r <Enabled>true</Enabled>\r </TimeTrigger>\r </Triggers>\r <Settings>\r <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\r <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>\r <StopIfGoingOnBatteries>false</StopIfGoingOnBatteries>\r <AllowHardTerminate>true</AllowHardTerminate>\r <StartWhenAvailable>false</StartWhenAvailable>\r <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>\r <IdleSettings>\r <Duration>PT10M</Duration>\r <WaitTimeout>PT1H</WaitTimeout>\r <StopOnIdleEnd>true</StopOnIdleEnd>\r <RestartOnIdle>false</RestartOnIdle>\r </IdleSettings>\r <AllowStartOnDemand>true</AllowStartOnDemand>\r <Enabled>true</Enabled>\r <Hidden>false</Hidden>\r <RunOnlyIfIdle>false</RunOnlyIfIdle>\r <WakeToRun>false</WakeToRun>\r <ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\r <Priority>7</Priority>\r </Settings>\r <Actions Context=\"Author\">\r <Exec>\r <Command>%systemroot%\\system32\\usoclient.exe</Command>\r <Arguments>StartUWork</Arguments>\r </Exec>\r </Actions>\r <Principals>\r <Principal id=\"Author\">\r <UserId>S-1-5-18</UserId>\r <RunLevel>LeastPrivilege</RunLevel>\r </Principal>\r </Principals>\r </Task>\"}
So far, I have tried spark.json.read and the net.liftweb library, but to no avail.
Any kind of help is highly appreciated.

Answer 1

The json output that you are getting might not be a valid json or if the json is valid then it has xml content in the TaskContent element which has xml tags with attributes and I think that is what is causing the issue. The idea I have is to remove the Double quotes from the XML attributes values and then parse. You could replace that double quotes with any specific value and once you have that `TaskContent' as dataframe column than you can again replace that to get the original content.
This might not be the perfect or efficient answer but based on how you are getting the json and if that json structure remains the same then you could do something as below :
Convert the json that you have to string.
Do some replaceAll operations on the string to make it look like valid json.
read the json into Dataframe.
//Source data copied from Question
val json = """{\"TimeCreated\":\"2021-01-09T04:29:21.413Z\",\"Computer\":\"WIN-10.atfdetonate.local\",\"Channel\":\"Security\",\"Provider\":\"Microsoft-Windows-Security-Auditing\",\"Category\":\"AUDIT_SUCCESS\",\"Version\":\"1\",\"EventID\":\"4698\",\"EventRecordID\":\"12956650\",\"SubjectUserSid\":\"S-1-5-18\",\"SubjectUserName\":\"WIN-10$\",\"SubjectDomainName\":\"ATFDETONATE\",\"SubjectLogonId\":\"0x3e7\",\"TaskName\":\"\\Microsoft\\Windows\\UpdateOrchestrator\\Universal Orchestrator Start\",\"TaskContent\":\"<?xml version=\"1.0\" encoding=\"UTF-16\"?>\r <Task version=\"1.2\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\r <RegistrationInfo>\r <URI>\\Microsoft\\Windows\\UpdateOrchestrator\\Universal Orchestrator Start</URI>\r <SecurityDescriptor>D:P(A;;FA;;;SY)(A;;FRFX;;;LS)(A;;FRFX;;;BA)</SecurityDescriptor>\r </RegistrationInfo>\r <Triggers>\r <TimeTrigger>\r <StartBoundary>2021-01-09T11:42:00.000Z</StartBoundary>\r <Enabled>true</Enabled>\r </TimeTrigger>\r </Triggers>\r <Settings>\r <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\r <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>\r <StopIfGoingOnBatteries>false</StopIfGoingOnBatteries>\r <AllowHardTerminate>true</AllowHardTerminate>\r <StartWhenAvailable>false</StartWhenAvailable>\r <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>\r <IdleSettings>\r <Duration>PT10M</Duration>\r <WaitTimeout>PT1H</WaitTimeout>\r <StopOnIdleEnd>true</StopOnIdleEnd>\r <RestartOnIdle>false</RestartOnIdle>\r </IdleSettings>\r <AllowStartOnDemand>true</AllowStartOnDemand>\r <Enabled>true</Enabled>\r <Hidden>false</Hidden>\r <RunOnlyIfIdle>false</RunOnlyIfIdle>\r <WakeToRun>false</WakeToRun>\r <ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\r <Priority>7</Priority>\r </Settings>\r <Actions Context=\"Author\">\r <Exec>\r <Command>%systemroot%\\system32\\usoclient.exe</Command>\r <Arguments>StartUWork</Arguments>\r </Exec>\r </Actions>\r <Principals>\r <Principal id=\"Author\">\r <UserId>S-1-5-18</UserId>\r <RunLevel>LeastPrivilege</RunLevel>\r </Principal>\r </Principals>\r </Task>\"}"""
//Modifying json to make it valid
val modifiedJson = json.replaceAll("\\\\\\\\","#").replaceAll("\\\\r","").replaceAll("\\\\","").replaceAll(" ","").replaceAll(" ","").replaceAll("> <","><").replaceAll("=\"","=").replaceAll("\">",">").replaceAll("#","\\\\\\\\").replaceAll("1.0\"","1.0").replaceAll("UTF-16\"?","UTF-16").replaceAll("1.2\"","1.2")
//creating a dataset out of json String
val ds = spark.createDataset(modifiedJson :: Nil)
//reading the dataset as json
val df = spark.read.json(ds)
you can see the output as below:
You can do some optimization for it to work in a more efficient way but this is how I made it work.

Answer 2

You can use replace escaped quotes \" in the json with " (except for those inside the xml content) using regexp_replace function then read into dataframe :
val jsonString = """
{\"TimeCreated\":\"2021-01-09T04:29:21.413Z\",\"Computer\":\"WIN-10.atfdetonate.local\",\"Channel\":\"Security\",\"Provider\":\"Microsoft-Windows-Security-Auditing\",\"Category\":\"AUDIT_SUCCESS\",\"Version\":\"1\",\"EventID\":\"4698\",\"EventRecordID\":\"12956650\",\"SubjectUserSid\":\"S-1-5-18\",\"SubjectUserName\":\"WIN-10$\",\"SubjectDomainName\":\"ATFDETONATE\",\"SubjectLogonId\":\"0x3e7\",\"TaskName\":\"\\Microsoft\\Windows\\UpdateOrchestrator\\Universal Orchestrator Start\",\"TaskContent\":\"<?xml version=\"1.0\" encoding=\"UTF-16\"?>\r <Task version=\"1.2\" xmlns=\"http://schemas.microsoft.com/windows/2004/02/mit/task\">\r <RegistrationInfo>\r <URI>\\Microsoft\\Windows\\UpdateOrchestrator\\Universal Orchestrator Start</URI>\r <SecurityDescriptor>D:P(A;;FA;;;SY)(A;;FRFX;;;LS)(A;;FRFX;;;BA)</SecurityDescriptor>\r </RegistrationInfo>\r <Triggers>\r <TimeTrigger>\r <StartBoundary>2021-01-09T11:42:00.000Z</StartBoundary>\r <Enabled>true</Enabled>\r </TimeTrigger>\r </Triggers>\r <Settings>\r <MultipleInstancesPolicy>IgnoreNew</MultipleInstancesPolicy>\r <DisallowStartIfOnBatteries>true</DisallowStartIfOnBatteries>\r <StopIfGoingOnBatteries>false</StopIfGoingOnBatteries>\r <AllowHardTerminate>true</AllowHardTerminate>\r <StartWhenAvailable>false</StartWhenAvailable>\r <RunOnlyIfNetworkAvailable>false</RunOnlyIfNetworkAvailable>\r <IdleSettings>\r <Duration>PT10M</Duration>\r <WaitTimeout>PT1H</WaitTimeout>\r <StopOnIdleEnd>true</StopOnIdleEnd>\r <RestartOnIdle>false</RestartOnIdle>\r </IdleSettings>\r <AllowStartOnDemand>true</AllowStartOnDemand>\r <Enabled>true</Enabled>\r <Hidden>false</Hidden>\r <RunOnlyIfIdle>false</RunOnlyIfIdle>\r <WakeToRun>false</WakeToRun>\r <ExecutionTimeLimit>PT72H</ExecutionTimeLimit>\r <Priority>7</Priority>\r </Settings>\r <Actions Context=\"Author\">\r <Exec>\r <Command>%systemroot%\\system32\\usoclient.exe</Command>\r <Arguments>StartUWork</Arguments>\r </Exec>\r </Actions>\r <Principals>\r <Principal id=\"Author\">\r <UserId>S-1-5-18</UserId>\r <RunLevel>LeastPrivilege</RunLevel>\r </Principal>\r </Principals>\r </Task>\"}
"""
val df = spark.read.json(
Seq(jsonString).toDS
.withColumn("value", regexp_replace($"value", """([:\[,{]\s*)\\"(.*?)\\"(?=\s*[:,\]}])""", "$1\"$2\""))
.as[String]
)
df.show
//+-------------+--------+--------------------+-------+-------------+--------------------+-----------------+--------------+---------------+--------------+--------------------+--------------------+--------------------+-------+
//| Category| Channel| Computer|EventID|EventRecordID| Provider|SubjectDomainName|SubjectLogonId|SubjectUserName|SubjectUserSid| TaskContent| TaskName| TimeCreated|Version|
//+-------------+--------+--------------------+-------+-------------+--------------------+-----------------+--------------+---------------+--------------+--------------------+--------------------+--------------------+-------+
//|AUDIT_SUCCESS|Security|WIN-10.atfdetonat...| 4698| 12956650|Microsoft-Windows...| ATFDETONATE| 0x3e7| WIN-10$| S-1-5-18|<?xml version="1....|\Microsoft\Window...|2021-01-09T04:29:...| 1|
//+-------------+--------+--------------------+-------+-------------+--------------------+-----------------+--------------+---------------+--------------+--------------------+--------------------+--------------------+-------+