Python processing nested json - json

I am feeding the following data to my python script as an argument:
source=Blabla||
name=TEST Error in Log - Error: 3345||
item_value={ "Error": 3345, "Message": "Some error occurred", "Status": 1 }
The script:
#!/usr/bin/env python
import sys, json
data=sys.argv[3]
if 'NOTE: Escalation cancelled' in data:
exit(0)
data=data.split('||')
datadictionary = {}
for item in data:
key, val = item.split("=", 1)
k=key.strip()
v=(((((val.strip())).replace('\n',"<br>")).replace("`", "&lsquo")))
datadictionary[ k ] = v
datajson = json.dumps(datadictionary)
datajson=datajson.encode('utf8')
print(datajson)
It gives the following output:
{"source": "Blabla", "name": "TEST Error in Log - Error: 3345", "item_value": "{ \"Error\": 3345, \"Message\": \"Some error occurred\", \"Status\": 1 }"}
Which is not a proper JSON as item_value is a string rather than a JSON object. How can I generate a nested JSON here?

I figured out that I need to first decode the JSON object first by the following line:
datadictionary['item_value']=json.loads(datadictionary['item_value'])
So the script looks like this:
#!/usr/bin/env python
import sys, json
data=sys.argv[3]
if 'NOTE: Escalation cancelled' in data:
exit(0)
data=data.split('||')
datadictionary = {}
for item in data:
key, val = item.split("=", 1)
k=key.strip()
v=val.strip().replace('\n',"<br>").replace("`", "&lsquo")
datadictionary[ k ] = v
datadictionary['item_value']=json.loads(datadictionary['item_value'])
datajson = json.dumps(datadictionary)
datajson=datajson.encode('utf8')
print(datajson)
And the output:
{"source": "Blabla", "name": "TEST Error in Log - Error: 3345", "item_value": {"Status": 1, "Message": "Some error occurred", "Error": 3345}}

Related

Seeding rails project with Json file

I'm at a lost and my searches have gotten me nowhere.
In my seeds.rb file I have the following code
require 'json'
jsonfile = File.open 'db/search_result2.json'
jsondata = JSON.load jsonfile
#jsondata = JSON.parse(jsonfile)
jsondata[].each do |data|
Jobpost.create!(post: data['title'],
link: data['link'],
image: data['pagemap']['cse_image']['src'] )
end
Snippet of the json file looks like this:
{
"kind": "customsearch#result",
"title": "Careers Open Positions - Databricks",
"link": "https://databricks.com/company/careers/open-positions",
"pagemap": {
"cse_image": [
{
"src": "https://databricks.com/wp-content/uploads/2020/08/careeers-new-og-image-sept20.jpg"
}
]
}
},
Fixed jsondata[].each to jasondata.each. Now I'm getting the following error:
TypeError: no implicit conversion of String into Integer
jsondata[] says to call the [] method with no arguments on the object in the jsondata variable. Normally [] would take an index like jsondata[0] to get the first element or a start and length like jsondata[0, 5] to get the first five elements.
You want to call the each method on jsondata, so jsondata.each.
So this is very specific to what you have posted:
require 'json'
file = File.open('path_to_file.json').read
json_data = JSON.parse file
p json_data['kind'] #=> "customsearch#result"
# etc for all the other keys
now maybe the json you posted is just the first element in an array:
[
{}, // where each {} is the json you posted
{},
{},
// etc
]
in which case you will indeed have to iterate:
require 'json'
file = File.open('path_to_file.json').read
json_data = JSON.parse file
json_data.each do |data|
p data['kind'] #=> "customsearch#result"
end

How to send json message body in aws SNS using console

I am doing a hands on where I want to add an SNS trigger to a lambda function which then sends a message to a slack channel. There is a blueprint for this lambda in python and also a template test event which looks like the following
{
"Records": [
{
"EventVersion": "1.0",
"EventSubscriptionArn": "arn:aws:sns:EXAMPLE",
"EventSource": "aws:sns",
"Sns": {
"SignatureVersion": "1",
"Timestamp": "1970-01-01T00:00:00.000Z",
"Signature": "EXAMPLE",
"SigningCertUrl": "EXAMPLE",
"MessageId": "12345",
"Message": {
"AlarmName": "SlackAlarm",
"NewStateValue": "OK",
"NewStateReason": "Threshold Crossed: 1 datapoint (0.0) was not greater than or equal to the threshold (1.0)."
},
"MessageAttributes": {
"Test": {
"Type": "String",
"Value": "TestString"
},
"TestBinary": {
"Type": "Binary",
"Value": "TestBinary"
}
},
"Type": "Notification",
"UnsubscribeUrl": "EXAMPLE",
"TopicArn": "arn:aws:sns:EXAMPLE",
"Subject": "TestInvoke"
}
}
]
The code in lambda handler from the blueprint is as follows
import boto3
import json
import logging
import os
from base64 import b64decode
from urllib.request import Request, urlopen
from urllib.error import URLError, HTTPError
HOOK_URL = os.environ['kmsEncryptedHookUrl']
SLACK_CHANNEL = os.environ['slackChannel']
logger = logging.getLogger()
logger.setLevel(logging.INFO)
def lambda_handler(event, context):
logger.info("Event: " + str(event))
message = event['Records'][0]['Sns']['Message']
logger.info("Message: " + str(message))
alarm_name = message['AlarmName']
new_state = message['NewStateValue']
reason = message['NewStateReason']
slack_message = {
'channel': SLACK_CHANNEL,
'text': "%s state is now %s: %s" % (alarm_name, new_state, reason)
}
req = Request(HOOK_URL, json.dumps(slack_message).encode('utf-8'))
try:
response = urlopen(req)
response.read()
logger.info("Message posted to %s", slack_message['channel'])
except HTTPError as e:
logger.error("Request failed: %d %s", e.code, e.reason)
except URLError as e:
logger.error("Server connection failed: %s", e.reason)
When I run the test event, the lambda runs successfully.
I wanted to publish a message in SNS topic from the console to see if the lambda is triggered correctly. But when I try to publish the JSON object as a message body, I am getting the error
[ERROR] TypeError: string indices must be integersTraceback (most recent call last):  File "/var/task/lambda_function.py", line 21, in lambda_handler    alarm_name = message['AlarmName']
I tried giving plain json
{
"AlarmName": "PublishedAlarm",
"NewStateValue": "OK",
"NewStateReason": "This alarm is published"
}
I tried giving a stringified JSON
"{\"AlarmName\": \"PublishedAlarm\",\"NewStateValue\": \"OK\",\"NewStateReason\": \"This alarm is published\"}"
I tried choosing Custom payload for each delivery message structure and then gave the following message body
{
"default": "Sample fallback message",
"email": "Sample message for email endpoints",
"sqs": "Sample message for Amazon SQS endpoints",
"lambda": "{\"AlarmName\": \"PublishedAlarm\",\"NewStateValue\": \"OK\",\"NewStateReason\": \"This alarm is published\"}",
"http": "Sample message for HTTP endpoints",
"https": "Sample message for HTTPS endpoints",
"sms": "Sample message for SMS endpoints",
"firehose": "Sample message for Amazon Kinesis Data Firehose endpoints",
"APNS": "{\"aps\":{\"alert\": \"Sample message for iOS endpoints\"} }",
"APNS_SANDBOX": "{\"aps\":{\"alert\":\"Sample message for iOS development endpoints\"}}",
"APNS_VOIP": "{\"aps\":{\"alert\":\"Sample message for Apple VoIP endpoints\"}}",
"APNS_VOIP_SANDBOX": "{\"aps\":{\"alert\": \"Sample message for Apple VoIP development endpoints\"} }",
"MACOS": "{\"aps\":{\"alert\":\"Sample message for MacOS endpoints\"}}",
"MACOS_SANDBOX": "{\"aps\":{\"alert\": \"Sample message for MacOS development endpoints\"} }",
"GCM": "{ \"data\": { \"message\": \"Sample message for Android endpoints\" } }",
"ADM": "{ \"data\": { \"message\": \"Sample message for FireOS endpoints\" } }",
"BAIDU": "{\"title\":\"Sample message title\",\"description\":\"Sample message for Baidu endpoints\"}",
"MPNS": "<?xml version=\"1.0\" encoding=\"utf-8\"?><wp:Notification xmlns:wp=\"WPNotification\"><wp:Tile><wp:Count>ENTER COUNT</wp:Count><wp:Title>Sample message for Windows Phone 7+ endpoints</wp:Title></wp:Tile></wp:Notification>",
"WNS": "<badge version=\"1\" value=\"42\"/>"
}
Nothing worked. I've also subscribed an email address to the topic and I'm getting emails without any issues.
How can I simulate the test event given in lambda event templates from the SNS?
When you send your plain json message using SNS, it will be delivered to lambda in in the format:
'Message': '{\n "AlarmName": "PublishedAlarm",\n "NewStateValue": "OK",\n "NewStateReason": "This alarm is published"\n}'
You can parse it using ast' literal_eval method:
import ast
#...
#...
def lambda_handler(event, context):
logger.info("Event: " + str(event))
message = event['Records'][0]['Sns']['Message']
logger.info("Message: " + str(message))
message = ast.literal_eval(event['Records'][0]['Sns']['Message'])
alarm_name = message['AlarmName']
new_state = message['NewStateValue']
reason = message['NewStateReason']
#...
#...

Databricks - explode JSON from SQL column with PySpark

New to Databricks. Have a SQL database table that I am creating a dataframe from. One of the columns is a JSON string. I need to explode the nested JSON into multiple columns. Have used this post and this post to get me to where I am at now.
Example JSON:
{
"Module": {
"PCBA Serial Number": "G7456789",
"Manufacturing Designator": "DISNEY",
"Firmware Version": "0.0.0",
"Hardware Revision": "46858",
"Manufacturing Date": "10/17/2018 4:04:25 PM",
"Test Result": "Fail",
"Test Start Time": "10/22/2018 6:14:14 AM",
"Test End Time": "10/22/2018 6:16:11 AM"
}
Code so far:
#define schema
schema = StructType(
[
StructField('Module',ArrayType(StructType(Seq
StructField('PCBA Serial Number',StringType,True),
StructField('Manufacturing Designator',StringType,True),
StructField('Firmware Version',StringType,True),
StructField('Hardware Revision',StringType,True),
StructField('Test Result',StringType,True),
StructField('Test Start Time',StringType,True),
StructField('Test End Time',StringType,True))), True) ,True),
StructField('Test Results',StringType(),True),
StructField('HVM Code Errors',StringType(),True)
]
#use from_json to explode json by applying it to column
df.withColumn("ActivityName", from_json("ActivityName", schema))\
.select(col('ActivityName'))\
.show()
Error:
SyntaxError: invalid syntax
File "<command-1632344621139040>", line 10
StructField('PCBA Serial Number',StringType,True),
^
SyntaxError: invalid syntax
As you are using pyspark then types should be StringType() instead of StringType and remove Seq replace it with []
schema = StructType([StructField('Module',ArrayType(StructType([
StructField('PCBA Serial Number',StringType(),True),
StructField('Manufacturing Designator',StringType(),True),
StructField('Firmware Version',StringType(),True),
StructField('Hardware Revision',StringType(),True),
StructField('Test Result',StringType(),True),
StructField('Test Start Time',StringType(),True),
StructField('Test End Time',StringType(),True)])), True),
StructField('Test Results',StringType(),True),
StructField('HVM Code Errors',StringType(),True)])

Can JSON String format be converted to Actual format using groovy?

I have the following JSON String format getting from external source:-
What kind of format is this actually?
{
id=102,
brand=Disha,
book=[{
slr=EFTR,
description=Grammer,
data=TYR,
rate=true,
numberOfPages=345,
maxAllowed=12,
currentPage=345
},
{
slr=EFRE,
description=English,
data=TYR,
rate=true,
numberOfPages=345,
maxAllowed=12,
currentPage=345
}]
}
I want to convert this into actual JSON format like this: -
{
"id": "102",
"brand": "Disha",
"book": [{
"slr": "EFTR",
"description": "Grammer",
"data": "TYR",
"rate": true,
"numberOfPages": 345,
"maxAllowed": "12",
"currentPage": 345
},
{
"slr": "EFRE",
"description": "English",
"data": "TYR",
"rate": true,
"numberOfPages": 345,
"maxAllowed": "12",
"currentPage": 345
}]
}
Is this achievable using groovy command or code?
Couple of things:
You do not need Groovy Script test step which is currently there as step3
For step2, Add a 'Script Assertion` with given below script
Provide step name for nextStepName in the script below for which you want to add the request.
//Provide the test step name where you want to add the request
def nextStepName = 'step4'
def setRequestToStep = { stepName, requestContent ->
context.testCase.testSteps[stepName]?.httpRequest.requestContent = requestContent
}
//Check the response
assert context.response, 'Response is empty or null'
setRequestToStep(nextStepName, context.response)
EDIT: Based on the discussion with OP on the chat, OP want to update existing request of step4 for a key and its value as step2's response.
Using samples to demonstrate the change input and desired outputs.
Let us say, step2's response is:
{
"world": "test1"
}
And step4's existing request is :
{
"key" : "value",
"key2" : "value2"
}
Now, OP wants to update value of key with first response in ste4's request, and desired is :
{
"key": {
"world": "test1"
},
"key2": "value2"
}
Here is the updated script, use it in Script Assertion for step 2:
//Change the key name if required; the step2 response is updated for this key of step4
def keyName = 'key'
//Change the name of test step to expected to be updated with new request
def nextStepName = 'step4'
//Check response
assert context.response, 'Response is null or empty'
def getJson = { str ->
new groovy.json.JsonSlurper().parseText(str)
}
def getStringRequest = { json ->
new groovy.json.JsonBuilder(json).toPrettyString()
}
def setRequestToStep = { stepName, requestContent, key ->
def currentRequest = context.testCase.testSteps[stepName]?.httpRequest.requestContent
log.info "Existing request of step ${stepName} is ${currentRequest}"
def currentReqJson = getJson(currentRequest)
currentReqJson."$key" = getJson(requestContent)
context.testCase.testSteps[stepName]?.httpRequest.requestContent = getStringRequest(currentReqJson)
log.info "Updated request of step ${stepName} is ${getStringRequest(currentReqJson)}"
}
setRequestToStep(nextStepName, context.request, keyName)
We can convert the invalid JSON format to valid JSON format using this line of code:-
def validJSONString = JsonOutput.toJson(invalidJSONString).toString()

i got empty value when print json key in lua code

this is the json response plus "\x00" in the end from server :
{
"STATUS": [{
"STATUS":"S",
"When":1470180059,
"Code":11,
"Msg":"Summary",
"Description":"nsgminer 0.9.2"
}],"SUMMARY": [{
"Elapsed":2061,
"MHS av":0.00,
"Found Blocks":0,
"Getworks":76,
"Accepted":0,
"Rejected":0,
"Hardware Errors":0,
"Utility":0.00,
"Discarded":209,
"Stale":0,
"Get Failures":3,
"Local Work":293,
"Remote Failures":0,
"Network Blocks":14,
"Total MH":0.0000,
"Work Utility":0.00,
"Difficulty Accepted":0.00000000,
"Difficulty Rejected":0.00000000,
"Difficulty Stale":0.00000000,
"Best Share":0
}],
"id":1
}\x00
i want to use the json in lua code :
local output = stdnse.output_table()
local json_string = tostring(result:sub(1, -2))
local pos, value = json.parse(json_string)
output["Description"] = value["STATUS"][0]["Description"]
return output
when i print it out, i got null value
i solve that with covert json to string and convert string into json table
local pos, value = json.parse(tostring(json_string))
output["Description"] = value["STATUS"][1]["Description"]