How to parse JSON data using Lua patterns? - json

I am integrating lua-module in nginx. I want to check parameter's value is empty or not. Whenever I use following code, I get true result against the JSON request
Request
{
"data": {
"user": {
"username": "Ethen",
"type": "PDF"
}
},
"passport": {
"user": "001"
},
}
Code
local arg = ngx.req.get_body_data();
L="return "..arg:gsub('("[^"]-"):','[%1]=')
T=loadstring(L)()
ngx.print(T.data.user.username)
if T.data.user.username == "" then
ngx.say("Empty username");
end
But when I use this request, I got error (attempt to call a nil value stack traceback)
Request
{
"reference" : "567456314",
"callback_url" : "http://www.example.com/",
"verification_mode" : "any",
"document" : {
"proof" : "data:image/png;base64,iVBOR=“,
"additional_proof": "data:image/png;base64,iVBORw0=",
"supported_types" : ["id_card","driving_license","passport"],
"expiry_date" : "",
"document_number" : ""
},
"address" : {
"proof" : "data:image/png;base64,iVBORw0KG=",
"supported_types" : ["id_card","bank_statement"],
"name" : "",
"issue_date":""
}
}
Code
local arg = ngx.req.get_body_data();
L="return "..arg:gsub('("[^"]-"):','[%1]=')
T=loadstring(L)()
ngx.print(T.document.proof)
if T.document.proof == "" then
ngx.say("Empty proof");
end
What's the problem and solution??....Thanks in advance!!

Related

Index a JSON file into elasticsearch command/mapping errors

I'm new to ELK and I want to import a JSON file into Elasticsearch. this is my file:
{
"news":{
"1":{
"_score":1.0,
"_index":"newsvit",
"_source":{
"content":" \u0641\u0647\u06cc\u0645\u0647 \u062d\u0633\u0646\u200c\u0645\u06cc\u0631\u06cc: \u0627\u06af\u0631\u0686\u0647 \u062f\u0631 \u0647\u06cc\u0627\u0647\u0648\u06cc \u0627\u0646\u062a\u062e\u0627\u0628\u0627\u062a \u0631\u06cc\u0627\u0633\u062a \u062c\u0645\u0647\u0648\u0631\u06cc\u060c \u0645\u0648\u0636\u0648\u0639\u06cc \u0645\u0627\u0646\u0646\u062f \u0645\u0639\u0631\u0641\u06cc \u06a9\u0627\u0646\u062f\u06cc\u062f\u0627\u0647\u0627\u06cc \u0634\u0648\u0631\u0627\u06cc \u0634\u0647\u0631 \u062f\u0631 \u062d\u0627\u0634\u06cc\u0647 \u0642\u0631\u0627\u0631 \u06af\u0631\u0641\u062a\u0647\u060c \u0627\u0645\u0627 \u0627\u0645\u0633\u0627\u0644 \u0628\u0647 \u0639\u0646\u0648\u0627\u0646 \u067e\u0646\u062c\u0645\u06cc\u0646 \u062f\u0648\u0631\u0647 \u0627\u0646\u062a\u062e\u0627\u0628 \u0627\u0639\u0636\u0627\u06cc \u0634\u0648\u0631\u0627\u06cc \u0634\u0647\u0631\u060c \u0627\u06cc\u0646 \u0631\u0648\u06cc\u062f\u0627\u062f \u0628\u0647 \u0646\u0633\u0628\u062a \u062f\u0648\u0631\u0647\u200c\u0647\u0627\u06cc \u0642\u0628\u0644\u060c \u0628\u06cc\u0634\u062a\u0631 \u0645\u0648\u0631\u062f \u062a\u0648\u062c\u0647 \u0648\u0627\u0642\u0639 \u0634\u062f\u0647. \u0627\u06cc\u0646 \u0627\u0642\u0628\u0627\u0644\u060c \u0686\u0647 \u0627\u0632 \u0633\u0648\u06cc \u0686\u0647\u0631\u0647\u200c\u0647\u0627\u06cc \u0645\u0637\u0631\u062d \u0628\u0631\u0627\u06cc \u062b\u0628\u062a \u0646\u0627\u0645 \u0648 \u0686\u0647 \u0627\u0632 \u0633\u0648\u06cc \u0645\u0631\u062f\u0645 \u0628\u0631\u0627\u06cc \u0645\u0634\u0627\u0631\u06a9\u062a \u062f\u0631 \u0627\u06cc\u0646 \u0631\u0648\u06cc\u062f\u0627\u062f\u060c \u0639\u0644\u062a\u200c\u0647\u0627\u06cc \u06af\u0648\u0646\u0627\u06af\u0648\u0646\u06cc \u0645\u06cc\u200c\u062a\u0648\u0627\u0646\u062f \u062f\u0627\u0634\u062a\u0647 \u0628\u0627\u0634\u062f \u06a9\u0647 \u062a\u0648\u062c\u0647 \u0628\u0647 \u0622\u0646\u060c \u0645\u06cc\u200c\u062a\u0648\u0627\u0646\u062f \u0631\u0627\u0647\u06af\u0634\u0627\u06cc \u0627\u0639\u0636\u0627\u06cc \u0631\u06",
"lead":"\u062c\u0627\u0645\u0639\u0647 > \u0634\u0647\u0631\u06cc - \u0645\u06cc\u0632\u06af\u0631\u062f\u06cc \u062f\u0631\u0628\u0627\u0631\u0647 \u0639\u0645\u0644\u06a9\u0631\u062f \u062f\u0648\u0631\u0647\u200c\u0647\u0627\u06cc \u06af\u0630\u0634\u062a\u0647 \u0634\u0648\u0631\u0627\u06cc \u0634\u0647\u0631\u060c \u0622\u0646\u0686\u0647 \u0627\u0639\u0636\u0627\u06cc \u062c\u062f\u06cc\u062f \u0628\u0627\u06cc\u062f \u0645\u062f \u0646\u0638\u0631 \u062f\u0627\u0634\u062a\u0647 \u0628\u0627\u0634\u0646\u062f \u0648 \u0647\u0645\u0686\u0646\u06cc\u0646 \u0645\u0627\u0647\u06cc\u062a \u0633\u06cc\u0627\u0633\u06cc \u0628\u0648\u062f\u0646 \u06cc\u0627 \u0646\u0628\u0648\u062f\u0646 \u0634\u0648\u0631\u0627\u06cc \u0634\u0647\u0631.",
"agency":"13",
"date_created":1494518193,
"url":"http://www.khabaronline.ir/(X(1)S(bud4wg3ebzbxv51mj45iwjtp))/detail/663749/society/urban",
"image":"uploads/2017/05/11/1589793661.jpg",
"category":"15"
},
"_type":"news",
"_id":"2981643"
},
"2": {
...
based on what I have learnt, at first, I tried to create a mapping system for it in DevTools of Kibana. I want to be able to perform queries and search on this file based on fields in _source, such as category, id and so on. this is my mapping:
PUT /main-news-test-data
{
"mappings": {
"properties": {
"_score": {"type":"integer"},
"_index": {"type":"keyword"},
"_type":{"type":"keyword"},
"_id":{"type":"keyword"}
},
"_source":{
"properties": {
"content":{"type":"text"},
"title":{"type":"text"},
"lead":{"type":"text"},
"agency":{"type":"keyword"},
"date_created":{"type":"date"},
"url":{"type":"keyword"},
"image":{"type":"keyword"},
"category":{"type":"keyword"}
}
}
}
}
HEAD main-news-test-data
GET /main-news-test-data/_search?q=*
but when I run this in Devtools I receive this error:
{
"error" : {
"root_cause" : [
{
"type" : "mapper_parsing_exception",
"reason" : "Mapping definition for [_source] has unsupported parameters: [properties : {image={type=keyword}, agency={type=keyword}, date_created={type=date}, title={type=text}, category={type=keyword}, content={type=text}, lead={type=text}, url={type=keyword}}]"
}
],
"type" : "mapper_parsing_exception",
"reason" : "Failed to parse mapping [_doc]: Mapping definition for [_source] has unsupported parameters: [properties : {image={type=keyword}, agency={type=keyword}, date_created={type=date}, title={type=text}, category={type=keyword}, content={type=text}, lead={type=text}, url={type=keyword}}]",
"caused_by" : {
"type" : "mapper_parsing_exception",
"reason" : "Mapping definition for [_source] has unsupported parameters: [properties : {image={type=keyword}, agency={type=keyword}, date_created={type=date}, title={type=text}, category={type=keyword}, content={type=text}, lead={type=text}, url={type=keyword}}]"
}
},
"status" : 400
}
I also tried to index my file into elasticsearch using this PowerShell command afterwards:
Invoke-RestMethod "http://localhost:9200/main-news-test-data/doc/_bulk?pretty" -Method Post -ContentType 'application/x-ndjson' -InFile "test.json"
but again I get this error from Powershell:
Invoke-RestMethod : {
"error" : {
"root_cause" : [
{
"type" : "json_e_o_f_exception",
"reason" : "Unexpected end-of-input: expected close marker for Object (start marker at [Source:
(org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 1])\n at
[Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 2, column: 1]"
}
],
"type" : "json_e_o_f_exception",
"reason" : "Unexpected end-of-input: expected close marker for Object (start marker at [Source:
(org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 1, column: 1])\n at
[Source: (org.elasticsearch.common.bytes.AbstractBytesReference$MarkSupportingStreamInputWrapper); line: 2, column: 1]"
},
"status" : 400
}
So what should I do? how do I import a JSON file into elasticsearch that is queryable by fields?
with what I read I can say that :
your mapping is strange
Just put :
PUT /main-news-test-data
{
"mappings": {
"properties": {
"content": {
"type": "text"
},
"title": {
"type": "text"
},
"lead": {
"type": "text"
},
"agency": {
"type": "keyword"
},
"date_created": {
"type": "date"
},
"url": {
"type": "keyword"
},
"image": {
"type": "keyword"
},
"category": {
"type": "keyword"
}
}
}
}
Your Json is wrong. A bulk don't use a valid json.
A file for the _bulk api will look like this :
{ "index" : { "_index" : "main-news-test-data", "_id" : "1" } }
{ "field1" : "value1" }
{ "index" : { "_index" : "main-news-test-data", "_id" : "2" } }
{ "field1" : "value2" }
Please also Note that "_score":1.0, has no reason to be in your request and that _type is deprecated (if you use a 7.0+, _type can only be _doc and should be ignored)

How to print this specific JSON in a Mat Table?

I have this specific JSON file I'm not allowed to change, which looks like this :
{
"computers" : {
"John" : {
"version" : "1.42.0",
"environment" : "Default",
"platform" : "x64",
"admin" : "true"
},
"Peter" : {
"version" : "1.43.6",
"environment" : "Default",
"platform" : "x64",
"admin" : "true"
},
"Eric" : {
"version" : "1.43.6",
"environment" : "Default",
"platform" : "x64",
"admin" : "false"
}
}
I use the JSON.parse() method to parse the file and I put it into a MatTableDataSource.
Problem is, when I need to display it in my MatTable, I can't really access it the way I want.
I have a column where I want to display all version parameters, so for this I can't say : this.dataSource.computers.????.version
Do you guys see where I'm getting at ? Do you have any idea of what I can do differently in order to solve this ?
Looking forward to reading you.
The Angular mat-table requires the input data to be in an array. First, we use Object.keys() to extract the keys, which will contain the list of names. Then, we can use Object.values() to other values within each key in array format. This is followed by mapping the above array objects with the name property from the list of names.
const data = {
"computers": {
"John": {
"version": "1.42.0",
"environment": "Default",
"platform": "x64",
"admin": "true"
},
"Peter": {
"version": "1.43.6",
"environment": "Default",
"platform": "x64",
"admin": "true"
},
"Eric": {
"version": "1.43.6",
"environment": "Default",
"platform": "x64",
"admin": "false"
}
}
};
const nameList = Object.keys(data.computers);
const dataList = Object.values(data.computers).map((obj, index) => {
obj['name'] = nameList[index];
return obj;
});
console.log(dataList);

Sub-records in Avro with Morphlines

I'm trying to convert JSON into Avro using the kite-sdk morphline module. After playing around I'm able to convert the JSON into Avro using a simple schema (no complex data types).
Then I took it one step further and modified the Avro schema as displayed below (subrec.avsc). As you can see the schema consist of a subrecord.
As soon as I tried to convert the JSON to Avro using the morphlines.conf and the subrec.avsc it failed.
Somehow the JSON paths "/record_type[]/alert/action" are not translated by the toAvro function.
The morphlines.conf
morphlines : [
{
id : morphline1
importCommands : ["org.kitesdk.**"]
commands : [
# Read the JSON blob
{ readJson: {} }
{ logError { format : "record: {}", args : ["#{}"] } }
# Extract JSON
{ extractJsonPaths { flatten: false, paths: {
"/record_type[]/alert/action" : /alert/action,
"/record_type[]/alert/signature_id" : /alert/signature_id,
"/record_type[]/alert/signature" : /alert/signature,
"/record_type[]/alert/category" : /alert/category,
"/record_type[]/alert/severity" : /alert/severity
} } }
{ logError { format : "EXTRACTED THIS : {}", args : ["#{}"] } }
{ extractJsonPaths { flatten: false, paths: {
timestamp : /timestamp,
event_type : /event_type,
source_ip : /src_ip,
source_port : /src_port,
destination_ip : /dest_ip,
destination_port : /dest_port,
protocol : /proto,
} } }
# Create Avro according to schema
{ logError { format : "WE GO TO AVRO"} }
{ toAvro { schemaFile : /etc/flume/conf/conf.empty/subrec.avsc } }
# Create Avro container
{ logError { format : "WE GO TO BINARY"} }
{ writeAvroToByteArray { format: containerlessBinary } }
{ logError { format : "DONE!!!"} }
]
}
]
And the subrec.avsc
{
"type" : "record",
"name" : "Event",
"fields" : [ {
"name" : "timestamp",
"type" : "string"
}, {
"name" : "event_type",
"type" : "string"
}, {
"name" : "source_ip",
"type" : "string"
}, {
"name" : "source_port",
"type" : "int"
}, {
"name" : "destination_ip",
"type" : "string"
}, {
"name" : "destination_port",
"type" : "int"
}, {
"name" : "protocol",
"type" : "string"
}, {
"name": "record_type",
"type" : ["null", {
"name" : "alert",
"type" : "record",
"fields" : [ {
"name" : "action",
"type" : "string"
}, {
"name" : "signature_id",
"type" : "int"
}, {
"name" : "signature",
"type" : "string"
}, {
"name" : "category",
"type" : "string"
}, {
"name" : "severity",
"type" : "int"
}
] } ]
} ]
}
The output on { logError { format : "EXTRACTED THIS : {}", args : ["#{}"] } } I output the following:
[{
/record_type[]/alert / action = [allowed],
/record_type[]/alert / category = [],
/record_type[]/alert / severity = [3],
/record_type[]/alert / signature = [GeoIP from NL,
Netherlands],
/record_type[]/alert / signature_id = [88006],
_attachment_body = [{
"timestamp": "2015-03-23T07:42:01.303046",
"event_type": "alert",
"src_ip": "1.1.1.1",
"src_port": 18192,
"dest_ip": "46.231.41.166",
"dest_port": 62004,
"proto": "TCP",
"alert": {
"action": "allowed",
"gid": "1",
"signature_id": "88006",
"rev": "1",
"signature" : "GeoIP from NL, Netherlands ",
"category" : ""
"severity" : "3"
}
}],
_attachment_mimetype=[json/java + memory],
basename = [simple_eve.json]
}]
UPDATE 2017-06-22
you MUST populate the data in the structure in order for this to work, by using addValues or setValues
{
addValues {
micDefaultHeader : [
{
eventTimestampString : "2017-06-22 18:18:36"
}
]
}
}
after debugging the sources of morphline toAvro, it appears that the record is the first object to be evaluated, no matter what you put in your mappings structure.
the solution is quite simple, but unfortunately took a little extra time, eclipse, running the flume agent in debug mode, cloning the source code and lots of coffee.
here it goes.
my schema:
{
"type" : "record",
"name" : "co_lowbalance_event",
"namespace" : "co.tigo.billing.cboss.lowBalance",
"fields" : [ {
"name" : "dummyValue",
"type" : "string",
"default" : "dummy"
}, {
"name" : "micDefaultHeader",
"type" : {
"type" : "record",
"name" : "mic_default_header_v_1_0",
"namespace" : "com.millicom.schemas.root.struct",
"doc" : "standard millicom header definition",
"fields" : [ {
"name" : "eventTimestampString",
"type" : "string",
"default" : "12345678910"
} ]
}
} ]
}
morphlines file:
morphlines : [
{
id : convertJsonToAvro
importCommands : ["org.kitesdk.**"]
commands : [
{
readJson {
outputClass : java.util.Map
}
}
{
addValues {
micDefaultHeader : [{}]
}
}
{
logDebug { format : "my record: {}", args : ["#{}"] }
}
{
toAvro {
schemaFile : /home/asarubbi/Development/test/co_lowbalance_event.avsc
mappings : {
"micDefaultHeader" : micDefaultHeader
"micDefaultHeader/eventTimestampString" : eventTimestampString
}
}
}
{
writeAvroToByteArray {
format : containerlessJSON
codec : null
}
}
]
}
]
the magic lies here:
{
addValues {
micDefaultHeader : [{}]
}
}
and in the mappings:
mappings : {
"micDefaultHeader" : micDefaultHeader
"micDefaultHeader/eventTimestampString" : eventTimestampString
}
explanation:
inside the code the first field name that is evaluated is micDefaultHeader of type RECORD. as there's no way to specify a default value for a RECORD (logically correct), the toAvro code evaluates this, does not get any value configured in mappings and therefore it fails at it detects (wrongly) that the record is empty when it shouldn't.
however, taking a look at the code, you may see that it requires a Map object, containing no values to please the parser and continue to the next element.
so we add a map object using the addValues and fill it with an empty map [{}]. notice that this must match the name of the record that is causing you an empty value. in my case "micDefaultHeader"
feel free to comment if you have a better solution, as this looks like a "dirty fix"

Matching data in JsonPath with wiremock

I'm trying to create mocks for my login procedure. I use POST method with a couple of fields and login object (with login, password, etc.)
For that I'm using JsonPath. Code below:
{
"request": {
"method": "POST",
"url": "/login",
"bodyPatterns" : [
{"matchesJsonPath" : "$.method"},
{"matchesJsonPath" : "$.params[?(#.clientVersion == "1")]"},
{"matchesJsonPath" : "$.params.login"},
{"matchesJsonPath" : "$.params.password"}
]
},
"response": {
"status": 200,
"bodyFileName": "login.json"
}
}
I'm checking the clientVersion because it's similar to the examples.
My problem is, that with te given POST JSON:
{
"method": "login",
"params": {
"clientVersion": "1",
"login": "test#test.com",
"password": "681819535da188b6ef2"
}
}
I receive 404.
However, when I change
{"matchesJsonPath" : "$.params[?(#.clientVersion == "1")]"},
to normal
{"matchesJsonPath" : "$.params.clientVersion"},
everything works just fine.
So - how to check inside wiremock, using matchesJsonPath if given field equals some value?
How to apply it to the root field like method in my case?
And while we're at it - I had similar problems with checking if the value is not null. I tried to apply regular expressions and such - no luck.
It's working in my case :
wiremock:
"request": {
"urlPathPattern": "/api/authins-portail-rs/authins/inscription/infosperso",
"bodyPatterns" : [
{"matchesJsonPath" : "$[?(#.nir == '123456789')]"},
{"matchesJsonPath" : "$[?(#.nomPatronyme == 'aubert')]"},
{"matchesJsonPath" : "$[?(#.prenoms == 'christian')]"},
{"matchesJsonPath" : "$[?(#.dateNaissance == '01/09/1952')]"}
],
"method": "POST"
}
Json:
{
"nir": "123456789",
"nomPatronyme": "aubert",
"prenoms": "christian",
"dateNaissance": "01/09/1952"
}
Following worked for me.
"matchesJsonPath" : "$.rootItem.itemA[0].item..[?(#.fieldName=='file')]"
Json :
{
"rootItem" : {
"itemA" : [
{
"item" : {
"fieldName" : "file",
"name" : "test"
}
}
]
}
}
Wiremock
{
"request" : {
"urlPattern" : "/testjsonpath",
"method" : "POST",
"bodyPatterns" : [ {
"matchesJsonPath" : "$.rootItem.itemA[0].item..[?(#.fieldName=='file')]"
} ]
},
"response" : {
"status" : 200,
"body" : "{\"result\": \"success\"}",
"headers" : {
"Content-Type" : "application/json"
}
}
}
Update Wiremock. It should work with newer versions >= 2.0.0-beta. Its JsonPath dependency was very outdated (GitHub #261).
Using the double dots operator is semantically not the same, as the filter will also match for elements with the same name deeper down the tree.
try with double dots operator (recursive)
$..params[?(#.clientVersion == "1")]

New Line output CloudFormation Json Template

This is my Output section for JSON script that I use for CloudFormation template.
"Outputs": {
"Value":{"Fn::Join": [" ",["Username : ",{"Ref": "Username"},"Password : ",{"Ref": "Pass"}]]},
}
How do I print Output in below format:
username : abc
password : abc
Right now I am getting this in same line (Username: abc Password abc).
Try appending new line. It should work. As the arguments are taken as string.
Why not print these out as separate outputs?
"Outputs" : {
"Username" : {
"Description" : "Username",
"Value": { "Ref": "Username" }
},
"Password" : {
"Description" : "Password",
"Value": { "Ref": "Password" }
}
}