I try to transform an Avro file to an SQL request. My file is like this :
{
"type" : "record",
"name" : "warranty",
"doc" : "Schema generated by Kite",
"fields" : [ {
"name" : "id",
"type" : "long",
"doc" : "Type inferred from '1'"
}, {
"name" : "train_id",
"type" : "long",
"doc" : "Type inferred from '21691'"
}, {
"name" : "siemens_nr",
"type" : "string",
"doc" : "Type inferred from 'Loco-001'"
}, {
"name" : "uic_nr",
"type" : "long",
"doc" : "Type inferred from '193901'"
}, {
"name" : "Configuration",
"type" : "string",
"doc" : "Type inferred from 'ZP28'"
}, {
"name" : "Warranty_Status",
"type" : "string",
"doc" : "Type inferred from 'Out_of_Warranty'"
}, {
"name" : "Warranty_Data_Type",
"type" : "string",
"doc" : "Type inferred from 'Real_based_on_preliminary_acceptance_date'"
}
and my code is :
import groovy.json.JsonSlurper
def ff = session.get()
if(!ff)return
//parse afro schema from flow file content
def schema = ff.read().withReader("UTF-8"){ new JsonSlurper().parse(it) }
//define type mapping
def typeMap = [
"string" : "varchar(255)",
"long" : "numeric(10)",
[ "null", "string" ]: "varchar(255)",
[ "null", "long" ] : "numeric(10)",
]
//build create table statement
def createTable = "create table ${schema.name} (" +
schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') +
"\n)"
//execute statement through the custom defined property
//SQL.mydb references http://docs.groovy-lang.org/2.4.10/html/api/groovy/sql/Sql.html object
SQL.mydb.execute(createTable)
//transfer flow file to success
REL_SUCCESS << ff
And i got this error :
ERROR nifi.processors.script.ExecuteScript ExecuteScript[id=e65b733e-0161-1000-45f0-3264d6fb51dd] ExecuteSc$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object); rolling back session: {} org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of m$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object)
Can someone help me plz
This references a script from another SO post, I commented there and I provided an answer on a different forum, which I will copy here for completeness:
The variable createTable is a GString, not a Java String. This causes invocation of Sql.execute(GString), which converts the embedded expressions into parameters, and you can't use a parameter for a table name. Use the following instead:
SQL.mydb.execute(createTable.toString())
This will cause the invocation of Sql.execute(String), which does not try to parameterize the statement.
Related
I created a avro schema to validate the messages that I publish on a kafka topic. These messages consist of complex types that contain other complex types. Not all fields are required, so I figured out I need to use union types in avro for that.
So basically at some point my avro schema looks like this:
"type" : "record",
"name" : "RecordA",
"fields" : [ {
"name" : "fieldX",
"type" : [ "null", {
"type" : "record",
"name" : "RecordB",
"fields" : [ {
"name" : "gemeente",
"type" : "string"
}, {
"name" : "nummer",
"type" : "string"
}, {
"name" : "postcode",
"type" : "string"
}, {
"name" : "straat",
"type" : "string"
} ]
} ]
Can someone give me an example of how a json message that adheres to this schema would look like? All the examples I found refer to simple union types that consist of primitive values.
I'm reading a CSV file with some strings and some integer columns and converting them to JSON. While this conversion, all fields and values seem to have double quotes around them. However I want the integer values NOT to have double quotes.
I'm using Jackson Fasterxml and here's my code snippet
File input = new File("/Users/name/1.csv");
File output = new File("/Users/name/output.json");
CsvSchema csvSchema = CsvSchema.builder().setUseHeader(true).build();
CsvMapper csvMapper = new CsvMapper();
// Read data from CSV file
List<Object> readAll = csvMapper.readerFor(Map.class).with(csvSchema).readValues(input).readAll();
ObjectMapper mapper = new ObjectMapper();
mapper.configure(JsonGenerator.Feature.QUOTE_NON_NUMERIC_NUMBERS, true);
// Write JSON formated data to output.json file
mapper.writerWithDefaultPrettyPrinter().writeValue(output, readAll);
Here's my expected output: Output, please note id and budget do not have double quotes on them
[ {
"id" : 120,
"name" : "Name 1",
"type" : "type1",
"budget" : 100
},
{
"id" : 130,
"name" : "Name 2",
"type" : "type2",
"budget" : 200
},
{
"id" : 140,
"name" : "Name 3",
"type" : "type2",
"budget" : 130
},
{
"id" : 150,
"name" : "Name 4",
"type" : "type4",
"budget" : 400
}
}]
However all fields and values have quotes after conversion
[ {
"id" : "120",
"name" : "Name 1",
"type" : "type1",
"budget" : "100"
},
{
"id" : "130",
"name" : "Name 2",
"type" : "type2",
"budget" : "200"
},
{
"id" : "140",
"name" : "Name 3",
"type" : "type2",
"budget" : "130"
},
{
"id" : "150",
"name" : "Name 4",
"type" : "type4",
"budget" : "400"
}
}]
Unfortunately right now it is not possible to read CSV and specify type for using schema only. You can create POJO with private int budget; and conversion will be done automatically. For other solutions take a look on this question: jackson-dataformat-csv: Mapping number value without POJO where you can see:
Usage of univocity-parsers
Trick with custom Map implementation.
I have a Json file like this :
{
"type" : "record",
"name" : "test",
"fields" : [ {
"name" : "y",
"type" : {
"type" : "array",
"items" : "double"
},
"doc" : "Type inferred from '[1.0,0.9766205557100867,0.907575419670957,0.7960930657056438,0.6473862847818277,0.46840844069979015,0.26752833852922075,0.05413890858541761,-0.16178199655276473,-0.37013815533991445,-0.5611870653623823,-0.7259954919231308,-0.8568571761675893,-0.9476531711828025,-0.9941379571543596,-0.9941379571543596,-0.9476531711828025,-0.8568571761675892,-0.7259954919231307,-0.5611870653623825,-0.37013815533991445,-0.16178199655276476,0.05413890858541758,0.267528338529221,0.4684084406997903,0.6473862847818279,0.796093065705644,0.9075754196709569,0.9766205557100867,1.0]'"
}, {
"name" : "x",
"type" : {
"type" : "array",
"items" : "double"
},
"doc" : "Type inferred from '[0.0,0.21666156231653746,0.43332312463307493,0.6499846869496124,0.8666462492661499,1.0833078115826873,1.2999693738992248,1.5166309362157622,1.7332924985322997,1.9499540608488373,2.1666156231653746,2.383277185481912,2.5999387477984497,2.8166003101149872,3.0332618724315243,3.249923434748062,3.4665849970645994,3.683246559381137,3.8999081216976745,4.116569684014212,4.333231246330749,4.549892808647287,4.766554370963824,4.983215933280362,5.199877495596899,5.416539057913437,5.6332006202299745,5.849862182546511,6.066523744863049,6.283185307179586]'"
} ]
}
and i want to create an sql query from this file. So so I wrote this code
def schema= new JsonSlurper().parseText(myAttr)
//build create table statement
def createTable = "create table if not exists ${schema.name} (" +
schema.fields.collectMany{ "\n ${it.name.padRight(39)} ${typeMap[it.type.collectMany{it.items}]}" }.join(',') +
"\n)"
but I think that I do not access the value items correctly. Can somebody help me please
Ok, guessing that typeMap is similar to:
def typeMap = [
double: 'DOUBLE'
]
You can change your code to:
String createTable = """create table if not exists ${schema.name} (
|${schema.fields.collect { " ${it.name.padRight(39)} ${typeMap[it.type.items]}" }.join(',\n')}
|)""".stripMargin()
I am pretty new to node, so might be the case I am using the JSON Schema not correctly, please correct me if I am wrong.
I have been using the npm module named jsonschema.
And for using validation I am using it like:
var Validator = require('jsonschema').Validator;
var v = new Validator();
var instance = {
"user_id" : "jesus",
"password" : "password"
};
var schema = {
"id" : "Login_login",
"type" : "object",
"additionalProperties" : false,
"properties" : {
"user_id": {
"type" : "string",
"required": true,
"minLenth" : 8,
"maxLength" : 10,
"description": "User id to login."
},
"password" : {
"type" : "string",
"required": true,
"minLength" : 8,
"maxLength" : 10,
"description" : "password to login."
}
}
};
var result = v.validate(instance, schema);
console.log('>>>>>> ' + result);
But the point is result is not having an error although the the minLength of user_id is kept as 8 but I have passed 5 characters, so if I am not wrong result should throw an error for the same, Why is it so
:(
The schema itself needs validation. Its "user_id" condition "minLength" spelled without a "g".
I am writing a software that can read the JSON data from a file. The file contains "person" - an object whose value is an array of objects. I am planning to use a JSON schema validating libraries to validate the contents instead of writing the code myself. What is the correct schema that conforms to JSON Schema Draf-4 which represents the below data?
{
"person" : [
{
"name" : "aaa",
"age" : 10
},
{
"name" : "ddd",
"age" : 11
},
{
"name" : "ccc",
"age" : 12
}
]
}
The schema that wrote down is given below. I am not sure whether it is correct or is there any other form?
{
"person" : {
"type" : "object",
"properties" : {
"type" : "array",
"items" : {
"type" : "object",
"properties" : {
"name" : {"type" : "string"},
"age" : {"type" : "integer"}
}
}
}
}
}
You actually only have one line in the wrong place, but that one line breaks the whole schema. "person" is a property of the object and thus must be under the properties keyword. By putting "person" at the top, JSON Schema interprets it as a keyword instead of a property name. Since there is no person keyword, JSON Schema ignores it and everything below it. Therefore, it is the same as validating against the empty schema {} which places no restrictions on what a JSON document can contain. Any valid JSON is valid against the empty schema.
{
"type" : "object",
"properties" : {
"person" : {
"type" : "array",
"items": {
"type" : "object",
"properties" : {
"name" : {"type" : "string"}
"age" : {"type" : "integer"}
}
}
}
}
}
By the way, there are several online JSON Schema testing tools out there that can help you out when crafting your schemas. This one is my goto http://jsonschemalint.com/draft4/#
Also, here is a great JSON Schema reference that might help you out as well: https://spacetelescope.github.io/understanding-json-schema/