I have a Json file like this :
{
"type" : "record",
"name" : "test",
"fields" : [ {
"name" : "y",
"type" : {
"type" : "array",
"items" : "double"
},
"doc" : "Type inferred from '[1.0,0.9766205557100867,0.907575419670957,0.7960930657056438,0.6473862847818277,0.46840844069979015,0.26752833852922075,0.05413890858541761,-0.16178199655276473,-0.37013815533991445,-0.5611870653623823,-0.7259954919231308,-0.8568571761675893,-0.9476531711828025,-0.9941379571543596,-0.9941379571543596,-0.9476531711828025,-0.8568571761675892,-0.7259954919231307,-0.5611870653623825,-0.37013815533991445,-0.16178199655276476,0.05413890858541758,0.267528338529221,0.4684084406997903,0.6473862847818279,0.796093065705644,0.9075754196709569,0.9766205557100867,1.0]'"
}, {
"name" : "x",
"type" : {
"type" : "array",
"items" : "double"
},
"doc" : "Type inferred from '[0.0,0.21666156231653746,0.43332312463307493,0.6499846869496124,0.8666462492661499,1.0833078115826873,1.2999693738992248,1.5166309362157622,1.7332924985322997,1.9499540608488373,2.1666156231653746,2.383277185481912,2.5999387477984497,2.8166003101149872,3.0332618724315243,3.249923434748062,3.4665849970645994,3.683246559381137,3.8999081216976745,4.116569684014212,4.333231246330749,4.549892808647287,4.766554370963824,4.983215933280362,5.199877495596899,5.416539057913437,5.6332006202299745,5.849862182546511,6.066523744863049,6.283185307179586]'"
} ]
}
and i want to create an sql query from this file. So so I wrote this code
def schema= new JsonSlurper().parseText(myAttr)
//build create table statement
def createTable = "create table if not exists ${schema.name} (" +
schema.fields.collectMany{ "\n ${it.name.padRight(39)} ${typeMap[it.type.collectMany{it.items}]}" }.join(',') +
"\n)"
but I think that I do not access the value items correctly. Can somebody help me please
Ok, guessing that typeMap is similar to:
def typeMap = [
double: 'DOUBLE'
]
You can change your code to:
String createTable = """create table if not exists ${schema.name} (
|${schema.fields.collect { " ${it.name.padRight(39)} ${typeMap[it.type.items]}" }.join(',\n')}
|)""".stripMargin()
Related
I created a avro schema to validate the messages that I publish on a kafka topic. These messages consist of complex types that contain other complex types. Not all fields are required, so I figured out I need to use union types in avro for that.
So basically at some point my avro schema looks like this:
"type" : "record",
"name" : "RecordA",
"fields" : [ {
"name" : "fieldX",
"type" : [ "null", {
"type" : "record",
"name" : "RecordB",
"fields" : [ {
"name" : "gemeente",
"type" : "string"
}, {
"name" : "nummer",
"type" : "string"
}, {
"name" : "postcode",
"type" : "string"
}, {
"name" : "straat",
"type" : "string"
} ]
} ]
Can someone give me an example of how a json message that adheres to this schema would look like? All the examples I found refer to simple union types that consist of primitive values.
I'm reading a CSV file with some strings and some integer columns and converting them to JSON. While this conversion, all fields and values seem to have double quotes around them. However I want the integer values NOT to have double quotes.
I'm using Jackson Fasterxml and here's my code snippet
File input = new File("/Users/name/1.csv");
File output = new File("/Users/name/output.json");
CsvSchema csvSchema = CsvSchema.builder().setUseHeader(true).build();
CsvMapper csvMapper = new CsvMapper();
// Read data from CSV file
List<Object> readAll = csvMapper.readerFor(Map.class).with(csvSchema).readValues(input).readAll();
ObjectMapper mapper = new ObjectMapper();
mapper.configure(JsonGenerator.Feature.QUOTE_NON_NUMERIC_NUMBERS, true);
// Write JSON formated data to output.json file
mapper.writerWithDefaultPrettyPrinter().writeValue(output, readAll);
Here's my expected output: Output, please note id and budget do not have double quotes on them
[ {
"id" : 120,
"name" : "Name 1",
"type" : "type1",
"budget" : 100
},
{
"id" : 130,
"name" : "Name 2",
"type" : "type2",
"budget" : 200
},
{
"id" : 140,
"name" : "Name 3",
"type" : "type2",
"budget" : 130
},
{
"id" : 150,
"name" : "Name 4",
"type" : "type4",
"budget" : 400
}
}]
However all fields and values have quotes after conversion
[ {
"id" : "120",
"name" : "Name 1",
"type" : "type1",
"budget" : "100"
},
{
"id" : "130",
"name" : "Name 2",
"type" : "type2",
"budget" : "200"
},
{
"id" : "140",
"name" : "Name 3",
"type" : "type2",
"budget" : "130"
},
{
"id" : "150",
"name" : "Name 4",
"type" : "type4",
"budget" : "400"
}
}]
Unfortunately right now it is not possible to read CSV and specify type for using schema only. You can create POJO with private int budget; and conversion will be done automatically. For other solutions take a look on this question: jackson-dataformat-csv: Mapping number value without POJO where you can see:
Usage of univocity-parsers
Trick with custom Map implementation.
I try to transform an Avro file to an SQL request. My file is like this :
{
"type" : "record",
"name" : "warranty",
"doc" : "Schema generated by Kite",
"fields" : [ {
"name" : "id",
"type" : "long",
"doc" : "Type inferred from '1'"
}, {
"name" : "train_id",
"type" : "long",
"doc" : "Type inferred from '21691'"
}, {
"name" : "siemens_nr",
"type" : "string",
"doc" : "Type inferred from 'Loco-001'"
}, {
"name" : "uic_nr",
"type" : "long",
"doc" : "Type inferred from '193901'"
}, {
"name" : "Configuration",
"type" : "string",
"doc" : "Type inferred from 'ZP28'"
}, {
"name" : "Warranty_Status",
"type" : "string",
"doc" : "Type inferred from 'Out_of_Warranty'"
}, {
"name" : "Warranty_Data_Type",
"type" : "string",
"doc" : "Type inferred from 'Real_based_on_preliminary_acceptance_date'"
}
and my code is :
import groovy.json.JsonSlurper
def ff = session.get()
if(!ff)return
//parse afro schema from flow file content
def schema = ff.read().withReader("UTF-8"){ new JsonSlurper().parse(it) }
//define type mapping
def typeMap = [
"string" : "varchar(255)",
"long" : "numeric(10)",
[ "null", "string" ]: "varchar(255)",
[ "null", "long" ] : "numeric(10)",
]
//build create table statement
def createTable = "create table ${schema.name} (" +
schema.fields.collect{ "\n ${it.name.padRight(39)} ${typeMap[it.type]}" }.join(',') +
"\n)"
//execute statement through the custom defined property
//SQL.mydb references http://docs.groovy-lang.org/2.4.10/html/api/groovy/sql/Sql.html object
SQL.mydb.execute(createTable)
//transfer flow file to success
REL_SUCCESS << ff
And i got this error :
ERROR nifi.processors.script.ExecuteScript ExecuteScript[id=e65b733e-0161-1000-45f0-3264d6fb51dd] ExecuteSc$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object); rolling back session: {} org.apache.nifi.processor.exception.ProcessException: javax.script.ScriptException: javax.script.ScriptException: groovy.lang.MissingMethodException: No signature of m$ Possible solutions: getId(), find(), grep(), each(groovy.lang.Closure), find(groovy.lang.Closure), grep(java.lang.Object)
Can someone help me plz
This references a script from another SO post, I commented there and I provided an answer on a different forum, which I will copy here for completeness:
The variable createTable is a GString, not a Java String. This causes invocation of Sql.execute(GString), which converts the embedded expressions into parameters, and you can't use a parameter for a table name. Use the following instead:
SQL.mydb.execute(createTable.toString())
This will cause the invocation of Sql.execute(String), which does not try to parameterize the statement.
I am pretty new to node, so might be the case I am using the JSON Schema not correctly, please correct me if I am wrong.
I have been using the npm module named jsonschema.
And for using validation I am using it like:
var Validator = require('jsonschema').Validator;
var v = new Validator();
var instance = {
"user_id" : "jesus",
"password" : "password"
};
var schema = {
"id" : "Login_login",
"type" : "object",
"additionalProperties" : false,
"properties" : {
"user_id": {
"type" : "string",
"required": true,
"minLenth" : 8,
"maxLength" : 10,
"description": "User id to login."
},
"password" : {
"type" : "string",
"required": true,
"minLength" : 8,
"maxLength" : 10,
"description" : "password to login."
}
}
};
var result = v.validate(instance, schema);
console.log('>>>>>> ' + result);
But the point is result is not having an error although the the minLength of user_id is kept as 8 but I have passed 5 characters, so if I am not wrong result should throw an error for the same, Why is it so
:(
The schema itself needs validation. Its "user_id" condition "minLength" spelled without a "g".
In Swagger, I'm able to create a parameter that is an array of any type like this:
"MyType" : {
"description" : "my example object type",
"type" : "object",
"properties" : {
"id" : {
"description" : "identifier",
"type" : "number"
},
"data" : {
"description" : "data container",
"type" : "array",
"items" : {
"type" : "string"
}
}
}
}
Which defines an object that might look like this:
{
"id" : 1,
"data" : ["a", "b", "c"]
}
But what I need to do is define an object that might look like this:
{
"id" : 1,
"data" : [
[0, 1, 2],
["a", "b"],
[true, "foo", 99, false]
]
}
The data property needs to be a multidimensional array, and ideally it could contain any number of "rows", each with any number of columns containing any type of data in each field. I'd even settle for the schema allowing for data to just be an array of anything, but I can't figure out how to get that to work.
All you need to do is change the type of the items schema to be array. The following schema means "data" is an array, whose items are arrays. There are no constraints on the inner arrays.
"data" : {
"description" : "data container",
"type" : "array",
"items" : {
"type" : "array",
"items": {}
}
}