I am trying to use SpringXD to stream some JSON metrics data to a Oracle database.
I am using this example from here: SpringXD Example
Http call being made: EarthquakeJsonExample
My shell cmd.
stream create earthData --definition "trigger|usgs| jdbc --columns='mag,place,time,updated,tz,url,felt,cdi,mni,alert,tsunami,status,sig,net,code,ids,souces,types,nst,dmin,rms,gap,magnitude_type' --driverClassName=driver --username=username --password --url=url --tableName=Test_Table" --deploy
I would like to capture just the properties portion of this JSON response into the given table columns. I got it to the point where it doesn't give me a error on the hashing but instead just deposits a bunch of nulls into the column.
I think my problem is the parsing of the JSON itself. Since really the properties is in the Features array. Can SpringXD distinguish this for me out of the box or will I need to write a custom processor?
Here is a look at what the database looks like after a successful cmd.
Any advice? Im new to parsing JSON in this fashion and im not really sure how to find more documentation or examples with SpringXD itself.
Here is reference to the documentation: SpringXD Doc
The transformer in the JDBC sink expects a simple document that can converted to a map of keys/values. You would need to add a transformer upstream, perhaps in your usgs processor or even a separate processor. You could use a #jsonPath expression to extract the properties key and make it the payload.
Related
I want to convert incoming JSON data from Kafka into a dataframe.
I am using structured streaming with Scala 2.12
Most people add a hard coded schema, but if the json can have additional fields, it requires changing the code base every-time, which is tedious.
One approach is to write it into a file and infer it with but I rather avoid doing that.
Is there any other way to approach this problem?
Edit: Found a way to turn a json string into a dataframe but cant extract it from the stream source, it is possible to extract it?
One way is to store the schema itself in the message headers (not in the key or value).
Though, this increases message size, it will be easy to parse the JSON value without the need for any external resource like a file or a schema registry.
New messages can have new schemas while at the same time old messages can still be processed using their old schema itself, because the schema is within the message itself.
Alternatively, you can version the schemas and include an id for every schema in the message headers (or) a magic byte in the key or value and infer the schema from there.
This approach is followed by Confluent Schema registry. It allows you to basically go through different versions of same schema and see how your schema has evolved over time.
Read the data as string and then convert it to map[string,String], this way you can process the any json without even knowing its schema
based on JavaTechnical answer , the best approach would be to use a schema registry and
avro data instead of json, there is no going around hardcoding a schema (for now).
include your schema name and id as a header and use them to read the schema from the schema registry.
use the from_avro fucntion to turn that data into a df!
I need to create a Logic Apps workflow with three steps:
When HTTP Request is received (JSON)
Convert Json from request to XML
Save XML file to FTP
What I have done so far:
Add action "When HTTP Request is received"
Add Liquid to Convert JSON to XML
(but i don't see option JSON to XML...Only Tranform JSON to JSON, JSON to
TEXT, XML to JSON, XML to TEXT)
Add action "FTP - Create file"
I also created Integration Account and try to add map for mapping JSON to XML, but I can't find any examples/templates to do this...
Is it possible at all ? Maybe there is another way to convert between these two formats ?
When you just want to convert a JSON payload to an XML file, without doing any transformation to the data, you can use the built-in xml() function of the Workflow Definition Language.
Detailed info in the docs: Workflow Definition Language reference #xml
I've made a small test Logic App to demo your usecase. It looks like this:
As you can see I use the xml function on the triggerbody #xml(triggerBody()) as an input for my FTP file content.
Remark: This will only work if your JSON message has a single rootnode. Otherwise the xml conversion will fail. You'll get this error:
The provided value cannot be converted to XML: 'JSON root object has multiple properties. The root object must have a single property in order to create a valid XML document. Consider specifying a DeserializeRootElementName.
You can work around that by concatenating a rootnode to your JSON payload. The function then would look like: #xml(json(concat('{\"rootnode\":',triggerBody(),'}')))
Good luck testing this out. Let me know if you need more help with this.
I currently have an API built with PL/SQL that uses Oracle 11g. It currently outputs the data in XML. I have been tasked to convert this output to JSON. Is this even possible to do with Oracle 11g? I have been researching the web and I see that JSON support did not arrive until Oracle 12c. Is there a way I can convert the output of this API from XML to JSON. Any help is appreciated. Thanks.
Here is the current XML output I have below:
<?xml version="1.0"?>
-<items>
-<CAGE_INFO>
<CAGE_CODE>21356</CAGE_CODE>
<ORG_NAME_ABBR>NASAJSC</ORG_NAME_ABBR>
<ORGANIZATION_NAME>NASA/ LYNDON B JOHNSON SPACE CENTER</ORGANIZATION_NAME>
</CAGE_INFO>
</items>
I need this JSON output using Oracle 11g:
{
"items": {
"CAGE_INFO": {
"CAGE_CODE": "21356",
"ORG_NAME_ABBR": "NASAJSC",
"ORGANIZATION_NAME": "NASA/ LYNDON B JOHNSON SPACE CENTER"
}
}
}
I guess it depends on what you mean by "convert".
If you literally mean convert and are looking for a tool that takes arbitrary XML and returns JSON, well, writing that would be a lot of work. Someone may have done that already, I don't know.
If you just need this output in JSON, you could find wherever your XML is generated and rewrite it (I assume it's backed by one or more SQL queries) and call a PL/SQL package that generates JSON. My first stop would be plsql-utils library and JSON_UTIL_PKG.
Or, take the function that generates your XML and rewrite it to construct JSON via string operations. JSON is just formatted text, after all. I've done this before and it might be the quickest way if your needs are simple.
A direct conversion might be difficult. Instead you can use XMLTYPE to first parse the XML and then convert it to JSON. The conversion to JSON can be either a custom piece of code or if you have APEX installed on the DB instance, then you can look at the the APEX_JSON package.
Check this out for a description of XMLTYPE in Oracle.
https://docs.oracle.com/cd/A97630_01/appdev.920/a96616/arxml24.htm
I am loading data from a mongodb collection to a mysql table through Kettle transformation.
First I extract them using MongodbInput and then I use json input step.
But since json input step has very low performance, I wanted to replace it with a
javacript script.
I am a beginner in Javascript and even though i tried somethings, the kettle javascript script is not recognizing any keywords.
can anyone give me sample code to convert Json data to different columns using javascript?
To solve your problem you need to see three aspects:
Reading from MongoDB
Reading from JSON
Reading from (probably) String
Reading from MongoDB Except if you changed the interface, MongoDB returns not JSON but BSON files (~binary JSON). You need to see the MongoDB documentation about reading and writing BSON: probably something like BSON.to() and BSON.from() but I don't know it by heart.
Reading from JSON Once you have your BSON in JSON format, you can read it using JSON.stringify() which returns a String.
Reading from (probably) String If you want to use the capabilities of JSON (why else would you use JSON?), you also want to use JSON.parse() which returns a JSON object.
My experience is that to send a JSON object from one step to the other, using a String is not a bad idea, i.e. at the end of a JavaScript step, you write your JSON object to a String and at the beginning of the next JavaScript step (can be further down the stream) you parse it back to JSON to work with it.
I hope this answers your question.
PS: writing JavaScript steps requires you to learn JavaScript. You don't have to be a master, but the basics are required. There is no way around it.
you could use the json input step to get the values of this json and put in common rows
It seems that Yahoo pipes are represented using JSON. I want to download these JSON objects for some research purpose. Usually a Yahoo pipe is rendered in a browser editor thru a url like this: http://pipes.yahoo.com/pipes/pipe.edit?_id=XgRo96h13BGtJWvS8SvLAg, but you can't get the corresponding JSON object to this Yahoo pipe. Does anyone know how to get JSON objects representing Yahoo pipes and store them in any persistent form?
It is possible to get hold of a JSON description of a Yahoo Pipe using a URL of the form:
http://pipes.yahoo.com/pipes/pipe.info?_out=json&_id=PIPE_ID
The pipe2py python library demonstrates how to grab the JSON description of a pipe and "compile" it to a Python equivalent that can be run on your own server.
The post Exporting Yahoo Pipe Definitions, Compiling Them to Python, and Running Them in Scraperwiki describes how you can use pipe2py in the Scraperwiki environment to compile and execute pipes on Scraperwiki using pipe definitions imported directly from Yahoo Pipes, or exported from Yahoo Pipes and then stored locally in a Scraperwiki database table.
When I load that page in a browser I can see that it makes an ajax request for:
http://pipes.yahoo.com/pipes/ajax.pipe.load?id=XgRo96h13BGtJWvS8SvLAg&_out=json&modinfo=true&rnd=7560&.crumb=MjvGjpzhPLl
That's your object but I'm not sure if I'm answering your question of how to "get it". If you need to get it through a program you would need a script that loges into pipes and extracts that url.
A quick way, while not automated, is to use an HTTP analyzer. Here's a process for getting the object using HttpFox (I use v0.8.9) for Firefox. With the analyzer running, load the edit page for a pipe, like the one you linked:
http://pipes.yahoo.com/pipes/pipe.edit?_id=XgRo96h13BGtJWvS8SvLAg
Look at the request with a URL that starts with:
http://pipes.yahoo.com/pipes/ajax.pipe.load?id=....
Next, explore the content of the request (there's a 'Content' tab in HttpFox). That's the JSON object representing the pipe structure.
Use pipe.run?[your pipe id here]&_render=json as opposed to pipe.edit
So in your case to get the json it would be - http://pipes.yahoo.com/pipes/pipe.run?_id=XgRo96h13BGtJWvS8SvLAg&_render=json
I guess how you implement the client is dependent on what you like writing in/what other functionality you need.
You could also do it the other way around and use the web service service module to post the data to a script that can extract the json and persist it to a database. You could check out json.org.