guys i am trying in convert my xml file to csv using hadoop so i am using the following code in mapper class
protected void map(LongWritable key, Text value,
#SuppressWarnings("rawtypes") Mapper.Context context)
throws
IOException, InterruptedException {
String document = value.toString();
System.out.println("‘" + document + "‘");
try {
XMLStreamReader reader =
XMLInputFactory.newInstance().createXMLStreamReader(new
ByteArrayInputStream(document.getBytes()));
String propertyName = "";
String propertyValue = "";
String currentElement = "";
while (reader.hasNext()) {
int code = reader.next();
switch (code) {
case XMLStreamConstants.START_ELEMENT: //START_ELEMENT:
currentElement = reader.getLocalName();
break;
case XMLStreamConstants.CHARACTERS: //CHARACTERS:
if (currentElement.equalsIgnoreCase("author")) {
propertyName += reader.getText();
} else if (currentElement.equalsIgnoreCase("price"))
{
String name=reader.getText();
name.trim();
propertyName += name;
propertyName.trim();
}
}
console.write(null,new Text(propertyName));
}
}
but the output i am getting is in this form
Gambardella, Matthew
XML Developer's Guide
44.95
2000-10-01
Ralls, Kim
Midnight Rain
5.95
2000-12-16
can u help me with this
The output of the program depends on how you are collecting/writing from mapper.
In this case you should be using TextOutputFormat & KeyOut will be NullWritable and ValueOut will be Text. The Value out should be a concatenation of the values which you extracted from CSV.
From your code it looks like you are writing output after reading each value from the XML.
Related
I have an JSON array as defined below:-
[
{"Name":"Ayush","Age":24,"Job":"Developer"},
{"Name":"Monika","Age":23,"Job":"Developer"},
{"Name":"Chinmay","Age":23,"Job":"Developer"}
]
I want to dump this into text file in following format:-
Name Age Job
Ayush 24 Developer
Monika 23 Developer
Chinmay 23 Developer
Is there any C# function to accomplish the above? If not, how can i achieve it with minimum memory consumption?
Thanks in advance
There is no such built-in function. You may achieve this by reading JTokens from input stream using JsonTextReader and writing their values into another stream. Stream input and output ensures minimal memory footprint.
using (var inputStream = File.OpenRead("input.json"))
using (var streamReader = new StreamReader(inputStream))
using (var jsonTextReader = new JsonTextReader(streamReader))
using (var outputStream = File.OpenWrite("output.csv"))
using (var streamWriter = new StreamWriter(outputStream))
{
var firstItem = true;
while (jsonTextReader.Read())
{
if (jsonTextReader.TokenType == JsonToken.StartObject)
{
var jObject = JObject.ReadFrom(jsonTextReader);
if (firstItem)
{
streamWriter.WriteLine(string.Join("\t",
jObject.Children().Select(c => (c as JProperty).Name)));
firstItem = false;
}
streamWriter.WriteLine(string.Join("\t",
jObject.Values().Select(t => t.ToString())));
}
}
}
Demo: https://dotnetfiddle.net/2fCRa6. (I used MemoryStream and Console instead of input and output file streams in this demo since .NET Fiddle does not allow file IO, but the idea is the same.)
You can create a class with Name, Age and Job as properties.
public class Info{
public string Name { get; set; }
public int Age { get; set; }
public string Job { get; set; }
}
Then in another function use we can use System.Web.Script.Serialization class(to use this class make sure you have referenced System.Web.Extensions in project references). Once done we can use JavaScriptSerializer class and get list of objects from the json data. Then we can iterate over each item and add it two our file with a tab as a delimeter.
public static void WriteDetailsInFile(string jsonData)
{
var list = new JavaScriptSerializer().Deserialize<List<Info>>(jsonData);
using (var streamWriter = File.AppendText("D:MyFile.txt"))
{
streamWriter.WriteLine("Name\tAge\tJob");
foreach (var item in list)
{
streamWriter.WriteLine(item.Name + "\t" + item.Age + "\t" + item.Job);
}
}
}
//driver
public static void Main()
{
string data = #"[
{ ""Name"":""Ayush"",""Age"":24,""Job"":""Developer""},
{ ""Name"":""Monika"",""Age"":23,""Job"":""Developer""},
{ ""Name"":""Chinmay"",""Age"":23,""Job"":""Developer""}
]";
WriteDetailsInFile(data);
}
I want to access a JSON of this structure in firebase
The structure
{
"questions":{
"English":{
"English_2002":[
{
"correct_ans":"A",
"OptionA":"a coder",
"OptionB":"a hacker",
"OptionC":"a writer",
"OptionD":"a programmer",
"Question":"Who build software"
},
{},
{}
],
"English_2003":[],
}
}
}
I want this structure. In the subject structure, other subjects will come after I exhaust 9 years of English.
My confusion is how to logically get each subject since firebase will only accept the root name questions.
Please I may sound dumb, but I have a very long questions thread almost 55000 lines. Because firebase accept one JSON tree.
Sorry i wasn't very clear i was asking from the stack phone app:
I have a question json tag of the structure above; my question is how will i be able to access the object subject like "english":{
// then accessing the first english array "english":[]
//since am now using firebase.
}
initially each array was individual json file, i have to recreate them into one for firebase sake. this is how i was parsing it then.
public class QuestionParser {
Context context;
public QuestionParser(Context c) {
this.context = c;
}
public ArrayList<Question> getJsonFromUrl(String url, String arrayName)
{
ArrayList<Question> arrayofQuestion = new ArrayList<>();
return arrayofQuestion;
}
// Processing question from JSon file in res > raw folder
public ArrayList<Question> parseQuestionJson(int rawJsonFileId, String arrayName) {
ArrayList<Question> questionList = new ArrayList<>();
String jsonstr = null;
try {
InputStream in = context.getResources().openRawResource(rawJsonFileId);
BufferedReader br = new BufferedReader(new InputStreamReader(in));
StringBuilder sb = new StringBuilder();
String line = null;
while ((line = br.readLine()) != null) {
sb.append(line + "\n");
}
jsonstr = sb.toString();
Log.d("REEEEADDD" + this.toString(), jsonstr);
//System.out.println(jsonstr);
} catch (Exception e) {
Log.e("Buffer Error", "Error converting result " + e.toString());
}
// If the JSON string is empty or null, then return early.
if (TextUtils.isEmpty(jsonstr)) {
return null;
}
try {
JSONObject jsonObject = new JSONObject(jsonstr);
JSONArray jsonArray = jsonObject.getJSONArray(arrayName);
JSONObject jobject;
for (int i = 0; i < jsonArray.length(); i++) {
// TEST
jobject = jsonArray.getJSONObject(i);
String ans = jobject.getString("correct_answer");
String graphic_name = jobject.getString("question_image");
String optionA = jobject.getString("optiona");
String optionB = jobject.getString("optionb");
String optionC = jobject.getString("optionc");
String optionD = jobject.getString("optiond");
String questionNo = jobject.getString("question_number");
String question = jobject.getString("question");
questionList.add(new Question(questionNo, graphic_name, question, optionA, optionB, optionC, optionD, ans));
Log.d("DDD" + this.toString(), String.valueOf(questionList.get(i)));
}
Log.i("ONE QUESTION", questionList.get(50).getQuestion());
} catch (Exception e) {
Log.e("JSON Parser", "Error parsing data " + e.toString());
}
return questionList;
}
}
So how can i parse it from firebase because initially, if a student chooses question and year i passes those value as parameter and use them for parsing. but in firebase now i have access to only root firebase name in the get reference e method
To access for example "correct_ans":"A" you would query your firebase like so:
your.firebase.domain/questions/English/English_2002/0/correct_ans
Notice that each level in the json object is represented by a / and the key you want to access whereas in case of an array you simple add the array index. JSON's simple structure also allows simple REST like access
I want to put a MySQL result set into a JsonArray using Gsons library. How can I best achieve this. I've read this:
resultset to json using gson
But it uses for some reason, the simple-Json library in addition. i dont want that if possible. Is there any way to achieve this easily with the gson library?
Thank you very much!
PlayerList.java:
/*
* To change this license header, choose License Headers in Project Properties.
* To change this template file, choose Tools | Templates
* and open the template in the editor.
*/
package de.freakyonline.ucone;
import de.freakyonline.ucone.Player;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.net.Socket;
import java.util.ArrayList;
import java.util.Iterator;
import javafx.application.Platform;
import javafx.collections.FXCollections;
import javafx.collections.ObservableList;
import javafx.scene.control.TextArea;
/**
*
* #author uwe
*/
public class PlayerList {
ObservableList<Player> playerList;
ObjectInputStream in;
ObjectOutputStream out;
Socket sock;
private Object obj = null;
private Object obj2 = null;
TextArea consoleOneTextArea;
public PlayerList(ObjectInputStream in, ObjectOutputStream out, Socket sock, TextArea consoleOneTextArea) {
this.in = in;
this.out = out;
this.sock = sock;
this.consoleOneTextArea = consoleOneTextArea;
getPlayersFromServer();
}
private void getPlayersFromServer() {
/* try {
out.writeObject("getplayers");
obj=in.readObject();
if(obj == null) {
System.out.println("ERROR! void getPlayersFromServer in PlayerList.java");
Platform.exit();
}
String command = obj.toString().toLowerCase();
String currentFromServer;
if(command.equalsIgnoreCase("getplayers")) {
while((obj2=in.readObject()) != null) {
currentFromServer = obj2.toString().toLowerCase();
for(String cell : currentFromServer.split(" ")) {
System.out.println(cell.toString());
}
if (currentFromServer.equalsIgnoreCase("done")) {
consoleOneTextArea.appendText("This is finished. Have fun!\n");
break;
}
consoleOneTextArea.appendText(currentFromServer + "\n");
}
} { System.out.println("ERROR"); }
} catch (Exception ex) { ex.printStackTrace(); }
*/
this.playerList = FXCollections.observableArrayList(
new Player("freakyy85","Owner","1810",31,"m", "missing..."),
new Player("Ky3ak","Owner","1920",34,"m", "missing...")
);
}
}
(ive commented out some parts, as they are not relevant anymore)
Player.java:
/*
* To change this license header, choose License Headers in Project Properties.
* To change this template file, choose Tools | Templates
* and open the template in the editor.
*/
package de.freakyonline.ucone;
import com.google.gson.stream.JsonReader;
import java.io.InputStreamReader;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.net.Socket;
import javafx.scene.control.TextArea;
/**
*
* #author uwe
*/
public class Remote implements Runnable {
private Object obj = null;
private Object obj2 = null;
private ObjectInputStream in;
private ObjectOutputStream out;
private Socket sock;
private TextArea consoleOneTextArea;
public Remote (ObjectInputStream in, ObjectOutputStream out, Socket sock) {
this.in = in;
this.out = out;
this.sock = sock;
}
public ObjectInputStream getIn() {
return in;
}
public ObjectOutputStream getOut() {
return out;
}
public Socket getSock() {
return sock;
}
public void setConsoleOneTextArea(TextArea consoleOneTextArea) {
this.consoleOneTextArea = consoleOneTextArea;
}
public void run() {
try {
while((obj=in.readObject()) != null && sock.isConnected()) {
String command = obj.toString().toLowerCase();
String currentFromServer;
switch(command) {
case "getplayers":
/* while((obj2=in.readObject()) != null) {
currentFromServer = obj2.toString().toLowerCase();
if (currentFromServer.equalsIgnoreCase("done")) {
consoleOneTextArea.appendText("This is finished. Have fun!\n");
break;
}
consoleOneTextArea.appendText(currentFromServer + "\n");
*/ }
JsonReader jsonReader = new JsonReader(new InputStreamReader(in, "UTF-8"));
jsonReader.close();
break;
}
} catch (Exception ex) { ex.printStackTrace(); }
}
}
Is there any way to achieve this easily with the gson library?
Not really. Gson and JDBC are too/two unrelated things so you have to implement a custom remapping function to "decode" JDBC result set rows/fields and "encode" them back to JSON array/object respectively. Accumulating a JsonArray instance may be expensive from the memory consumption point of view, or even crash the application with OutOfMemoryError for huge result sets. Nonetheless they are good if the result sets are known to be small or LIMITed.
Accumulating JsonArray
static JsonArray resultSetToJsonArray(final ResultSet resultSet)
throws SQLException {
final ResultSetMetaData metaData = resultSet.getMetaData();
// JsonArray is a Gson built-in class to hold JSON arrays
final JsonArray jsonArray = new JsonArray();
while ( resultSet.next() ) {
jsonArray.add(resultSetRowToJsonObject(resultSet, metaData));
}
return jsonArray;
}
private static JsonElement resultSetRowToJsonObject(final ResultSet resultSet, final ResultSetMetaData metaData)
throws SQLException {
final int columnCount = metaData.getColumnCount();
// Every result set row is a JsonObject equivalent
final JsonObject jsonObject = new JsonObject();
// JDBC uses 1-based loops
for ( int i = 1; i <= columnCount; i++ ) {
jsonObject.add(metaData.getColumnName(i), fieldToJsonElement(resultSet, metaData, i));
}
return jsonObject;
}
private static JsonElement fieldToJsonElement(final ResultSet resultSet, final ResultSetMetaData metaData, final int column)
throws SQLException {
final int columnType = metaData.getColumnType(column);
final Optional<JsonElement> jsonElement;
// Process each SQL type mapping a value to a JSON tree equivalent
switch ( columnType ) {
case Types.BIT:
case Types.TINYINT:
case Types.SMALLINT:
throw new UnsupportedOperationException("TODO: " + JDBCType.valueOf(columnType));
case Types.INTEGER:
// resultSet.getInt() returns 0 in case of null, so it must be extracted with getObject and cast, then converted to a JsonPrimitive
jsonElement = Optional.ofNullable((Integer) resultSet.getObject(column)).map(JsonPrimitive::new);
break;
case Types.BIGINT:
case Types.FLOAT:
case Types.REAL:
case Types.DOUBLE:
case Types.NUMERIC:
case Types.DECIMAL:
case Types.CHAR:
throw new UnsupportedOperationException("TODO: " + JDBCType.valueOf(columnType));
case Types.VARCHAR:
jsonElement = Optional.ofNullable(resultSet.getString(column)).map(JsonPrimitive::new);
break;
case Types.LONGVARCHAR:
case Types.DATE:
case Types.TIME:
case Types.TIMESTAMP:
case Types.BINARY:
case Types.VARBINARY:
case Types.LONGVARBINARY:
case Types.NULL:
case Types.OTHER:
case Types.JAVA_OBJECT:
case Types.DISTINCT:
case Types.STRUCT:
case Types.ARRAY:
case Types.BLOB:
case Types.CLOB:
case Types.REF:
case Types.DATALINK:
case Types.BOOLEAN:
case Types.ROWID:
case Types.NCHAR:
case Types.NVARCHAR:
case Types.LONGNVARCHAR:
case Types.NCLOB:
case Types.SQLXML:
case Types.REF_CURSOR:
case Types.TIME_WITH_TIMEZONE:
case Types.TIMESTAMP_WITH_TIMEZONE:
throw new UnsupportedOperationException("TODO: " + JDBCType.valueOf(columnType));
default:
throw new UnsupportedOperationException("Unknown type: " + columnType);
}
// If the optional value is missing, assume it's a null
return jsonElement.orElse(JsonNull.INSTANCE);
}
final JsonArray jsonArray = resultSetToJsonArray(resultSet);
System.out.println(jsonArray);
Don't forget to close the resultSet, of course.
JSON streaming
If the JsonArray is supposed to be written elsewhere, JsonWriter can be a better solution being able to process huge result sets reading row by row and writing JSON element by JSON element.
#SuppressWarnings("resource")
static void resultSetToJsonArrayStream(final ResultSet resultSet, final JsonWriter jsonWriter)
throws SQLException, IOException {
// Write the [ token
jsonWriter.beginArray();
final ResultSetMetaData metaData = resultSet.getMetaData();
while ( resultSet.next() ) {
// Write row by row
writeRow(resultSet, jsonWriter, metaData);
}
// Finish the array with the ] token
jsonWriter.endArray();
}
#SuppressWarnings("resource")
private static void writeRow(final ResultSet resultSet, final JsonWriter jsonWriter, final ResultSetMetaData metaData)
throws SQLException, IOException {
final int columnCount = metaData.getColumnCount();
// Similarly to the outer array: the { token starts a new object representing a row
jsonWriter.beginObject();
for ( int i = 1; i <= columnCount; i++ ) {
// Write the column name and try to resolve a JSON literal to be written
jsonWriter.name(metaData.getColumnName(i));
writeField(resultSet, jsonWriter, metaData, i);
}
// Terminate the object with }
jsonWriter.endObject();
}
#SuppressWarnings("resource")
private static void writeField(final ResultSet resultSet, final JsonWriter jsonWriter, final ResultSetMetaData metaData, final int column)
throws SQLException, IOException {
final int columnType = metaData.getColumnType(column);
switch ( columnType ) {
case Types.BIT:
case Types.TINYINT:
case Types.SMALLINT:
throw new UnsupportedOperationException("TODO: " + JDBCType.valueOf(columnType));
case Types.INTEGER:
jsonWriter.value((Integer) resultSet.getObject(column));
break;
case Types.BIGINT:
case Types.FLOAT:
case Types.REAL:
case Types.DOUBLE:
case Types.NUMERIC:
case Types.DECIMAL:
case Types.CHAR:
throw new UnsupportedOperationException("TODO: " + JDBCType.valueOf(columnType));
case Types.VARCHAR:
jsonWriter.value((String) resultSet.getObject(column));
break;
case Types.LONGVARCHAR:
case Types.DATE:
case Types.TIME:
case Types.TIMESTAMP:
case Types.BINARY:
case Types.VARBINARY:
case Types.LONGVARBINARY:
case Types.NULL:
case Types.OTHER:
case Types.JAVA_OBJECT:
case Types.DISTINCT:
case Types.STRUCT:
case Types.ARRAY:
case Types.BLOB:
case Types.CLOB:
case Types.REF:
case Types.DATALINK:
case Types.BOOLEAN:
case Types.ROWID:
case Types.NCHAR:
case Types.NVARCHAR:
case Types.LONGNVARCHAR:
case Types.NCLOB:
case Types.SQLXML:
case Types.REF_CURSOR:
case Types.TIME_WITH_TIMEZONE:
case Types.TIMESTAMP_WITH_TIMEZONE:
throw new UnsupportedOperationException("TODO: " + JDBCType.valueOf(columnType));
default:
throw new UnsupportedOperationException("Unknown type: " + columnType);
}
}
Example of writing to System.out, but, of course, it can be written anywhere just supplying an appropriate OutputStream instance:
final JsonWriter jsonWriter = new JsonWriter(new OutputStreamWriter(System.out))
resultSetToJsonArrayStream(resultSet, jsonWriter);
Similarly to ResultSet, JsonWriter must be closed as well.
I've written the above code for SQLite, but it should work for MySQL too. For example, the test database created and populated with the following SQL statements:
CREATE TABLE `table` (i NUMBER NOT NULL, s TEXT NOT NULL);
INSERT INTO `table` (i, s) VALUES (1, 'foo');
INSERT INTO `table` (i, s) VALUES (2, 'bar');
INSERT INTO `table` (i, s) VALUES (3, 'baz');
will result in
[{"i":1,"s":"foo"},{"i":2,"s":"bar"},{"i":3,"s":"baz"}]
for both object model and streaming approaches.
I am using Google cloud end point for my rest service. I am consuming this data in a GWT web client using RestyGWT.
I noticed that cloud end point is automatically enclosing a long datatype in double quotes which is causing an exception in RestyGWT when I try to convert JSON to POJO.
Here is my sample code.
#Api(name = "test")
public class EndpointAPI {
#ApiMethod(httpMethod = HttpMethod.GET, path = "test")
public Container test() {
Container container = new Container();
container.testLong = (long)3234345;
container.testDate = new Date();
container.testString = "sathya";
container.testDouble = 123.98;
container.testInt = 123;
return container;
}
public class Container {
public long testLong;
public Date testDate;
public String testString;
public double testDouble;
public int testInt;
}
}
This is what is returned as JSON by cloud end point. You can see that testLong is serialized as "3234345" rather than 3234345.
I have the following questions.
(1) How can I remove double quotes in long values ?
(2) How can I change the string format to "yyyy-MMM-dd hh:mm:ss" ?
Regards,
Sathya
What version of restyGWT are you using ? Did you try 1.4 snapshot ?
I think this is the code (1.4) responsible for parsing a long in restygwt, it might help you :
public static final AbstractJsonEncoderDecoder<Long> LONG = new AbstractJsonEncoderDecoder<Long>() {
public Long decode(JSONValue value) throws DecodingException {
if (value == null || value.isNull() != null) {
return null;
}
return (long) toDouble(value);
}
public JSONValue encode(Long value) throws EncodingException {
return (value == null) ? getNullType() : new JSONNumber(value);
}
};
static public double toDouble(JSONValue value) {
JSONNumber number = value.isNumber();
if (number == null) {
JSONString val = value.isString();
if (val != null){
try {
return Double.parseDouble(val.stringValue());
}
catch(NumberFormatException e){
// just through exception below
}
}
throw new DecodingException("Expected a json number, but was given: " + value);
}
return number.doubleValue();
}
I want to teach my domain class to automatically convert the results of JSON.parse(someJSON) into a member that is also a custom domain class.
Given these domain classes:
class Person {
Long id
String name
static hasMany = [aliases: PersonAlias]
}
class PersonAlias {
Person person
Long id
String name
}
And this JSON representing a Person with some PersonAliases:
{
"id":20044397,
"name":"John Smith",
"aliases":[{"id":13376,"name":"Johnny Smith"},{"id":13377,"name":"J. Smith"}]
}
I want to keep the controller simple like:
class PersonController {
def saveViaAjax = {
def props = JSON.parse(params.JSON)
Person p = Person.get(props.id)
p.properties = props
p.save(flush: true)
}
}
But sadly I get this error:
Failed to convert property value of type
'org.codehaus.groovy.grails.web.json.JSONArray' to required type
'java.util.Set' for property 'aliases'; nested exception is
java.lang.IllegalStateException: Cannot convert value of type
[org.codehaus.groovy.grails.web.json.JSONObject] to required type
[heavymeta.PersonAlias] for property 'aliases[0]': no matching editors
or conversion strategy found
So, I want to teach my domain class to how to convert the JSON data into PersonAlias instances automatically. I'd like to avoid formatting the data in the controller before passing it to the Domain object. How do I accomplish these goals?
You can use the bindUsing annotation and provide your custom binding code to convert the json to the property being bound.
class Person {
Long id
String name
#BindUsing({obj, source ->
List retVal = []
def aliases = source['aliases']
if(aliases) {
aliases.each {
retVal << new PersonAlias(name:it.name)
}
}
return retVal
})
List<PersonAlias> aliases
static hasMany = [aliases: PersonAlias]
}
I think this plugin: https://github.com/pedjak/grails-marshallers might do what you're looking for? I have not tried it myself though.
I also encountered this problem - I did my best to document the fix on my website - See http://dalelotts.com/software-architect/grails
In general the solution is to convert the JSON to a parameter map that can be used for data binding. More info on the site, including an annotation driven DomainClassMarshaller for JSON
protected Object readFromJson(Class type, InputStream entityStream, String charset) {
def mapper = new ObjectMapper();
def parsedJSON = mapper.readValue(entityStream, typeRef);
Map<String, Object> map = new HashMap<>();
parsedJSON.entrySet().each {Map.Entry<String, Object> entry ->
if (List.isAssignableFrom(entry.getValue().getClass())) {
List values = (List) entry.getValue();
int limit = values.size()
for (int i = 0; i < limit; i++) {
final theValue = values.get(i)
map.put(entry.key + '[' + i + ']', theValue)
appendMapValues(map, theValue, entry.key + '[' + i + ']' )
}
} else {
map.put(entry.key, entry.value);
}
}
def result = type.metaClass.invokeConstructor(map)
// Workaround for http://jira.codehaus.org/browse/GRAILS-1984
if (!result.id) {
result.id = idFromMap(map)
}
result
}
private void appendMapValues(Map<String, Object> theMap, Object theValue, String prefix) {
if (Map.isAssignableFrom(theValue.getClass())) {
Map<String, Object> valueMap = (Map<String, Object>) theValue;
for (Map.Entry<String, Object> valueEntry : valueMap.entrySet()) {
theMap.put(prefix + '.' + valueEntry.key, valueEntry.value)
appendMapValues(theMap, valueEntry.value, prefix + '.' + valueEntry.key)
}
}
}