I am stuck in parsing a data response receiving from some third party vendor.
response is something like:-
data: ()(responseCode='A01', responseMessage='Approved', accountNumber='qwerty');
I have tried several ways of parsing/stringify but it does not provide me a JSON response. I tried weird combinations of Querystring functions as well but that did not help.
I am badly stuck in this.
I will post a workaround it might not be efficient but will give you the result.
var data = "data: ()(responseCode='A01', responseMessage='Approved', accountNumber='qwerty');";
var temp = data.substring(8,);
temp = temp.replace("(","{");
temp = temp.replace(")","}");
temp = temp.replace(/=/g,":");
temp = temp.replace(";","");
temp = eval('(' + temp + ')');
var Result = JSON.stringify(temp)
Result : {"responseCode":"A01","responseMessage":"Approved","accountNumber":"qwerty"}
You can use regex to convert it to a valid JSON structure.
let data = `data: ()(responseCode='A01', responseMessage='Approved', accountNumber='qwerty');`;
let modified = data
.replace(/\s/g,'')
.replace("data:()(",'{\'')
.replace(");",'}')
.replace(/=/g,"':")
.replace(/,/g,",'")
.replace(/'/g,'"')
let json = JSON.parse(modified);
console.log(json)
Related
QueryRequest req=new QueryRequest(solrQuery);
NoOpResponseParser responseParser = new NoOpResponseParser();
responseParser.setWriterType("csv");
searcherServer.setParser(responseParser);
NamedList<Object> resp=searcherServer.request(req);
QueryResponse res = searcherServer.query(solrQuery);
responseString = (String)resp.get("response");
I use the above code to get the output in CSV format. The data I am trying to fetch is huge (In billions). So I want to include deep pagination of SOLR and get chunks of CSV output. Is there a way to do? Also, with the current version of SOLR (I cannot upgrade) I have to use the above code to get CSV output.
I tried the below way to fetch the results.
searcherServer = new HttpSolrServer(url);
SolrQuery solrQuery = new SolrQuery();
solrQuery.setQuery(query);
solrQuery.set("fl","field1");
solrQuery.setParam("wt", "csv");
solrQuery.setStart(0);
solrQuery.setRows(1000);
solrQuery.setSort(SolrQuery.SortClause.asc("field2"));
In the output from the above code has wt as javabin. So I cannot get the CSV output.
Any suggestions?
You have two ways.
use Solr export request handler (or add it) and wt=csv parameter. Just to be clear, this is an Implicit Request Handler usually available even in older Solr versions and specifically designed to handle scenarios that involve exporting millions of records.
implement deep paging correctly. I suggest Yonic post paging and deep paging, it easier than you think. But after you'll have correctly implement, you also need to create the csv file by yourself.
The solution I found was:
SolrQuery solrQuery = new SolrQuery();
solrQuery.setQuery(query); //what you want to fetch
QueryResponse res = searcherServer.query(solrQuery);
int numFound = (int)res.getResults().getNumFound();
int rowsToBeFetched = (numFound > 1000 ? (int)(numFound/6) : numFound);
for(int i=0; i< numFound; i=i+rowsToBeFetched ){
solrQuery.set("fl","fieldToBeFetched");
solrQuery.setParam("wt", "csv");
solrQuery.setStart(i);
solrQuery.setRows(rowsToBeFetched);
QueryRequest req=new QueryRequest(solrQuery);
NoOpResponseParser responseParser = new NoOpResponseParser();
responseParser.setWriterType("csv");
searcherServer.setParser(responseParser);
NamedList<Object> resp=searcherServer.request(req);
responseString = (String)resp.get("response"); //This is in CSV format
}
Pros:
Since I don't get the result at once, it was faster.
The output was csv.
Hitting solr multiple items isn't costly.
Cons:
The result is not unique, meaning there can be repeated data based on what you are fetching.
To get unique results, you can use facets.
Thanks!
This code seems to work fine when I used Django console and just print it.
reference = FloodHazard.objects.filter(hazard='High')
ids = reference.values_list('id', flat=True)
for myid in ids:
getgeom = FloodHazard.objects.get(id=myid).geom
response = BuildingStructure.objects.filter(geom__intersects=getgeom).values(
'brgy_locat').annotate(counthigh=Count('brgy_locat'))
print response
I was able to show all the values, but when using HttpResponse, it returns an empty set. What is the proper way of returning JSON data from a queryset? So far, tried this:
reference = FloodHazard.objects.filter(hazard='High')
ids = reference.values_list('id', flat=True)
response = {}
for myid in ids:
getgeom = FloodHazard.objects.get(id=myid).geom
response['high'] = BuildingStructure.objects.filter(geom__intersects=getgeom).values(
'brgy_locat').annotate(counthigh=Count('brgy_locat'))
json_post = ujson.dumps(list(response))
return HttpResponse(json_post, content_type='application/json')
There is no much sense in your code. You assign all querysets to the single key in the response dict. You should use a list for this purpose:
As far as I understand the code should be something like this:
response = []
for myid in ids:
getgeom = FloodHazard.objects.get(id=myid).geom
response.extend(BuildingStructure.objects.filter(geom__intersects=getgeom)
.values('brgy_locat')
.annotate(counthigh=Count('brgy_locat')))
json_post = ujson.dumps(response)
If you want to return a hazard level as well as the list of buildings then you can return a dict:
json_post = ujson.dumps({'hazard': 'high', 'buildings': response})
On node.js, are JSON methods (parse, stringify) the same as Query Sting methods (parse, stringify) ?
Thank you.
Let's try:
var qs = require('querystring');
var obj = { 'foo' : '1 + 2' };
console.log(qs.stringify(obj));
// result: foo=1%20%2B%202
console.log(JSON.stringify(obj));
// result: {"foo":"1 + 2"}
So, no, they aren't :-)
Talking names. The query string methods return strings suitable for use in a URL query string. I. e., they are percent-escaped. That's not at all related to JSON.
I'm working on a T4 file that generates .cs classes based on an entity model, and one of the things I'm trying to get to is the mapping info in the model. Specifically, for each field in the model I'm trying retrieve the database field name it is mapped to.
I've found that the mapping info is apparently stored in StorageMappingItemCollection, but am having an impossible time figuring out how to query it and retrieve the data I need. Has anyone worked with this class and can maybe provide guidance?
The code I have so far goes something like this (I've pasted everything up to the problematic line):
<#
System.Diagnostics.Debugger.Launch();
System.Diagnostics.Debugger.Break();
#>
<## template language="C#" debug="true" hostspecific="true"#>
<## include file="EF.Utility.CS.ttinclude"#>
<## output extension=".cs"#><#
CodeGenerationTools code = new CodeGenerationTools(this);
MetadataLoader loader = new MetadataLoader(this);
CodeRegion region = new CodeRegion(this, 1);
MetadataTools ef = new MetadataTools(this);
string inputFile = #"MyModel.edmx";
EdmItemCollection ItemCollection = loader.CreateEdmItemCollection(inputFile);
StoreItemCollection storeItemCollection = null;
loader.TryCreateStoreItemCollection(inputFile, out storeItemCollection);
StorageMappingItemCollection storageMappingItemCollection = null;
loader.TryCreateStorageMappingItemCollection(
inputFile, ItemCollection, storeItemCollection, out storageMappingItemCollection);
var item = storageMappingItemCollection.First();
storageMappingItemCollection has methods like GetItem() and such, but I can't for the life of me get it to return data on fields that I know exist in the model.
Thx in advance!
Parsing the MSL isn't really that hard with Linq to XML
string mslManifestResourceName = GetMslName(ConfigurationManager.ConnectionStrings["Your Connection String"].ConnectionString);
var stream = Assembly.GetExecutingAssembly().GetManifestResourceStream(mslManifestResourceName);
XmlReader xreader = new XmlTextReader(stream);
XDocument doc = XDocument.Load(xreader);
XNamespace xmlns = "http://schemas.microsoft.com/ado/2009/11/mapping/cs";
var items = from entitySetMap in doc.Descendants(xmlns + "EntitySetMapping")
let entityTypeMap = entitySetMap.Element(xmlns + "EntityTypeMapping")
let mappingFragment = entityTypeMap.Element(xmlns + "MappingFragment")
select new
{
EntitySet = entitySetMap.Attribute("Name").Value,
TypeName = entityTypeMap.Attribute("TypeName").Value,
TableName = mappingFragment.Attribute("StoreEntitySet").Value
};
It may be easier to parse the EDMX file as XML rather than using the StorageMappingItemCollection.
I am new to Mondrian. I am using it in my project for OLAP operations.
I am testing it with Foodmart database.
The problem is that I need the OLAP operations results in JSON format.
I know that mondrian has the same structure as JSON in the form of hierarchies.
I want to generate a JSON file as an output from the result of mondrian MDX query.
The result should be similar to OLAP operations.
I don't know how to iterate over the result generated from MDX query.
Here is the code.
String connStr = "Provider=mondrian;" +
"Catalog=/WEB-INF/FoodMart.xml;" +
"JdbcDrivers=com.mysql.jdbc.Driver;" +
"Jdbc=jdbc:mysql://localhost/foodmart;" +
"jdbcUser=root;" +
"jdbcPassword=;";
String queryStr ="select {[Measures].[Unit Sales], [Measures].[Store Cost], [Measures].>Store Sales]} ON COLUMNS,"+"Crossjoin(Hierarchize(Union({[Promotion Media].[All Media]}, >[Promotion Media].[All Media].Children)), {[Product].[All Products]})
ON ROWS"+" from [Sales]"+"where [Time].[1997]";
Connection connection = DriverManager.getConnection(connStr, null);
Query query = connection.parseQuery(queryStr);
Result result = connection.execute(query);
result.print(new PrintWriter(System.out));
Actually I need to perform OLAP operations on data warehouse which is stored in MySQL.
The resulted data should be in JSON format which I will pass to D3 http://mbostock.github.com/d3 for visualizations.
For data format I have to use JSON format.
Please any suggestions how to iterate MDX result and convert it in JSON file.
I am using Pentaho Mondrian for this purpose.
Thanks.
if you are working with PHP you could use this library to transform the xmla result into Json
http://www.ibm.com/developerworks/xml/library/x-xml2jsonphp/
Here's an example of what i suppose you want to do:
Class.forName("mondrian.olap4j.MondrianOlap4jDriver"); //load the driver
Connection connection = DriverManager.getConnection("Provider=mondrian;" +
"Catalog=/WEB-INF/FoodMart.xml;" +
"JdbcDrivers=com.mysql.jdbc.Driver;" +
"Jdbc=jdbc:mysql://localhost/foodmart;" +
"jdbcUser=root;" +
"jdbcPassword=;");
OlapWrapper wrapper = (OlapWrapper) connection;
OlapConnection olapConnection = wrapper.unwrap(OlapConnection.class);
CellSet cellSet = statement.executeOlapQuery(query);
CellSetAxis rows = cellSet.getAxes().get(1); //cube rows
CellSetAxis columns = cellSet.getAxes().get(0); //cube columns
int resultSize = rows.getPositionCount() * columns.getPositionCount();
String resultValues[] = new String[resultSize];
int valueIndex = 0;
for (Position row : rows) {
for (Position column : columns) {
Cell cell = cellSet.getCell(column, row);
String cellValue = cell.getFormattedValue();
resultValues[valueIndex++] = cellValue;
}
}
Gson gson = new Gson(); //gson library instance
String resultString = gson.toJson(responseValues); //json string
olapConnection.close();
connection.close();