Preprocessing Mongoose documents right after querying - json

Abstract
Hi, I have two models City and Country:
var Country = new Schema({
name: String,
population: String
});
var City = new Schema({
name: String,
timeZone: { type: Number, min: -12, max: 12 },
summerTime: Boolean,
country: { type: Schema.ObjectId, ref: 'Country'}
});
When I'm querying data from mongo using the following way:
function (request, response) {
var CityModel = mongoose.model('City', City);
CityModel.find().lean().populate('country').exec(function(err, docs){
response.send(JSON.stringify(docs));
});
}
But on client when I parse JSON, I get multiple instances of the same country:
var cities = $.get('/cities').then(function(citiesJson){
var cities = JSON.parse(citiesJson);
var washingtonIndex = ...;
var californiaIndex = ...;
var washington = cities[washingtonIndex];
var california = cities[californiaIndex];
washington.country.population = "300 000 000";
california.country.population = "500 000 000";
console.log([california.country.population, california.country.population]);
});
Resulting with two values ["300 000 000", "500 000 000"] in the console.
The problem
To prevent this kind of behaviour and preserve object references I'm doing JSON.decycle before object serialization:
...
response.send(JSON.stringify(JSON.decycle(docs)));
...
It works better than it should be. As a result I get the following JSON on the client:
// City:
{
name: "Bost",
deleted: "0",
country: {
$ref: "$[0]["city"]["country"]"
},
lastModified: "2013-08-06T23:44:11.000Z",
_id: {
_bsontype: "ObjectID",
id: "Rm1'"
},
approved: "1"
}
Notice the _id field which is getting serialized by reference so I don't get the actual ObjectId's string representation on the client, instead I'm getting some internal mongo representation of the id which is not usable on the client.
The question
1) Is there a way to setup per model or per schema preprocessing for all queried documents so that I convert ObjectId's into string
2) Probably there is some other more efficient way of dealing with this problem?
Thanks

Related

Dart| json formatting

I have been trying to fetch data out of JSON in flutter application.
I have tested the API request and formatting in python before making the application and all seems right
Python code:
for data in results['sections']:
for restaurant in data['items']:
restaurantName = restaurant['title']
restaurantImage = restaurant['image']['url']
for detail in restaurant:
if detail == 'venue':
restaurantAddress = restaurant['venue']['address']
restaurantDeliveryPrice = restaurant['venue']['delivery_price']
restaurantDeliveryETA = restaurant['venue']['estimate_range']
restaurantOnline = restaurant['venue']['online']
restaurantTags = restaurant['venue']['tags']
but when trying to get the data in flutter it doesn't return any feedback after
"for detail in restaurant:"
Dart code :
for (var data in jsonResponse['sections']) {
for (var restaurant in data['items']) {
var restaurantName = restaurant['title'];
var restaurantImage = restaurant['image']['url'];
print(restaurant);
}
}
Example restaurant json:
{filtering: {filters: [{id: primary, values: [hummus, kosher-for-passover, kosher]}]}, image: {blurhash: j6RIaL4iPuN3XKHdlktkX;OWgOmY, url: https://prod-wolt-venue-images-cdn.wolt.com/5fc619b23e62b6b24328928f/e122a0f8-3897-11eb-8333-de04af82ec5b_list1.jpg, variants: [xs, sm, md, frontpage]}, link: {target: 5fc619b23e62b6b24328928f, target_sort: default, target_title: , title: , type: venue-id, venue_mainimage_blurhash: j4OXin0g6J00Tl;;;mch7vgPV2WX}, overlay: Temporarily offline, sorting: {sortables: [{id: delivery-price, value: 1013}, {id: rating, value: 846}, {id: delivery-estimate, value: 946}, {id: distance, value: 1016}]}, template: venue-large, title: Hummus Hakerem | Shoken, track_id: venue-hummus-hakerem, venue: {address: שוקן 30 תל אביב , badges: [{text: KOSHER, variant: secondary}], categories: [], city: , country: ISR, currency: ILS, delivers: false, delivery_price: ₪18.00, delivery_price_highlight: false, delivery_price_int: 1800, estimate: 30, estimate_range: 25-35, franchise: , id: 5fc619b23e62b6b24328928f, location: [34.7720054, 32.0515629], name: Hummus Hakerem | Shoken, online: false, price_range: 1, product_line: restaurant, promotions: [], rating: {rating: 4, score: 9.2}, short_description: Simply great hummus, show_wolt_plus: false, slug: hummus-hakerem, tags: [kosher]}}
when trying to program the "for detail in restaurant:" loop in dart the app just loop endlessly

Typescript convert an array to JSON

I have a complicated data structure that I need to convert to JSON. The problem is that my field names and values are in an array.
For instance, I have the following (simplified from my code base):
let SampleData = [
{ Field: 'Key', Value: '7'},
{ Field: 'City', Value: 'Some City'},
{ Field: 'Description', Value: 'Some Description'}
];
Basically my data is an array where the first element is the database column name, and the second element is the data in the column. I am trying to get a JSON object that is:
{ Key: 7, City: 'Some City', Description: 'Some Description' }
My real code has the fields and data is structures within the object, so I cannot simply use an Object.create() or Object.assign() as far as I can get working.
I have tried looping through to build a simple string and then use the JSON.parse to break it apart, but this seems like a lot of overhead for something I would have thought would be simpler.
As you asked, here's how to do it:
Mapping the array to an object
Converting the object to JSON
let array = [{
Field: 'Key',
Value: '7'
},
{
Field: 'City',
Value: 'Some City'
},
{
Field: 'Description',
Value: 'Some Description'
}
];
// #1 Mapping the array to an object...
let obj = {};
array.forEach(item => obj[item.Field] = item.Value);
// #2 Converting the object to JSON...
let json = JSON.stringify(obj);
console.log(json);
Bonus (ES6 + reduce):
const obj = array.reduce((acc, { Field, Value }) => ({ ...acc, [Field]: Value }), {});
you can try the below approach . I have used spread operator(ES6) and Object.assign to create the object ,then converted it into json string.
let SampleData = [
{ Field: 'Key', Value: '7'},
{ Field: 'City', Value: 'Some City'},
{ Field: 'Description', Value: 'Some Description'}
];
let obj = Object.assign(...SampleData.map( x => Object.values(x)).map(y => ({[y[0]]: y[1]})));
console.log(obj);
//{ Key: "7", City: "Some City", Description: "Some Description" }
console.log(JSON.stringify(obj));
I had a similar requirement and here is how I achieved it.
var ranges: segmentRange[] = new Array(2);
ranges[0] = { minimumPercentage: 50, maximumPercentage: 60 };
ranges[1] = { minimumPercentage: 30, maximumPercentage: 40 };
const segmentRanges = { segmentRanges: ranges };
return JSON.stringify(segmentRanges);
Output:
{"segmentRanges":[{"minimumPercentage":50,"maximumPercentage":60},{"minimumPercentage":30,"maximumPercentage":40}]}
HTH,

Deep level xml parsing to csv using NodeJS

I have a medium sized xml ~ 5mb that needs to be converted to csv.
Obviously wont go for reinventing the wheel,
so a two layer approach -
1> xml to json
2> json to csv
My current code is :
const xml_obj = {}
const htt = require('http-status-code-node');
var fs = require('fs');
var xml2js = require('xml2js');
var converter = require('json-2-csv');
xml_obj["convert"] = (req, res, next) => {
var parser = new xml2js.Parser();
fs.readFile(__dirname + '/directoryexport.xml', function (err, data) {
parser.parseString(data, function (err, result) {
console.log('Done');
var callback = function (err, ycsv) {
if (err) return console.log(err);
///
res.setHeader('Content-Disposition', 'attachment; filename=testing.csv');
res.set('Content-Type', 'text/csv');
res.status(200).send(result);
///
}
var documents = [];
documents.push(result)
converter.json2csv(documents, callback);
})
});
}
module.exports = xml_obj.convert
However the xml being nested gives a multi layered json which the yields a single string instead of a proper delimited csv..
The current output CSV
The Original xml
The XML structure
The Json I get on converting xml
Also as per the documentation of the json to csv converter
if the input json is in a proper structure like :
[
{
Make: 'Nissan',
Model: 'Murano',
Year: '2013',
Specifications: {
Mileage: '7106',
Trim: 'S AWD'
}
},
{
Make: 'BMW',
Model: 'X5',
Year: '2014',
Specifications: {
Mileage: '3287',
Trim: 'M'
}
}
];
This yields a very nicely formatted csv like this : Example Perfect CSV From JSON
Edit 1 :
The format I'm looking for is somewhat like,
It's important to capture all parent organization and organizationalUnit details for each person node.
For example,
organizationalUnit UUID "b3b05b77-a8a7-43ed-ab74-b7d898c60296" should
produce a CSV lines like:
"Mr Shayne Howard","Howard","Shayne","Mr","","Branch Manager","(02) 6121 5492","","Level 1, 12 Mort Street, Canberra, ACT, 2601","Shayne.Howard#employment.gov.au","","b43e0864-1b9a-40f0-8049-c90af5f9141c","","GPO Box 9880 CANBERRA ACT 2601 Australia",1392,"","Department of Employment","","1300 488 064","","","","http://www.employment.gov.au","GPO Box 9880, Canberra ACT 2601","EMPLOYMENT"
"Mr Luke de Jong","De Jong","Luke","Mr","","Branch Manager, General Counsel","(02) 6240 0909",""(02) 6123 5100"","","Luke.deJong#employment.gov.au","","58a503a8-ce8b-41c0-b690-b9f9efd98a89","","GPO Box 9880 CANBERRA ACT 2601",1393,"","Department of Employment","","1300 488 064","","","","http://www.employment.gov.au","GPO Box 9880, Canberra ACT 2601","EMPLOYMENT"
Edit 2 :
Flattening out the json is a good idea, but its not capturing the entire data.
Using the camaro nodejs module with the following template :
persons: ['//person', {
root_organization_name: '../../../../name',
main_organization_name: '../../../name',
main_organization_website: '../../../website',
fullName: 'fullName',
familyName: 'familyName',
firstName: 'firstName',
personalTitle: 'personalTitle',
title: 'title',
person_phone: 'phone',
person_location: 'location',
person_fax: 'fax',
otherRolesDN: 'otherRolesDN',
person_mail: 'mail',
informationPublicationScheme: '../informationPublicationScheme',
publications: '../../publications',
annualReport: '../../annualReport',
mediaReleases: '../../mediaReleases',
organizationUnit_1_name: '../../name',
organizationUnit_1_description: '../../description',
organizationUnit_1_location: '../../location',
organizationUnit_1_phone: '../../phone',
organizationUnit_1_fax: '../../fax',
organizationUnit_1_website: '../../website',
organizationUnit_2_name: '../name',
organizationUnit_2_location: '../location',
organizationUnit_2_phone: '../phone',
organizationUnit_2_fax: '../fax',
organizationUnit_2_website: '../website',
occupantName: './role/occupantName',
roleName: './role/roleName',
occupantUUID: './role/occupantUUID',
role_phone: './role/phone',
role_fax: './role/fax',
role_location: './role/location',
role_mail: './role/ mail'
}]
How could I also get the roles array.
Also current csv gets some rows of data in wrong columns :
Wrong csv after camaro
Any tips on how to make this work with my input.
Because the output of json structure is deep, if you want it to convert to csv properly, you would have to flatten it.
Seems like you only interest in deepest level. here's an example, if you want to add more data, feel free to add to the template
const transform = require('camaro')
const tocsv = require('json2csv')
const fs = require('fs')
const xml = fs.readFileSync('so.xml', 'utf-8')
const template = {
persons: ['//person', {
root_organization_name: '../../../../name',
main_organization_name: '../../../name',
main_organization_website: '../../../website',
fullName: 'fullName',
familyName: 'familyName',
firstName: 'firstName',
personalTitle: 'personalTitle',
title: 'title',
person_phone: 'phone',
person_location: 'location',
person_fax: 'fax',
otherRolesDN: 'otherRolesDN',
person_mail: 'mail',
informationPublicationScheme: '../informationPublicationScheme',
publications: '../../publications',
annualReport: '../../annualReport',
mediaReleases: '../../mediaReleases',
organizationUnit_1_name: '../../name',
organizationUnit_1_description: '../../description',
organizationUnit_1_location: '../../location',
organizationUnit_1_phone: '../../phone',
organizationUnit_1_fax: '../../fax',
organizationUnit_1_website: '../../website',
organizationUnit_2_name: '../name',
organizationUnit_2_location: '../location',
organizationUnit_2_phone: '../phone',
organizationUnit_2_fax: '../fax',
organizationUnit_2_website: '../website',
roles: ['../role', {
occupantName: 'occupantName',
roleName: 'roleName',
occupantUUID: 'occupantUUID',
role_phone: 'phone',
role_fax: 'fax',
role_location: 'location',
role_mail: ' mail'
}]
}]
}
const result = transform(xml, template)
console.log(JSON.stringify(result.roles, null, 4))
Example of output json (use json2csv to convert to csv if you want)

ScriptDB auto numbering

I have a database like this one:
var db = ScriptDb.getMyDb();
var ob = {
type: "employee",
employee_id: 1,
name: {
first: "Fatima",
initial: "S",
last: "Pauli"
},
address: {
street: "4076 Washington Avenue",
city: "Jackson", state: "MS", zip: "39201"
},
department_id: 52
};
var stored = db.save(ob);
Now I want the employee_id to have auto-increment without looping trough the entire existing database.
What is the best way to do this?
A solution is to store a current max value of employee_id in a separate ScriptDB object and use the LockService to provide atomicity, as shown here, during reading and incrementing operations.

Serializing/Deserializing nested Backbone Relational models to JSON

Given the following RelationalModel model:
var Server = Backbone.RelationalModel.extend({
relations: [{
type: Backbone.HasMany,
key: 'databases',
relatedModel: 'Database',
collectionType: 'DatabaseCollection',
includeInJSON: 'id'
}],
defaults: {
databases: []
},
});
var Database = Backbone.RelationalModel.extend({});
var DatabaseCollection = Backbone.Collection.extend({
model: Database
});
And these objects:
new Database({
id: 1,
name: 'DB1'
});
new Database({
id: 2,
name: 'DB2'
});
var s1 = new Server({
id: 3,
name: 'S1',
databases: [1,2]
});
What would be the easiest/recommended way to serialize/deserialize this model to something aproaching this JSON structure?:
{
databases: [
{ id: 1, name: 'DB1' }
{ id: 2, name: 'DB2' }
],
servers: [
{ id: 3, name: 'S1', databases: [1, 2] }
]
}
Such that the data can be sent to / read from the server in a single request.
Thanks!
Tim
I was able to produce the JSON you described using your example with some minor changes in this fiddle I just created Example.
I made these changes due to some warnings that were being shown in the debugger and to get the result you described. Hope this helps.
Moved the declaration of Database Model and DatabaseCollection to top before Server since Servers relatedModel and CollectionType point to those Models.
For relatedModel and collectionType instead of using Strings used the reference to Database and DatabaseCollection
Created a collection for Servers called ServerCollection
Added a few more examples
Here is the code you end up with, I just created a plain old Backbone model to combine the two collection into one. Calling toJSON gives you the single JSON object to transmit to the server.
var Database = Backbone.RelationalModel.extend({});
var DatabaseCollection = Backbone.Collection.extend({
model: Database
});
var Server = Backbone.RelationalModel.extend({
relations: [{
type: Backbone.HasMany,
key: 'databases',
relatedModel: Database,
collectionType: DatabaseCollection,
includeInJSON: 'id'
}],
defaults: {
databases: []
}
});
var ServerCollection = Backbone.Collection.extend({
model: Server
});
var allDatabases = new DatabaseCollection();
allDatabases.add([
new Database({ id: 1, name: 'DB1' }),
new Database({ id: 2, name: 'DB2' }),
new Database({ id: 3, name: 'DB3' }),
new Database({ id: 4, name: 'DB4' })
]);
var allServers = new ServerCollection();
allServers.add([
new Server({
id: 30,
name: 'S1',
databases: [
allDatabases.get(1),
allDatabases.get(2)
]
}),
new Server({
id: 40,
name: 'S2',
databases: [
allDatabases.get(3),
allDatabases.get(4)
]
})
]);
// combine into an object to transfer to server as one
var serverObject = new Backbone.Model({
'servers': allServers,
'databases': allDatabases
});
console.log(serverObject.toJSON());