I am trying to master the principles/syntax of using a list function in CouchDB 1.6.1 to output specific fields to a csv file.
I have set up a simple output to html and this seems easier to do and it works well.
What I want to do is have a view, which the list function requires, to output selected fields from the database and output the data to a csv file.
What I can't seem to be able to do is get the list function to 'read' the specific fields from the view output, which I succeded in doing to obtain html output.
The view function looks something like this:
function(doc){
emit({'A':doc.a, 'B':doc.b, 'C':doc.c.d .....}, null);}
The html list function would look something like this:
"function(head, req){
start({'headers': {
'Content-Type': 'text/html' }});
send('<html><body><table>');
send('<tr><th>A</th><th>B</th><th>C</th></tr>');
while(row=getRow()){
send(''.concat( '<tr>', '<td>' + toJSON(row.key.A) + '</td>','<td>' + toJSON(row.key.B) + '</td>','<td>' + toJSON(row.key.C) + '</td>', '</tr>' ));}
send('</table></body></html>');}"
A similar list function for csv output for the same view should look like:
"function(head, req){
start({'headers': { 'Content-Type': 'text/csv' }});
send('A' +','+ 'B' +','+'C' + '\\n');
while(row=getRow()){
send(''.concat( toJSON(row.key.A) , toJSON(row.key.B) , toJSON(row.key.C) ));};}"
this results in " "error":"compilation_error","reason":"Expression does not eval to a function ...."
I have tried numerous variations of the csv function without success except for a jumbled lot of incorrectly formatted text.
A recommended starting point for a csv list function was, on a certain website, given as:
function (head, req) {
start({
“headers”: {
“Content-Type”: “text/csv”
}
});
send(‘Username, Name, Email\n’);
while(row = getRow()) {
send(row.value.username+’,’+row.value.email+’,’+row.value.metadata.name+’\n’);
}
}
I cannot get this structure to work at all.
I would appreciate some input on the correct syntax to use please.
I was able to find a guideline here. A slideshare by Oliver Kurowski.
The basic principles are:
"views": {
"byPrice": {
"map": "function(doc){emit(doc.price, [doc.make,doc.year]);};"
}
}
"lists": {
"csv": "function(head,req){start({'headers':{'Content-Type':'text/csv'}});send('Make'+','+'Year'+','+'Price'+'\\n');while(row=getRow()){send(row.value+','+row.key+'\\n');}};"
}
It worked fine.
Thank you Oliver.
Related
I am new to both JSON and Postman(as of yesterday).
I'm trying to do something very simple, I've created a GET request which pulls in a list of forms in a JSON response. I want to take this response and get the first "id" token and place it in a variable.
I am using a global variable but would like to use a collection variable if possible. Either way here is what I am doing.
I've tried several things, most recently this:
var jsonData = JSON.parse(responseBody);
postman.setGlobalVariable("id", jsonData.args.id);
As well as this:
pm.test("GetId", function () {
var jsonData = pm.response.json();
pm.globals.set("id", jsonData.id);
});
Response code looks like this:
{
"forms":[
{
"id":"3197239",
"created":"2018-09-18 11:37:39",
"db":"1",
"deleted":"0",
"folder":"151801",
"language":"en",
"name":"Contact Us",
"num_columns":"2",
"submissions":"0",
"submissions_unread":"0",
"updated":"2018-09-18 12:02:13",
"viewkey":"xxxxxx",
"views":"1",
"submissions_today":0,
"url":"https://xxx",
"data_url":"",
"summary_url":"",
"rss_url":"",
"encrypted":false,
"thumbnail_url":null,
"submit_button_title":"Submit Form",
"inactive":false,
"timezone":"US/Eastern",
"permissions":150
},
{
"id":"3197245",
"created":"2018-09-18 11:42:02",
"db":"1",
"deleted":"0",
"folder":"151801",
"language":"en",
"name":"Football Draft",
"num_columns":"1",
"submissions":"0",
"submissions_unread":"0",
"updated":"2018-09-18 12:11:54",
"viewkey":"xxxxxx",
"views":"1",
"submissions_today":0,
"url":"https://xxxxxxxxx",
"data_url":"",
"summary_url":"",
"rss_url":"",
"encrypted":false,
"thumbnail_url":null,
"submit_button_title":"Submit Form",
"inactive":false,
"timezone":"US/Eastern",
"permissions":150
}
]
}
This would get the first id:
pm.globals.set('firstId', _.first(pm.response.json().forms).id)
That would get the first in the array each time so it would set a different variable it that response changed.
The test that you created was nearly there but the reference needed to go down a level into the forms array:
pm.test("GetId", function () {
var jsonData = pm.response.json()
pm.expect(jsonData.forms[0].id).to.equal("3197239")
pm.globals.set("id", jsonData.forms[0].id)
})
The [0]is referencing the first id in the first object within the array. For example [1] would get the second one and so on.
You currently cannot set a collection level variable using the pm.* API - These can only be added manually and referenced using the pm.variables.get('var_name') syntax.
Edit:
In the new versions of the desktop app you can set variables at the Collection level using pm.collectionVariables.set().
Based on the name or any other attribute if you want to set the id as a global variable then this is the way.
for(var i=0; i<jsonData.forms.length; i++)
{
if (jsonData.forms[i].name==="Contact Us")
{
pm.environment.set("id", jsonData.forms[i].id);
}
}
I'm doing a SQL query in Node-Red to output a load of time/value data. This data is then passed to a web page for display in a graph.
Previously I've used php to do the SQL query, which I'm trying to replace. However SQL queries in php are delivered in a different format.
With Node-Red, I get:
[
{
"Watts": 1018,
"Time": 1453825454
},
{
"Watts": 1018,
"Time": 1453825448
},
{
"Watts": 1010,
"Time": 1453825442
}]
With PHP, I get:
[
[1453819620000,962],
[1453819614000,950],
[1453819608000,967],
[1453819602000,947]
]
I think I'm getting an array from php and an array of JSON objects from Node-Red. How do I convert the Node-Red object to be output from Node-Red in the same format as the PHP is? (Ie: I want to handle the processing at the server, rather than the client.)
A function node can be used to generate something in the same format.
var array = msg.payload;
var phpFormat = "[";
for (var i=0; i<array.length; i++) {
phpFormat += "[" +
// time format differ, NodeJS is in seconds
// php is in milliseconds
(array[i].Time * 1000 ) +
"," +
array[i].Watts + "],";
}
//take the last "," off
phpFormat = phpFormat.substring(0,phpFormat.lenght - 1);
phpFormat += "]";
msg.payload = phpFormat;
return msg;
I've had a bit of help from a chap at work and here is what he's come up with, modified for node-red by me:
var outputArray = [];
for(var i in msg.payload){
var entryData = [msg.payload[i]['Time']];
for(var attr in msg.payload[i]) {
if(attr!='Time') {
entryData.push(msg.payload[i][attr])}
};
outputArray.push(entryData); }
var returnMsg={"payload":outputArray};
return returnMsg;
I know, I know, this question is over 2 years old... however, for the next 500 people seeking an answer to a similar problem, I'd like to highlight the new JSONata expression feature built-in to the change node. Using this simple expression:
payload.[Time, Watts]
transforms your JS objects into the requested output of an array of arrays. In fact, much of my old repetitive looping through arrays has been replaced with some simpler (to me) expressions like this.
The magic of the lambda syntax evaluator is documented on the JSONata site. There you will also find the online exerciser where you can build an expression against your own data and immediately see the resulting structure.
Note: in order to use a jsonata expression in your change node, be sure to select the J: pulldown next to the input field (not the {} JSON option)... two totally different things!
I am using json2csv package to convert the queried data to csv and allow the user to download the file. Everything works fine, except that the output header and the corresponding data are not aligned properly. This is what I get
But this is what I need.
Code:
var fields = ['firstname', 'surname']
Users.find().select().exec(function(err, users){
json2csv({data: users, fields: fields}, function(err, csv){
if(err)
console.log(err)
else{
var filename = 'users.csv'
var mimetype = 'application/csv'
res.setHeader('Content-disposition', 'attachment; filename=' + filename)
res.setHeader('Content-type', mimetype)
res.end(csv)
}
})
})
How to align the headers and the data properly in their columns? Thanks
EDIT:
CSV output in the browser console.
json2csv have option
quotes - String, quotes around cell values and column names. Defaults to " if not specified.
you can try to set empty space or some one else instead quotes
Very new to d3 and javascript, but have done a good four or so hours trying to resolve this issue and haven't found a solution yet.
My .csv dataset I'd like to use has both string and float numbers (see sample set here: http://individual.utoronto.ca/lannajin/d3/pcadummyset.csv), which I'm currently having issues importing.
d3.csv("http://individual.utoronto.ca/lannajin/d3/pcadummyset.csv")
.row(function (d) {
return {
metric: d.metric,
Var: d.Var,
param: d.param,
PC1: +d.PC1, // convert "PC1" column to number
PC2: +d.PC2, // convert "PC2" column to number
Type: d.Type
};
})
.get(function (error, rows) {
console.log(rows);
});
Where I've tried variations (for line 7 & 8) as:
PC1: d.parseFloat("PC1"),
PC1: parseFloat(d.PC1),
PC1: d.parseFloat.PC1,
etc.
The main problem is that my float characters (PC1 and PC2) are in the form 1.0e-02 rather than 0.001, which means I can't simply use the "+" operator to convert my data to float. Instead, I have to use the parseFloat function, which, I unfortunately am not sure how to use.
When I change the structure to the solution offered by Importing CSV without headers into D3 - data with both numbers and strings,
data = d3.csv.parseRows("http://individual.utoronto.ca/lannajin/d3/pcadummyset.csv").map(function(row) {
return row.map(function(value, index) {
if(index == 2) {
return value;
} else {
return +value;
}
});
});
It's clear that the data is being read, but not plotted since the PC1 and PC2 values are probably being read in as strings. I tried to integrate the parseFloat function, but it still won't plot my values.
Help please!
I''m using EF to query the database using anonymous type.
here the code I use for EF
public JsonResult OverdueEventsCustom()
{
var eventCustomOverdue = _eventCustomRepository.FindOverdueEventsCustom();
return Json(eventCustomOverdue, JsonRequestBehavior.AllowGet);
}
public IQueryable<dynamic> FindOverdueEventsCustom()
{
DateTime dateTimeNow = DateTime.UtcNow;
DateTime dateTomorrow = dateTimeNow.Date.AddDays(1);
return db.EventCustoms.Where(x => x.DateTimeStart < dateTomorrow)
.Select(y => new { y.EventId, y.EventTitle, y.DateTimeStart});
}
Inspecting using the debugger I see the properties is in this format
Date = {16/08/2012 00:00:00}
The resultfor the JSON is
[{
"EventId": 1,
"EventTitle": "Homework Math",
"DateTimeStart": "\/Date(1345108269310)\/"
}, {
"EventId": 4,
"EventTitle": "Homework help with Annie",
"DateTimeStart": "\/Date(1345108269310)\/"
}, {
"EventId": 6,
"EventTitle": "Physic laboratory",
"DateTimeStart": "\/Date(1345108269310)\/"
}]
I need the the json in this format
"DateTimeStart": "(16/08/2012)"
Any idea what i'm doing wrong here? thanks for your help
Related articles
http://www.hanselman.com/blog/OnTheNightmareThatIsJSONDatesPlusJSONNETAndASPNETWebAPI.aspx
How do I format a Microsoft JSON date?
"\/Date(1345108269310)\/" is the correct way to pass a Date to javascript. The way I see it, you have two options here:
If you do not explicitly need the value as a date, you could just pass a string to the JSON variable, containing the pretty-printed date.
Something along the lines of:
DateTimeStart: String.Format("{0: dd-MM-yyyy}", myDate)
If you will still need to use the variable a a date in javascript (for calculations for example), the most consice and readably way would be to create a javascript function that converts said date into the pretty-printed string you want (I don't know if such a function already exists. It isn't too hard to create though:
function prettyDate(date) {
return date.getDate() + "-" + date.getMonth() + "-" + date.getFullYear();
}
I would suggest passing it along as a string from you code behind, as it is more readable. But that only works if you do not need to use the date except for displaying.