Vega statistics Get Transform result as an object - vega-lite

I want to use transform in Vega and get results as an object. Similar to the example in the link:
https://vega.github.io/vega/docs/transforms/aggregate/
I want to get the output, example: [{"v": 2, "s": 6, "m": 2}], and use it in other calculation. So I don't want to load to visualization ex. chart. I want to get the resulted object and work on it. Can I do that.


Here are the steps:
For Vega spec, just include the data block with transforms.
Run Vega the usual way to create a Vega "view".
In javascript, use Vega View API's data method to get the transformed dataset object.
Here is a complete working example (html and javascript) using the example from Vega aggregate.
<!DOCTYPE html>
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=utf-8" />
<title>Test Vega5</title>
</head>
<body>
<div id="vis"></div>
</body>
<script src="https://vega.github.io/vega/vega.js"></script>
<script>
let vegaSpec = {
"$schema": "https://vega.github.io/schema/vega/v5.json",
"data": [
{
"name": "mydata",
"values" : [
{"foo": 1, "bar": 1},
{"foo": 1, "bar": 2},
{"foo": null, "bar": 3}
],
"transform": [
{
"type": "aggregate",
"fields": ["foo", "bar", "bar"],
"ops": ["valid", "sum", "median"],
"as": ["v", "s", "m"]
}
]
}
]
};
let vegaView = new vega.View(vega.parse(vegaSpec))
.logLevel(vega.Warn)
.initialize("#vis")
.renderer("svg")
.hover()
.run();
let my_transformed_data = vegaView.data("mydata"); // vega view API
console.log(my_transformed_data);
</script>
</html>
console.log output:

Related

Vega-Lite Calculated Scale domainMax

I'm trying to calculate a value for domainMax on the Y-axis scale. I tried the following example where I want the Y-axis domainMax to be one greater than the maximum value in the dataset field named "value". The example produces the error 'Unrecognized signal name: "domMax"'. How can I get it to work?
{
"data": {
"values": [
{"date": "2021-03-01T00:00:00", "value": 1},
{"date": "2021-04-01T00:00:00", "value": 3},
{"date": "2021-05-01T00:00:00", "value": 2}
]
},
"transform": [
{ "calculate": "max(datum.value)+1","as": "domMax"}
],
"mark": "line",
"encoding": {
"x": {
"field": "date",
"type": "temporal"
},
"y": {"field": "value", "type": "quantitative",
"scale": {"domainMax": {"expr": "domMax"}}
}
}
}

This transform
"transform": [
{ "calculate": "max(datum.value)+1","as": "domMax"}
]
adds a new column to your data set - it does not create a new signal. You can check that in the editor. Go to the DataViewer tab and select data_0 from the drop down. Can you see the new domMax column?
Signals are a different thing entirely - have a look here in the documentation. Note that the link points to Vega, not Vega-Lite. (Vega-Lite specifications are compiled to Vega.)
Vega-Lite does not let you declare signals; you declare parameters instead. Here is another example using the domMax parameter. Vega-Lite parameters are translated to Vega signals.
It looks like you are trying to derive the value of your parameter/signal from the data. I am not sure you can do that in Vega-Lite.
On the other hand it's very easy in Vega. For example you could use the extent transform:
https://vega.github.io/vega/docs/transforms/extent/
Side comment - while Vega specifications are more verbose you can sometimes find their primitives simpler and a good way to understand how the visualisation works. (You can see compiled Vega in the editor.)

I tried to get a custom domain based on the data but hit the same limitations as you did.
In my case, I update the data from the outside a bit like the streaming example. I compute the domain from the outside and modify them in the visualization with params. This is quite easy as vega-lite params are exposed as vega signals.
This is the gist of the layout:
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"params": [
{
"name": "lowBound",
"value": -10
},
{
"name": "highBound",
"value": 100
}
],
../..
"vconcat": [
{
"name": "detailed",
../..
"layer": [
{
../..
"mark": "line",
"encoding": {
"y": {
"field": "value",
"title": "Temperature",
"type": "quantitative",
"scale": {
"domainMin": {
"expr": "lowBound"
},
"domainMax": {
"expr": "highBound"
}
}
},
...
The lowBound and highBound are dynamically changed through vega signals. I change them with the regular JS API.

You can add a param to pan and zoom in case your hard coded values are less than ideal.
"params": [{"name": "grid", "select": "interval", "bind": "scales"}],
Open the Chart in the Vega Editor

Reading CSVs w/o Headers in DataWeave 2.0

How do you parse a CSV without a header in DataWeave 2.0?
I have the following CSV:
Chris,Doe,Student
Bob,Smith,Teacher
and am trying to merely convert it to JSON like this:
[
[ "Chris", "Doe", "Student" ],
[ "Bob", "Smith", "Teacher" ]
]
or even this:
[
{"0": "Chris", "1": "Doe", "2": "Student" },
{"0": "Bob", "1": "Smith", "2": "Teacher" }
]
Here's my DataWeave:
%dw 2.0
input payload application/csv header=false
output application/json
---
payload
But this is the payload being returned from the DW script:
[
{
"Chris": "Bob",
"Doe": "Smith",
"Student": "Teacher"
}
]
I've tried messing around with the metadata, specifying CSV metadata type that include/exclude headers, but no luck.

This doesn't work at runtime because input directive doesn't work in mule reader properties need to be configured on the source of the value. Go to the source of your value (file:read, http:listener, etc) click on MimeType in there go an pick application/csv and under that there are the mimeType properties. In there go and select header | false

The script that you published in there works for me. Modified it a bit and am attaching the screenshot below.

How to format json file view on html

current json view on html page
{"result": [{"id": "103300640", "sms_phone": "730276061", "corrected to":
"27730276061"}, {"id": "103277733", "sms_phone": "716125197", "corrected
to": "27716125197"}]}
what I want it to be formatted display like
{
"result":[
{
"id":"103300640",
"sms_phone":"730276061",
"corrected to":"27730276061"
},
{
"id":"103277733",
"sms_phone":"716125197",
"corrected to":"27716125197"
}]
}
JS code:
#app.route('/<path:req_path>')
def dir_listing(req_path):
abs_path = os.path.join(UPLOAD_FOLDER, req_path)
# Check if path is a file and serve
if os.path.isfile(abs_path):
return send_file(abs_path, mimetype="application/json")
return render_template('/index.html')
HTML code:
<ul>
{% for file in file_list %}
<li>{{ file }}</li>
{% endfor %}
</ul>

You can try this out:
Use <pre> tag for showing code itself in HTML page and with JSON.stringify().
var data = {"result": [{"id": "103300640", "sms_phone": "730276061", "corrected to": "27730276061"}, {"id": "103277733", "sms_phone": "716125197", "corrected to": "27716125197"}]}
document.getElementById("beautified").innerHTML = JSON.stringify(data, undefined, 2);
<pre id="beautified"></pre>
The JSON.stringify() method converts a JavaScript object or value to a JSON string, optionally replacing values if a re-placer function is specified or optionally including only the specified properties if a re-placer array is specified.

You have to use JS to achieve this.
Please have a look at the below code.
Use the <pre> tag to achieve this.
<html>
<head>
</head>
<body onload="myFunction()">
<script>
function myFunction() {
var data = {
"data": {
"x": "1",
"y": "1",
"url": "http://test.com"
},
"event": "start",
"show": 1,
"id": 50
}
document.getElementById("json").innerHTML = JSON.stringify(data, undefined, 2);
}
</script>
<pre id="json"></pre>
</body>
<html>

You can use JSON.stringify
See example at: https://codepen.io/adrianparr/pen/VeyeVP

Live code examples of NVD3 does not accept my JSON

I am trying to uses my JSON file in the Live code editor of NVD3 for the stacked area chart: http://nvd3.org/livecode/index.html#codemirrorNav
Here is my JSON file:
[{"key":"SSB","values":[]},{"key":"GEN","values":[]},{"key":"LYM","values":[]},{"key":"LUD","values":[]},{"key":"GCC","values":[[1433116800000,1],[1434326400000,2],[1434499200000,2],[1434931200000,1],[1435536000000,1],[1437436800000,1],[1437523200000,1],[1439164800000,1],[1439251200000,1],[1439942400000,1],[1440288000000,1],[1465257600000,3]]},{"key":"MAC","values":[[1465257600000,1]]},{"key":"MMB","values":[[1420156800000,2],[1420416000000,1],[1420502400000,4],[1420588800000,1],[1420675200000,2],[1420761600000,1],[1421020800000,2],[1421107200000,6],[1421193600000,3],[1421280000000,3],[1421366400000,2],[1421625600000,7],[1421712000000,4],[1421798400000,1],[1421884800000,3],[1421971200000,4],[1422230400000,3],[1422316800000,5],[1422403200000,3],[1422489600000,3],[1422576000000,6],[1422921600000,8],[1423008000000,10],[1423094400000,8],[1423180800000,17],[1423267200000,1],[1423353600000,2],[1423440000000,3],[1423526400000,2],[1423699200000,5],[1423785600000,3],[1424131200000,6],[1424217600000,4],[1424304000000,5],[1424390400000,2],[1424563200000,1],[1424736000000,2],[1424822400000,2],[1424995200000,8],[1425340800000,7],[1425427200000,4],[1425513600000,2],[1425600000000,5],[1425772800000,1],[1425859200000,3],[1425945600000,4],[1426032000000,6],[1426118400000,3],[1426204800000,2],[1426377600000,1],[1426550400000,5],[1426636800000,2],[1426723200000,1],[1426809600000,14],[1426982400000,1],[1427068800000,1],[1427155200000,4],[1427241600000,2],[1427328000000,8],[1427414400000,13],[1427500800000,1],[1427587200000,1],[1428624000000,1],[1429833600000,2],[1436313600000,1],[1458000000000,1],[1458864000000,1],[1465257600000,3]]}]
I replace the sample JSON with mine, NVD3 recognizes the key elements but not the data.
I am confused because I have used online JSON formatters to look at the structures of both JSON files and didn't see a difference. I am pretty sure that I am making an obvious error. I would greatly appreciate the community's feedback! Thank you!
EDIT:
The graph that I am trying to achieve:

The problem is that JSON expected by nvd3 is in this form:
values:[{x:0, y:10},{x:1, y:20} ... ]
you are passing is in this form
,{"key":"GCC","values":[[1433116800000,1],[1434326400000,2], ...
That is the reason why its not working:
FIX: You will need to change your JSON in the expected format:
var myData = [{
"key": "SSB",
"values": []
}, {
"key": "GEN",
"values": []
}, {
"key": "LYM",
"values": []
}, {
"key": "LUD",
"values": []
}, {
"key": "GCC",
"values": [
[1433116800000, 1],
[1434326400000, 2],
[1434499200000, 2],
[1434931200000, 1],
[1435536000000, 1], ...
//change into the format expected by nvd3
myData.forEach(function(d){
d.values = d.values.map(function(value){
return {x:value[0], y:value[1]};
})
});
working code here

How to get the parent elements by quering the child elements in the json data.Explanation is given below

I am new to Apigee,I had tried a query to extract the value from the json data child and i need to get the parent element from the json data.
JSON is Here:
{ "Booksstall": [
{ "serialId": "10123456",
"Name": "magic"
"books": [
{ "Order": 3,
"Name": "Supermax"
}
],
"NormalserialIds": [
{"serialId": "1234556",
"Status": "InStock",
"books": [
{"type": "400001623",
"Code": "PATR"
}
]
},
{"serialId": "789101",
"Status": "OutoffStock",
"books": [
{"type": "400001623",
"Code": "NFES"
}
]
}
]
}
]
}
I tried this query in online json validator ,but if I tried same in the Apigee extract variable policy by doing substitution Its not returning me any values.
$.Booksstall[?(#.NormalserialIds[0].serialId=="1234556")].serialId
The above query result is: 10123456.
Please suggest how can I try this.

You can also use Javascript Policy to extract same. It should be simple and straight forward with Javascript Policy.
Cheers,
Anil Sagar