I'm trying to build a chart from live json data but the format has the date data as the key for other nested data, the essential structure is:
{
"Meta": {
"Frequency": "Monthly",
"Time zone": "EST"
},
"Monthly data": {
"2022-01-01": {
"valueA": "13.44",
"valueB": "97.68"
},
"2021-12-01": {
"valueA": "11.39",
"valueB": "99.81"
},
"2021-11-01": {
"valueA": "12.31",
"valueB: "93.56"
},
"2022-10-01": {
"valueA": "15.42",
"valueB": "98.54"
}
}}
I'm trying to get into a format where i can encode the date on the x-axis, but can't as there is no column name/key and it isn't an array so cannot flatten. Then would be looking to encode any normal nested field (eg valueA) on the y-axis, but can't find any documentation or examples on this specific case. I don't need any information in the metadata part so can be ignored.
The only similar comment on this I've found is https://github.com/vega/vega/issues/1360, I can get this to work as a test but typing out each monthly date to map of course removes the benefit of live data and wouldn't allow high-frequency data.
Is there a way for this to be mapped automatically?
No, I don't believe this is currently possible. You could potentially do it with the flatten transform if there were a vega expression for extracting keys from an object, but support for this has not yet been added to vega expressions (see https://github.com/vega/vega/issues/3207 for a similar request).
Unfortunately, I believe the best you can do currently is use the fold transform while specifying the keys explicitly. For example:
{
"data": {
"values": {
"2022-01-01": {"valueA": "13.44", "valueB": "97.68"},
"2021-12-01": {"valueA": "11.39", "valueB": "99.81"},
"2021-11-01": {"valueA": "12.31", "valueB": "93.56"},
"2022-10-01": {"valueA": "15.42", "valueB": "98.54"}
}
},
"transform": [
{
"fold": ["2021-11-01", "2021-12-01", "2022-01-01", "2022-10-01"],
"as": ["date", "value"]
},
{"calculate": "datum.value.valueA", "as": "valueA"},
{"calculate": "datum.value.valueB", "as": "valueB"}
],
"mark": "bar",
"encoding": {
"y": {"field": "date"},
"x": {"field": "valueA"},
"x2": {"field": "valueB"}
}
}
Related
I'm trying to calculate a value for domainMax on the Y-axis scale. I tried the following example where I want the Y-axis domainMax to be one greater than the maximum value in the dataset field named "value". The example produces the error 'Unrecognized signal name: "domMax"'. How can I get it to work?
{
"data": {
"values": [
{"date": "2021-03-01T00:00:00", "value": 1},
{"date": "2021-04-01T00:00:00", "value": 3},
{"date": "2021-05-01T00:00:00", "value": 2}
]
},
"transform": [
{ "calculate": "max(datum.value)+1","as": "domMax"}
],
"mark": "line",
"encoding": {
"x": {
"field": "date",
"type": "temporal"
},
"y": {"field": "value", "type": "quantitative",
"scale": {"domainMax": {"expr": "domMax"}}
}
}
}
This transform
"transform": [
{ "calculate": "max(datum.value)+1","as": "domMax"}
]
adds a new column to your data set - it does not create a new signal. You can check that in the editor. Go to the DataViewer tab and select data_0 from the drop down. Can you see the new domMax column?
Signals are a different thing entirely - have a look here in the documentation. Note that the link points to Vega, not Vega-Lite. (Vega-Lite specifications are compiled to Vega.)
Vega-Lite does not let you declare signals; you declare parameters instead. Here is another example using the domMax parameter. Vega-Lite parameters are translated to Vega signals.
It looks like you are trying to derive the value of your parameter/signal from the data. I am not sure you can do that in Vega-Lite.
On the other hand it's very easy in Vega. For example you could use the extent transform:
https://vega.github.io/vega/docs/transforms/extent/
Side comment - while Vega specifications are more verbose you can sometimes find their primitives simpler and a good way to understand how the visualisation works. (You can see compiled Vega in the editor.)
I tried to get a custom domain based on the data but hit the same limitations as you did.
In my case, I update the data from the outside a bit like the streaming example. I compute the domain from the outside and modify them in the visualization with params. This is quite easy as vega-lite params are exposed as vega signals.
This is the gist of the layout:
{
"$schema": "https://vega.github.io/schema/vega-lite/v5.json",
"params": [
{
"name": "lowBound",
"value": -10
},
{
"name": "highBound",
"value": 100
}
],
../..
"vconcat": [
{
"name": "detailed",
../..
"layer": [
{
../..
"mark": "line",
"encoding": {
"y": {
"field": "value",
"title": "Temperature",
"type": "quantitative",
"scale": {
"domainMin": {
"expr": "lowBound"
},
"domainMax": {
"expr": "highBound"
}
}
},
...
The lowBound and highBound are dynamically changed through vega signals. I change them with the regular JS API.
You can add a param to pan and zoom in case your hard coded values are less than ideal.
"params": [{"name": "grid", "select": "interval", "bind": "scales"}],
Open the Chart in the Vega Editor
Can anyone please help me a JOLT spec to convert my matrix type json to denormalized json. Please find the below my input json and my expected josn output.
Input Json:
[
{
"attributes": [
{
"name": "brand",
"value": "Patriot Lighting"
},
{
"name": "color",
"value": "Chrome"
},
{
"name": "price",
"value": "49.97 USD"
}
]
},
{
"attributes": [
{
"name": "brand",
"value": "Masterforce"
},
{
"name": "color",
"value": "Green"
},
{
"name": "price",
"value": "99.0 USD"
}
]
}
]
Expected Json output:
[
{
"brand": "Patriot Lighting",
"color": "Chrome",
"price": "49.97 USD"
},
{
"brand": "Masterforce",
"color": "Green",
"price": "99.0 USD"
}
]
I was trying to build JOLT spec to convert this json. But challenge is the json which I have multiple tables with "attributes" tag.
Thanks in advance!
JOLT is not easy to use but I get a lot out of some other StackOverflow questions floating around and I just started reading up on the source code comments
[
{
"operation": "shift",
"spec": {
// for each element in the array
"*": {
"attributes": {
// for each element in the attribute
"*": {
// grab the value
// - put it in an array
// - but it must be indexed by the "positions" found four steps back
// - put the value in a key
// - that is determined by moving one step back and looking at member name
"value": "[#4].#(1,name)"
}
}
}
}
}
]
This is seems very obscure at first glance but I hope the comments explain everything.
Please go read on JOLT transformation to copy single value along an array
Also this is almost mandatory for JOLT beginners https://docs.google.com/presentation/d/1sAiuiFC4Lzz4-064sg1p8EQt2ev0o442MfEbvrpD1ls/edit#slide=id.g9a487080_011
If you need another example, I just answered a question here Nifi JOLT: flat JSON object to a list of JSON object
And, probably your best friend can be found at https://jolt-demo.appspot.com
The "line chart" demo works fine, and is expressed in javascript, as in getting_started.html.
... but when I replace "data" field from URL to inline values, it shows only the axys, no chart into.
It works: "data":{"url":"https://vega.github.io/vega-lite/examples/data/stocks.csv"},
It not works:
var vlSpec = {
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
"description": "Google's stock price over time.",
"data": {"values": [
{"symbol":"MSFT","date":"Jan 1 2000","price":39.81},
{"symbol":"MSFT","date":"Feb 1 2000","price":36.35},
{"symbol":"MSFT","date":"Mar 1 2000","price":43.22},
{"symbol":"MSFT","date":"Apr 1 2000","price":28.37},
{"symbol":"MSFT","date":"May 1 2000","price":25.45}
]},
"transform": [{"filter": "datum.symbol==='GOOG'"}],
"mark": "line",
"encoding": {
"x": {"field": "date", "type": "temporal"},
"y": {"field": "price", "type": "quantitative"}
}
};
There are an error at console, that not make sense:
WARN Infinite extent for field "date": [Infinity, -Infinity]
Your specification only contains rows with symbol="MSFT". It also contains a filter transform, {"filter": "datum.symbol==='GOOG'"}, which removes all rows except those with symbol="GOOG". The result is there is no data left to plot.
The warning you see comes from the fact that axis extent is determined from the data, and when there is no data, the extent is left at [-Infinity, Infinity].
If you remove or modify this filter statement, the chart will work. For example:
vlSpec = {
"$schema": "https://vega.github.io/schema/vega-lite/v4.json",
"description": "Google's stock price over time.",
"data": {"values": [
{"symbol":"MSFT","date":"Jan 1 2000","price":39.81},
{"symbol":"MSFT","date":"Feb 1 2000","price":36.35},
{"symbol":"MSFT","date":"Mar 1 2000","price":43.22},
{"symbol":"MSFT","date":"Apr 1 2000","price":28.37},
{"symbol":"MSFT","date":"May 1 2000","price":25.45}
]},
"transform": [{"filter": "datum.symbol==='MSFT'"}],
"mark": "line",
"encoding": {
"x": {"field": "date", "type": "temporal"},
"y": {"field": "price", "type": "quantitative"}
}
};
The result looks like this (Open the Chart in the Vega Editor):
Need only to comment the "magic line"
// "transform": [{"filter": "datum.symbol==='GOOG'"}],
Please explain! This is a Wiki answer
According to the Vega docs for the Date type, the date needs to be a valid Javascript Date object or timestamp.
A valid JavaScript Date object or timestamp. As JSON does not support date values natively, within a Vega specification a date-time value can be expressed either as a numeric timestamp (the number of milliseconds since the UNIX epoch, as produced by the Date.getTime() method) or using a signal expression (such as {"signal": "datetime(2001, 2, 3)"})
I can't find how to define a device where its entity contains nested attributes.
The need is to pass gps coordinates from a device to an entity having geo:json (like urn:ngsi-ld:Shelf:unit001 in tutorial https://github.com/FIWARE/tutorials.Entity-Relationships)
Following the https://fiware-tutorials.readthedocs.io/en/latest/iot-over-mqtt/index.html the below
curl -iX POST \
'http://localhost:4041/iot/devices' \
...
-d '{
"devices": [
{
"device_id": "motion001",
"entity_name": "urn:ngsi-ld:Motion:001",
{ "object_id": "c", "name": "count", "type": "Integer" }
],
"static_attributes": [
...
creates the count attribute
{
"id": "Motion:001",
"type": "Motion",
"TimeInstant": {
"type": "DateTime",
"value": "2020-04-04T07:52:29.00Z",
"metadata": {}
},
...
"count": {
"type": "Integer",
"value": "12",
"metadata": {
"TimeInstant": {
"type": "DateTime",
"value": "2020-04-04T07:52:29.00Z"
}
}
},
How to replace the count above in order to get the below outcome ?
"type": "geo:json",
"value": {
"type": "Point",
"coordinates": [13.3986,52.5547]
},
In all examples and documentation the attributes are defined in one level (not nested) https://iotagent-node-lib.readthedocs.io/en/latest/api/index.html.
Tses
Device measures are usually just numbers so for structured elements you either use the expression library or pass data as structured JSON using the IoT Agent for JSON.
That being said, improved GeoJSON support has recently been added to the IoT Agent library, so the following will work on *:latest IoT Agent docker images. According to the Documentation various input formats are now possible.
The defined type of any GeoJSON attribute can be any set to any of
the standard NGSI-v2 GeoJSON types - (e.g. geo:json, geo:point).
NGSI-LD formats such as GeoProperty, Point and LineString are
also accepted type values. If the latitude and longitude are
received as separate measures, the expression
language can be used to concatenate them.
"entity_type": "GPS",
"resource": "/iot/d",
"protocol": "PDI-IoTA-JSON", ..etc
"attributes": [
{
"name": "location",
"type": "geo:json",
"expression": "${#lng}, ${#lat}"
}
] }
For attributes and static_attributes which need to be formatted as
GeoJSON values, three separate input formats are accepted. Provided
the type is provisioned correctly, the value may be defined using
any of the following formats:
a comma delimited string
{ "name": "location", "value": "23, 12.5" }
an array of numbers
"name": "location",
"value": [23, 12.5]
}
an fully formatted GeoJSON object
{
"name": "location",
"value": {
"type": "Point",
"coordinates": [23, 12.5]
}
}
So if your device can send a comma separated long/lat string you should be fine.
I am trying to filter some data from the geoJSON data structure shown as below:
"features": [
{
"type": "Feature",
"properties": {
"#id": "node/7071544593",
"addr:city": "Joensuu",
"addr:housenumber": "12",
"addr:postcode": "80100",
"addr:street": "Siltakatu",
"addr:unit": "C 33",
"alt_name": "Crasman Oy Joensuu",
"alt_name_1": "Crasman Oy",
"name": "Crasman Joensuu",
"short_name": "Crasman",
"website": "https://www.crasman.fi"
},
"geometry": {
"type": "Point",
"coordinates": [
29.7621398,
62.6015236
]
},
"id": "node/7071544593"
},
{
"type": "Feature",
"properties": {
"#id": "node/7117872562",
"amenity": "car_rental",
"operator": "avis"
},
"geometry": {
"type": "Point",
"coordinates": [
29.7630643,
62.6036656
]
},
"id": "node/7117872562"
}
]
What I am trying to do is iterate through this array of features, look into the properties object to check if it contains website, if Yes, then I can print its coordinates from geometry object.
This is what I tried:
Features[*].properties[?contains(#,'website')=='true'].geometry.coordinates
It gives me null value
Try this:
features[?contains(keys(properties),'website')].geometry.coordinates
E.g.:
$ jp "features[?contains(keys(properties),'website')].geometry.coordinates" <input.json
[
[
29.7621398,
62.6015236
]
]
With regard to why your example didn't work:
Identifiers are case-sensitive, so you need features, not Features.
properties is an object, not an array, so you can't apply a filter expression to it.
Even if you could, it's not properties that you want to filter. You are trying to filter whole features.
contains tests if an array contains an item (or if a string contains a substring), not whether an object has a key. You can use keys() to get the keys of an object in an array.
You don't need to compare the result of contains() to true, it's already a boolean.
Even if you were trying to compare to true, you'd need to use backticks: `true`, not quotes 'true'.