I have some json on a website that i want to convert to excel using the power query option from web. But I ran into a small problem. My json looks like this:
[
{
"id" : 1,
"visitors" : 26,
"some_number" : 1,
"value" : 3500
},
{
"id" : 2,
"visitors" : 21,
"some_number" : 5,
"value" : 2000
}
]
but when i use from web i get this:
I can drill down into a record,convert it to a table, transpose and use first row as header but then i get just one row. How can i get all of my data to the table and not just one row?
First I would use the List Tools / Transform menu (it should be automatically selected) and click the To Table button. This will give you a single-column table with 2 rows. Then I would click the small Expand button - it will appear in the column headings, just to the right of "Column1". Uncheck the Use original column name ... option and you will get a table of 4 columns and 2 rows.
Here's the full script I generated:
let
Source = Json.Document(File.Contents("C:\Users\Mike.Honey\Downloads\json2.json")),
#"Converted to Table" = Table.FromList(Source, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column2" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"id", "visitors", "some_number", "value"}, {"id", "visitors", "some_number", "value"})
in
#"Expanded Column2"
The Table.FromRecords() function is suitable for that sample data:
let
Source = Json.Document("[{""id"": 1, ""visitors"": 26, ""some_number"": 1, ""value"": 3500}, {""id"": 2, ""visitors"": 21, ""some_number"": 5, ""value"": 2000}]"),
AsTable = Table.FromRecords(Source)
in
AsTable
I have Excel Professional 2016 and I don't see the JSON option, but I can get to it via the "Other Sources" query option. Here are all the steps to convert a JSON array to a table.
Data > New Query > From Other Sources > From Web
Enter URL : "file:///C:/temp/document.json", (or a http web url) Press OK
A row is displayed with a "items" property and List type,
Click on "List", the items in list are displayed
Press "To Table" button in upper left corner, Press OK in next
dialog
A table with one column named "Column1" is displayed
Press the button next to the column name (has left and right arrows
on it)
Properties in the row object are selected
Uncheck "User Original column name as prefix", Press "OK"
Press "Close & Load" button in upper left corner
my file looks like
{
"items": [{
"code": "1",
"name": "first"
}, {
"code": "2",
"name": "second"
}, {
"code": "3",
"name": "third"
},
]
}
You need to convert the list to a table first, then you can expand the record column and proceed from there. If no luck, then you can take a look at this video I created recently for a similar question.
Related
I am using Big Query for as a cloud data warehouse and DataStudio for vizualisation.
In Big Query I have a table with a column named data written in JSON. I only want to extract what is inside the field "city".
This formula below that someone gave me worked to extract what is inside the field "title". I used it to create a field in DataStudio.
REPLACE(REGEXP_EXTRACT(data, '"title":(.+)","ur'), "\"", "")
So, I tried in multiple ways to reuse this formula for the "city" field, but it hasn't worked. I don't understand this code.
What's inside my column data:
{
"address":{
"city":"This is what i want",
"country":"blablabla",
"lineAdresse":"blablabla",
"region":"blablabla",
"zipCode":"blablabla"
},
"contract":"blablabla",
"dataType":"blablabla",
"description":"blablabla",
"endDate":{
"_seconds":1625841747,
"_nanoseconds":690000000
},
"entreprise":{
"denomination":"blabla",
"description":"1",
"logo":"blablabla",
"blabla":"blablabla",
"verified":"false"
},
"id":"16256R8TOUHJG",
"idEntreprise":"blablabla",
"jobType":"blablabla",
"listInfosRh":null,
"listeCandidats":[
],
"field":0,
"field":0,
"field":14,
"field":"1625834547690",
"field":true,
"field":"",
"field":"ref1625834547690",
"skills":[
"field",
"field",
"field"
],
"startDate":{
"_seconds":1625841747,
"_nanoseconds":690000000
},
"status":true,
"title":"this I can extract",
"urlRedirection":"blablabla",
"validated":true
}
If anyone knows the formula to put in Data Studio to extract what's inside city and can explain it to me, this would help a lot.
Here's the formula I tried but where I got "null" result:
REPLACE(REGEXP_EXTRACT(data,'"city":/{([^}]*)}/'),"\"","") >>null
I tried this one but it wouldn't stop at the city. I got the address, the region, zipcode and all the rest after:
REPLACE(REGEXP_EXTRACT(data, '"city":(.+)","ur'), "\"", "")
It is possible to parse a JSON in a text field by ignoring any hierarchy and only looking for a specific field. In your case the field names were title and city . Please be aware that this approach is not save for user entered data: By setting the value of the "city":"\" hide \"" the script cannot extract the city.
select *,
REGEXP_EXTRACT(data, r'"title":\s*"([^"]+)') as title,
REGEXP_EXTRACT(data, r'"city":\s*"([^"]+)') as city
from(
Select ' { "address":{ "city":"This is what i want", "country":"blablabla", "lineAdresse":"blablabla", "region":"blablabla", "zipCode":"blablabla" }, "contract":"blablabla", "dataType":"blablabla", "description":"blablabla", "endDate":{ "_seconds":1625841747, "_nanoseconds":690000000 }, "entreprise":{ "denomination":"blabla", "description":"1", "logo":"blablabla", "blabla":"blablabla", "verified":"false" }, "id":"16256R8TOUHJG", "idEntreprise":"blablabla", "jobType":"blablabla", "listInfosRh":null, "listeCandidats":[ ], "field":0, "field":0, "field":14, "field":"1625834547690", "field":true, "field":"", "field":"ref1625834547690", "skills":[ "field", "field", "field" ], "startDate":{ "_seconds":1625841747, "_nanoseconds":690000000 }, "status":true, "title":"this I can extract", "urlRedirection":"blablabla", "validated":true }' as data
)
I have a sqlite database and in one of the fields I have stored complete json object . I have to make some json select requests . If you see my json
the ALL key has value which is an array . We need to extract some data like all comments where "pod" field is fb . How to extract properly when sqlite json has value as an array ?
select json_extract(data,'$."json"') from datatable ; gives me entire thing . Then I do
select json_extract(data,'$."json"[0]') but i dont want to do it manually . i want to iterate .
kindly suggest some source where i can study and work on it .
MY JSON
{
"ALL": [{
"comments": "your site is awesome",
"pod": "passcode",
"originalDirectory": "case1"
},
{
"comments": "your channel is good",
"data": ["youTube"],
"pod": "library"
},
{
"comments": "you like everything",
"data": ["facebook"],
"pod": "fb"
},
{
"data": ["twitter"],
"pod": "tw",
"ALL": [{
"data": [{
"codeLevel": "3"
}],
"pod": "mo",
"pod2": "p"
}]
}
]
}
create table datatable ( path string , data json1 );
insert into datatable values("1" , json('<abovejson in a single line>'));
Simple List
Where your JSON represents a "simple" list of comments, you want something like:
select key, value
from datatable, json_each( datatable.data, '$.ALL' )
where json_extract( value, '$.pod' ) = 'fb' ;
which, using your sample data, returns:
2|{"comments":"you like everything","data":["facebook"],"pod":"fb"}
The use of json_each() returns a row for every element of the input JSON (datatable.data), starting at the path $.ALL (where $ is the top-level, and ALL is the name of your array: the path can be omitted if the top-level of the JSON object is required). In your case, this returns one row for each comment entry.
The fields of this row are documented at 4.13. The json_each() and json_tree() table-valued functions in the SQLite documentation: the two we're interested in are key (very roughly, the "row number") and value (the JSON for the current element). The latter will contain elements called comment and pod, etc..
Because we are only interested in elements where pod is equal to fb, we add a where clause, using json_extract() to get at pod (where $.pod is relative to value returned by the json_each function).
Nested List
If your JSON contains nested elements (something I didn't notice at first), then you need to use the json_tree() function instead of json_each(). Whereas the latter will only iterate over the immediate children of the node specified, json_tree() will descend recursively through all children from the node specified.
To give us some data to work with, I have augmented your test data with an extra element:
create table datatable ( path string , data json1 );
insert into datatable values("1" , json('
{
"ALL": [{
"comments": "your site is awesome",
"pod": "passcode",
"originalDirectory": "case1"
},
{
"comments": "your channel is good",
"data": ["youTube"],
"pod": "library"
},
{
"comments": "you like everything",
"data": ["facebook"],
"pod": "fb"
},
{
"data": ["twitter"],
"pod": "tw",
"ALL": [{
"data": [{
"codeLevel": "3"
}],
"pod": "mo",
"pod2": "p"
},
{
"comments": "inserted by TripeHound",
"data": ["facebook"],
"pod": "fb"
}]
}
]
}
'));
If we were to simply switch to using json_each(), then we see that a simple query (with no where clause) will return all elements of the source JSON:
select key, value
from datatable, json_tree( datatable.data, '$.ALL' ) limit 10 ;
ALL|[{"comments":"your site is awesome","pod":"passcode","originalDirectory":"case1"},{"comments":"your channel is good","data":["youTube"],"pod":"library"},{"comments":"you like everything","data":["facebook"],"pod":"fb"},{"data":["twitter"],"pod":"tw","ALL":[{"data":[{"codeLevel":"3"}],"pod":"mo","pod2":"p"},{"comments":"inserted by TripeHound","data":["facebook"],"pod":"fb"}]}]
0|{"comments":"your site is awesome","pod":"passcode","originalDirectory":"case1"}
comments|your site is awesome
pod|passcode
originalDirectory|case1
1|{"comments":"your channel is good","data":["youTube"],"pod":"library"}
comments|your channel is good
data|["youTube"]
0|youTube
pod|library
Because JSON objects are mixed in with simple values, we can no longer simply add where json_extract( value, '$.pod' ) = 'fb' because this produces errors when value does not represent an object. The simplest way around this is to look at the type values returned by json_each()/json_tree(): these will be the string object if the row represents a JSON object (see above documentation for other values).
Adding this to the where clause (and relying on "short-circuit evaluation" to prevent json_extract() being called on non-object rows), we get:
select key, value
from datatable, json_tree( datatable.data, '$.ALL' )
where type = 'object'
and json_extract( value, '$.pod' ) = 'fb' ;
which returns:
2|{"comments":"you like everything","data":["facebook"],"pod":"fb"}
1|{"comments":"inserted by TripeHound","data":["facebook"],"pod":"fb"}
If desired, we could use json_extract() to break apart the returned objects:
.mode column
.headers on
.width 30 15 5
select json_extract( value, '$.comments' ) as Comments,
json_extract( value, '$.data' ) as Data,
json_extract( value, '$.pod' ) as POD
from datatable, json_tree( datatable.data, '$.ALL' )
where type = 'object'
and json_extract( value, '$.pod' ) = 'fb' ;
Comments Data POD
------------------------------ --------------- -----
you like everything ["facebook"] fb
inserted by TripeHound ["facebook"] fb
Note: If your structure contained other objects, of different formats, it may not be sufficient to simply select for type = 'object': you may have to devise a more subtle filtering process.
I'm totally new to Power BI so I'm tried different approached to convert my JSON file to a table but unsuccessful so far.
{
"Family": [
{
"Father": "F1",
"Age": 50,
"Mother": "M1",
"MAge": 49,
"Children": [
{
"Name": "C1"
},
{
"Name": "C2"
}
]
},
{
"Father": "F2",
"Age": 55,
"Mother": "M2",
"MAge": 53,
"Children": [
{
"Name": "Cc1"
},
{
"Name": "Cc2"
}
]
}
]
}
I'm trying to convert this into the table below
Father Age Mother MAge
F1 50 M1 49
F2 55 M2 53
I tried like convert table and transpose which is not working I always get an error like
Expression.Error: We cannot convert a value of type Record to type
Let me share a step by step guide for those who want to do it without writing Power Query and jsut using Power BI user interface (Desktop/Web).
Open Power BI Desktop
Get Data → More ... → Select JSON from the list and click on Connect (You can also use WEB API or other sources which gives you JSON data)
Choose the JSON file and open it.
In the data panel, you see Family|List. Click on List link to add a navigation step.
From Transform tab, click on To Table and from dialog click on OK.
From the header of "Column1", click on Expand columns button to expand columns and from the menu, uncheck Use original column name as prefix and check the columns that you want and click on OK.
You will see the columns and data in table format. You can change data type of columns by click on data type button .
And finally you will have something like this:
Finally if you want to see the generated query, click on Advanced Editor button and see the code:
let
Source = Json.Document(File.Contents("C:\Users\rag\Desktop\data.json")),
Family = Source[Family],
#"Converted to Table" = Table.FromList(Family, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Father", "Age", "Mother", "MAge"}, {"Father", "Age", "Mother", "MAge"}),
#"Changed Type" = Table.TransformColumnTypes(#"Expanded Column1",{{"Age", type number}, {"MAge", type number}})
in
#"Changed Type"
More information:
Get started with Power BI Desktop
Shape and combine data in Power BI Desktop
Perform common query tasks in Power BI Desktop
I'm missing the step of ExpandRecordColumn Function. After I put that in my Query. It's worked.
let
Source = Json.Document(File.Contents("C:\Users\hp\Desktop\File.JSON")),
Family = Source[Family],
#"Converted to Table" = Table.FromList(Family, Splitter.SplitByNothing(), null, null, ExtraValues.Error),
#"Expanded Column1" = Table.ExpandRecordColumn(#"Converted to Table", "Column1", {"Father", "Age", "Mother", "MAge", "Children"}, {"Father", "Age", "Mother", "MAge", "Children"})
in
#"Expanded Column1"
I just followed this article to get data from a JSON file. When I clicked the Transform button, the Power Query engine automatically show me the final table. So I think Now Power bi(latest version 2.107. 683.0) has auto capability features to do this job for us.
Thanks.
Working with the repeating grids through the form builder.
I have a custom control that has a string value represented in json.
{
"data": {
"type": "File",
"itemID": "12345",
"name": "Annual Summary",
"parentFolderID": "fileID",
"owner": "Owner",
"lastModifiedDate": "2016-10-17 22:48:05Z"
}
}
In the controls outside of the repeating grid, i need to check if name = "Annual Summary"
Previously, i had a drop down control and using Calculated Value $dropdownControl = "Annual Summary" it was able to return true if any of the repeated rows contained the value. My understanding is that using the = operator, it will validate against all rows.
Now with the json output of the control, I am attempting to use
contains($jsonStringValue, 'Annual Summary')
However, this only works with one entry and will be null if there are multiple rows.
2 questions:
How would validate whether "Annual Summary" (or any other text) is present within any of the repeated rows?
Is there any way to navigate the json or parse it to XML and navigate it?
Constraint:
within the Calculated Value or Visibility fields within form builder
manipulating the source that is generated by the form builder
You probably want to parse the JSON string first. See also this other Stackoverflow question.
Until Orbeon Forms 2016.3 is released, you would write:
(
for $v in $jsonStringValue
return converter:jsonStringToXml($v)
)//name = 'Annual Summary'
With the above, you also need to scope the namespace:
xmlns:converter="org.orbeon.oxf.json.Converter"
Once Orbeon Forms 2016.3 is released you can switch to:
$jsonStringValue/xxf:json-to-xml()//name = 'Annual Summary'
I have a setup in angular which displays a json string called 'items'. Each item contains an array of field ids. By matching the field ids, it pulls information for the fields using a seperate 'fields' json string.
{
"data": [
{
"id": "1",
"title": "Item 1",
"fields": [
1,
1
]
},
{
"id": "2",
"title": "Item 2",
"fields": [
1,
3
]
},
{
"id": 3,
"title": "fsdfs"
}
]
}
You can copy or delete either the items or fields, which will modify the 'items' json.
Everything works except when I copy one item (and its fields), and then choose to delete a field for that specific item.
What happens is that it deletes the field for both the copied item AND the original.
Plunker -
http://plnkr.co/edit/hN8tQiBMBhQ1jwmPiZp3?p=preview
I've read that using 'track by' helps to index each item as unique and prevent duplicate keys, but this seems to be having no effect.
Any help appreciated, thanks
Edit -
Credit to Eric McCormick for this one, using angular.copy with array.push solved this issue.
Instead of -
$scope.copyItem = function (index) {
items.data.push({
id: $scope.items.data.length + 1,
title: items.data[index].title,
fields: items.data[index].fields
});
}
This worked -
$scope.copyItem = function (index) {
items.data.push(angular.copy(items.data[index]));
}
I recommend using angular.copy, which is a "deep copy" of the source object. This is a unique object from the source one.
It may seem slightly counter-intuitive, but a direct reference (as you're observing) interacts with the original object. If you inspect the element's scope after it's instantiated in the DOM, you can see there's a $id property assigned to the object in memory. Basically, by using angular.copy(source, destination), you ensure a copying of all the properties/values and having a unique object.
Example:
//inside the controller, a function to instantiate new copy of selected object
this.selectItem = function(item){
var copyOfItem = angular.copy(item);
//new item with same properties and values but unique object!
}
Egghead.io has a video on angular.copy.