I am using AdminDirectory.Users.list() to pull a list of Directory attributes including some custom schema fields. Logging the output shows that all of the data is there however the custom schema fields are nested in the following format.
customSchemas={TVL={Date=1942-03-31, Vehicle_Reg=[{type=work, value=AB05CDE}, {type=work, value=FG06HIJ}]}},
How can I extract the individual values so that I can push them into a sheet?
Many thanks,
James P
Related
I want to convert this Excel file which contains two tables in a single worksheet
Into this JSON format
{
parent:
{
"P1":"x1",
"P2":"y1",
"P3":"z1"
}
children: [
{"C1":"a1", "C2":"b1", "C3":"c1", "C4":"d1"},
{"C1":"a2", "C2":"b2", "C3":"c2", "C4":"d2"},
...
]
}
And then post the JSON to a REST endpoint.
How to perform the mapping and posting to REST service?
Also, it appears that I need to sink the JSON to a physical JSON file before I can post as a payload to REST service - is this physical sink step necessary or can it be held in memory?
I cannot use Lookup activity to read in the Excel file because it is limited to 5,000 rows and 4MB.
I managed to do it in ADF, the solution is a bit long, but you can use azure functions to do it programmatically.
Here is a quick demo that i built:
the main idea is to split data, add headers as requested and then re-join data and add relevant keys like parents and children.
ADF:
added Conditional join to split data (see attached pictures).
add surrogate key for each table.
filtered first row to get red off the headers in the csv.
map children/parents' columns: renaming columns using derived column activity
added constant value in children data flow so i can aggregate by it and convert the CSV into a complex data type.
childrenArray: in a derived column,added subcolumn to a new column named Children and in values i added relevant columns.
aggregated children Jsons by using the constant value.
in parents dataFlow: after mapping columns , i created jsons using derived column.(please see attached pictures).
joined the children array and parents jsons into one table so it will be converted to the requested Json.
wrote to cached sink(here you can do the post request instead of writing to sink).
DataFlow:
![enter image description here
Activities:
Conditional Split:
AddSurrogateKey:
(it's the same for parents data flow just change the name of incoming stream as shown in dataflow above)
FilterFirstRow:
MapChildrenColumns:
MapParentColumns:
AddConstantValue:
PartentsJson:
Here i added subcolumn in Expression Builder and sent column name as value,this will build the parents json.
ChildrenArray:
Again in a derived column, added column with a name "children"
and in Expression Builder i added relevant columns.
Aggregate:
the purpose of this activity is to aggregate children Json's and build the array, without it you will not get an array.
the aggregation function is collect().
Join Activity:
Here i added an outer join to join the parents json and the children array.
Select Relevant columns:
Output:
Not sure how to ask this but here goes. I have a collection of 500+ JSON files that I need to import into PowerBI. Each JSON has four different levels of information that I need to parse out. I converted the JSON top-level info into a table and transposed it so I had one row like the attached screenshot.
enter image description here
My first question is: can I easily add the filename to the JSON record? I would like to use the filename as a unique key in later queries.
Thanks!
It looks like you may be connecting to each JSON file individually? If I'm correct, assuming all the JSON files can be in a single folder, you can use the "Folder" connection. That then allows you to right-click on the original folder query and choose "reference" to then create various transformations for each JSON file, and it includes the file name.
Related details:
https://powerbi.tips/2016/06/loading-data-from-folder/
https://learn.microsoft.com/en-us/power-bi/guidance/power-query-referenced-queries
Hoping that helps!
I'm new to Azure Data Factory. I've been able to generate a set of JSON files from a REST API source using a Pipeline. Each file consists of one top level JSON object with an array of up to 100 child objects. The output is saved to an Azure Blob Storage container.
I now want to use a Mapping Data Flow to modify the JSON before I write it to Azure SQL, however I'm struggling with the syntax. I've configured the source to point to the directory containing the JSON files. The Source Projection tab displays the correct schema. I can preview the data and I see a row for each file and I can expand the child objects to see the full structure.
However, when I add a Derived Column action, the Input Schema is blank in the Expression Builder. I can refer to the top level elements in the source using the byName and byPosition functions, but I don't know how I can reference the child elements.
The examples that I have been able to find online use a SQL table or CSV file as a source. I can't find any examples that use hierarchical data as the source for a derived column.
Am I missing something? Is this scenario supported?
I found a way to achieve what I want. This may not be the best approach, but it works.
It seems that it is difficult to deal with JSON that has multiple hierarchies as a source for copy data activities. You can choose one level of repeating data to map to a table structure (the Collection Reference property on the Mapping tab).
In my scenario, there was additional repeating data within the data I was mapping to my table. I updated the mapping to write the child JSON data to a text field in my SQL table. To do this, I needed to use the Azure Data Factory JSON editor for my pipeline. You can access this from the "Code" link in the top right corner of the pipeline visual editor.
I added the following line after the closing bracket for the "mappings" array for my copy activity:
"mapComplexValuesToString": true
The full path to the mapping array in the activity definition is typeProperties - translator - mappings. Make sure your commas are correct after you add the new element.
With this approach, I had a row in my SQL table for each array item in my Collection Reference. The scalar child elements in the array items are mapped to table columns and the child JSON element is written to a data column in the same table.
To extract the values I need within the child JSON, I created a SQL view that uses the CROSS APPLY OPENJSON syntax. This allows me to treat the JSON in the data field similar to a related table. You can specify the structure that your JSON is in. If you have nested data in your JSON, you can apply the same approach for each level.
The OPENJSON command is only supported by more recent versions of SQL Server. I'm using Azure SQL, so that works for me.
i want to update properties in the relationship by import the csv data in Neo4j.
i have created some labels and relationships from csv data like this:
node,name
1,leo
2,sun
3,wang
4,hi
now i would like to add a property "descripe" in the table.
descripe
leader
pro-leader
crew
crew
how can i add this property into the graph? Just only this new property, i dont want to create the new four labels.
THX
It would be useful if you posted the cypher statements that you used to perform your initial graph creation. Without this it is difficult to know exactly how the update should be written. Essentially though you want to load your csv file containing the new properties, perform a MATCH on the nodes in the graph and then set the properties on these nodes. Assuming that you have created a schema index against your labels it would be something like:
LOAD CSV WITH HEADERS FROM "file:///C:/temp/myfile.csv" AS csvLine
MATCH (n:`Label` { indexedproperty : csvLine.value })
SET n.newproperty = csvLine.newpropertyvalue
where Label is the label you applied to your nodes on creation, indexedproperty is the name of the property you added and indexed, csvLine.value is the lookup value of the indexed property read from the .csv file, csvLine.newpropertyvalue is the new property you wish to add (read from the .csv file).
If you post more details on your graph we can help more precisely.
I'm working on a C# program that retrieves data from a ServiceNow database and converts that data into C# .NET objects. I'm using the JSON Web Service to return my data in JSON format.
What I want to achieve is as follows: If there is a relational mapping between a value (for
example: I have a table called Company, where CEO is not a TEXT field but an sys_id to a Employee Table) I want to be able to output that data not with an sys_id (or just displaying the name property by using the 'displayvariable' parameter) but by an object displayed in JSON.
This means that the value of a property should be an object in JSON instead of just a single value.
A few examples:
// I don't want the JSON like this
{"Company":{"CEO":"b181e841c9212c008aeb36850331fab2"}}
// Or by displaying the name of the sys_id table
{"Company":{"CEO":"James Henderson" }}
// I want the data as follows, so I can have all the data I need inside a single JSON record.
{"Company":{"CEO":{"name":"James Henderson", "age":34, "sex":"male", "office":"SBN Left Floor 23"}}}
From reading the documentation I couldn't find anything in the JSON Web Service that allowed me to display the information like this nor
find any other alternative. It should have something to do with joining the tables and displaying it all in the right format.
I have been using SNC for almost three years and have not found you can automatically join tables in a web service. Your best option would be to use a scripted web service which possibly takes a query parameter and table parameter. Then you can json serialized your result as you see fit.
Or, another option would be to generate a new processor that will traverse the GlideRecord object. The ?JSON parameter you pass in to the URL is merely a flag to pass your request to a particular processor. Unfortunately the OOB one I believe is a Java class not a JS script, so you would need to write a script much like I mentioned earlier to traverse the object path serializing the object graph as far down as your want to go.