I have REST call in a Copy data activity which gives me a json response
My goal is to fetch the "hasNextPage" value and put it into the hasNext variable
I want to set it as a value in a "Set variable" activity that is connected to the "Copy data" activity, where I expected to acess the output in a way like this: #activity('Timesheets').output.data.timesheets.pageinfo.hasNext
I also want to be able to fetch the value of "cursor" from the last element in the "edges" array[]
I couldn't find any documentation on how to do this
Json response that I get from the Timesheets activity
[
{
"data": {
"timesheets": {
"pageInfo": {
"hasNextPage": true
},
"edges": [
{
"cursor": "81836000243260.81836000243275.",
"node": {
"parameter1": "2019-11-04",
"parameter2": "81836000243260"
}
},
{
"cursor": "81836000243252.81836000243260.81836000243275",
"node": {
"parameter1": "2019-11-04",
"parameter2": "81836000243260"
}
}
]
}
}
}
]
According to this, the output of an copy data activity don't have a data property you can access.
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview
Copy Activity are made for copying large data, and it doesn't copy all rows in one go.
So it would not make sense to have an output dataset for a Copy Activity.
If your response from your REST service contains limited element, you can use an Web Activity to consume the REST service.
This have an output dataset you can access.
Followed by a foreach activity to iterate the data set. Remember to take into consideration parallel vs sequential iteration of you data set in the foreach activity.
Note in your service response, you get an array of "data" objects, so you need to address the first "data" element.
Related
I am syncing a data set with about 300k rows to Phonograph2 and need to make those records available via REST (End Point: /phonograph2/api/search/tables).
My requests looks as following (retrieving records after a certain timestamp):
{
"tableRids": [
"ri.phonograph2.main.table.xxxxxxxxxxxxxxx"
],
"filter": {
"type" : "range",
"range": {
"field": "reco_timestamp",
"gte": "1634408219000"
}
}
}
}
The response ends with:
"nextPageToken": "xxxxxxx"
This leads me to the following questions:
How do I use the "nextPageToken" to retrieve the next set of results?
Can the consumer get a list/array of pages to consume?
Can the number of hits which are displayed until the nextPageToken is written be configured?
We discussed this with our Palantir project support and will use Objects Gateway - as suggested. Thanks for pointing us into this direction.
I have a JSON file that I need to move to Cosmos DB. I currently have a PowerShell script that will modify this file to a proper format to be used in a Data Flow or Copy activity in Azure Data Factory. However, I was wondering if there is a way to do all these modification in Azure Data Factory without using the Powershell script.
The Powershell script can manipulate a 50MB file in a matter of seconds. I would also like a similar speeds if we build something directly in Azure Data Factory.
Without the modification, I get a error because of the "#" sign. Furthermore, if I want to use companyId as my partition key, it is not allowed because it is inside of an array.
The current JSON file looks similar to the below:
{
"Extract": {
"consumptionInfo": {
"Name": "Test Stuff",
"createdOnTimestamp": "20200101161521Z",
"Version": "1.0",
"extractType": "Incremental",
"extractDate": "20200101113514Z"
},
"company": [{
"company": {
"#action": "create",
"companyId": "xxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb",
"Status": "1",
"StatusName": "1 - Test - Calendar"
}
}]
}
}
I would like to be converted to the below:
{
"action": "create",
"companyId": "xxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb",
"Status": "1",
"StatusName": "1 - Test - Calendar"
}
Create a new data flow that reads in your JSON file. Add a Select transformation to choose the properties you wish to send to CosmosDB. If some of those properties are embedded inside of an array, then first use Flatten. You can also use the Select transformation rename "#action" to "action".
Data Factory or Data Flow doesn't works well with nested JSON file. Per my experience, the workaround may be a little complexed but works well:
Source 1 + Flatten active 1 to flat the data in key 'Extract'.
Source 2(same with source 1) + Flatten active 2 to flat the data in
key 'company'.
Add a Union active 1 in source 1 flow to join the data after
flatten active 2
create a Dervied Column to filter the column/key you want after
union active1
Then create the Azure Cosmos DB as sink.
The Data flow overview should like this:
I have a REST call that is working properly. I can pass parameters, it returns data.
The app uses TRESTClient, TRESTResponse, TRESTRequest, TRESTAdapter, feeding into TClientDataSet and TDataSource.
The end result is that when the JSON data comes in, I can iterate through it as if it was a table. With simple JSON, I can get this working.
I am now querying a REST service which is providing data that is one level deeper than normal. See JSON below.
Everything I need to access is UNDER the mycursor element, which is under the items element.
I can't change the REST service, so how can I tell one of these components to ignore the items level and look at the mycursor level?
The data I am looking to parse has a first element of id.
{
"next":
{
"$ref":"https://<internal URL>/?page=1"
},
"items":
[
{
"mycursor":
[
{
"id":13372,
…
},
{
"id":13373,
…
},
{
"id":13374,
…
},
{
"id":13375,
…
},
{
"id":13376,
…
}
]
}
]
}
Would it be possible, in any way, to create json code that zabbix can understand and recreate on a graph?
Eg:
I have this json:
{
"response:" {
"success": true,
"server": {
"name": "Test Server",
"alive": true,
"users": 25
}
}
}
And I would like to have a simple graph where I can see the value of users.
I might be asking a nonsense here but I was reading about the URL element and it looks like it is possible but couldn't find any type template or any info on how to send the data.
Create a Zabbix trapper item and send such values with the zabbix_sender. The values will be processed as any normal item values by Zabbix, and graphs will be available as well.
In my project I got a JSON response via GET request. The subTopics will be selected by the user and stored. Afterwards I send a POST request to the server with the selected ids.
Example JSON1: (from GET request)
{
"TopicList" :
[{
"id": "1234",
"name": "topic1",
"number": "1",
"subTopics": [
{
"id": "4567",
"name": "subTopic1.1",
"number": "1.1"
},
{
"id": "9876",
"name": "subTopic1.2",
"number": :1.2"
}
]
}]
}
In the POST response I get another JSON object from the server, which I have to show in my HTML view as a table. In the response JSON I have the subTopics id (selected by the user) but I do not have the subTopic name associated with the id.
I have to show the subTopic name in my table which is available in a separate object(see above JSON file). I don't know how to access the first JSON object while working with another.
My table view looks like this,
<tr ng-repeat-start="tableRow in searchCtrl.tableViewData" ng-click="tableRow.expanded = !tableRow.expanded">
<td>{{tableRow.project.name}}</td>
<td>{{tableRow.project.number}}</td>
<td>{{tableRow.project.endDate | date}}</td>
<td>{{tableRow.topicIds[0]}}</td>
<td>{{tableRow.matching.score}}</td>
</tr>
As you can see the 4th row: <td>{{tableRow.topicIds[0]}}</td> shows the id. How can I show the topicName?
Any help would be appreciable.
EDIT
In my controller this variable contains the above JSON object.
if (!self.topic) {
searchService.getTopic().then(
function (response) {
self.topic = response.data;
},
function (error) {
alert("Server is not found");
}
);
}
So, the topic variable contains the response JSON object. Maybe it will help.
You can create a function that takes an id and returns the subTopic.
$scope.getSubTopic = function(id) {
var selectedSubTopic = {};
angular.forEach(subTopics, function(subTopic) {
// loop through subTopics until a matching id is found
if (subTopic.id === id) {
selectedSubTopic = subTopic;
return;
}
});
return selectedSubTopic;
};
then you can update your fourth row to:
<td>{{getSubTopic(tableRow.topicIds[0]).name}}</td>
This assumes you have an array named subTopics.
Edit
As mentioned in my comment this will end up performing pretty slow for heavy pages and/or large datasets. You will likely want to generate a map object for the subTopics for quick access. The downside being you have to generate this each time the TopicList is modified.
function generateSubTopicMap(topics) {
var map = {};
angular.forEach(topics, function(topic) {
angular.forEach(topic.subTopics, function(subTopic) {
// use this if you want the map to reference the same data
// (i.e. updating subTopic.name will update the map at the same time)
map[subTopic.id] = subTopic;
// use this if you don't want the map to reference the same data
// map[subTopic.id] = {};
// angular.copy(subTopic, map[subTopic.id]);
// you can also add the parent id here if you need access to it
// this will modify the original object if you use the first method!
// map[subTopic.id].parentId = topic.id
});
});
return map;
}
The output looks like:
{
"4567": {
"id": "4567",
"name": "subTopic1.1",
"number": "1.1"
},
"9876": {
"id": "9876",
"name": "subTopic1.2",
"number": :1.2"
}
}
With this you would call it after every GET request and pass it the array of topics.
// where topics is the response from the GET request
$scope.subTopics = generateSubTopicMap(topics);
And finally to display you just need:
<td>{{subTopics[tableRow.topicIds[0])].name}}</td>
Edit 2
Here is a jsfiddle showing how to use the second method. All you have to do is pass the array containing your TopicList to generateSubTopicMap and it returns an object with the keys as subTopic ids and the value as the subTopic itself.
I wouldn't worry about my first solution. It isn't going to be performant inside an ng-repeat or grabbing 2nd level objects.