Would it be possible, in any way, to create json code that zabbix can understand and recreate on a graph?
Eg:
I have this json:
{
"response:" {
"success": true,
"server": {
"name": "Test Server",
"alive": true,
"users": 25
}
}
}
And I would like to have a simple graph where I can see the value of users.
I might be asking a nonsense here but I was reading about the URL element and it looks like it is possible but couldn't find any type template or any info on how to send the data.
Create a Zabbix trapper item and send such values with the zabbix_sender. The values will be processed as any normal item values by Zabbix, and graphs will be available as well.
Related
I have a JSON file that I need to move to Cosmos DB. I currently have a PowerShell script that will modify this file to a proper format to be used in a Data Flow or Copy activity in Azure Data Factory. However, I was wondering if there is a way to do all these modification in Azure Data Factory without using the Powershell script.
The Powershell script can manipulate a 50MB file in a matter of seconds. I would also like a similar speeds if we build something directly in Azure Data Factory.
Without the modification, I get a error because of the "#" sign. Furthermore, if I want to use companyId as my partition key, it is not allowed because it is inside of an array.
The current JSON file looks similar to the below:
{
"Extract": {
"consumptionInfo": {
"Name": "Test Stuff",
"createdOnTimestamp": "20200101161521Z",
"Version": "1.0",
"extractType": "Incremental",
"extractDate": "20200101113514Z"
},
"company": [{
"company": {
"#action": "create",
"companyId": "xxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb",
"Status": "1",
"StatusName": "1 - Test - Calendar"
}
}]
}
}
I would like to be converted to the below:
{
"action": "create",
"companyId": "xxxxxxx-yyyy-zzzz-aaaa-bbbbbbbbbbbb",
"Status": "1",
"StatusName": "1 - Test - Calendar"
}
Create a new data flow that reads in your JSON file. Add a Select transformation to choose the properties you wish to send to CosmosDB. If some of those properties are embedded inside of an array, then first use Flatten. You can also use the Select transformation rename "#action" to "action".
Data Factory or Data Flow doesn't works well with nested JSON file. Per my experience, the workaround may be a little complexed but works well:
Source 1 + Flatten active 1 to flat the data in key 'Extract'.
Source 2(same with source 1) + Flatten active 2 to flat the data in
key 'company'.
Add a Union active 1 in source 1 flow to join the data after
flatten active 2
create a Dervied Column to filter the column/key you want after
union active1
Then create the Azure Cosmos DB as sink.
The Data flow overview should like this:
I have a requirement to convert the json into csv(or a SQL table) or any other flatten structure using Data Flow in Azure Data Factory. I need to take the property names at some hierarchy and values of the child properties at lower of hierrarchy from the source json and add them both as column/row values in csv or any other flatten structure.
Source Data Rules/Constraints :
Parent level data property names will change dynamically (e.g. ABCDataPoints,CementUse, CoalUse, ABCUseIndicators names are dynamic)
The hierarchy always remains same as in below sample json.
I need some help in defining Json path/expression to get the names ABCDataPoints,CementUse, CoalUse, ABCUseIndicators etc. I am able to figure out how to retrieve the values for the properties Value,ValueDate,ValueScore,AsReported.
Source Data Structure :
{
"ABCDataPoints": {
"CementUse": {
"Value": null,
"ValueDate": null,
"ValueScore": null,
"AsReported": [],
"Sources": []
},
"CoalUse": {
"Value": null,
"ValueDate": null,
"AsReported": [],
"Sources": []
}
},
"ABCUseIndicators": {
"EnvironmentalControversies": {
"Value": false,
"ValueDate": "2021-03-06T23:22:49.870Z"
},
"RenewableEnergyUseRatio": {
"Value": null,
"ValueDate": null,
"ValueScore": null
}
},
"XYZDataPoints": {
"AccountingControversiesCount": {
"Value": null,
"ValueDate": null,
"AsReported": [],
"Sources": []
},
"AdvanceNotices": {
"Value": null,
"ValueDate": null,
"Sources": []
}
},
"XYXIndicators": {
"AccountingControversies": {
"Value": false,
"ValueDate": "2021-03-06T23:22:49.870Z"
},
"AntiTakeoverDevicesAboveTwo": {
"Value": 4,
"ValueDate": "2021-03-06T23:22:49.870Z",
"ValueScore": "0.8351945854483925"
}
}
}
Expected Flatten structure
Background:
After having multiple calls with ADF experts at Microsoft(Our workplace have Microsoft/Azure partnership), they concluded this is not possible with out of the box activities provided by ADF as is, neither by Dataflow(need not to use data flow though) nor Flatten feature. Reasons are Dataflow/Flatten only unroll the Array objects and there are no mapping functions available to pick the property names - Custom expression are in internal beta testing and will in PA in near future.
Conclusion/Solution:
We concluded with an agreement based on calls with Microsoft emps ended up to go multiple approaches but both needs the custom code - with out custom code this is not possible by using out of box activities.
Solution-1 : Use some code to flatten as per requirement using a ADF Custom Activity. The downside of this you need to use an external compute(VM/Batch), the options supported are not on-demand. So it is little bit expensive but works best if have continuous stream workloads. This approach also continuously monitor if input sources are of different sizes because the compute needs to be elastic in this case or else you will get out of memory exceptions.
Solution-2 : Still needs to write the custom code - but in a function app.
Create a Copy Activity with source as the files with Json content(preferably storage account).
Use target as Rest Endpoint of function(Not as a function activity because it has 90sec timeout when called from an ADF activity)
The function app will takes Json lines as input and parse and flatten.
If you use the above way so you can scale the number of lines cane be send in each request to function and also scale the parallel requests.
The function will do the flatten as required to one file or multiple files and store in blob storage.
The pipeline will continue from there as needed from there.
One problem with this approach is if any of the range is failed the copy activity will retry but it will run the whole process again.
Trying something very similar, is there any other / native solution to address this?
As mentioned in the response above, has this been GA yet? If yes, any reference documentation / samples would be of great help!
Custom expression are in internal beta testing and will in PA in near future.
I have REST call in a Copy data activity which gives me a json response
My goal is to fetch the "hasNextPage" value and put it into the hasNext variable
I want to set it as a value in a "Set variable" activity that is connected to the "Copy data" activity, where I expected to acess the output in a way like this: #activity('Timesheets').output.data.timesheets.pageinfo.hasNext
I also want to be able to fetch the value of "cursor" from the last element in the "edges" array[]
I couldn't find any documentation on how to do this
Json response that I get from the Timesheets activity
[
{
"data": {
"timesheets": {
"pageInfo": {
"hasNextPage": true
},
"edges": [
{
"cursor": "81836000243260.81836000243275.",
"node": {
"parameter1": "2019-11-04",
"parameter2": "81836000243260"
}
},
{
"cursor": "81836000243252.81836000243260.81836000243275",
"node": {
"parameter1": "2019-11-04",
"parameter2": "81836000243260"
}
}
]
}
}
}
]
According to this, the output of an copy data activity don't have a data property you can access.
https://learn.microsoft.com/en-us/azure/data-factory/copy-activity-overview
Copy Activity are made for copying large data, and it doesn't copy all rows in one go.
So it would not make sense to have an output dataset for a Copy Activity.
If your response from your REST service contains limited element, you can use an Web Activity to consume the REST service.
This have an output dataset you can access.
Followed by a foreach activity to iterate the data set. Remember to take into consideration parallel vs sequential iteration of you data set in the foreach activity.
Note in your service response, you get an array of "data" objects, so you need to address the first "data" element.
Hi I would like to create html tabe in Azure Logic App.
I have data loaded from blob via Blob connector (Get blob content using path)
I used Compose connector based on answer in this post
But I get error -
Unable to process template language expressions in action 'Create_HTML_table' inputs at line '1' and column '1747': 'The template language function 'json' parameter is not valid. The provided value '[{"ServiceName":"routingsf","SubServiceName":"roadinfo/supportedmaps","ErrorType":"System.AggregateException","ErrorMessage":"One or more errors occurred. (Object reference not set to an instance of an object.)","Count":4} ]' cannot be parsed: 'Unexpected character encountered while parsing value: . Path '', line 0, position 0.'. Please see https://aka.ms/logicexpressions#json for usage details.'.
Code of Compose and Create HTML table connector is like :
"Compose": {
"inputs": "#base64ToString(body('Get_blob_content_using_path').$content)",
"runAfter": {
"Get_blob_content_using_path_2": [
"Succeeded"
]
},
"type": "Compose"
},
"Create_HTML_table": {
"inputs": {
"format": "HTML",
"from": "#json(outputs('Compose'))"
},
"runAfter": {
"Compose": [
"Succeeded"
]
},
"type": "Table"
},
Can you help?
Maybe you could try my way to create table. After get the content, use the Parse JSON , the content input json(body('Get_blob_content_using_path')), the Scheme choose Use sample payload to generate schema and just copy and paste your json file content.
Then create HTML table , the From choose the expression array(body('Parse_JSON')).
Here is the Logic flow and my result.
Hope this could help you, if you still have other questions, please let me know.
Update: I copy your json content tomy json file and test again. And It works.
So please make sure you logic app flow is right. Or you could share you flow.
Make sure these two steps are right:
I'm having trouble properly formatting one particular soap parameter using the node-soap module for node.js as a client, to a 3rd-party SOAP service.
The client.describe() for this method says this particular input should be in the shape of:
params: { 'param[]': {} }
I have tried a bunch of different JSON notations to try to fit my data to that shape.
Examples of formats that do NOT work:
"params": { "param": [ {"myParameterName": "myParameterValue"} ] }
"params": [ "param": { "name": "myParameterName", "_": "myParameterValue"} ]
"params": { "param" : [ {"name": "myParameterName", "_": "myParameterValue"} ] }
"params": { "param[]": {"myParameterName": "myParameterValue" } }
"params": { "param[myParameterName]": {"_": "myParameterValue" } }
I must be overlooking something, and I suspect I'm going to feel like Captain Obvious when some nice person points out what I'm doing wrong.
Here is what DOES work, using other soap clients, and how they handle the "named parameter with a value"
soapUI for this method successfully accepts this particular input via XML in the shape of:
<ns:params>
<ns:param name="myParameterName">myParameterValue</ns:param>
</ns:params>
Also, using PHP, I can successfully make the call by creating a stdClass of arrays like so:
$parms = new stdClass;
$parms->param = array(
array(
"name"=>"myParameterName","_"=>"myParameterValue"
)
);
and then eventually passing
'params' => $parms
to the PHP soap client
Many thanks!
To get a better look at what XML was being generated by node-soap, I added a console.log(message) statement to the node_modules/soap/lib/client.js after the object-to-XML encoding. I then began experimenting with various JSON structures to figure out empirically how they were mapping to XML structures.
I found a JSON structure for node-soap to generate the XML in my 3rd-party's required named-parameter-with-value format. I was completely unaware of the "$value" special keyword. Looks like this may have been added in the 0.4.6 release from mid-June 2014. See the change history
"params": [
{
"param": {
"attributes": {
"name": "myParameterName"
},
$value: "myParameterValue"
}
}
]
(note the outer array, which gives me the luxury of specifying multiple "param" entries, which is sometimes needed by this particular 3rd-party API)
generates this XML:
<tns:params>
<tns:param name="myParameterName">myParameterValue</tns:param>
</tns:params>
which perfectly matches the structure in soapUI (which I already knew worked) of:
<ns:params>
<ns:param name="myParameterName">myParameterValue</ns:param>
</ns:params>