I'm using Head plugin for ELASTICSEARCH for running queries.
I want to convert in a table the output of the query.
The part that I need is just the "hits" object array
where the columns are the fields that I have specified into the query:
"http.date","src_shift","#timestamp","src_tz".
is there any tool or plugin to do that?
below a brief output of query:
"took": 2418,
"timed_out": false,
"_shards": {
"total": 3503,
"successful": 3503,
"failed": 0
},
"hits": {
"total": 2524,"max_score": 9.194927,"hits": [
{
"_index": "$002555","_type": "pcap","_id": "AVAJJphp2MeWtoWCbQYG","_score": 9.194927,"fields": {
"src_shift": [
1],"http.date": [
"Fri, 12 Jun 2015 22:40:54 GMT"],"#timestamp": [
1434147980397],"src_tz": [
"Europe/Warsaw"]}},{
"_index": "$002555","_type": "pcap","_id": "AVAJJphp2MeWtoWCbQYH","_score": 9.194927,"fields": {
"src_shift": [
1],"http.date": [
"Fri, 12 Jun 2015 22:40:54 GMT"],"#timestamp": [
1434147980397],"src_tz": [
"Europe/Warsaw"]}},...
In the head plugin, on your Any Request tab, you can use the Result Transformer section located just below the Query section. By default it returns the whole JSON response.
You can modify that and massage the response to return whatever you want. In your case, if you replace the default return root; by the code below, you'll get what you want:
return root.hits.hits.map(function(hit) {
var values = [];
for (var field in hit.fields) {
values.push(hit.fields[field]);
}
return values.join(",");
});
The output should be
1,"Fri, 12 Jun 2015 22:40:54 GMT",1434147980397,"Europe/Warsaw"
1,"Fri, 12 Jun 2015 22:40:54 GMT",1434147980397,"Europe/Warsaw"
...
There's a utility in Kibana called tabify that converts ElasticSearch results into tabular form. You can find its implementation here: https://github.com/elastic/kibana/blob/master/src/ui/public/agg_response/tabify/tabify.js
Related
I have used REST to get data from API and the format of JSON output that contains arrays. When I am trying to copy the JSON as it is using copy activity to BLOB, I am only getting first object data and the rest is ignored.
In the documentation is says we can copy JSON as is by skipping schema section on both dataset and copy activity. I followed the same and I am the getting the output as below.
https://learn.microsoft.com/en-us/azure/data-factory/connector-rest#export-json-response-as-is
Tried copy activity without schema, using the header as first row and output files to BLOB as .json and .txt
Sample REST output:
{
"totalPages": 500,
"firstPage": true,
"lastPage": false,
"numberOfElements": 50,
"number": 0,
"totalElements": 636,
"columns": {
"dimension": {
"id": "variables/page",
"type": "string"
},
"columnIds": [
"0"
]
},
"rows": [
{
"itemId": "1234",
"value": "home",
"data": [
65
]
},
{
"itemId": "1235",
"value": "category",
"data": [
92
]
},
],
"summaryData": {
"totals": [
157
],
"col-max": [
123
],
"col-min": [
1
]
}
}
BLOB Output as the text is below: which is only first object data
totalPages,firstPage,lastPage,numberOfElements,number,totalElements
500,True,False,50,0,636
If you want to write the JSON response as is, you can use an HTTP connector. However, please note that the HTTP connector doesn't support pagination.
If you want to keep using the REST connector and to write a csv file as output, can you please specify how you want the nested objects and arrays to be written ?
In csv files, we can not write arrays. You could always use a custom activity or an azure function activity to call the REST API, parse it the way you want and write to a csv file.
Hope this helps.
My JSON response has multiple values for a single attribute, as copied below.
{
"version": "10.2.2.48",
"systemMessages": [ {
"code": -8010,
"error": "",
"type": "error",
"module": "BROKER"
}],
"locations": [ {
"id": "10101102",
"name": "Bus Stop",
"disassembledName": "Bus",
"coord": [
3755258,
4889121
],
"type": "stop",
"matchQuality": 1000,
"isBest": true,
"modes": [
1,
5,
11
],
"parent": {
"id": "95301001|1",
"name": "Sydney",
"type": "locality"
},
"assignedStops": [ {
"id": "10101102",
"name": "Bus Stop",
"type": "stop",
"coord": [
3755258,
4889121
],
"parent": {
"name": "Sydney",
"type": "locality"
},
"modes": [
1,
5,
11
],
"connectingMode": 100
}]
}]
}
Observe that "modes" has 3 values. If I try to extract the value of modes through JsonSlurper script assertion, it gives the value as [1,5,11] and count as just 1. I want to split them into 3 array elements or variables and need the count as 3. What's the possible script assertion code?
Assertion:
import groovy.json.JsonSlurper
def resp = messageExchange.response.responseContent;
def jsl = new JsonSlurper().parseText(resp);
def modes = jsl.locations.modes
log.info modes
log.info modes.size()
Result:
Wed Feb 13 10:50:49 AEDT 2019:INFO:[[1, 5, 11]]
Wed Feb 13 10:50:49 AEDT 2019:INFO:1
What you are dealing with in this example is a shorthand version of Groovy's spread operator and your code returns a valid result. When you call jsl.locations you actually access a list of all locations objects (a singleton list in your example). When you call
jsl.locations.modes
you use a shorthand version of
jsl.locations*.modes
which is an equivalent of the following code:
jsl.locations.collect { it.modes }
This code means: iterate locations and transform a list of locations into a list of lists of modes of these locations - [[1,5,11]].
Applying the correct solution depends on a few more factors. For instance, you need to consider locations list containing multiple locations - in this case, transformation jsl.locations.modes may produce a result like [[1,5,11],[1,5,11],[2,4,9]] - a list of 3 modes lists.
If you assume that there is always a single location returned then you can simply flatten the final list like:
def modes = jsl.locations.modes.flatten()
assert modes == [1,5,11]
assert modes.size() == 3
However, if locations contains another JSON object, let's say, with exactly the same modes, then it will produce a completely different result:
def modes = jsl.locations.modes.flatten()
assert modes == [1,5,11,1,5,11]
assert modes.size() == 6
In this case, it might be better to use assertions like:
def modes = jsl.locations.modes
assert modes == [[1,5,11],[1,5,11]]
assert modes*.size() == [3,3]
which means:
modes stores 2 lists [1,5,11] and [1,5,11],
and the size of the first list is 3, and the size of the second list is also 3.
I have a stored procedure in SQL Server 2017 that outputs a result set as JSON. It works to output the set just fine, but I want it in a different format.
According to the MS documentation, the format it's sending out is as expected, but it seems counter-intuitive to me. I'm not JSON expert by any means but I've always assumed it to be a single object representation of a structure.
The query is:
SELECT
e.EventID AS 'Event.ID',
EventDate AS 'Event.Date',
ea.ActivityID AS 'Event.Activity.ID',
ea.CreateDate AS 'Event.Activity.Date',
ea.Notes AS 'Event.Activity.Notes'
FROM
Events e
JOIN
dbo.EventActivities ea ON e.EventID = ea.EventID
FOR JSON PATH
This returns an output of:
[
{"Event": {
"ID":"236",
"Date":"2019-03-01",
"Activity": {"ID": 10,
"Date":"2019-01-02T11:47:33.2800000",
"Notes":"Event created"}
}
},
{"Event": {
"ID":"236",
"Date":"2019-03-01",
"Activity": {"ID":20,
"Date":"2019-01-02T11:47:34.3933333",
"Notes":"Staff selected"}
}
},
{"Event": {
"ID":"236",
"Date":"2019-03-01",
"Activity": {"ID":20,
"Date":"2019-01-02T11:47:34.3933333",
"Notes":"Staff selected"}
}
}
]
When I format this manually (to visualise it better, it's giving me an array of 3 identical events, for each activity. This is consistent with what MS say in Format Nested JSON
I was expecting (or rather hoping) to see something like:
[
{
"Event": {
"ID": "236",
"Date": "2019-03-01",
"Activity": [
{
"ID": 10,
"Date": "2019-01-02T11:47:33.2800000",
"Notes": "Event created"
},
{
"ID": 20,
"Date": "2019-01-02T11:47:34.3933333",
"Notes": "Staff selected"
},
{
"ID": 20,
"Date": "2019-01-02T11:47:34.3933333",
"Notes": "Staff selected"
}
]
}
}
]
Is it possible to get an output formulated like this? or would this be invalid?
To start, you can test if a JSON String is valid with ISJSON. The expected output you indicated does not pass validation, but is close. It is missing a "[]" for the inner array.
However, I see where you were going with this. To better explain what I think the issue you are running into is, I am going to beautify the format of the output JSON from your query to match your expected JSON.
Original output as follows:
[
{"Event":
{"ID":"236","Date":"2019-03-01",
"Activity":{
"ID":10,"Date":"2019-01-02T11:47:33.2800000","Notes":"Event created"
}
}
},
{"Event":
{"ID":"236","Date":"2019-03-01",
"Activity":{
"ID":20,"Date":"2019-01-02T11:47:34.3933333","Notes":"Staff selected"}
}
},
{"Event":
{"ID":"236","Date":"2019-03-01",
"Activity":{
"ID":20,"Date":"2019-01-02T11:47:34.3933333","Notes":"Staff selected"
}
}
}
]
Based on your ideal format, a possible valid JSON string would be as follows:
{"Event":
[
{"ID":236,"Date":"2019-03-01",
"Activity":
[
{"ID":10,"Date":"2019-01-02T11:47:33.2800000","Notes":"Event created"},
{"ID":20,"Date":"2019-01-02T11:47:34.3933333","Notes":"Staff selected"},
{"ID":20,"Date":"2019-01-02T11:47:34.3933333","Notes":"Staff selected"}
]
}
]
}
You can achieve this by adjusting your table alias for the second table and using FOR JSON AUTO and ROOT. It will return an output with the "Event" attributes not repeated for each of its "EventActivities". For each "Event" it will put its related "EventActivities" into an array instead.
The following SQL will return the desired output:
SELECT [Event].EventID AS 'ID',
[Event].EventDate AS 'Date',
Activity.ActivityID AS 'ID',
Activity.CreateDate AS 'Date',
Activity.Notes AS 'Notes'
FROM #Events [Event]
JOIN #EventActivities Activity
ON [Event].EventID = Activity.EventID
FOR JSON AUTO, ROOT('Event')
The exact output for this will be as follows:
{"Event":[{"ID":236,"Date":"2019-03-01","Activity":[{"ID":10,"Date":"2019-01-02T11:47:33.2800000Z","Notes":"Event created"},{"ID":20,"Date":"2019-01-02T11:47:33.2800000Z","Notes":"Staff selected"},{"ID":20,"Date":"2019-01-02T11:47:33.2800000Z","Notes":"Staff selected"}]}]}
It will not return in a beautified format, but spaces and indentation can be added without compromising the fact that it is valid JSON, while also achieving the intended array requirements.
Your expected output is not a valid JSON. But, assuming you are referring Activity field as an array of activities, you can use nested queries.
SELECT
E.EventID AS 'Event.ID',
E.EventDate AS 'Event.Date',
(
SELECT
A.ActivityID AS 'ID',
A.CreateDate AS 'Date',
A.Notes AS 'Notes'
FROM dbo.EventActivities AS A
WHERE A.EventID = E.EventID
FOR JSON PATH
) AS 'Event.Activities'
FROM dbo.Events AS E
FOR JSON PATH
I'm kind of confused and after spending a lot of time on it, I have not found any good solution to my problem. So without even wasting my time I'm explaining my question.
I have a JSON data, with a data[] having different objects which looks like this :
{
"data": [
{
"id": 1,
"name": "XYZ",
"course": {
"id": 25,
"name": "XYZ",
"description": "",
"teacher": {
"id": 4,
"name": "",
"email": "",
"role": "",
"token": "",
"about": "Blah blah blah ",
"phone": "2222222222",
"image_url": "",
"payment_information": {}
},
"image": "",
"cover_image": "",
"course_type": "",
"ongoing": true,
"price": 10,
"students_count": 4,
"lesson_price": 2.5,
"drop_in_price": 12,
"lessons_data": {
"first_lesson_at": "",
"last_lesson_at": "",
"next_lesson_at": ""
},
"data": {
"number_of_classes": "",
"start_time": "06:45",
"day_of_week": "Tuesday",
"duration": "-300",
"start_date": "2017-11-07T06:45:04.000-0800",
"end_time": "06:50"
}
},
"start_time": "2018-01-23T14:45:00.000Z",
"end_time": "2018-01-23T14:50:00.000Z",
"zoom_url": "http://zoom.us/j/5124648907"
}
]
}
This is only one object which I just showed it to you but there are many other objects present inside the data array.
I have a widget which has a block and accepts the title and the , we will look at the later. So for this first I want to loop through the data object and check if the timestamps' date is matching with the other one or not and then get the particular amount of that widget after using *ngFor in my html
For now I have 7 object inside the JSON data and after getting the result from the timestamps I have 4 objects which has similar start_time (2 same and other 2 same).
More pieces of the information :
JSON data
1 with Wednesday from Timestamp
2 objects with Thursday from Timestamp
2 Objects with Friday from Timestamp
1 with Saturday from Timestamp
So according to the above data we have to get only 4 blocks, but I have tried finding some solution to it but failed.
I have tried doing this
<div *ngFor="les lesson of lessons">
<widget-app-block [title]="lesson.start_time | date: 'EEEE, MMMM d'"></widget-app-block>
</div>
But it prints all the 7 blocks and prints the Time for all the 7 blocks. But what we need here to get only 4 blocks which would be sorted after checking.
POSSIBLE ATTEMPTS
I have read about using *ngIF-else condition from HERE, so what I have done is I have tried getting the results from my TYPESCRIPT file and tried matching it with the lesson variable from the foreach loop in my html and then pass the title else just print the timestamp for individual objects but didn't work out.
TYPESCRIPT
ngAfterViewInit() {
this.http.get("/lessons").subscribe(data => {
this.lessons = castCollection(data['data'], Lesson)
for(let time of this.lessons){
this.timeTitle = time.start_time
}
})
}
and then comparing it to the lesson variable in my html
HTML
<div *ngFor="let lesson of lessons">
<div *ngIf="lesson.start_time === timeTitle; else elseTimeBlock">
<widget-app-block [title]="timeTitle | date: 'EEEE, MMMM d'"></widget-app-block>
</div>
<ng-template #elseTimeBlock>
<widget-app-block [title]="lesson_start_time | date: 'EEEE, MMMM d'"></widget-app-block>
</ng-template>
</div>
But no luck. Since my data fetching is working fine so, now I have to just focus on the data itself. Could any one please help me with this. I'd be grateful to learn new thing. Since this is very tricky.
It sounds like what you need is to filter your list to match a particular timestamp. I have a blog post here on filtering data in Angular: https://blogs.msmvps.com/deborahk/filtering-in-angular/
Something like this:
performFilter(filterBy: string): IProduct[] {
filterBy = filterBy.toLocaleLowerCase();
return this.products.filter((product: IProduct) =>
product.productName.toLocaleLowerCase().indexOf(filterBy) !== -1);
}
I've filtering on a string, but you could change this to filter on a timestamp instead.
See the blog post for the full code.
I have just set up a Cachet status page but i am struggling to push updates to the components via it's API.
I am looking to take an existing JSON feed from a partner site and use this to update the status on my own page.
Here is a sample of the JSON data I need to pull:
{
"state":"online",
"message":"",
"description":"",
"append":"false",
"status":true,
"time":"Sat 23 Apr 2016 10:51:23 AM UTC +0000"
}
and below is the format Cachet uses in it's API.
{
"data": {
"id": 1,
"name": "Component Name",
"description": "Description",
"link": "",
"status": 1,
"order": 0,
"group_id": 0,
"created_at": "2015-08-01 12:00:00",
"updated_at": "2015-08-01 12:00:00",
"deleted_at": null,
"status_name": "Operational"
}
}
Never dealt with JSON stuff before, but I guess i need a script that i can run every X mins to grab the original data and do the following:
Convert the "state" from the original feed into Cachet ones.
Update the "updated_at" time with the time the script was last run.
Any help or tutorial's would be really appreciated.
Thanks!
I'm the Lead Developer of Cachet, thanks for trying it out!
All you need to do is update the Component status. Cachet will take care of the updated_at timestamp for you.
I'm unable to write the script for you, but you'd do something like this:
// This will be a lookup of the states from the service you're watching to Cachet.
serviceStates = [
'online' => 1,
'issues' => 2,
'offline' => 4,
]
// The id of the component we're updating
componentId = 1
// The state that is coming back from the service you're watching
state = 'online'
request({url: 'demo.cachethq.io/api/v1/components/'+componentId, data: { status: serviceStates[state] }})
Pseudo code, but you should be able to work from that.