Extract Url from access Log in Splunk - extract

This is the sample accessLog which is coming up in the Splunk ui.
{"timestamp":"2021-10-17T15:03:56,763Z","level":"INFO","thread":"reactor-http-epolpl-20","message":"method=GET, uri=/api/v1/hello1, status=200, duration=1, "logger":"reactor.netty.http.server.AccessLog"}
{"timestamp":"2021-10-17T15:03:56,763Z","level":"INFO","thread":"reactor-http-epolpl-20","message":"method=GET, uri=/api/v1/dummy1, status=200, duration=1, "logger":"reactor.netty.http.server.AccessLog"}
I want to extract the url and make a count for all the API's like how many times an API is hitted from the uri part(uri=/api/v1/dummy1)
(index=dummy OR index=dummy1) source=*dummy-service* logger=reactor.netty.http.server.AccessLog
| rex field=message "(?<url>uri.[\/api\/v1\/hello]+)"
| chart count by url
But it's not giving URL in a proper format. I tried various regex, but couldn't get the proper URL count.
I wanted to use this query to show the API counts in the Splunk dashboard.

The problem appears to be with the regular expression in the rex command. Square brackets ([]) in a regex denote a set from which any character can match, in any order. The example regex should match "/api/v1/hello", but also will match "iap/1v/ohell", "philo", and any other permutation of those characters. It will not, however, match "/api/v1/dummy1".
Try this query. The rex command here takes everything between "uri=" and the following comma as the url.
index=dummy OR index=dummy1 source=*dummy-service* logger=reactor.netty.http.server.AccessLog
| rex field=message "uri=(?<url>[^,]+)"
| chart count by url

Related

while searching in elasticsearch,if we search for a word "brother" which is present more than once in array of json

then if we write a nested query to get brother then will we get the both occurences present in the array or json or only single occurence?
for example
if we are querying to get the word "brother" and its present more than twice in the array of json
will it show only one occurence of brother or multiple occurences?
hence we are handling multiple files we need the file id in which the word is present and if single word is present in a file more than once is it possible to get the word in that single file more than once?
i tried to get the single file id more than once because the single file contained the word more than once in multiple jsons but im getting the file id only once ...
If you are using kibana sql, you will search any occurence in each elasticsearch doc. If you search by "brother" you will get 1 occurence from each doc contains the word "brother". If you have 3 docs and once has 3 times the word you are searching for and the other 2 have the word once each, you will get 3 hits.
We have to use a for loop in any programming language and we will get only max 3 hits in inner hits. We have to access the json file and load it in Python for example and then use for loop and we can have upto 3 hits.by this process we can access upto 3 inner hits in a same file.
the code:-
we can fetch em in this way
for i in range(some_value):
try:
uniq_id = data["hits"]["hits"][i]["_id"]
start = data["hits"]["hits"][i]["inner_hits"]["key"]["hits"]["hits"][0]["fields"]["key.start"][0]
check = data["hits"]["hits"][i]["inner_hits"]["key"]["hits"]["total"]["value"]

Google Drive API: find a file by name using wildcards?

When using the Google Drive API v3, can one search for a file by its name using wildcards or regular expressions? The docs don't mention anything.
I am trying to match a set of files whose names have the format
backup_YYYY-MM-DD-XXXX_NameOfWebsite_xxxxxxxxxx.zip
And am wondering what's the best way to construct a pattern that might match it. Of course, I could follow the docs and just do something like:
q="name contains 'backup' and name contains 'NameOfWebsite'"
But if I need to match a different pattern, or something with more than 2 distinctive strings in its filename ("backup_" and "NameOfWebsite"), you can quickly see what a pain would be to construct a query in that way:
q="name contains 'string1' and name contains 'string2' and name contains...
Answer:
You can't use a wildcard in the middle of a file name when making a Drive.list request with a q parameter.
More Information:
The name field only takes three operators - =, != and contains:
The = operator is regular equivalence and with this you can not use a wildcard.
name = 'backup*' will return no results.
The != operator is not equivalence, not relevant here but does the opposite of =
The contains operator. You can use wildcards with this, but with restrictions:
name contains 'backup*' will return all files with filenames starting with the string backup.
name contains '*NameOfWebsite' will return all files with filenames that have a word starting with the string NameOfWebsite. The file name backup0194364-NameOfWebsite.zip will not be returned, because there is no space before the string.
Therefore, the only way for this to work is if you do it the way you have already started to realise; string chaining:
name contains 'backup' and name contains 'NameOfWebsite' and name contains ...
References:
Files: list | Google Drive API | Google Developers
Search for files | Google Drive API | Google Developers

Trying to pull the Name and/or ID of the code below, but can only pull the Job-Base-Cost

Below is the code I have now. It pulls the Job-Base-Cost just fine, however I cannot get it to pull the ID and or Name of the item. Can you help?
Link to the sites XML pull.
=importxml("link","//job-base-cost")
This is a sample of one line of the OP's XML file
<job-base-cost id="24693" name="Abaddon Blueprint">109555912.69</job-base-cost>
The OP wants to use the IMPORTXML function to report the ID and Name as well as the Job Cost from the XML data. Presently, the OP's formula is:
=importxml("link","//job-base-cost")
There are two options:
1 - One long column
=importxml("link","//#id | //#name | //job-base-cost")
Note //#id and //#name in the xpath query: // indicate nodes in the document (at any level, not just the root level) and # indicate attributes. The pipe | operator indicates AND. So the plain english query is to display the id, name and job-base-cost.
2 - Three columns (table format)
={IMPORTXML("link","//#name"),IMPORTXML("link","//job-base-cost"),IMPORTXML("link","//#id")}
This creates a series that will display the fields in each of three columns.
Note: there is an arrayformula that uses a single importXML function described in How do I return multiple columns of data using ImportXML in Google Spreadsheets?. Readers may want to look at whether that option can be implemented.
My thanks to #Tanaike for his comment which spurred me to look at how xpath works.

How to paginate with string provided in JSON result?

I am using Feedly API to access feeds from a particular rss feed. Take a look at this link (1). As you see, it only returns newest 20 items but I think it provides sort of a link to paginate to the next result. There is 'continuation' key provided in the result but it is a string and not a link.
How can I use that to fetch the next result? Is this even possible?
JSON response from the server
See http://developer.feedly.com/v3/streams/
You can pass the continuation key to get the next batch of results.
For example: https://cloud.feedly.com/v3/streams/contents?streamId=feed/http://feeds.engadget.com/weblogsinc/engadget?continuation=14de41de03e:f7bda:87649ed8

How the google charts query language works?

I have implemented my own datasource in Go, which returns a JSON string. The data source is working fine and returning the correct JSON format expected by the chart (which is actually a table object to make the tests easier).
Now, I'd like to be able to use the query language features for my chart and something I could not figure out is how exactly the query language works.
Let's take the table below as example:
Name | Age | Phone
-------------------------------
John | 23 | 12341234
Chris | 47 | 54223452
Sam | 36 | 69694356
When called, my datasource will return a JSON representation of the entire table above.
In theory, I should be able to do something like this from my Javascript
query.setQuery('select Name, Age');
So, the result would ignore the column "Phone".
Now, my question is:
Is the setQuery() method applied to the JSON response only, or my datasource should be able to handle the query on request and return the correct data (only Name and Age columns). I'm not sure if the query language will act on the JSON response or if it's just an interface to tell the server what to do and the server should be able to prepare the correct data.
I'm asking that because as I said, my JSON response works fine, however the setQuery() method is being ignored. My table always shows the entire Dataset, no matter what I put on the setQuery method. Even if I define a column which does not exist, it does not cause any error.
I did some tests using a google spreadsheet and it works just fine. I should add that the structure of the JSON response from my app and the one from the google spreadsheet looks exactly the same.
Any help would be very much appreciated.
Thanks,
JB