Assuming there is a blog which doesnt have RSS feed association and I am not registered to any notification regarding this blog , I want to be able to "know" each time a new post is published on it.
Is there any way I can do that? I'm looking for a generic solution ...
Pseudo cron script, run /1 * * * *
curl http://thatguysblog.html -> latest.blog;
grep latest.blog "posted at $1\n";
readf previous.time;
If (unixtime($1) > $previoustime) {
send SMS me#myphone.com "OMG, a new blog post!!!";
}
printf $1 previous.time;
Related
On every pull request that has a specific label our github-actions bot makes an automatic comment that includes a template which needs to be edited by the PR approver, before it can be merged.
The template the bot comments is as follows:
**Hello, #${{ github.actor }}!**
**This component is critical!**
**Edit and input the following BEFORE approving the PR!**
___
* Description: XXXXXXXXXX
* Justification: XXXXXXXXXX
The last change was authored by:
#${{ github.event.pull_request.user.login }}
The Description and Justification fields are replaced with actual values manually by the PR approver (it is not possible to know these in advance). Their values need to be saved as: change_description and justification to be stored in a .json file like so:
{"change_date": "<Date when PR Merged>", "change_id": "<Pull Request Link>", "change_description": "<from PR Comment>", "component": "<from terragrunt.hcl file>", "justification": "<from PR Comment>", "team": "<from terragrunt.hcl file>"}
I've gotten as far as saving the comment body as an environment variable in a new Github action, where I want to parse the comment and save it into a .json file:
env:
BODY: ${{ github.event.comment.body }}
All of the logic so far as been simple enough, but now I cannot find a way to pull the values of the Description and Justification from the comment body after they are edited manually. Is this not possible via Github Actions?
Yes it's possible in GitHub Actions.
You have to use regular expressions and bash to parse desired values.
Here is example of regexp to get one of your variables:
\*\*Hello, (.*)\*\*
https://regex101.com/r/Wfww5Y/1
End then in bash you can do:
[[ $body =~ $regex ]]
$ echo ${BASH_REMATCH[1]}
I need to automatically move new cases (TheHive-Project) to LimeSurvey every 5 minutes. I have figured out the basis of the API script to add responses to LimeSurvey. However, I can't figure out how to add only new cases, and how to parse the Hive case data for the information I want to add.
So far I've been using curl to get a list of cases from hive. The following is the command and the output.
curl -su user:pass http://myhiveIPaddress:9000/api/case
[{"createdBy":"charlie","owner":"charlie","createdAt":1498749369897,"startDate":1498749300000,"title":"test","caseId":1,"user":"charlie","status":"Open","description":"testtest","tlp":2,"tags":[],"flag":false,"severity":1,"metrics":{"Time for Alert to Handler Pickup":2,"Time from open to close":4,"Time from compromise to discovery":6},"updatedBy":"charlie","updatedAt":1498751817577,"id":"AVz0bH7yqaVU6WeZlx3w","_type":"case"},{"createdBy":"charlie","owner":"charlie","title":"testtest","caseId":3,"description":"ddd","user":"charlie","status":"Open","createdAt":1499446483328,"startDate":1499446440000,"severity":2,"tlp":2,"tags":[],"flag":false,"id":"AV0d-Z0DqHSVxnJ8z_HI","_type":"case"},{"createdBy":"charlie","owner":"charlie","createdAt":1499268177619,"title":"test test","user":"charlie","status":"Open","caseId":2,"startDate":1499268120000,"tlp":2,"tags":[],"flag":false,"description":"s","severity":1,"metrics":{"Time from open to close":2,"Time for Alert to Handler Pickup":3,"Time from compromise to discovery":null},"updatedBy":"charlie","updatedAt":1499268203235,"id":"AV0TWOIinKQtYP_yBYgG","_type":"case"}]
Each field is separated by the delimiter },{.
In regards to parsing out specific information from each case, I previously tried to just use the cut command. This mostly worked until I reached "metrics"; it doesn't always work for metrics because they will not always be listed in the same order.
I have asked my boss for help, and he told me this command might get me going in the right direction to adding only new hive cases to the survey, but I'm still very lost and want to avoid asking too much again.
curl -su user:pass http://myhiveIPaddress:9000/api/case | sed 's/},{/\n/g' | sed 's/\[{//g' | sed 's/}]//g' | awk -F '"caseId":' {'print $2'} | cut -f 1 -d , | sort -n | while read line; do echo '"caseId":'$line; done
Basically, I'm in way over my head and feel like I have no idea what I'm doing. If I need to clarify anything, or if it would help for me to post what I have so far in my API script, please let me know.
Update
Here is the potential logic for the script I'd like to write.
get list of hive cases (curl ...)
read each field, delimited by },{
while read each field, check /tmp/addedHiveCases to see if caseId of field already exists
--> if it does not exist in file, add case to limesurvey and add caseId to /tmp/addedHiveCases
--> if it does exist, skip to next field
why are you thinking that the fields are separated by a "},{" delimiter?
The response of the /api/case API is a valid JSON format, that lists the cases.
Can you use a Python script to play with the API? If yes, I can help you write the script you need.
I'm using rsyslog to watch over my syslogs and send them over to Logstash+Kibana.
My syslogs messages are logged as JSON. They can look something like this:
{"foo":"bar", "timegenerated": 43843274834}
rsyslog configuration as so:
module(load="omelasticsearch")
#define a template to print all fields of the message
template(name="messageToES" type="list" option.json="on") {
property(name="msg")
}
*.* action(type="omelasticsearch"
server="localserverhere"
serverport="80"
template="messageToES")
The Kibana is fine, since if I run a CURL command to it, it receives the record. The code as below:
curl -XPOST myserver/test/bar -d '{"test": "baz", "timegenerated":1447145221519}'
When I run rsyslogs and point it to a dummy server, I can see the incoming requests with the valid json. However, when I point it back to my logstash server, it doesn't show up in logstash or kibana.
Does anyone know how to send syslogs as json into Kibana/logstash?
I've never used it, but it looks like you are missing things from your config file. The docs have a pretty thorough example:
module(load="omelasticsearch")
template(name="testTemplate"
type="list"
option.json="on") {
constant(value="{")
constant(value="\"timestamp\":\"") property(name="timereported" dateFormat="rfc3339")
constant(value="\",\"message\":\"") property(name="msg")
constant(value="\",\"host\":\"") property(name="hostname")
constant(value="\",\"severity\":\"") property(name="syslogseverity-text")
constant(value="\",\"facility\":\"") property(name="syslogfacility-text")
constant(value="\",\"syslogtag\":\"") property(name="syslogtag")
constant(value="\"}")
}
action(type="omelasticsearch"
server="myserver.local"
serverport="9200"
template="testTemplate"
searchIndex="test-index"
searchType="test-type"
bulkmode="on"
queue.type="linkedlist"
queue.size="5000"
queue.dequeuebatchsize="300"
action.resumeretrycount="-1")
Based on what you are trying to do, it looks like you need to plug in localserverhere where it shows myserver.local. It also looks like you have ES accepting stuff on port 80, so you'd put in 80 instead of 9200.
We've recently moved our centralized logging from Splunk to an ELK solution, and we have a need to export search results - is there a way to do this in Kibana 4.1? If there is, it's not exactly obvious...
Thanks!
This is very old post. But I think still someone searching for a good answer.
You can easily export your searches from Kibana Discover.
Click Save first, then click Share
Click CSV Reports
Then click Generate CSV
After a few moments, you'll get download option bottom right side.
This works with Kibana v 7.2.0 - export query results into a local JSON file. Here I assume that you have Chrome, similar approach may work with Firefox.
Chrome - open Developer Tools / Network
Kibana - execute your query
Chrome - right click on the network call and choose Copy / Copy as cURL
command line - execute [cURL from step 3] > query_result.json . The query response data is now stored in query_result.json
Edit: To drill down into the source nodes in the resulting JSON file using jq:
jq '.responses | .[] | .hits | .hits | .[]._source ' query_result.json
If you want to export the logs (not just the timestamp and counts), you have a couple of options (tylerjl answered this question very well on the Kibana forums):
If you're looking to actually export logs from Elasticsearch, you
probably want to save them somewhere, so viewing them in the browser
probably isn't the best way to view hundreds or thousands of logs.
There are a couple of options here:
In the "Discover" tab, you can click on the arrow tab near the bottom to see the raw request and response. You could click "Request"
and use that as a query to ES with curl (or something similar) to
query ES for the logs you want.
You could use logstash or stream2es206 to dump out the contents of a index (with possible query parameters to get the
specific documents you want.)
#Sean's answer is right, but lacks specifics.
Here is a quick-and-dirty script that can grab all the logs from ElasticSearch via httpie, parse and write them out via jq, and use a scroll cursor to iterate the query so that more than the first 500 entries can be captured (unlike other solutions on this page).
This script is implemented with httpie (the http command) and fish shell, but could readily be adapted to more standard tools like bash and curl.
The query is set as per #Sean's answer:
In the "Discover" tab, you can click on the arrow tab near the bottom
to see the raw request and response. You could click "Request" and
use that as a query to ES with curl (or something similar) to query ES
for the logs you want.
set output logs.txt
set query '<paste value from Discover tab here>'
set es_url http://your-es-server:port
set index 'filebeat-*'
function process_page
# You can do anything with each page of results here
# but writing to a TSV file isn't a bad example -- note
# the jq expression here extracts a kubernetes pod name and
# the message field, but can be modified to suit
echo $argv | \
jq -r '.hits.hits[]._source | [.kubernetes.pod.name, .message] | #tsv' \
>> $output
end
function summarize_string
echo (echo $argv | string sub -l 10)"..."(echo $argv | string sub -s -10 -l 10)
end
set response (echo $query | http POST $es_url/$index/_search\?scroll=1m)
set scroll_id (echo $response | jq -r ._scroll_id)
set hits_count (echo $response | jq -r '.hits.hits | length')
set hits_so_far $hits_count
echo "Got initial response with $hits_count hits and scroll ID "(summarize_string $scroll_id)
process_page $response
while test "$hits_count" != "0"
set response (echo "{ \"scroll\": \"1m\", \"scroll_id\": \"$scroll_id\" }" | http POST $es_url/_search/scroll)
set scroll_id (echo $response | jq -r ._scroll_id)
set hits_count (echo $response | jq -r '.hits.hits | length')
set hits_so_far (math $hits_so_far + $hits_count)
echo "Got response with $hits_count hits (hits so far: $hits_so_far) and scroll ID "(summarize_string $scroll_id)
process_page $response
end
echo Done!
The end result is all of the logs matching the query in Kibana, in the output file specified at the top of the script, transformed as per the code in the process_page function.
If you have troubles making your own request with curl or you don't need automatic program to extract logs from Kibana, just click 'Response' and get what you need.
After having troubles like 'xsrf token missing' when using curl,
I found this way is more easier and simple!
Like others said, Request button appears after clicking the arrow tab near the bottom.
Only the Timestamp and the count of messages at that time are exported, not the log information:
Raw:
1441240200000,1214
1441251000000,1217
1441261800000,1342
1441272600000,1452
1441283400000,1396
1441294200000,1332
1441305000000,1332
1441315800000,1334
1441326600000,1337
1441337400000,1215
1441348200000,12523
1441359000000,61897
Formatted:
"September 3rd 2015, 06:00:00.000","1,214"
"September 3rd 2015, 09:00:00.000","1,217"
"September 3rd 2015, 12:00:00.000","1,342"
"September 3rd 2015, 15:00:00.000","1,452"
"September 3rd 2015, 18:00:00.000","1,396"
"September 3rd 2015, 21:00:00.000","1,332"
"September 4th 2015, 00:00:00.000","1,332"
"September 4th 2015, 03:00:00.000","1,334"
"September 4th 2015, 06:00:00.000","1,337"
"September 4th 2015, 09:00:00.000","1,215"
"September 4th 2015, 12:00:00.000","12,523"
"September 4th 2015, 15:00:00.000","61,897"
Sure, you can export from Kibana's Discover (Kibana 4.x+).
1. On the discover page click the "up arrow" here:
Now, on the bottom of the page, you'll have two options to export search results
At logz.io (the company I work for), we'll be releasing scheduled reports based on specific searches.
I can´t find a solution to reading the amount of likes from a facebook page. It is not possible to pass the variable $name correctly. Would highly appreciate help!
name=$(grep "NAME_OF_FACEBOOK_PAGE" output3.txt |
echo "$name"
curl -s "http://graph.facebook.com/?ids=https://www.facebook.com/$name/likes" -o output3.txt
cat output3.txt | grep "\"likes\"" -A1 -B0
If you just want to read the number of likes of a certain page, why don't you use
https://graph.facebook.com/{page_name}?fields=id,name,likes,talking_about_count
Your call to the Graph API is incorrect.