Get all ips from json log Grafana loki - json

I have a log file named test.log
{"app":"My app", "date":"12/12/2022", "ip_source":="192.168.1.1"},
{"app":"My app1", "date":"12/12/2022", "ip_source":="192.168.1.3"},
{"app":"My app2", "date":"12/12/2022", "ip_source":="192.168.1.5"},
And I want to get the dates and the ip's to make a graph that shows how many ip_sources attemp to connect to the server on the specific date.
I can get all the ip's and dates with the following query:
{filename="/var/log/test.log"} | json | line_format "Date: {{.date}} - IP source: {{.ip_source}} "
In order to show the graph I know I have to convert the date to an acceptable grafana format date with transform and I have tried with "convert field type" but I haven't understand how to do this.

Related

node js Sequelize with MySQL date formate issue

I developed node js services with Sequelize with MySQL Everything working good. I am storing data in DB with date and time with service but if I get the same data with service, node js automatically converting time (a wrong format which is not in the database), I want same time which is present in the database. to solve this issue I set the timezone in services and in the database but none of them is working. can anyone provide me with the solution.
suggestion
you could include time format function in mysql
for example
SELECT TIME_FORMAT("19:30:10", "%H %i %s");
result
TIME_FORMAT("19:30:10", "%H %i %s")
19 30 10
ref
That's not the issue of node js , Sequelize by default gives the time in UTC , so that date time will be correct
Check : Just compare your date time with UTC , and check is that you are getting.
Solution : So what you can do is , just convert returned date and time to your timezone and you are good to go.
And that's a good practice if you are using UTC everywhere , you can get benefits when you are getting / showing in many countries.

how can i input json file on mysql database?

[
{"link":"https://twitter.com/GreenAddress/status/550793651186855937",
"pDate":"2015 01 1",
"title":"GreenAddress",
"description": "btcarchitect coinkite blockchain circlebits coinbase bitgo some maybe some are oracle cosigners which require lesszero trust"},
{"link":"https://twitter.com/Bit_Swift/status/550765718581411840",
"pDate":"2015 01 1",
"title":"Bitswift™",
"description": "swiftstealth offers you privacy in bitswift v2 swiftstealth enables stealth address use on the bitswift blockchain swift"},
{"link":"https://twitter.com/allenday/status/550741133500772352",
"pDate":"2015 01 1",
"title":"Allen Day, PhD",
"description": "all in one article bitcoin blockchain 3dprinting drones and deeplearninghttp simondlr compost101071618938adecentralizedaivia simondlr"}
]
my test.json file like this
and my mysql db table is here
i can input text file with csv type, but i have no idea how input json text file on mysql
i try [create table test ( data json);] and
[insert into test values ( '{json type}'); but when i try input data with csv type LOAD DATA INFILE 'test.txt' made it possible
so I wonder json has the same functionality
thanks for any advice
MySQL does have JSON data field. However, it will not work with your file and current table structure as it request a field to be JSON. To solve your data, will require a little bit of programming work. Depending on your current ability, you will need to write codes that does the following:
Open a database connection
Read the JSON and loop through each value
Store each value using the following INSERT query:
INSERT INTO news(link, date, title, description) VALUES($link, $pDate, $title, $description);
Depending on your language and database connection feature, close the database connection.

Grails default timezone nature while building API

I am using grails 2.4.4 for building a restful API. I am having a lot of trouble figuring out grails default time-zone settings.
The premise of the problem is I am using default grails setting without using and dateBinders in config.groovy and the timezone of my both mysql and application server is in NPT.
So I am posting a json something like
{"clusterStartTime":"2015-10-15T17:45:00Z"}
Now this is a ISO 8601 format. So how can this date format represent my timezone so grails can interpret it ??
In database it is being saved as 2015-10-15 17:45:00.
But when I get this resource I am getting :
{"clusterStartTime": "2015-10-15T12:00:00Z"}
So I am getting a time with offset of 5:45 which is NPT offset. So how is grails doing all of this ??
So in production I have a server in NPT timezone so how can I save all my dates in UTC timezone and spit out dates in UTC timezone using grails only without setting Java enviroment variables ??
In your bootstrap.groovy set the timezone like this:
TimeZone.setDefault(TimeZone.getTimeZone("Asia/Katmandu"))
NTP is not supported directly. To get a list of supported timezones see the output of TimeZone.getAvailableIDs().
In order to interpret correctly any ISO 8601 date you have to use a good parser. The one provided by javax.xml.bind.DatatypeConverter should be good enough. The method parseDateTime expects a lexicalXSDDateTime, which must be ISO 6801 compliant.
Therefore try this:
def cal = javax.xml.bind.DatatypeConverter.parseDateTime('2015-10-15T17:45:00Z')
def d = cal.getTime()
def t1 = new TestClass(myDate: d).save(failOnError:true)
println t1.myDate //Thu Oct 15 17:45:00 NPT 2015
The date gets saved on the database (e.g. in a DATETIME field) considering the time zone you defined, so it should be correct when you retrieve it.

Regarding output to mysql database using barnyard 2

While trying to output unified2 format into mysql database using Barnyard2, the Source or destination IP addresses in the database come in the format "3232240901" instead of 192.168.1.0(just in case)..How do I fix it?

What is the easiest way to extract sprint start and end dates from the JIRA db?

I'm trying to extract the start and end days of my sprints from the Jira database. This would seem like a simple task, but it really (as far as I have found out at least) is not.
While trying to figure this out I found one solution, but it seems to me so cumbersome and difficult that I have problems thinking this is the only way.
Here is what I have found out:
Sprints are not a native Jira expression and the Greenhopper plugin uses the projectversion table to represent sprints.
The projectversion table contains some information about the sprint, like name, what project it belongs to and the release date. The release date can be thought of as a sprint end date, but the start date is missing.
If you run back to back sprints maybe the start date of a sprint can be set to the release date of the previous sprint plus one day? But that is really not a good solution.
So I searched through the Jira data model and the best and only reference I found to a sprint start date was in the property structure.
You can define properties and assign them values. In the main table of this structure, the propertyentry table, I found entries like this:
ID ENTITY_NAME ENTITY_ID PROPERTY_KEY propertytype
----- -------------- ------------ ------------------------------------ ------------
10288 GreenHopper 10010 CONFIGURATION 6
10304 GreenHopper 10012 CONFIGURATION 6
10312 GreenHopper 10013 CONFIGURATION 6
10449 GreenHopper 10014 CONFIGURATION 6
So GreenHopper have added a property with the key set to CONFIGURATION. The etity_id field references project.id and the configuration property is a project configuration. The property_type is set to 6 which tells us to look for the value in the propertytext table.
The value stored in the propertytext table reveals it self as a XML string containing different information about the project, among it entries like this:
<entry>
<string>BURNDOWN_START_DATE_10074</string>
<long>1316988000000</long>
</entry>
There it is. The best equivalent I have found to a sprint start date. Hidden in an xml string in a property table.
My question is: Is this really the only way to find my sprint starting dates?
There seems to be no way of getting end and start date of sprint via Jira SOAP/REST API.
You can extract the start and end dates of your sprints using:
com.pyxis.greenhopper.jira.configurations.ProjectConfiguration#getVersionStartDate
com.pyxis.greenhopper.jira.configurations.ProjectConfiguration#getVersionEndDate
To use this class you can write a Jira plugin - Developing with the Atlassian Plugin SDK
Another option is to write GreenHopper module - GreenHopper Developer Documentation
I don't recommend accessing the JIRA database directly if it can be avoided. The undocumented JIRA Agile REST API such as rest/greenhopper/1.0/rapid/charts/sprintreport.json?rapidViewId=652&sprintId=577 where rapidViewId is the board id,
gives you the Sprint information. This and other REST resources can be seen in the jira-python library at http://jira-python.readthedocs.org/en/latest/
The easiest way to find start date and end date of sprint in Agile jira is to hit jiraschema.AO_60DB71_SPRINT table. This stores start_date, end_date as big int. Jira for some reasons stores these date formats as int data type. To convert the int to date data type, here is the query in MS Sql. You can easily change it to some other database if required.
###This query pulls start_date, end_date, completed_date of all non active sprints in MS SQL.###
select ID, Name SprintName
,START_DATE / 60 / 60 / 24 / 1000 + CAST('12/31/1969' as datetime)+1 StartDate
,END_DATE / 60 / 60 / 24 / 1000 + CAST('12/31/1969' as datetime)+1 EndDate
,COMPLETE_DATE / 60 / 60 / 24 / 1000 + CAST('12/31/1969' as datetime)+1 CompletedDate
FROM
AO_60DB71_SPRINT as sprint
where COMPLETE_DATE is not null
SELECT * FROM a0_60db71_sprint
This is a MySQL question, not java.
Connect to your JIRA MySQL Database and look for a table that matches *_sprint
The fields on the above table are:
Closed (boolean)
ID (key)
Start_Date (timestamp)
End_Date (timestamp)
Complete_Date (timestamp).
I was given a task recently to get list of sprints with dates for specific project. First I needed to find project ID from Project table and customfield ID for field Sprint from tables customfield/customfieldvalue.
Here is the result
select
p.pname as "Project Name",
s.NAME as "Sprint Name",
from_unixtime(s.START_DATE / 1000) as "Start Date",
from_unixtime(s.END_DATE / 1000) as "End Date",
from_unixtime(s.COMPLETE_DATE / 1000 ) as "Complete Date"
from
customfieldvalue as c,
jiraissue as i,
project as p,
AO_60DB71_SPRINT as s
where
p.id = <project ID> and p.id = i.project and
c.issue = i.id and c.customfield = <customfield ID> and
c.stringvalue = s.id
group by s.name
;
Our mysql server is in different time zone so I had to modify output time.
...
from_unixtime(s.START_DATE / 1000) - interval 1 hour as "Start Date",
...
Maybe it will help somebody
Not sure! why JIRA doesn't provide a very simple Rest Endpoint to just spit all sprints info. Why I have to deal with board/boardID to find sprints in that board, why I have to iterate over all sprints.
I'm administrator user and still hitting some of the sprint # gives me, Sprint does not exist.
Anyways, here's a work-around script.
#!/bin/bash
JIRA_URL="http://my_jira_server:8080"
users_sprint_limit_cmd_line_arg="$1"
# First parameter passed to the script is a NUMBER (for how many sprints a user wants to iterate over.
## I know!! it's a work-around for dealing with "Sprint does not exist" and
## becasue there's no shitty direct JIRA Rest API that exist, to query JIRA server, to spit all SPRINTS with info (start/end date) in just one call.
## You can use API token (or base64 hash). I'm just going rouge here.
user="a_user_user_who_can_read_any_sprint_or_serviceuser_or_admin"
pass="D00M4u!"
## Set build number variable
b_no=${BUILD_NUMBER:="999999"}
## At the end, you'll have a Temp file will store all sprints info, Valid will contain only valid sprints.
temp_sprint_file="/tmp/all_sprints_startdates_${b_no}_temp.txt"
valid_sprint_file="/tmp/all_sprints_startdates_${b_no}.txt"
## Clean files
rm ${temp_sprint_file} ${valid_sprint_file} || true;
## Sprint counter
sprint_no=1
result="ToBeSet"
## Iterate over all sprints and find their start/stop dates.
## -- This is one-odd way to find sprint's start/end dates, but it works!!
## -- A user can pass a larger value in while condition "-lt value" via cmd line 1st param.
while [[ $sprint_no -lt ${users_sprint_limit_cmd_line_arg} ]];
do
## assumes 'jq' is installed. --OR run: sudo yum install jq
## --------------------------
result="$(curl -s -u $user:$pass -X GET -H 'Content-Type: application/json' "${JIRA_URL}/rest/agile/1.0/sprint/${sprint_no}" | \
jq | \
egrep "name|startDate|endDate" | \
cut -d'"' -f4 | \
sed "s/T[0-9][0-9]:[0-9][0-9].*$//" | \
tr '\012' ',' | \
sed "s/,$//")";
echo "${result}" >> ${temp_sprint_file}
((sprint_no++));
done
## Find valid sprints which have valid start/end dates.
grep "[A-Za-z],[0-9].*,[0-9]" ${temp_sprint_file} > ${valid_sprint_file};
echo -e "\n\n-- Sprints and Start/End Date file is available here: ${valid_sprint_file}\n\n"
Running cat command on this generated sprint data file will give you, something like:
1 Trumpy Trump,2019-01-09,2019-01-23
2 Magical Modi,2019-01-18,2019-02-01
Where, you can add a line in the above script, to use it as a pure CSV file by having a header line i.e. Sprint_Name,Sprint_Start_Date,Sprint_End_Date, I just didn't do that as my use case was to use just this file as a reference file.
A related post regarding dates: BASH: How to find no. of days (considering only "Network / Business Days") between two dates (i.e. exclude weekends Saturday/Sunday)