I am using a webhook to post the message to mattermost everytime a post is created in an ITSM. I want to post the message as a thread everytime somebody does some activity on the same post.
{"text": " | Incident Update | |\n| --- | --- |\n| Incident | {#subject#} |\n| Downtime | {#createdTime#} |\n| Uptime | {#request.lastClosedTime#} |\n| Root Cause | {#Problem Domain#} |\n| Duration | |\n| Escalation Level | 1 |\n| Escalation To | Network Operations |\n| Area | {#Area#} |\n| Priority | {#priority#} |\n| Impact | {#impact#} |\n| Category | {#category#} |\n| Affected Area | |\n| Remarks | {#comment#} {#Solution#} |\n" }```
Related
i am working on a MySQL database with the following schema:
In which we save information from different ontologies (exmp). Some of the terms have relationships (exmp: MS:1000004, sample mass) which are indicated by relationship: or is_a:. For the sake of this question let's focus on the is_a relationships.
We now want to provide an option to search along these is_a relationships. So a function which we give MS:1000004/sample mass as input and get in return all Terms which are connected to it by an is_a relationship. But not only all terms directly connected to MS:1000004, but also all terms which are connected to the children and so on. This image describes this a bit better, where thing would be MS:1000004/sample mass and everything below is what i want as result.
Currently we are using a rather unoptimized recursive function as a stored procedure to do this:
DELIMITER $$
CREATE DEFINER=`root`#`localhost` PROCEDURE `getAllTermsByParentTerm`(IN `parentOntology` varchar(512))
BEGIN
WITH RECURSIVE previous (accession, FK_OntologyName, name, definition, xrefvaluetype, isobsolete, fk_termAccession, relationshiptype, fk_termAccession_related, depth_level) AS (
SELECT
t.accession,
t.FK_OntologyName,
t.name,
t.definition,
t.xrefvaluetype,
t.isobsolete,
trt.fk_termAccession,
trt.relationshiptype,
trt.fk_termAccession_related,
0 depth_level
FROM Term t
INNER JOIN (TermRelationship AS trt, Term AS ref) ON(
t.Accession = trt.FK_TermAccession
AND trt.FK_TermAccession_Related = ref.Accession
AND (trt.RelationshipType = `is_a` OR trt.RelationshipType = `part_of`)
AND
(
trt.FK_TermAccession_Related = ref.Accession
AND ref.Name = parentOntology
)
)
UNION All
SELECT
t2.accession,
t2.FK_OntologyName,
t2.name,
t2.definition,
t2.xrefvaluetype,
t2.isobsolete,
trt2.fk_termAccession,
trt2.relationshiptype,
trt2.fk_termAccession_related,
(previous.depth_level+1) depth_level
FROM Term t2
INNER JOIN (TermRelationship AS trt2, previous) ON(
t2.Accession = trt2.FK_TermAccession
AND trt2.FK_TermAccession_Related = previous.Accession
)
)
SELECT
t.Accession,
t.FK_OntologyName,
t.Name,
t.Definition,
t.xRefValueType,
t.IsObsolete,
p.depth_level
FROM previous p
Inner JOIN Term AS t ON (
p.Accession = t.Accession
);
END$$
DELIMITER ;
To the question: i am not that experienced with MySQL so are there any options to optimize this function, or is MySQL the wrong tool overall?
Example rows:
Term
| Accession | FK_OntologyName | Name | Definition | XRefValueType | IsObsolete |
|------------|-----------------|----------------------------------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------|------------|
| TEST:0000000 | TEST | Proteomics Standards Initiative Mass Spectrometry Vocabularies | "Proteomics Standards Initiative Mass Spectrometry Vocabularies." [PSI:MS] | | 0 |
| TEST:1000001 | TEST | sample number | "A reference number relevant to the sample under study." [PSI:MS] | value-type:xsd\:string "The allowed value-type for this CV term." | 0 |
| TEST:1000002 | TEST | sample name | "A reference string relevant to the sample under study." [PSI:MS] | value-type:xsd\:string "The allowed value-type for this CV term." | 0 |
| TEST:1000003 | TEST | sample state | "The chemical phase of a pure sample, or the state of a mixed sample." [PSI:MS] | | 0 |
| TEST:1000004 | TEST | sample mass | "Total mass of sample used." [PSI:MS] | value-type:xsd\:float "The allowed value-type for this CV term." | 0 |
| TEST:1000005 | TEST | sample volume | "Total volume of solution used." [PSI:MS] | value-type:xsd\:float "The allowed value-type for this CV term." | 0 |
| TEST:1000006 | TEST | sample concentration | "Concentration of sample in picomol/ul, femtomol/ul or attomol/ul solution used." [PSI:MS] | value-type:xsd\:float "The allowed value-type for this CV term." | 0 |
| TEST:1000007 | TEST | inlet type | "The nature of the sample inlet." [PSI:MS] | | 0 |
| TEST:1000008 | TEST | ionization type | "The method by which gas phase ions are generated from the sample." [PSI:MS] | | 0 |
| TEST:1000011 | TEST | mass resolution | "Smallest mass difference between two equal magnitude peaks so that the valley between them is a specified fraction of the peak height." [PSI:MS] | value-type:xsd\:string "The allowed value-type for this CV term." | 0 |
| TEST:1000012 | TEST | resolution measurement method | "Which of the available standard measures is used to define whether two peaks are separate." [PSI:MS] | | 0 |
| TEST:1000014 | TEST | accuracy | "Accuracy is the degree of conformity of a measured mass to its actual value." [PSI:MS] | value-type:xsd\:float "The allowed value-type for this CV term." | 0 |
| TEST:1000015 | TEST | scan rate | "Rate in Th/sec for scanning analyzers." [PSI:MS] | value-type:xsd\:float "The allowed value-type for this CV term." | 0 |
| TEST:1000016 | TEST | scan start time | "The time that an analyzer started a scan, relative to the start of the MS run." [PSI:MS] | value-type:xsd\:float "The allowed value-type for this CV term." | 0 |
| TEST:1000018 | TEST | scan direction | "Direction in terms of m/z of the scan for scanning analyzers (low to high, or high to low)." [PSI:MS] | | 0 |
| TEST:1000019 | TEST | scan law | "Describes the function in control of the m/z scan (for scanning instruments). Commonly the scan function is linear, but in principle any function can be used." [PSI:MS] | | 0 |
| TEST:1000021 | TEST | reflectron state | "Status of the reflectron, turned on or off." [PSI:MS] | | 0 |
| TEST:1000022 | TEST | TOF Total Path Length | "The length of the field free drift space in a time of flight mass spectrometer." [PSI:MS] | value-type:xsd\:float "The allowed value-type for this CV term." | 0 | | value-type:xsd\:int "The allowed value-type for this CV term." | 0 |
TermRelationship
| ID | FK_TermAccession | RelationshipType | FK_TermAccession_Related | FK_OntologyName |
|--------|------------------|------------------|--------------------------|-----------------|
| 0 | TEST:1000001 | is_a | TEST:0000000 | TEST |
| 1 | TEST:1000002 | is_a | TEST:0000000 | TEST |
| 2 | TEST:1000003 | is_a | TEST:1000002 | TEST |
| 3 | TEST:1000004 | is_a | TEST:1000002 | TEST |
| 4 | TEST:1000005 | is_a | TEST:1000002 | TEST |
| 5 | TEST:1000006 | is_a | TEST:1000002 | TEST |
| 6 | TEST:1000007 | is_a | TEST:1000002 | TEST |
| 7 | TEST:1000008 | is_a | TEST:1000007 | TEST |
| 8 | TEST:1000011 | is_a | TEST:1000007 | TEST |
| 9 | TEST:1000022 | is_a | TEST:0000000 | TEST |
Expected outcome is to execute the function for TEST:0000000/Proteomics Standards Initiative Mass Spectrometry Vocabularies and get all the following terms, because they are child/grandchild terms to TEST:0000000.
TEST:1000001
TEST:1000002
TEST:1000003
TEST:1000004
TEST:1000005
TEST:1000006
TEST:1000007
TEST:1000008
TEST:1000011
TEST:1000022
Some of these may help performance:
t: INDEX(Accession)
trt: INDEX(FK_TermAccession_Related, relationshiptype, FK_TermAccession)
trt: INDEX(FK_TermAccession, FK_TermAccession_Related, relationshiptype)
ref: INDEX(Accession, Name)
previous: INDEX(Accession, depth_level)
Please use JOIN...ON instead of (TermRelationship AS trt, Term AS ref)
What table is parentOntology in?
trt.FK_TermAccession_Related = ref.Accession show up redundantly.
Consider following list of connections:
+----------+---------+------+------------------------+
| ID | COMMAND | TIME | STATE |
+----------+---------+------+------------------------+
| 87997796 | Sleep | 15 | cleaned up |
| 90850182 | Sleep | 105 | cleaned up |
| 88009697 | Sleep | 38 | delayed commit ok done |
| 88000267 | Sleep | 6 | delayed commit ok done |
| 88009819 | Sleep | 38 | delayed commit ok done |
| 90634882 | Sleep | 21 | cleaned up |
| 90634878 | Sleep | 21 | cleaned up |
| 90634884 | Sleep | 21 | cleaned up |
| 90634875 | Sleep | 21 | cleaned up |
+----------+---------+------+------------------------+
After some short time under minute:
+----------+---------+------+------------------------+
| ID | COMMAND | TIME | STATE |
+----------+---------+------+------------------------+
| 87997796 | Sleep | 9 | cleaned up |
| 88009697 | Sleep | 32 | delayed commit ok done |
| 88000267 | Sleep | 9 | delayed commit ok done |
| 88009819 | Sleep | 31 | delayed commit ok done |
| 90634882 | Sleep | 14 | cleaned up |
| 90634878 | Sleep | 14 | cleaned up |
| 90634884 | Sleep | 14 | cleaned up |
| 90634875 | Sleep | 14 | cleaned up |
+----------+---------+------+------------------------+
8 rows in set (0.02 sec)
enter code here
After I finished writing this stackoverflow post:
+----------+---------+------+------------------------+
| ID | COMMAND | TIME | STATE |
+----------+---------+------+------------------------+
| 87997796 | Sleep | 0 | cleaned up |
| 88009697 | Sleep | 53 | delayed commit ok done |
| 88000267 | Sleep | 0 | delayed commit ok done |
| 88009819 | Sleep | 52 | delayed commit ok done |
| 90634882 | Sleep | 5 | cleaned up |
| 90634878 | Sleep | 5 | cleaned up |
| 90634884 | Sleep | 5 | cleaned up |
| 90634875 | Sleep | 5 | cleaned up |
+----------+---------+------+------------------------+
Context:
This is some 3rd vendor app opening connections (source code isn't available to us, so we don't know details). We know that their connection management is awful , they know it as well. It is awful because connections leak which you can see in first table - 90850182. If others have their timers reset, then this one starts to age infinitely. In older versions of the app it would stay forever. In newer version it is eventually captured by a "patch" which vendor introduced , which effectively cleans connections after the x seconds you specify. So it's "a leak healing patch".
The problem:
We are hosting hundreds of such vendor apps and most of them have much more than 8 connections as they have more traffic. That results in disgusting number(talking thousands) of connections we have to maintain. About 80% of connections sit in "cleaned up" state and under 120 seconds (cleaned eventually by aforementioned configurable app parameter).
This is all handled by Aurora RDS and AWS engineers told us that if the app doesn't close properly connections the standard "wait_timeout" isn't going to work. Well, "wait_timeout" becomes useless decoration in AWS Aurora, but let us take it with Jeff in other thread/topic.
So regardless, we have this magic configurable parameter from third party vendor set on this obscure app which controls eviction of stale connections and it works.
The questions:
Is it safe to evict connections which are in "cleaned up" state immediately?
At the moment this happens after 120 seconds which results in huge number of such connections. Yet in the tables above you can see that the timers are reset meaning that something is happening to these connections and they are not entirely stale. I.e. connection pooling of the app "touches" them for further re-use?
I don't posses knowledge of connection pools inner guts as how they are seen from within database. Are all reserved connections of a connection pool by default are "sleeping" in "cleaned up" state?
So say if you start cleaning too much, you will fight connection pool aggressively creating more to replenish?
Or reserved connections have some different state?
Even if you don't fully understand the context I'd expect veteran DBA or connection pool library maintainer to help with such questions. Otherwise will get my hands dirty and answer this myself eventually, would try apache connection pool, hikari, observe them and try to kill their idle connections (simulating magic parameter) and try this 3rd party app connection with 0 seconds magic parameter, see if it still works.
Appreciate your time :bow:.
The Answer
Yes, from AWS forum (https://forums.aws.amazon.com/thread.jspa?messageID=708499)
In Aurora the 'cleaned up' state is the final state of a connection
whose work is complete but which has not been closed from the client
side. In MySQL this field is left blank (no State) in the same
circumstance.
Also from the same post:
Ultimately, explicitly closing the connection in code is the best
solution here
From my personal experience as a MySQL DBA, and knowing that "cleaned up" represents a blank state, I'd definitely kill those connections.
I have a Tomcat and Mysql installed on the same machine, its not been recently updated so we are using Tomcat 7.0.31 and MySql 5.0.95 on Linux.
When a user makes a purchase, its processed by Paypal and then they contact our server and we create the license and store it in the database, but unfortunately it doesn't always work giving errors such as:
The last packet successfully received from the server was 44,533,707
milliseconds ago. The last packet sent successfully to the server was
44,533,707 milliseconds ago. is longer than the server configured
value of 'wait_timeout'. You should consider either expiring and/or
testing connection validity before use in your application, increasing
the server configured values for client timeouts, or using the
Connector/J connection property 'autoReconnect=true' to avoid this
problem.
But I dont think MySql is down since I have never ever had a problem connecting to it, the errors occur randomly about 5% of the time.
Mysql contains two db instances and in my web.xml file I have
<resource-ref>
<description>DB Connection</description>
<res-ref-name>jdbc/jaikoz</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>
<resource-ref>
<description>DB Connection2</description>
<res-ref-name>jdbc/songkong</res-ref-name>
<res-type>javax.sql.DataSource</res-type>
<res-auth>Container</res-auth>
</resource-ref>
and in context.xml (I have changed username and password) I have
<Context path="/store" privileged="true">
<Resource name="jdbc/jaikoz" auth="Container" type="javax.sql.DataSource"
maxActive="-1" maxIdle="-1" maxWait="10000"
username="usrnm" password="pwd" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/jaikoz?autoReconnect=true"/>
<Resource name="jdbc/songkong" auth="Container" type="javax.sql.DataSource"
maxActive="-1" maxIdle="-1" maxWait="10000"
username="usernm" password="pwd" driverClassName="com.mysql.jdbc.Driver"
url="jdbc:mysql://localhost:3306/songkong?autoReconnect=true"/>
</Context>
Also this configuration is from my Store web application, I also a have a jaikoz and a songkong web application both of these define one of the connections. I had to introduce 'store' since payment provider required a single url to send all successful payments (be it from songkong or jaikoz)
I wonder if having two database connections is breaking things or having two applications define the same connection is breaking things since I don't think i had such a problem when I only the jaikoz application.
The errors say I could use AutorReconnect=true, but I am already doing that.
Ran Show Process List as suggested and got the following:
mysql> show processlist
-> ;
+--------+------+-----------------+----------+---------+-------+-------+------------------+
| Id | User | Host | db | Command | Time | State | Info |
+--------+------+-----------------+----------+---------+-------+-------+------------------+
| 127681 | paul | localhost:40360 | jaikoz | Sleep | 5 | | NULL |
| 127682 | paul | localhost:40361 | jaikoz | Sleep | 7 | | NULL |
| 127683 | paul | localhost:40362 | jaikoz | Sleep | 11 | | NULL |
| 127684 | paul | localhost:40363 | jaikoz | Sleep | 2 | | NULL |
| 127685 | paul | localhost:40364 | jaikoz | Sleep | 17 | | NULL |
| 127754 | paul | localhost:40664 | jaikoz | Sleep | 20 | | NULL |
| 127755 | paul | localhost:40665 | jaikoz | Sleep | 8 | | NULL |
| 127756 | paul | localhost:40666 | jaikoz | Sleep | 25 | | NULL |
| 128444 | paul | localhost:41250 | jaikoz | Sleep | 14 | | NULL |
| 128445 | paul | localhost:41251 | jaikoz | Sleep | 10 | | NULL |
| 134807 | paul | localhost:56829 | jaikoz | Sleep | 226 | | NULL |
| 134849 | paul | localhost:38795 | songkong | Sleep | 475 | | NULL |
| 143552 | paul | localhost:35811 | jaikoz | Sleep | 19338 | | NULL |
| 145211 | paul | localhost | jaikoz | Query | 0 | NULL | show processlist |
+--------+------+-----------------+----------+---------+-------+-------+------------------+
14 rows in set (0.00 sec)
Maybe shoud not be so many Jaikoz processes ?
How do i resolve this ?
Edited
Speaking to someone else since it is only connections from the Store web application that is failing and that only does anything when a purchase is made which is currently only a few times a day it sounds like the connection is being timed out by MySql itself without telling the jdbc pool, so when try to use the connection via the pool we get the error.
I added testWhileIdle="true" to my context.xml and removed the autoreconnect=true hoping this will remove connections from pool before mysql just gives up on them.
Unfortunately still failing sporadically but now giving a slightly different error message
JDBC begin transaction failed
Turn off connection pooling.
Turn off auto-reconnect
Be sure your code catches transaction errors, and replays them if necessary -- especially in the case of deadlocks.
Those tend to be issues relating to integrity of financial transactions.
If those do not clear up the problem, then provide
Queries per second
SHOW VARIABLES LIKE '%timeout';`
Sample of the transaction (SQL, please, not Java)
SHOW CREATE TABLE
As for the PROCESSLIST...
I see one connection (possibly pooled) that performed some SQL 5 seconds ago.
I see one connection (possibly pooled) that has not done anything for 19338 seconds ago (over 5 hours).
I would guess that you have not needed more than 12 simultaneous connections in the past 5 hours.
The list looks "normal".
The key thing was that I needed to provide a validationQuery in order for the validation done by testForIdle to work, since then had no problems.
<Context path="/store" privileged="true">
<Resource name="jdbc/myapp" auth="Container" type="javax.sql.DataSource"
maxActive="10" maxIdle="10"
testWhileIdle="true" validationQuery="select 1" validationQueryTimeout="5"
username="usrnm" password="pwd"
driverClassName="com.mysql.jdbc.Driver" testOnBorrow="true"
url="jdbc:mysql://localhost:3306/myapp"/>
</Context>
I see frequently when I run mysqladmin proc or when I review the MySQL Server process list a user marked with: unauthenticated user trying to connect.
+-----+----------------------+--------------+-----------------+---------+------+------------------+------------------+-----------+---------------+-----------+
| Id | User | Host | db | Command | Time | State | Info | Rows_sent | Rows_examined | Rows_read |
+-----+----------------------+--------------+-----------------+---------+------+------------------+------------------+-----------+---------------+-----------+
| 40 | unauthenticated user | x.x.x.x:xxxx | | Connect | | Reading from net | | 0 | 0 | 0 |
What may causes such thing?
Is that normal, or should I investigate my system for any vulnerability or security breach?
Thanks
unauthenticated user is the user connected and not yet sent authentication credentials. Doesn't look like a hack attempt to me.
I created a replication between two computers (a laptop I use to add new datas in my db and a server that save everything I do on the laptop) and it works fine.
But today, my laptop was online so I was not able to update my server.
Result : I updated some lines and I created a lot of lines and when my laptop was back online, the replication did not works on the datas I've been working on offline.
Could anybody give me an advice to update datas (on the server) that have been modified on the laptop while I was not connected ? I don't get why it doesn't works !
Thx !
update : here is my show slave status :
mysql> show slave status;
+----------------------------------+-------------+-------------+-------------+--
-------------+------------------+---------------------+-------------------------
---+---------------+-----------------------+------------------+-----------------
--+-----------------+---------------------+--------------------+----------------
--------+-------------------------+-----------------------------+------------+--
----------+--------------+---------------------+-----------------+--------------
---+----------------+---------------+--------------------+--------------------+-
-------------------+-----------------+-------------------+----------------+-----
------------------+-------------------------------+---------------+-------------
--+----------------+----------------+-----------------------------+-------------
-----+
| Slave_IO_State | Master_Host | Master_User | Master_Port | C
onnect_Retry | Master_Log_File | Read_Master_Log_Pos | Relay_Log_File
| Relay_Log_Pos | Relay_Master_Log_File | Slave_IO_Running | Slave_SQL_Runnin
g | Replicate_Do_DB | Replicate_Ignore_DB | Replicate_Do_Table | Replicate_Ignor
e_Table | Replicate_Wild_Do_Table | Replicate_Wild_Ignore_Table | Last_Errno | L
ast_Error | Skip_Counter | Exec_Master_Log_Pos | Relay_Log_Space | Until_Conditi
on | Until_Log_File | Until_Log_Pos | Master_SSL_Allowed | Master_SSL_CA_File |
Master_SSL_CA_Path | Master_SSL_Cert | Master_SSL_Cipher | Master_SSL_Key | Seco
nds_Behind_Master | Master_SSL_Verify_Server_Cert | Last_IO_Errno | Last_IO_Erro
r | Last_SQL_Errno | Last_SQL_Error | Replicate_Ignore_Server_Ids | Master_Serve
r_Id |
+----------------------------------+-------------+-------------+-------------+--
-------------+------------------+---------------------+-------------------------
---+---------------+-----------------------+------------------+-----------------
--+-----------------+---------------------+--------------------+----------------
--------+-------------------------+-----------------------------+------------+--
----------+--------------+---------------------+-----------------+--------------
---+----------------+---------------+--------------------+--------------------+-
-------------------+-----------------+-------------------+----------------+-----
------------------+-------------------------------+---------------+-------------
--+----------------+----------------+-----------------------------+-------------
-----+
| Waiting for master to send event | ***.***.***.*** | masterRepli | 3306 |
60 | mysql-bin.000027 | 454717 | aofr19072-relay-bin.0000
02 | 1227 | mysql-bin.000027 | Yes | Yes
| | | |
| | | 0 |
| 0 | 454717 | 1387 | None
| | 0 | No | |
| | | |
0 | No | 0 |
| 0 | | |
2 |
+----------------------------------+-------------+-------------+-------------+--
-------------+------------------+---------------------+-------------------------
---+---------------+-----------------------+------------------+-----------------
--+-----------------+---------------------+--------------------+----------------
--------+-------------------------+-----------------------------+------------+--
----------+--------------+---------------------+-----------------+--------------
---+----------------+---------------+--------------------+--------------------+-
-------------------+-----------------+-------------------+----------------+-----
------------------+-------------------------------+---------------+-------------
--+----------------+----------------+-----------------------------+-------------
-----+
1 row in set (0.00 sec)
BRUTE FORCE SYNC
I wrote out a detailed description on how to resync a Slave to its Master in a Circular Replication setup
PRECISION STRIKE SYNC
If you simply want to find the differences and update only differences, you must use mk-table-checksum and mk-table-sync.
Give it a Try !!!
Is it possible that the slave server just doesn't have the slave process running?
See if the slave process is running by typing SHOW SLAVE STATUS; at the MySQL command prompt on the slave.
Then try starting it with START SLAVE;
If it still isn't working, post the output of the SHOW SLAVE STATUS; command, and we'll try to help you further :)