OpenResty / Lua - Maintain mysql database connection in worker threads - mysql

I have a simple module called "firewall.lua" that I wrote that has a function
firewall.check_ip(ip) which connects to localhost mysql and performs a query and returns the result. The function gets called from within Location / blocks in nginx sites via access_by_lua_block . The module gets initialized by init_worker_by_lua firewall.init().
Everything works as expected.
What I'd like to do however is maintain the database connection on the worker thread(s) so that I don't have to re-connect every time the function is called but instead re-use the existing connection established by the worker during initialization.
I'm not quite sure how to do this or if its actually doable in openresty/lua. I tried initializing the database connection variables outside of the function to give them scope within the module instead of function and I get various API errors that did not point me in the right direction.
Thank you!

This is possible using the OpenResty cosocket API, which gives you the ability to use a pool of non-blocking connections. There's already one MySQL driver (lua-resty-mysql) which uses the cosocket API. Since you didn't provide a code sample, I'm assuming you're not using it.
Example of a connection and query using lua-resty-mysql (untested):
access_by_lua_block {
local mysql = require "resty.mysql";
local db, err = mysql:new()
db:set_timeout(1000) -- 1 second
local ok, err, errcode, sqlstate = db:connect{
host = "127.0.0.1",
port = 3306,
database = "my_db",
user = "my_user",
password = "my_pwd",
}
if not ok then
ngx.say("Connection to MySQL failed: ", err)
return
end
result, err, errcode, sqlstate = db:query("select ...")
if not result then
ngx.say("MySQL error: ", err, ".")
return
end
db:close()
}
In case, e.g., you want to control the pool name or use other options, you can pass on additional parameters to connect:
...
local ok, err, errcode, sqlstate = db:connect{
host = "127.0.0.1",
port = 3306,
database = "my_db",
user = "my_user",
password = "my_pwd",
pool = "my_connection_pool",
}
...
You can find more information in the official docs:
lua-resty-mysql: https://github.com/openresty/lua-resty-mysql
Cosocket API: https://openresty-reference.readthedocs.io/en/latest/Lua_Nginx_API/#ngxsockettcp

Related

Qt & MySQL first connections fail

I using Qt 5.15 and I make a standard connection to MySQL
bool open() {
QMutexLocker ml(&fMutex);
fDatabaseNumber = QString::number(++fDatabaseCounter);
fSqlDatabase = QSqlDatabase::addDatabase("QMYSQL", fDatabaseNumber);
fSqlDatabase.setHostName(host);
fSqlDatabase.setDatabaseName(schema);
fSqlDatabase.setUserName(username);
fSqlDatabase.setPassword(password);
if (!fSqlDatabase.open()) {
qDebug() << fSqlDatabase.lastError().databaseText();
return false;
}
fQuery = new QSqlQuery(fSqlDatabase);
return true;
}
Like usually :). I call this code from 10 different threads at same time. And always several of the first connections to database fails with "Can't connect to MySQL server on '127.0.0.1'" message. All of next connections successful, no mater how many connections threads make at the time. I don't know, what I'm doing wrong.

How to connect to specific database using JDBC and Google Apps Script

I'm trying to connect to a Google Cloud MySQL 5.7 instance using GAS and JDBC. I'm able to run the following without error:
var conn = Jdbc.getCloudSqlConnection("jdbc:google:mysql://my-instance-111111:us-central1:mydbname", user,userPwd)
But this doesn't connect to a specific database, i.e. when I run
var stmt = conn.createStatement();
var results = stmt.executeQuery('SELECT col FROM myTable;');
I get Error Exception: No database selected.
One approach to try to fix:
The GAS docs indicate that I can use advanced parameters to include the DB name, but when I run var conn = Jdbc.getCloudSqlConnection("jdbc:google:mysql://my-instance-111111:us-central1:mydbname", user,userPwd,mydbname) ,
I get Exception: The parameters (String,String,String,String) don't match the method signature for Jdbc.getCloudSqlConnection.
A collection of getCloudSqlConnection statements that I've tried based on a number of StackOverflow posts:
var conn = Jdbc.getCloudSqlConnection("jdbc:google:mysql://my-instance-111111:us-central1:mydbname/mydbname", user,userPwd)
var conn = Jdbc.getCloudSqlConnection("jdbc:google:mysql://my-instance-111111:us-central1:mydbname:3306/mydbname", user,userPwd)
var conn = Jdbc.getCloudSqlConnection("jdbc:google:mysql://my-instance-111111:us-central1:mydbname:3307/mydbname", user,userPwd)
For the above statements, I also tried replacing my-instance-111111:us-central1:mydbname with the public IP supplied on the GC overview webpage. All return Exception: Failed to establish a database connection. Check connection string, username and password.
I'm using user root and the appropriate password. I can enter the DB via the command line with the user and password that I'm supplying in GAS, so I don't think I'm supplying the wrong user and password.
Edit:
I went back to the docs like #AddonDepot mentioned and realized that I need to pass an object, but I still can't get this work....
var obj = {
connectTimeoutSeconds: 15,
database: "mydbname",
instance: "my-instance-111111:us-central1:mydbname",
password: userPwd,
queryTimeoutSeconds: 15,
user: user };
var conn = Jdbc.getCloudSqlConnection("jdbc:google:mysql://my-instance-111111:us-central1:mydbname", obj);
returns
Exception: The following connection properties are unsupported: database,instance,connectTimeoutSeconds,queryTimeoutSeconds. .
Did I do something wrong in creating the object? I'm guessing not, because GAS recognizes user and password. Why wouldn't it recognize the other advanced parameters?
We have to distinguish between database system instances (sometimes referred simply as database; I'll use instances) and databases within them. That means that you may have an instance but may not have created any database in it.
You can use Apps Script to create a new database in your instance:
const connectionName = 'connection-name:for-your:instance'
const dbName = 'database'
const username = 'user'
const password = 'password'
function createDB() {
const conn = Jdbc.getCloudSqlConnection(`jdbc:google:mysql://${connectionName}`, username, password)
conn.createStatement().execute('CREATE DATABASE ' + dbName)
}
Once you have the database created in your instance you can use it:
function useDB() {
const conn = Jdbc.getCloudSqlConnection(`jdbc:google:mysql://${connectionName}/${dbName}`, username, password)
// Use conn
}
References
JDBC (Google Apps Script guide)

Node js mysql stored procedure call in a for loop

So, I'm currently using mysql npm package https://www.npmjs.com/package/mysql. I have a requirement where I'll have to call a stored procedure multiple times with the caveat that subsequent stored procedure calls depend on the previous call. Pseudo code will be as follows:
let mysql = require("mysql");
let mysqlPoolConnection = mysql.createPool({
connectionLimit: 20,
host: '0.0.0.0',
port: '3306',
user: 'user1',
password: 'pwd',
database: 'mysql_db'
});
for (let index = 0; index < params.length; index++) {
let sp_ProcedureCall = "CALL sp_StoredProcedure(?, ?, ?, ?)";
let sp_ProcedureParams = [params[index].firstParam, params[index].secondParam, params[index].thirdParam, params[index].fourthParam];
// This is where the issue is. I'd like to call the stored procedure once and then once I get a response from it then make subsequent calls. Basically, depending on the previous stored procedure result, I decide whether I continue with subsequent calls or not.
mysqlPoolConnection.query(sp_ProcedureCall, sp_ProcedureParams, (errorObj, responseObj, fieldsObj) => {
}
}
NodeJS is asynchronous, meaning your for-loop iterates without waiting for the result of the call. You need to control the flow of your program if you want to "wait" for previous results.
Look at a library like async to enabled that type of control flow. From your description, eachSeries() might be a good fit - to run a function for each value in an array, and only run one at a time. (Or reduce, depending what you need - there are other options too)

Error of update(conn,tablename,colnames,data,whereClause) by using matlab connect ODBC and mySQL server 5.6

Isn't my coding typing wrong way? I need create an update button so user can edit the information by using Matlab. After update, the button need connect to mySQL server 5.6 and ODBC connector.
This is my code:
% --- Executes on button press in update.
function update_Callback(hObject, eventdata, handles)
% hObject handle to update (see GCBO)
% eventdata reserved - to be defined in a future version of MATLAB
% handles structure with handles and user data (see GUIDATA)
%Display dialog box to confirm save
choice = questdlg('Confirm update to database?', ...
'', ...
'Yes','No','Yes');
% Handle dialog box response
switch choice
case 'Yes'
%Set preferences with setdbprefs.
setdbprefs('DataReturnFormat', 'cellarray');
%Make connection to database.
conn = database('animal_cbir', '', '');
%Test if database connection is valid
testConnection = isconnection(conn);
disp(testConnection);
fileID = getappdata(0,'namevalue');
imageID = fileID;
name = get(handles.edit11,'String');
commonName = get(handles.edit1,'String');
scientificName = get(handles.edit2,'String');
class = get(handles.edit3,'String');
diet = get(handles.edit4,'String');
habitat = get(handles.edit5,'String');
lifeSpan = get(handles.edit6,'String');
size = get(handles.edit7,'String');
weight = get(handles.edit8,'String');
characteristic = get(handles.edit10,'String');
tablename = 'animal';
colnames ={'imageID','name','commonName','scientificName','class','diet','habitat','lifeSpan','size','weight','characteristic'};
data = {imageID,name,commonName,scientificName,class,diet,habitat,lifeSpan,size,weight,characteristic};
disp (data);
whereClause = sprintf(['where imageID = "%s"'],fileID);
update(conn,tablename,colnames,data,whereClause);
updateSuccess = helpdlg('Existing animal species successfully updated in database.');
commit(conn);
case 'No'
end
Error I am getting:
No method 'setInt' with matching signature found for class 'sun.jdbc.odbc.JdbcOdbcPreparedStatement'.
Hope that anyone can help me solve it.

mysql - node - Row inserted and queryable via server connection lost on DB restart

I ran into an issue testing today that occurred during or after an insert via a connection on my node server. The code where the insert is performed looks something like this:
// ...
username = esc(username);
firstname = esc(firstname);
lastname = esc(lastname);
var values = [username, firstname, lastname].join(',');
var statement = 'INSERT INTO User(Username,FirstName,LastName) VALUES({0});\n'+
'SELECT FirstName, LastName, Username, Id, IsActive FROM User WHERE Username={1};'
statement = merge( statement, [ values, username ] );
conn.query(statement, function(e, rows, fields){
e ? function() {
res.status(400);
var err = new Error;
err.name = 'Bad request';
err.message = 'A problem occurred during sign-up.';
err.details = e;
res.json(err);
}() : function(){
res.json( rows[1] );
}();
}
A quick note on esc() and merge(), these are simply util functions that help prepare the database statement.
The above code completed successfully, ie. the response was a 200 with the newly inserted user row in the body. The row inserted was queryable via the same connection throughout the day. I only noticed this afternoon when running the following generic query as root via shell, the row was missing.
SELECT Id, FirstName, LastName FROM User;
So at that point I restarted the database and the node server. Unfortunately. Now it would appear the row is gone entirely, as well as any reliable path to troubleshoot.
Here are some details of interest to my server setup. As of yet, no idea how (if at all) any of these could be suspect.
Uses only single connection as opposed to conn pool (for now)
multipleStatements=true in the connection config (obviously above snippet makes use of this)
SET autocommit = 0; START TRANSACTION; COMMIT; used elsewhere in the codebase to control rollback
Using poor man's keep-alive every 30 seconds to avoid connection timing out: SELECT 1;
I've been reading up all evening and am running out of ideas. Any suggestions? Is this likely an issue of uncommitted data? If so, is there a reliable way to debug this? Better yet, is there any way to prevent it? What could be the cause? And finally, if in debugging my server I find data in this state, is there a way to force commit at least so that I don't lose my changes?