I set up a class with this class property:
##db : DB::Database = DB.open "mysql://root:root#localhost/db_name"
Then, in any class method I can use ##db:
##db.query("select * from thing") do |rs|
rs.each do
# make object
end
end
I was hoping this way I could DB.open only when the app starts, and not again for each request. This works great... until I run wrk -c10 and test multiple connections at the same time. Here is some error feedback:
Unhandled exception on HTTP::Handler Closed stream (IO::Error)
0x10e69a440: *raise:NoReturn at ?? 0x10e6a8d57:
*IO::FileDescriptor+#IO::Buffered#read:Int32 at ?? 0x10e7030db: *MySql::ReadPacket#read:Int32 at ??
0x10e74d89b: *DB::ResultSet+#DB::Disposable#close:(Bool | Nil) at ??
0x10e745ba2: *DefaultController::database:Nil
at ?? 0x10e7500c4:
*HTTP::Server::RequestProcessor#process<(OpenSSL::SSL::Socket::Server | TCPSocket+), (OpenSSL::SSL::Socket::Server | TCPSocket+),
IO::FileDescriptor>:Nil at ?? 0x10e6995fd:
*Fiber#run:(IO::FileDescriptor | Nil) at ??
I "fixed" the problem by doing this on each request:
DB.open("mysql://root:root#localhost/db_name") do |db|
db.query("select * from thing") do |rs|
rs.each do
# make object
end
end
end
This works, but with awful performance. What am I doing wrong?
Related
I have seen this issue many times happening to many people (here). I am still struggling trying to validate whether what my dictionary captures from a JSON is "None" or not but I still get the following error.
This code is supposed to call a CURL looking for the 'closed' value in the 'status' key until it finds it (or 10 times). When payment is done by means of a QR code, status changes from opened to closed.
status = (my_dict['elements'][0]['status'])
TypeError: 'NoneType' object is not subscriptable
Any clue of what am I doing wrong and how can I fix it?
Also, if I run the part of the script that calls the JSON standalone, it executes smoothly everytime. Is it anything in the code that could be affecting the CURL execution?
By the way, I have started programming 1 week ago so please excuse me if I mix concepts or say something that lacks of common sense.
I have tried to validate the IF with "is not" instead of "!=" and also with "None" instead of "".
def show_qr():
reference_json = reference.replace(' ','%20') #replaces "space" with %20 for CURL assembly
url = "https://api.mercadopago.com/merchant_orders?external_reference=" + reference_json #CURL URL concatenate
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer MY_TOKEN"
pygame.init()
ventana = pygame.display.set_mode(window_resolution,pygame.FULLSCREEN) #screen settings
producto = pygame.image.load("qrcode001.png") #Qr image load
producto = pygame.transform.scale(producto, [640,480]) #Qr size
trials = 0 #sets while loop variable start value
status = "undefined" #defines "status" variable
while status != "closed" and trials<10: #to repeat the loop until "status" value = "closed"
ventana.blit(producto, (448,192)) #QR code position setting
pygame.display.update() #
response = requests.request("GET", url, headers=headers) #makes CURL GET
lag = 0.5 #creates an incremental 0.5 seconds everytime return value is None
sleep(lag) #
json_data = (response.text) #Captures JSON response as text
my_dict = json.loads(json_data) #creates a dictionary with JSON data
if json_data != "": #Checks if json_data is None
status = (my_dict['elements'][0]['status']) #If json_data is not none, asigns 'status' key to "status" variable
else:
lag = lag + 0.5 #increments lag
trials = trials + 1 #increments loop variable
sleep (5) #time to avoid being banned from server.
print (trials)
From your original encountered error, it's not clear what the issue is. The problem is that basically any part of that statement can result in a TypeError being raised as the evaluated part is a None. For example, given my_dict['elements'][0]['status'] this can fail if my_dict is None, or also if my_dict['elements'] is None.
I would try inserting breakpoints to better assist with debugging the cause. another solution that might help would be to wrap each part of the statement in a try-catch block as below:
my_dict = None
try:
elements = my_dict['elements']
except TypeError as e:
print('It possible that my_dict maybe None.')
print('Error:', e)
else:
try:
ele = elements[0]
except TypeError as e:
print('It possible that elements maybe None.')
print('Error:', e)
else:
try:
status = ele['status']
except TypeError as e:
print('It possible that first element maybe None.')
print('Error:', e)
else:
print('Got the status successfully:', status)
I have this code for connect to MySQL through a SSH, inside of a python class:
def executeQuery(self, query_string):
print("connecting to database " + self.sql_main_database)
with SSHTunnelForwarder(
(
self.ssh_host,
self.ssh_port),
ssh_username = self.ssh_user,
ssh_pkey = self.pkey,
remote_bind_address=(self.sql_hostname, self.sql_port)
) as tunnel:
print("performing connection")
conn = pymysql.connect(
host="127.0.0.1",
user=self.sql_username,
password=self.sql_password,
db=self.sql_main_database,
port=tunnel.local_bind_port)
query = query_string
print("Querying")
data = pd.read_sql_query(query, conn)
print("Done!")
conn.close()
return data
The code is working well, but when the query is not well defined, the notebook freezes.
Then, I tried to use a try/catch, and the code ended like this
def executeQuery(self, query_string):
try:
with SSHTunnelForwarder(
(
self.ssh_host,
self.ssh_port
),
ssh_username = self.ssh_user,
ssh_pkey = self.pkey,
remote_bind_address=(self.sql_hostname, self.sql_port)
) as tunnel:
try:
conn = pymysql.connect(
host = "127.0.0.1",
user = self.sql_username,
password = self.sql_password,
db = self.sql_main_database,
port = tunnel.local_bind_port
)
try:
query = query_string
data = pd.read_sql_query(query, conn)
return data
except DatabaseError as e:
Log.debug(self,str(e))
raise DatabaseError
except pymysql.err.InternalError as e:
Log.debug(self, str(e))
raise DataError
except Exception as e:
Log.debug(self, "[Error]Setting up database: \'" + self.sql_main_database + "\'")
raise DataError
The issue is that pd.read_sql_query never stops so the except is never called, the try won't fail, and the process will just continue forever
The timeout workaround is not possible, because the queries don't have defined execution times and some of them can stay in processing during a couple of hours.
I'm not sure how to fix it.
Indeed the problem was not on the connector, just updating the jupyter version was needed.
I am using http.get() to get a JSON from an API I am using, but it's not getting the data. I have the suspicion that this JSON is too big for the NodeMCU. I only need the information in the subpart "stats:". Is it possible to only http.get() that part of the JSON?
EDIT:
This is my code
function getstats()
http.get("https://api.aeon-pool.com/v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit", nil, function(code, pool)
if (code < 0) then
print("can't get stats")
else
h = cjson.decode(pool)
hashrate = h[1]["hashrate"]
print(hashrate)
dofile('update_display.lua')
end
end)
end
I also have another function getting data from another api above getstats()
function getaeonrate()
http.get("https://api.coinmarketcap.com/v1/ticker/aeon/?convert=EUR", nil, function(code, dataaeon)
if (code < 0) then
print("can't get aeon")
else
-- Decode JSON data
m = cjson.decode(dataaeon)
-- Extract AEON/EUR price from decoded JSON
aeonrate = string.format("%f", m[1]["price_eur"]);
aeonchange = "24h " .. m[1]["percent_change_24h"] .. "% 7d " .. m[1]["percent_change_7d"] .. "%"
dofile('update_display.lua')
end
end)
end
But now the weird thing is, when I want to access 'pool' from getstats() I get the json data from getaeonrate(). So "hashrate" isn't even in the json because I am getting the json from another function.
I tried making a new project only with getstats() and that doesn't work at all I always get errors like this
HTTP client: Disconnected with error: -9
HTTP client: Connection timeout
HTTP client: Connection timeout
Yesterday I thought that the response was too big from api.aeon-pool.com, I if you look at the json in your webbrowser you can see that the top entry is 'stats:' and I only need that, none of the other stuff. So If the request is to big It would be nice to only http.get() that part of the json, hence my original question. At the moment I am not even sure what is not working correctly, I read that the nodemcu firmware generally had problems with http.get() and that it didn't work correctly for a long time, but getting data from api.coinmarketcap.com works fine in the original project.
The problems with the HTTP module are with near certainty related to https://github.com/nodemcu/nodemcu-firmware/issues/1707 (SSL and HTTP are problematic).
Therefore, I tried with the more bare-bone TLS module on the current master branch. This means you need to manually parse the HTTP response including all headers looking for the JSON content. Besides, you seem to be on an older NodeMCU version as you're still using CJSON - I used SJSON below:
Current NodeMCU master branch
function getstats()
buffer = nil
counter = 0
local srv = tls.createConnection()
srv:on("receive", function(sck, payload)
print("[stats] received data, " .. string.len(payload))
if buffer == nil then
buffer = payload
else
buffer = buffer .. payload
end
counter = counter + 1
-- not getting HTTP content-length header back -> poor man's checking for complete response
if counter == 2 then
print("[stats] done, processing payload")
local beginJsonString = buffer:find("{")
local jsonString = buffer:sub(beginJsonString)
local hashrate = sjson.decode(jsonString)["stats"]["hashrate"]
print("[stats] hashrate from aeon-pool.com: " .. hashrate)
end
end)
srv:on("connection", function(sck, c)
sck:send("GET /v1/stats_address?address=WmsGUrXTR7sgKmHEqRNLgPLndWKSvjFXcd4soHnaxVjY3aBWW4kncTrRcBJJgUkeGwcHfzuZABk6XK6qAp8VmSci2AyGHcUit HTTP/1.1\r\nHost: api.aeon-pool.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
end)
srv:connect(443, "api.aeon-pool.com")
end
Note that the receive event is fired for every network frame: https://nodemcu.readthedocs.io/en/latest/en/modules/net/#netsocketon
NodeMCU fails to establish a connection to api.coinmarketcap.com due to a TLS handshake failure. Not sure why that is. Otherwise your getaeonrate() could be implemented likewise.
Frozen 1.5.4 branch
With the old branch the net module can connect to coinmarketcap.com.
function getaeonrate()
local srv = net.createConnection(net.TCP, 1)
srv:on("receive", function(sck, payload)
print("[aeon rate] received data, " .. string.len(payload))
local beginJsonString = payload:find("%[")
local jsonString = payload:sub(beginJsonString)
local json = cjson.decode(jsonString)
local aeonrate = string.format("%f", json[1]["price_eur"]);
local aeonchange = "24h " .. json[1]["percent_change_24h"] .. "% 7d " .. json[1]["percent_change_7d"] .. "%"
print("[aeon rate] aeonrate from coinmarketcap.com: " .. aeonrate)
print("[aeon rate] aeonchange from coinmarketcap.com: " .. aeonchange)
end)
srv:on("connection", function(sck, c)
sck:send("GET /v1/ticker/aeon/?convert=EUR HTTP/1.1\r\nHost: api.coinmarketcap.com\r\nConnection: close\r\nAccept: */*\r\n\r\n")
end)
srv:connect(443, "api.coinmarketcap.com")
end
Conclusion
The HTTP module and TLS seem a no-go for your APIs due to a bug in the firmware (1707).
The net/TLS module of the current master branch manages to connect to api.aeon-pool.com but not to api.coinmarketcap.com.
With the old and frozen 1.5.4 branch it's exactly the other way around.
There may (also) be issues with cipher suits that don't match between the firmware and the API provider(s).
-> :( no fun like that
I want to the mysql-proxy lua script to handle interleaving accesses to a website (e.g. two different browser windows/users) but being able to pause/delay one of the two without influencing the other.Handling sessions interleavingly is possible in mysql-proyx lua (so it seems regarding the later listed output) but as soon as I start delaying the script it blocks everything and the other session cannot advance either.
-- the query which indicates the session/connection that shall be delayed at that execution
local qoi = "SELECT loginattempts,uid FROM mybb_users WHERE username='user1' LIMIT 1"
function read_query(packet)
if string.byte(packet) == proxy.COM_QUERY then
query = packet:sub(2)
start_time = os.time()
if query == qoi then
print("busy wait")
while os.time() < start_time + 20 do
--nothing
end
print("busy wait end")
end
print("Connection id: " .. proxy.connection.server.thread_id)
end
end
However this script ends up with output:
Connection id: 36
busy wait
busy wait end
Connection id: 36
Connection id: 36
Connection id: 36
Connection id: 37
Connection id: 37
Connection id: 36
Connection id: 36
Connection id: 36
Connection id: 37
and not the expected
Connection id: 36
busy wait
connection id: 37
connection id: 37
busy wait end
Connection id: 36
Is my intention even achievable and if so how?
It seems to be impossible to delay the session within lua but it works just as fine if I outsource the delay to mysql server as this will force the interleaving as well.
local DEBUG = true
local qoi = "SELECT loginattempts,uid FROM mybb_users WHERE username='user1' LIMIT 1"
function read_query(packet)
ret = nil
comp_query = qoi
if string.byte(packet) == proxy.COM_QUERY then
query = packet:sub(2)
if query == comp_query then
if DEBUG then
print("found matching query " .. packet:sub(2))
print("insert sleep")
end
inj_query = "SELECT sleep(30);"
new_packet = string.char(proxy.COM_QUERY) .. inj_query
proxy.queries:append(1, new_packet, { resultset_is_needed = true })
proxy.queries:append(2, packet, { resultset_is_needed = true })
ret = proxy.PROXY_SEND_QUERY
end
end
return ret
end
function read_query_result(inj)
if inj.id == 1 then
if DEBUG then
print("sleep query returns")
end
return proxy.PROXY_IGNORE_RESULT
end
if inj.id == 2 then
if DEBUG then
print("regular query returns")
end
return
end
return
end
I created a script for the game Garry's Mod, but once is loaded on some servers, it gets the next error:
[ERROR] addons/ulib-master/lua/ulib/shared/hook.lua:110: addons/applysystem/lua/applysystem/init.lua:13: bad argument #1 to 'pairs' (table expected, got nil)
fn - [C]:-1
unknown - addons/ulib-master/lua/ulib/shared/hook.lua:110
How can i fix it? this is the line 13:
for _, row in pairs(results[1].data) do
If needed, theres the entire function where the error is created:
db:Query("SELECT * FROM "..ApplySystem.MySQL.TableName.." WHERE delivered=0 AND status='Accepted.'", function(results)
for _, row in pairs(results[1].data) do
local steamid64 = row.steamid
if steamid64 != "" or steamid64 != nil then
local TransfSteamID = util.SteamIDFrom64(steamid64)
RunConsoleCommand("ulx","adduserid",TransfSteamID,ApplySystem.MySQL.DefaultRank)
db:Query("UPDATE "..ApplySystem.MySQL.TableName.." SET delivered=1 WHERE steamid='"..row.steamid.."' ")
end
end
end)
Fixed, thanks guys, it was because i were trying to retrieve nil values.