Exception handling using mysql with twisted adbapi and scrapy - mysql

I'm using this scrapy pipeline. If there is any error in the sql in the insert_record function, it fails silently. For example, if a column name is miss-spelled, like this
def _insert_record(self, tx, item):
print "before tx.execute"
result = tx.execute(
""" INSERT INTO table(col_one, col_typo, col_three) VALUES (1,2,3)"""
)
print "after tx.execute"
if result > 0:
self.stats.inc_value('database/items_added')
then nothing is output after "before execute". There is a handle_error method but that's not called either. How can I catch and handle such errors?

Just needed to surround it with try...except
try:
result = tx.execute(
"""INSERT INTO table(col_one, col_typo, col_three) VALUES (1,2,3)"""
)
except Exception,e:
print str(e)

Related

Why am I not being able of detecting a None value from a dictionary

I have seen this issue many times happening to many people (here). I am still struggling trying to validate whether what my dictionary captures from a JSON is "None" or not but I still get the following error.
This code is supposed to call a CURL looking for the 'closed' value in the 'status' key until it finds it (or 10 times). When payment is done by means of a QR code, status changes from opened to closed.
status = (my_dict['elements'][0]['status'])
TypeError: 'NoneType' object is not subscriptable
Any clue of what am I doing wrong and how can I fix it?
Also, if I run the part of the script that calls the JSON standalone, it executes smoothly everytime. Is it anything in the code that could be affecting the CURL execution?
By the way, I have started programming 1 week ago so please excuse me if I mix concepts or say something that lacks of common sense.
I have tried to validate the IF with "is not" instead of "!=" and also with "None" instead of "".
def show_qr():
reference_json = reference.replace(' ','%20') #replaces "space" with %20 for CURL assembly
url = "https://api.mercadopago.com/merchant_orders?external_reference=" + reference_json #CURL URL concatenate
headers = CaseInsensitiveDict()
headers["Authorization"] = "Bearer MY_TOKEN"
pygame.init()
ventana = pygame.display.set_mode(window_resolution,pygame.FULLSCREEN) #screen settings
producto = pygame.image.load("qrcode001.png") #Qr image load
producto = pygame.transform.scale(producto, [640,480]) #Qr size
trials = 0 #sets while loop variable start value
status = "undefined" #defines "status" variable
while status != "closed" and trials<10: #to repeat the loop until "status" value = "closed"
ventana.blit(producto, (448,192)) #QR code position setting
pygame.display.update() #
response = requests.request("GET", url, headers=headers) #makes CURL GET
lag = 0.5 #creates an incremental 0.5 seconds everytime return value is None
sleep(lag) #
json_data = (response.text) #Captures JSON response as text
my_dict = json.loads(json_data) #creates a dictionary with JSON data
if json_data != "": #Checks if json_data is None
status = (my_dict['elements'][0]['status']) #If json_data is not none, asigns 'status' key to "status" variable
else:
lag = lag + 0.5 #increments lag
trials = trials + 1 #increments loop variable
sleep (5) #time to avoid being banned from server.
print (trials)
From your original encountered error, it's not clear what the issue is. The problem is that basically any part of that statement can result in a TypeError being raised as the evaluated part is a None. For example, given my_dict['elements'][0]['status'] this can fail if my_dict is None, or also if my_dict['elements'] is None.
I would try inserting breakpoints to better assist with debugging the cause. another solution that might help would be to wrap each part of the statement in a try-catch block as below:
my_dict = None
try:
elements = my_dict['elements']
except TypeError as e:
print('It possible that my_dict maybe None.')
print('Error:', e)
else:
try:
ele = elements[0]
except TypeError as e:
print('It possible that elements maybe None.')
print('Error:', e)
else:
try:
status = ele['status']
except TypeError as e:
print('It possible that first element maybe None.')
print('Error:', e)
else:
print('Got the status successfully:', status)

Python 3 psycopg2 COPY from stdin failed: error in .read()

I am trying to apply the code found on this page, in particular part 'Copy Data from String Iterator' of the Table of Contents, but run into an issue with my code.
Since not all lines coming from the generator (here log_lines) can be imported into the PostgreSQL database, I try to filter the correct lines (here row) using itertools.filterfalse like in the codeblock below:
def copy_string_iterator(connection, log_lines) -> None:
with connection.cursor() as cursor:
create_staging_table(cursor)
log_string_iterator = StringIteratorIO((
'|'.join(map(clean_csv_value, (
row['date'],
row['time'],
row['cs_uri_query'],
row['s_contentpath'],
row['sc_status'],
row['s_computername'],
...
row['sc_substates'],
row['s_port'],
row['cs_version'],
row['c_protocol'],
row.update({'cs_cookie':'x'}),
row['timetakenms'],
row['cs_uri_stem'],
))) + '\n')
for row in filterfalse(lambda line: "#" in line.get('date'), log_lines)
)
cursor.copy_from(log_string_iterator, 'log_table', sep = '|')
When I run this, cursor.copy_from() gives me the following error:
QueryCanceled: COPY from stdin failed: error in .read() call
CONTEXT: COPY log_table, line 112910
I understand why this error happens, it is because in the test file I use there are only 112909 lines that meet the filterfalse condition. But why does it try to copy line 112910 and throw the error and not just stop?
Since Python doesn't have a coalescing operator, add something like:
(map(clean_csv_value, (
row['date'] if 'date' in row else None,
:
row['cs_uri_stem'] if 'cs_uri_stem' in row else None,
))) + '\n')
for each of your fields so you can handle any missing fields in the JSON file. Of course the fields should be nullable in the db if you use None otherwise replace with None with some default value for that field.

Proper way to (get from /insert into) table using Erlang Mysql Driver

I am trying to get erlang-mysql-driver working, I managed to set it up and make queries but there are two things I cannot do.(https://code.google.com/archive/p/erlang-mysql-driver/issues)
(BTW, I am new to Erlang)
So Here is my code to connect MySQL.
<erl>
out(Arg) ->
mysql:start_link(p1, "127.0.0.1", "root", "azzkikr", "MyDB"),
{data, Result} = mysql:fetch(p1, "SELECT * FROM messages").
</erl>
1. I cannot get data from table.
mysql.erl doesn't contain any specific information on how to get table datas but this is the farthest I could go.
{A,B} = mysql:get_result_rows(Result),
B.
And the result was this:
ERROR erlang code threw an uncaught exception:
File: /Users/{username}/Sites/Yaws/index.yaws:1
Class: error
Exception: {badmatch,[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]}
Req: {http_request,'GET',{abs_path,"/"},{1,1}}
Stack: [{m181,out,1,
[{file,"/Users/{username}/.yaws/yaws/default/m181.erl"},
{line,18}]},
{yaws_server,deliver_dyn_part,8,
[{file,"yaws_server.erl"},{line,2818}]},
{yaws_server,aloop,4,[{file,"yaws_server.erl"},{line,1232}]},
{yaws_server,acceptor0,2,[{file,"yaws_server.erl"},{line,1068}]},
{proc_lib,init_p_do_apply,3,[{file,"proc_lib.erl"},{line,240}]}]
I understand that somehow I need to get second element and use foreach to get each data but strings are returned in different format like queried string is Success but returned string is <<"Success">>.
{badmatch,[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]}
First Question is: How do I get datas from table?
2. How to insert values into table using variables?
I can insert data into table using this method:
Msg = "Hello World",
mysql:prepare(add_message,<<"INSERT INTO messages (`message`) VALUES (?)">>),
mysql:execute(p1, add_message, [Msg]).
But there are two things I am having trouble,
1. I am inserting data without << and >> operators, because When I do Msg = << ++ "Hello World" >>, erlang throws out an exception (I think I am doing something wrong), i don't know wether they are required but without them I am able to insert data into table except this error bothers me after execution:
yaws code at /Users/{username}/Yaws/index.yaws:1 crashed or ret bad val:{updated,
{mysql_result,
[],
[],
1,
[]}}
Req: {http_request,'GET',{abs_path,"/"},{1,1}}
returned atom is updated while I commanded to insert data.
Question 2 is: How do I insert data into table in a proper way?
Error:
{badmatch,[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]}
Tells you that returned values is:
[[4,0,<<"This is done baby!">>,19238],
[5,0,<<"Success">>,19238],
[6,0,<<"Hello">>,19238]]
Which obviously can't match with either {data, Data} nor {A, B}. You can obtain your data as:
<erl>
out(Arg) ->
mysql:start_link(p1, "127.0.0.1", "root", "azzkikr", "MyDB"),
{ehtml,
[{table, [{border, "1"}],
[{tr, [],
[{td, [],
case Val of
_ when is_binary(Val) -> yaws_api:htmlize(Val);
_ when is_integer(val) -> integer_to_binary(Val)
end}
|| Val <- Row
]}
|| Row <- mysql:fetch(p1, "SELECT * FROM messages")
]}
]
}.
</erl>

Jython: test to prevent exception "Cannot create PyString with non-byte value"?

Jython 2.7.0 (final release). OS: W7 (64-bit)
this code:
keys = javax.swing.UIManager.getDefaults().keys()
while keys.hasMoreElements():
key = keys.nextElement()
logger.info( "=== key %s" % str( key ) )
try:
value = javax.swing.UIManager.get(key)
except java.lang.Throwable, t:
logger.error( "=== thrown %s" % str( t ) )
produces all sorts of keys... until it outputs
=== key PasswordField.echoChar
it then throws
java.lang.IllegalArgumentException: Cannot create PyString with
non-byte value
I'm aware this is a known bug in Jython ... just wondering if there is a way of testing for this before the exception is thrown?
For me this gets triggered when using print() directly on a Java HashMap that contains any value with a Unicode character. A simple Python version of isBytes from PyString class is one way to detect it, but frankly, I don't think that is a good option unless you (a) know what element of the data that is triggering the issue and/or (b) intend to mask or fix the values triggering the issue. Probably the best solution is to just catch the exception.
This definitely effects Jython >= 2.7 and is a very annoying bug when troubleshooting. For me I just commented out the IllegalArgumentException in the PyString class code and recompiled. Now Jython will happily print out HashMaps directly and replace the Unicode characters with ? just like it did in previous versions. I would guess that this causes issues somewhere, maybe with code dealing with a lot of Unicode or something, but I haven't found any issues as of yet.
Catch Exception:
from java.lang import IllegalArgumentException
keys = javax.swing.UIManager.getDefaults().keys()
while keys.hasMoreElements():
key = keys.nextElement()
logger.info( "=== key %s" % str( key ) )
try:
value = javax.swing.UIManager.get(key)
except java.lang.Throwable, t:
try:
logger.error( "=== thrown %s" % str( t ) )
except IllegalArgumentException:
# fix it or w/e
Python version of isBytes:
def test_if_char(value):
for e in value:
if ord(e) > 255: return False
return True
In case you use Jython 2.7.0, one could use the following code to use any of Unicode strings within your code:
PyString str = Py.newStringOrUnicode("颜军")

How do I check for multiple exceptions in Python?

DNASequence = "laksjfklsajdfklsajfklasjfklsad"
while True:
lMerLength = input("Please enter the length of the l-mers of the universal array :")
try:
if len(DNASequence) >= lMerLength > 0:
break
except SyntaxError:
pass
#This is not working. How do I check for multiple exceptions in Python?
except NameError:
pass
print "ERROR: Please check your input. You entered an invalid input."
Here is how you check for multiple exceptions.
try:
..............
except (SyntaxError, NameError, ...):
..............
finally:
.............
The problem is, that input returns a string and you compare that string in your if to an int. And in python 2.x you should use raw_input instead of input:
DNASequence = "laksjfklsajdfklsajfklasjfklsad"
while True:
try:
lMerLength = int(raw_input("Please enter the length of the l-mers of the universal array :"))
except ValueError:
print "ERROR: Please check your input. You entered an invalid input."
continue
if len(DNASequence) >= lMerLength > 0:
break
print "ERROR: Please check your input. You entered an invalid input."