I am trying to read xlsx file from unread mail and convert it to data frame finally it will insert into MySQL DB.To avoid duplication while inserting each row of data frame i check if the data already present in db,for this duplication i check mails one by one.
My issue is when two or more unread mail is present in inbox this duplication check fails.
detach_dir = os.path.dirname(os.path.abspath(__file__)) +
'/attachments'
user = "abc#outlook.in"
pwd = "xyz#123*"
m = imaplib.IMAP4_SSL("outlook.office365.com")
m.login(user,pwd)
# Select the mailbox
m.select("folder name in mail")
n = 0
resp, items = m.search(None, '(UNSEEN)')
items = items[0].split()
for emailid in items:
resp, data = m.fetch(emailid, "(RFC822)")
email_body = data[0][1]
mail = email.message_from_bytes(email_body)
if mail.get_content_maintype() != 'multipart':
continue
att_path = os.path.join(detach_dir, filename)
if not os.path.isfile(att_path) :
fp = open(att_path, 'wb')
fp.write(part.get_payload(decode=True))
fp.close()
df_mail = pd.read_excel(att_path,skiprows=
[0,2,3,4,5,6,7,8,9],skip_blank_lines=True,skipfooter=1,index=False)
df_mail = df_mail.fillna(0)
df_mail.dropna(how="all", inplace=True)
for i, row in df_mail.iterrows():
sql = 'SELECT * FROM `tablename WHERE condition for duplicate'
extist=con.execute(sql)
duplicate=extist.fetchall()
if len(duplicate) == 0:
df_mail.iloc[i:i+1].to_sql('table', con = engine, if_exists = 'append', chunksize = 1000,index=False)
else:
print("duplicate data")
Can you please share your code and a list of things you have tried? It is very hard to help without those.
Related
I am using smack on the client-side. I tried to search registered users on the client-side because before creating a new user I want to know that the id is registered on server or not if not then create the user either log in the user but ejabberd server crashed with the error. Here are the crash logs of ejabberd server.
Failed to process iq:
#iq{
id = <<"Hh6AJ-28">>,type = set,lang = <<"en">>,
from =
#jid{
user = <<"admin">>,server = <<"faiqkhan-virtualbox">>,
resource = <<"92526029764259513741138">>,luser = <<"admin">>,
lserver = <<"faiqkhan-virtualbox">>,
lresource = <<"92526029764259513741138">>},
to =
#jid{
user = <<>>,server = <<"vjud.faiqkhan-virtualbox">>,resource = <<>>,
luser = <<>>,lserver = <<"vjud.faiqkhan-virtualbox">>,lresource = <<>>},
sub_els =
[#xmlel{
name = <<"query">>,
attrs = [{<<"xmlns">>,<<"jabber:iq:search">>}],
children =
[#xmlel{
name = <<"x">>,
attrs = [{<<"xmlns">>,<<"jabber:x:data">>},{<<"type">>,<<"submit">>}],
children =
[#xmlel{
name = <<"field">>,
attrs = [{<<"var">>,<<"user">>},{<<"type">>,<<"text-single">>}],
children =
[#xmlel{
name = <<"value">>,attrs = [],
children = [{xmlcdata,<<"wasiq#faiqkhan-virtualbox">>}]}]}]}]}],
meta = #{ip => {0,0,0,0,0,65535,49320,11092}}}
exception error: {module_not_loaded,mod_vcard_mnesia,
<<"faiqkhan-virtualbox">>}
in function gen_mod:get_module_opts/2 (src/gen_mod.erl, line 338)
in call from gen_mod:get_module_opt/3 (src/gen_mod.erl, line 318)
in call from mod_vcard_mnesia:filter_fields/3 (src/mod_vcard_mnesia.erl, line 200)
in call from mod_vcard_mnesia:search/4 (src/mod_vcard_mnesia.erl, line 78)
in call from mod_vcard:search_result/4 (src/mod_vcard.erl, line 479)
in call from mod_vcard:process_search/1 (src/mod_vcard.erl, line 264)
in call from gen_iq_handler:process_iq/3 (src/gen_iq_handler.erl, line 131)
in call from gen_iq_handler:process_iq/4 (src/gen_iq_handler.erl, line 109)
I used the following code to get a registered user from the client-side:
InetAddress address = InetAddress.getByName(HOST);
DomainBareJid serviceName = JidCreate.domainBareFrom("faiqkhan-VirtualBox"); XMPPTCPConnectionConfiguration.Builder builder = XMPPTCPConnectionConfiguration.builder();
builder.setPort(PORT);
builder.setSendPresence(true);
builder.setHostAddress(address);
builder.setServiceName(serviceName);
builder.setUsernameAndPassword("admin", "123456");
builder.setSecurityMode(ConnectionConfiguration.SecurityMode.disabled);
XMPPTCPConnection connection = new XMPPTCPConnection(builder.build());
try {
connection.connect();
connection.login("admin", "123456");
Logger.showError("21560-connection created to: " + connection.getHost());
Roster roster = Roster.getInstanceFor(connection);
Set<RosterEntry> entities = roster.getEntries();
UserSearchManager search = new UserSearchManager(connection);
DomainBareJid s = JidCreate.domainBareFrom("vjud.".concat("faiqkhan-VirtualBox"));
Form searchForm = search.getSearchForm(s);
Form answerForm = searchForm.createAnswerForm();
answerForm.setAnswer("user", "wasiq#faiqkhan-virtualbox");
ReportedData data = search.getSearchResults(answerForm, s);
if (data.getRows() != null) {
for (ReportedData.Row row : data.getRows()) {
for (CharSequence value : row.getValues("jid")) {
Log.i("Iteartor values......", " " + value);
}
}
}
} catch (SmackException | IOException | XMPPException | InterruptedException e) {
Logger.showError("21560-error on account creation: " + e.getMessage());
}
Server crashed on ReportedData data = search.getSearchResults(answerForm, s); line of code.
You found a bug in ejabberd:
It was already fixed in 2018 in
https://github.com/processone/ejabberd/commit/1be21126342d503205798605725ba5ceef9de42b
but the bug got reintroduced in 2019 in
https://github.com/processone/ejabberd/commit/a02cff0e780bb735531594c4ece81e8628f79782#diff-71f613241ed580c3e5dad2b4526503f2R12
and I've now applied a temporary fix in
https://github.com/processone/ejabberd/commit/92913389a51ddc564f11e0712b0d82fca8c9aecb
But even if that bug is fixed, the feature you want to use probably doesn't help: a user must set his vCard in order for his account to be provided in vJud search. In other words, vJUD searches in stored vCards, not in registered accounts. If I register an account but don't set a vCard, then vJUD will not find it.
before creating a new user I want to know that the id is registered on server or not
And why don't you simply attempt to register the account first?
On the other hand, if you would like to check in advance if an account is available for registration or not (like some web registration sites do, to allow the user modify the username until one is available), then I think XMPP doesn't provide that possibility for security reasons: it is not allowed to know if an account exists or not.
A solution for that would be to use the check_account ejabberd command
https://docs.ejabberd.im/developer/ejabberd-api/admin-api/#check-account
executing it with some frontend, like ejabberdctl, ReST or XMLRPC
https://docs.ejabberd.im/admin/guide/managing/#ejabberd-commands
$ ejabberdctl check_account user1 localhost
$ ejabberdctl check_account user6666 localhost
Error: false
~ 1
I have a large csv file with two columns like this:
Id and vehicle
and I like to replicate the rows and if the vehicle is "truck", but instead put "car".
I have this code, but there is an error
which says
_csv.Error: iterator should return strings, not bytes (did you open the file in text mode?)
what does it mean? where I am wrong?
infilename = r'external carriers.csv'
outfilename = r'outputCSV.csv'
with open(infilename, 'rb') as fp_in, open(outfilename, 'wb') as fp_out:
reader = csv.reader(fp_in, delimiter=",")
writer = csv.writer(fp_out, delimiter=",")
for row in reader:
if len(row) == 2:
if row == "truck":
writer.writerow = "car"
It's obvious, you have opened the file in text mode you need rt :
with open(infilename, 'rt') as fp_in, open(outfilename, 'wt') as fp_out:
Also if you want to check the vehicle type you need to check the row[1] which preserve your car name and then reassign it and write the row to your output file.Also note that you don't need to check the length of your rows since calling the len function can be terrible in term if your performance which has O(n) and for large files (specially with large rows) is very inefficient.
infilename = r'external carriers.csv'
outfilename = r'outputCSV.csv'
with open(infilename, 'rt') as fp_in, open(outfilename, 'wt') as fp_out:
reader = csv.reader(fp_in, delimiter=",")
writer = csv.writer(fp_out, delimiter=",")
for row1,row2 in reader:
if row2 == "truck":
writer.writerow([row1,'car'])
How one should do image file uploads with Pyramid, SQLAlchemy and deform? Preferably so that one can easily get image thumbnail tags in the templates. What configuration is needed (store images on the file system backend, so on).
This question is by no means specifically asking one thing. Here however is a view which defines a form upload with deform, tests the input for a valid image file, saves a record to a database, and then even uploads it to amazon S3. This example is shown under the links to the various documentation I have referenced.
To upload a file with deform see the documentation.
If you want to learn how to save image files to disk, see this article see the official documentation
Then if you want to learn how to save new items with SQLAlchemy see the SQLAlchemy tutorial.
If you want to ask a better question where a more precise detailed answer can be given for each section, then please do so.
#view_config(route_name='add_portfolio_item',
renderer='templates/user_settings/deform.jinja2',
permission='view')
def add_portfolio_item(request):
user = request.user
# define store for uploaded files
class Store(dict):
def preview_url(self, name):
return ""
store = Store()
# create a form schema
class PortfolioSchema(colander.MappingSchema):
description = colander.SchemaNode(colander.String(),
validator = Length(max=300),
widget = text_area,
title = "Description, tell us a few short words desribing your picture")
upload = colander.SchemaNode(
deform.FileData(),
widget=widget.FileUploadWidget(store))
schema = PortfolioSchema()
myform = Form(schema, buttons=('submit',), action=request.url)
# if form has been submitted
if 'submit' in request.POST:
controls = request.POST.items()
try:
appstruct = myform.validate(controls)
except ValidationFailure, e:
return {'form':e.render(), 'values': False}
# Data is valid as far as colander goes
f = appstruct['upload']
upload_filename = f['filename']
extension = os.path.splitext(upload_filename)[1]
image_file = f['fp']
# Now we test for a valid image upload
image_test = imghdr.what(image_file)
if image_test == None:
error_message = "I'm sorry, the image file seems to be invalid is invalid"
return {'form':myform.render(), 'values': False, 'error_message':error_message,
'user':user}
# generate date and random timestamp
pub_date = datetime.datetime.now()
random_n = str(time.time())
filename = random_n + '-' + user.user_name + extension
upload_dir = tmp_dir
output_file = open(os.path.join(upload_dir, filename), 'wb')
image_file.seek(0)
while 1:
data = image_file.read(2<<16)
if not data:
break
output_file.write(data)
output_file.close()
# we need to create a thumbnail for the users profile pic
basewidth = 530
max_height = 200
# open the image we just saved
root_location = open(os.path.join(upload_dir, filename), 'r')
image = pilImage.open(root_location)
if image.size[0] > basewidth:
# if image width greater than 670px
# we need to recduce its size
wpercent = (basewidth/float(image.size[0]))
hsize = int((float(image.size[1])*float(wpercent)))
portfolio_pic = image.resize((basewidth,hsize), pilImage.ANTIALIAS)
else:
# else the image can stay the same size as it is
# assign portfolio_pic var to the image
portfolio_pic = image
portfolio_pics_dir = os.path.join(upload_dir, 'work')
quality_val = 90
output_file = open(os.path.join(portfolio_pics_dir, filename), 'wb')
portfolio_pic.save(output_file, quality=quality_val)
profile_full_loc = portfolio_pics_dir + '/' + filename
# S3 stuff
new_key = user.user_name + '/portfolio_pics/' + filename
key = bucket.new_key(new_key)
key.set_contents_from_filename(profile_full_loc)
key.set_acl('public-read')
public_url = key.generate_url(0, query_auth=False, force_http=True)
output_dir = os.path.join(upload_dir)
output_file = output_dir + '/' + filename
os.remove(output_file)
os.remove(profile_full_loc)
new_image = Image(s3_key=new_key, public_url=public_url,
pub_date=pub_date, bucket=bucket_name, uid=user.id,
description=appstruct['description'])
DBSession.add(new_image)
# add the new entry to the association table.
user.portfolio.append(new_image)
return HTTPFound(location = route_url('list_portfolio', request))
return dict(user=user, form=myform.render())
I am new to programming and F# is my first language.
Here are the relevant parts of my code:
let internal saveJsonToFile<'t> (someObject:'t) (filePath: string) =
use fileStream = new FileStream(filePath, FileMode.OpenOrCreate)
(new DataContractJsonSerializer(typeof<'t>)).WriteObject(fileStream, someObject)
let dummyFighter1 = { id = 1; name = "Dummy1"; location = "Here"; nationality = "Somalia"; heightInMeters = 2.0; weightInKgs = 220.0; weightClass = "Too fat"}
let dummyFighter2 = { id = 2; name = "Dummy2"; location = "There"; nationality = "Afghanistan"; heightInMeters = 1.8; weightInKgs = 80.0; weightClass = "Just Nice"}
let filePath = #"G:\User\Fighters.json"
saveJsonToFile dummyFighter1 filePath
saveJsonToFile dummyFighter2 filePath
When I run "saveJsonToFile dummyFighter1 filePath", the information is successfully saved. My problem is this: Once I run "saveJsonToFile dummyFighter2 filePath", it immediately replaces all the contents that are already in the file, i.e., all the information about dummyFighter1.
What changes should I make so that information about dummyFighter2 is appended to the file, instead of replacing information about dummyFighter1?
Change the way you open a file setting FileMode.OpenOrCreate to FileMode.Append. Append means "create or append" :
use fileStream = new FileStream(filePath, FileMode.Append)
From MSDN (https://msdn.microsoft.com/fr-fr/library/system.io.filemode%28v=vs.110%29.aspx) :
FileMode.Append opens the file if it exists and seeks to the end of the file, or
creates a new file. This requires FileIOPermissionAccess.Append
permission. FileMode.Append can be used only in conjunction with
FileAccess.Write. Trying to seek to a position before the end of the
file throws an IOException exception, and any attempt to read fails
and throws a NotSupportedException exception.
I have written the following code.
The issue that I am having is that the database is not being updated and a new record is not being made. I am assuming that the way I am retrieving my POST are incorrect but that's why I am asking where do I begin to debug? I have created template variables for quantity,action,building, and test. When I try to call them in the html {{ test }} for example nothing ever shows up even the one that is hardcoded. I used firebug and the form is indeed posting the values that should be posted.
the form consists of two drop down menus one for action and one for building. A numerical input quantity, a text box and a submit button. If anyone can offer me some advice it would be appreciated. I really don't understand why the hardcoded one doesn't show up
If you need any more information please let me know.
def update(request,Type_slug, slug, id):
error = False
Slug = slug
ID = id
Type = Type_slug
test = 'test'
quantity = request.POST['Qty']
action = request.POST['Action']
building = request.POST['Building']
comments = request.POST['Comments']
if Type == 'Chemicals':
item = Chemicals.objects.filter(S_field=Slug, id= ID)
New_Record = ChemicalRecord(Name=item.Name,Barcode=item.Barcode,Cost=item.Cost,Action=action,Building=building)
if building == 'Marcus':
building_two = 'Pettit'
elif building =='Pettit':
building_two ='Marcus'
Record_one = ChemicalRecord.objects.filter(Barcode=New_Record.Barcode).filter(Building=New_Record.Building)
if Record_one:
Record_one = ChemicalRecord.objects.filter(Barcode=New_Record.Barcode).filter(Building=New_Record.Building).latest('id')
New_Record.On_hand = Record_one.On_hand
else:
New_Record.On_hand = 0
if action == 'Receiving':
New_Record.On_hand = New_Record.On_hand+quantity
elif action == 'Removing':
New_Record.On_hand = New_Record.On_hand-quantity
Record_two = ChemicalRecord.objects.filter(Barcode=New_Record.Barcode).filter(Building=building_two)
if Record_two:
Record_two = ChemicalRecord.objects.filter(Barcode=New_Record.Barcode).filter(Building=building_two).latest('id')
Record_two_qty = Record_two.On_hand
else:
Record_two_qty = 0
New_qty = New_Record.On_hand+Record_two_qty
Chemicals.objects.filter(Barcode=obj.Barcode).update(On_hand=New_qty)
New_Record.save()
You can use import pdb;pdb.set_trace() for debugging.