TL;DR - Any webapp devs out there able to squeeze performance out of using gmail as SMTP server without a 3-10 minute delay to delivery?
I'm looking for some feedback from developers who are using gmail as a secure SMTP server for use with Webapps.
Within the past 2 weeks, my company has switched to gmail to host email for our domains.
I have a few webapps that send from our ISP server, but I'd like to refactor the code to use gmail with our domain name. I know the ISP can rig things up, but I don't want relay issues.
I have gmail sending for the Webapps and it's more or less working. On ASP/IIS, I'm using SSL and on PHP Linux, I'm using TLS.
It seems that TLS (port 587) works better than plain old SSL (port 465), but the overall performance difference seems minimal.
What concerns me is that there is often a significant delay (10-20 minutes at times) to get email to a recipient no matter which method I use to send via gmail.
While I understand this can happen in any environment, I am used to fast delivery. Use case is salespeople talking to customers and needing to get email to them very quickly using the Webapp. Before the gmail switch, usually less than a minute to deliver. Now, it is 3-4 minutes minimum.
Am I crazy to try to use gmail for this purpose? There seem to be a lot of devs using it, so just wanting to know how it works out for them/you.
Thanks for any feedback!
Ive found gmail problematic for another reason... Grey listing. My application happy works away using gmail smtp and then every now and then just stops working. Manually logging into the gmail web interface and entering a captcha "unlocks" the account and my application can send mail again. Seems at some point google determines that I may be sending spam and locks my account...? Feels like sending automated mail through gmail is a risky idea IMHO
Just a shot in the dark, but may it have to do with additional outgoing mail spam checking and therefore long output queues on GMail's side and (at least in some cases) greylisting?
To elaborate on that:
Google is known to take things seriously. Hosting a mail service that may be used by everyone nowadays is a bit of a hassle. One user posting spam/viruses through your service and bam! your servers are on someones blacklist. Therefore, additional checking on outgoing mail would be a good idea, but the mail may take longer (especially in a distributed server environment where the mail has to pass several message queues ... from the incoming ESMTP server to the spam checker to the virus checker to the message storage to the outgoing mail queue to the outgoing mail SMTP service).
If a recipient uses greylisting, his mail server first rejects a mail by stating a temporary problem, remembering the sender address and/or the message id and telling the sending server to try again later. Most spammers do not try again. But GMail's servers will try again, but due to long message queues and/or long retry intervals it will be delayed further.
And, as Ryan already stated, try to get one of the most delayed mails into your hands, including all mail headers. Check the timestamps on the Received: headers and find out which server is the bottleneck. Then you'll have at least an idea where it goes wrong.
Gmail has a pretty nice step-by-step resolution process on their support site.
Here is a direct link
You will want to do some more digging to try and isolate the issue. I'm rather surprised you are seeing 10 - 20 minute delays. One thing to look at is the email message header. This will give you a good idea how long each hop is taking. There could be an issue somewhere else that you don't know about.
I doubt you are going to be able to get your email sending as fast as it was in-house. It just comes with the territory.
On my server, debian with plesk and postfix mailserver, i seen that using gmail took a bit of time to sent the mail (~2-3 seconds to send the mail via php using phpmailer), but the mails usually arrive in 1~2 minutes.
Seldom happens that the deliver took about 10 minutes.
Using posix, instead, php send email faster, but they take more times to be delivered (in true, php is faster delivering the mail at the mail server, but then the mail server is slower to deliver the mail to the recipient).. and is a nightmare with spam.
This is my experience, my 2 cents ;)
Related
In developing our docusign application, we need to use real emails addresses (to test the addressing logic, etc.) but we do not want those users to get all the development envelopes. In our own code, we handle this using mailtrap.io, a custom SMTP service that traps emails and sends them all to a common inbox that developers can review. Is there a way to do something similar in your docusign developer account, where you would be using real email addresses, but they would not actually go the intended user.
thanks,
~Bill
Thanks for any help
No, you cannot. The emails are sent from DocuSign.
If you want to test the real thing, completely real, then it also includes emails being sent.
If you are ok making any change, so it's not 100% like the real thing, I would suggest to take your email address and change the domain (mailinator.com or your own internal domain) this is a change, yes, but also changing the SMTP server is a change, so in either case - you are not doing 100% what the real app will do.
The emails can also be suppressed (by changing account settings in DocuSign), but again, that won't test the real thing if that's what you need.
I have tried to have my SMTP server with exim4 for a while to send ecommerce registration and transaction confirms.
I followed all the good rules, like SPF, DKIM, DMARC, SMTP encryption. The postmaster tool by google said the IP has a good reputation and there are not spam traffic.
I reached a antispam test result of 10/10 using https://www.mail-tester.com/ or some similar tools.
But everything was not enough, my ecommerce registration messages are sent to spam by Gmail and it is the same with some other famous providers. I have understood I can't do nothing against.
Then, what is a good solution for user registration for an ecommerce? How can I reduce the messages delivered to spam folder?
I'm afraid you haven't provided enough information to identify the problem.
Hundreds of factors can contribute to deliverability outcomes; it's not as simple as setting up authentication, reverse DNS, etc. In addition, no free mail tester can accurately tell you how your deliverability will be at Gmail, Outlook, Yahoo, etc., because mail testers don't have the same data those ISPs use to make spam filtering decisions.
If you want more reliable deliverability, it's generally easiest to use a transactional email service to send email, rather than trying to run an SMTP server yourself. There are occasional exceptions to this, but because most senders will have an easier time using one of these services, it's almost always the first thing to try.
(How do email services provide better deliverability? A number of ways, but I think the biggest difference is that they can more closely manage the email sending reputation of their IP ranges. Having anti-spam systems built into the service enables them to resolve problems that much faster, compared to hosting providers which tend to have fewer tools and less data available to stop spammers, so their IP ranges' sending reputation is usually worse.)
Here is something you can try that may help.
Use a 3rd-party delivery provider (AWS SES, SendGrid, Mailgun) with a Good IP reputation. Here's a list of places you can check their IP reputation https://www.helloinbox.email/#reputation (Talos and Barracuda).
Use a subdomain to send transactional emails (email.example.com).
Let me know if that doesn't work.
I use Mandrill for sending emails via SMTP, however, sometimes it can take a long time, as in 30-60minutes, before I receive my email.
If I look into the Outbound-list, it lists my emails as 'delivered', yet I often have to wait quite a bit before I actually see it in my inbox.
It seems to be mostly when I send to gmail and yahoo, but this could be just a coincidence.
Any thoughts on this?
Although the message is released from your ESP(Mandrill) it has not been processed by the recipients ISP server yet. This can take up to 48 hours to process, but usually only takes minutes. The ISP delay is caused by things such as large attachments or file size, large volume from same domain, large influx of messaging hitting server at same time, error in email header, etc.
More info
Let's imagine that I have a site with some features that require email validation (eg. user registration). Of course I will validate emails using regex but once upon a time I saw SMTP validation in someone else's code. What good and bad sides has SMTP validation?
I can assume the following pros and cons:
Pros:
We can check out some email addresses for existence (not all SMTP servers allows this feature).
Cons:
Our site can be blocked.
Maybe we can use this to make webserver going down. Webserver will take domain part of email address and make request to it. We can create fake SMTP server that will receive request but will make great delay to response. Maybe if we force webserver to make dozens of such checks it will not be able to answer other clients.
Not all SMTP servers provide this capability.
Please be constructive.
"SMTP Validation" I assume refers to the SMTP VRFY command, and in this day and age you would be a fool to trust in it. It's a great feature if you're a spammer because it allows you to enumerate email addresses for a given server.
No one in their right mind will expose this command to the internet and have it respond with anything other than 252 send some mail, i'll try my best, aka "I'm only going to validate email addresses during an actual mail transaction".
The only sane use of VRFY these days is as an internal interface between a spam filtering appliance and the mail server hiding behind it.
TL;DR Don't use "SMTP Validation", it will be horribly unreliable at best.
If you want to validate that a user has entered a valid email address, then send it an email with an activation link/code.
In addition to Sammitch's answer, there are email validation services available via API that can provide SMTP validation - sometimes down to the mailbox level - for reasonable prices.
You can find these with a simple Google search, but I'm told LeadSpend and QAS (an Experian subsidiary) are among the best of the bunch.
Some friends with the help of various sites check and know when i'm invisible on yahoo messenger and keep bragging about this.
Being curious about this I've tested lots of sites that check if a user is invisible on yahoo messenger and all of them sent me a C1 packet type.
From what i've tested I'm(my ymsgr client) not sending anything back. So i only receive 1 packet from the bot that performs the check and that's it, they know if i'm invisible or not.
Next i thought that if i'm not sending anything back then maybe the yahoo server sends something back to i tested on a friend of mine and i sent him a c1 packet but i did not received anything back from nobody (neither the server neither my friend).
So how do they do that? I'm just looking for some hints, not expecting for code or someone else to solve it for me. I just like the thrill in learning and discovering by myself just that now i'm stuck with no idea :)
Thanks.
This is a new answer because it's too long to be a comment.
I looked around a bit and it would seem that the older versions of yahoo had a deal where if you were invisible and someone tried to start a voice chat with you, it would give you away. The ping packet could be trying that.
One thing I noticed on gtalk was that often a user appears online even after they have disconnected until someone sends them a ping packet. Also, if you are invisible and then you come back, the length of time that you've been online gives away the fact that you were invisible.
Could it be, then, that when you are pinged, your status, online time, etc all update on the yahoo server if you are online and what these services are doing then are checking to see if yahoo takes any action? I would presume that all yahoo is doing is not broadcasting your info if you're online, but you can still see that the server does something.
Instead of monitoring for an incoming/outgoing packet, why don't you check your registry on the yahoo server? I believe there's a url for that.
hope this helps,
Mechko
When you are invisible, basically you are online, but simply their server do not broadcast your real status (online), but a fake one (offline).
Yahoo is not fully able to "hide" your real status, but it has some "holes" in how aggressively tries: see here for details.
Actually there are lot of websites and possible spyware that make it even easier for an end-user to check.
Is it possible that sending such a packet to someone who's not online results in an error? In that case, sending the packet and not getting an error means that that person is online.
From what you are saying, I'd suggest that the bot actually checks somewhere else (if it is true that there are no packets sent at all)
Could it be that there is an online registry of users who are online and the packet that you are sent is just an artifact?
I hadn't used Yahoo Messenger is a long time, but something you said made me wonder. You said these testing websites sent you a C1 packet. I'm wondering if it is as simple a firewall rules to block all but the Yahoo servers. I don't know if chat/voice/file transfers are done P2P, if so you would have to update your firewall rules.
Back in the day I used to use my firewall to block the ad banners on the IM clients because they were served on a different URL and failure to connect to the ad server didn't disconnect my IM channel.
Maybe you can have the firewall ask for each connection and see how few connections you need in order to have the messenger work. Then have the firewall drop unsolicited packets.
If that doesn't work, you can use the firewall to block the checking sites.
Good luck.
Here are some of the methods you can use to detect if a user is invisible or really offline (some of these depend on earlier versions of the messenger application):
Start a voice chat with the user you want to check. If you get an error, the user if really offline.
Send a message to the user, then change the IMEnviorment. If you see a message saying “waiting for your friend to load the theme”, then the user if really offline.
Download (and pay for, unfortunately) Buddy Check.
Navigate to this URL in your browser: http://opi.yahoo.com/online?m=g&t=2&u=userid (change userid to the user you are testing). (This did not work for me).
There are websites that let you can check the status online, but there is no guarantee they'll work (they're only doing what I am showing you above). Basically, you can use these VIA screen scraping. Here are some of the sites:
http://www.ydetector.com/
http://www.imvisible.info/
http://www.invisible.ir/
http://www.invisible-scanner.com/ (This one has worked best for me)