Does order in email headers indicate which MTA inserted each? - smtp

I'm trying to analyze a mail header to figure which element (MUA, MTA...) creates each. My supposition is:
The sender MUA composes the body (including Content-Type, Mime, Content-Transfer-Encoding), and sends it via SMTP to the sender border MTA. MUA provides info for the the initial headers (From, To, Reply-To, ) which are inserted by this MTA
This MTA inserts MessageID and Return_Path (and all extra headers and X-Headers it desires)
Then it starts 'hoping'. On each hop the receiving MTA inserts 'Received:' header and each other header it considers
If order is kept and each MTA inserts its headers AT THE TOP of the message, it should be easy to figure which MTA has inserted each header... but I can't find a valid scheme
Fields as DKIM-Signature, Authentication-Results, Received-SPF... appear in different places. Which MTA creates each? Who DKIM-signs the email (I suppose it must be the sender border MTA) Who authenticates SPF-DKIM-DMARC?
A lot of X-Headers are added, many related to spam control, and I cant find which MTA (on with hop) created each
Could you help me, please?

The MUA (Mail User Agent, RFC term for mail client) will add most headers that you can see in email clients, in particular From: and To:. (You can sometimes see this in the To: header when someone has "funny" nicknames for someone in their address book. Or when "From:" and SMTP envelope from (as in the MAIL FROM: command) disagree, even though may servers will reject such messages, as it's often a spam indicator.) The MUA will set the Reply-To: header as well, especially if the MUA is really a customer management system. The MTA (Mail Transfer Agent, actual SMTP server) may include bounce information (VERP).
The MUA then submits the message to the MTA. Every program (mail server, firewall, etc) that touches the email can add headers. For example, the MTA would normally do the DKIM signing (and add the relevant headers). It must prepend Received: header (i.e., put it at the top) and it must not mess with other Received: headers. Still some programs like firewalls may mess with headers.
You can see multiple DKIM signatures (see DKIM selectors), for example, when an email gets resent in the context of a mailing list. You could see a DKIM signature from the original sender to the list, then more headers including an additional DKIM header when the mailing list server sends its to the final recipients.
As for X-... headers: these are non-standard headers (hence the X- prefix). Everything is off here. Some sending MTAs insert them for tracking purposes (e.g., to catch spammers among their customer), some mail receivers put their spam assessments in a special x-header. Even the MUA may put some x-headers in the emails, for example, Claws Mail puts in a header for the account the mail came from, when it downloaded it, and so on. You may or may not trust these, and in fact, MUAs may have settings to indicate which x-headers they should trust.
To that regards, even the Authentication results header and such can come form anywhere along the road. Even the initial sender could add one, like some anti virus programs do, to indicate that they scanned the outgoing mail. Again, it's up to the receiver to decide whether to trust these headers. Clearly you want the mail server closest to you to do DKIM checks and anything related to authentication, as you can hopefully trust that server's verdict.
So, does order indicate which mailer inserted a header? Yes, but... mail servers may or may not conform to all parts of the RFCs, and many mail servers follow Postel's law and be somewhat lenient in what they accept.

Related

Are the spam filters legal?

Quick question:
If the recipient's mail server generates the command to sender:
250 message accepted for delivery
but for some reasons (series of spam filters) doesn't deliver it to INBOX, instead deletes or puts it in "junk' or 'spam' folders, is it a violation of any SMTP, RFC or other messaging exchange rules?
So, how far the legal is use of spam and other filters, after the agreement above (250...) is made with sender
are there any resources where it is clearly indicated or regulated?
From an SMTP RFC-compliance point of view, it is perfectly legal to accept the email first, then cancel the delivery or re-route the email, simply because SMTP is not concerned with these latter actions. SMTP is a transmission protocol with the sole focus on transmission only. As such, it does not and cannot make any guarantees as to what happens to the email once it has been transmitted.
If the filtering happens during transmission, you can reasonably expect an 550-type code, though. RFC2821 7.7 says:
[...] When mail is rejected for these or other policy reasons, a 550 code SHOULD be used in response to EHLO, MAIL, or RCPT as appropriate.
Note that in RFC terminology, "SHOULD" indicates a desired, but optional feature, so this is "soft" requirement.
A harder requirement is made by RFC2821 6.1:
When the receiver-SMTP accepts a piece of mail (by sending a "250 OK"
message in response to DATA), it is accepting responsibility for
delivering or relaying the message. [...] If there is a delivery
failure after acceptance of a message, the receiver-SMTP MUST
formulate and mail a notification message."
This mandates a DSN to be sent in case the email cannot be delivered. However, this also explicitly allows for delivery failures (such as those due to email filtering) to happen after accepting the email.

DMARC Anti-Spoofing Error When Sending Email with Sender Yahoo Domain Other Than Server Domain

I am using our email server at [mydomainhere].com to send emails through a web site UI. I just used the UI to send an email from [myemail]#yahoo.com. And received an Undeliverable message at my yahoo email address.
mta1400.mail.ne1.yahoo.com rejected your message to the following e-mail addresses:
[myemail]#yahoo.com
mta1400.mail.ne1.yahoo.com gave this error:
Message not accepted for policy reasons. See http://postmaster.yahoo.com/errors/postmaster-28.html
More information can be found here:
http://www.pcworld.com/article/2141120/yahoo-email-antispoofing-policy-breaks-mailing-lists.html
Any help would be appreciated.
Yes Allan, you are correct in assuming that the anti-spoofing that Yahoo (and now AOL) have turned on is permanent. The technology they are using is called DMARC. Yahoo has published a DMARC record in their DNS:
$ dig TXT _dmarc.yahoo.com. +short
"v=DMARC1; p=reject; sp=none; pct=100; rua=mailto:dmarc-yahoo-rua#yahoo-inc.com, mailto:dmarc_y_rua#yahoo.com;"
Every mail server that supports DMARC will look up that record in Yahoo's DNS and apply Yahoo's p=reject policy. In effect, what Yahoo has done is stated to the world, "if the email does not originate from this list of IPs (SPF) or bear this crytographic signature (DKIM), then reject (p=reject) it." Since your mail server is not in their list of mail servers IPs nor are the messages signed with Yahoo's DKIM key, a substantial and growing portion of the mail servers on the internet are going to reject it or deliver it to the Spam folder (Gmail).
In addition to the SPF & DKIM checks, DMARC also introduces the concept of alignment. In addition to passing SPF checks (which apply to the Envelope Sender), DMARC requires that the domain in the messages 'From' header passes SPF. This prevents you (and bad actors) from sending messages with a header From domain of Yahoo.com and an Envelope Sender domain of attacker.com, which the recipient will never see. This alignment also extends to the DKIM signature, requiring not just that the message is signed with DKIM, but also that the dkim signature domain (d= property) matches the From header domain.
we will just have to prevent users from using their yahoo email address in the sender email field
Coding in a check for the yahoo.com domain is a hack that won't last long. AOL has already joined the thousands of domain owners with DMARC p=reject policies. They won't be the last of the Very Large Email providers to publish p=reject DMARC policies. A much safer approach is to evaluate SPF against your mail servers public IP and the domain in the users selected email address. If the SPF check fails, then choose an option:
Inform the user that their choice of domains doesn't permit 3rd party senders and they should choose another.
Alter the From header to send from a domain you control:
From: "user#yahoo.com via" <my-app#my-domain.com>
As already stated, you could define Reply-To if you wish for replies to expose the senders real email address.
Set up local usernames that forward to the sender's real email address. If you've used Craigslist, you're familiar with the idea. You maintain a mapping of local addresses and the email address they forward to.
Based on what you've said about your web application, it seems like #2 is the best fit.
I have had a similar problem with mailing lists that I maintain (e-mails with a From address something#yahoo.com bounced). I solved my problem by changing the From, Reply-to and Errors-to fields of the e-mails' headers as follows:
From: Organization name <no-reply#somedomain.org>
Reply-to: my-email-address#yahoo.com
Errors-to: my-email-address#yahoo.com
I suspect that similar changes will fix your problem.

Sending email to same/different domains

I am doing a Networks course and as I was reading about SMTP I thought about the following matter:
If a user sends an email to user1#example.com, user2#example.com, user3#example.com, does it take the same amount of time to send the emails separately as it does to send it as a multi-recipient email?
Or when the first one is sent, the next two are easier to send since the connection is already established with that server? Or does the connection close immediately thus making no difference?
What about when the emails belong to different domains?
It depends on how the MTA is programmed. The SMTP protocol allows the sender to send multiple RCPT TO commands, to list all the recipients of a message, and any reasonably-designed MTA will make use of this to consolodate them.
For the multiple-domain case, your mail client will use a single connection to upload the message to its submission server. That server will then use one connection per destination domain for the recipients.

Does every SMTP server that can receive mail have to be able to send failure notices?

I am thinking about implementing my own SMTP server that can receive mail and store it. As this is a small hobby project, I'd prefer to keep the code simple – so, for example, I don't want to implement logic for sending mail. However, the following section from RFC 5321 makes me worry:
The server must give special treatment to cases in which the
processing following the end of mail data indication is only
partially successful. This could happen if, after accepting several
recipients and the mail data, the SMTP server finds that the mail
data could be successfully delivered to some, but not all, of the
recipients. In such cases, the response to the DATA command MUST be
an OK reply. However, the SMTP server MUST compose and send an
"undeliverable mail" notification message to the originator of the
message.
Does this mean that, even if I normally only return 250 OK when I'm sure the message has been stored safely, I am forced to implement stuff for sending failure notices just in case someone decides to send a mail to both an existing and a non-existing mail address at the same time? Is there any way to get around it without violating the standard? If there isn't, how bad would it be to just send back 452 Too many recipients (my RCPT limit is 1, not 100 as required by RFC 5321) whenever someone tries to send a mail to multiple recipients?
In such cases, the response to the DATA command MUST be an OK reply.
However, the SMTP server MUST compose and send an "undeliverable mail"
notification message to the originator of the message.
That seems rather unambiguous to me. If you are to be standards-compliant, you MUST both return an OK response (because the message could be delivered to some recipients) and send failure notification to the originator (because it could not be delivered to other recipients).
On the other hand, your question also mentioned that you will only accept delivery for a single recipient, so a) you've already resigned yourself to not being standards-compliant anyhow by not allowing 100 recipients and b) the failure mode of "mail data could be successfully delivered to some, but not all, of the recipients" will not be possible if there can only be a single recipient.
After re-thinking the situation, I think that it's actually possible to do this in an RFC-compliant way if the server can assure that the mail can be delivered to all recipients at RCPT command time – after all, individual recipients can be rejected when the corresponding RCPT command was received. Of course, this means that the server e.g. has to ensure that enough harddisk space is available for each recipient and the maximum mail size. And a server could just deny mail to a recipient if his current mailbox size plus the maximum mail size is more than the maximum mailbox size.
Of course, the actual implementation is a bit more complicated in order to support concurrency.

How does a mail relayer doesn't send a mail to already received recipients?

i was wondering.
Suppose i have an exchange server, (the domain can be me#company.com),
also, i have a relay server, that accepts requests to internet domains (gmail/yahoo/etc).
suppose i send a mail to: (you#company.com; him#gmail.com) - from a user in the domain (me#company.com)
The mail is sent to the exchange server, then the exchange groups the domains (company.com and gmail.com) => it puts the mail into you#company.com mailbox.
then, the exchange sees that it needs to send into him#gmail.com, so it will send it to the relay server.
the relay server will get the SMTP request:
from: me#company.com
to: you#company.com; him#gmail.com
the question is - how does the smtp relay server knows not to send it to you#company.com - thus causing duplicate mails received?
also, how does the gmail server knows not to try to send the mails it receives again? (because they only get smtp requests, similar to a client that sends the mail)
thanks in advanced!
The content of the mail is not parsed by either exchange or the relay for determining where the mail should go. Everyone in this context is talking SMTP with one another, and specify recipients with the RCPT protocol message. So your mail client tells exchange two RCPT, exchange decides one of them is remote, then says one RCPT to the relay, and so on and so forth. This document specifies the SMTP protocol where you may read about the RCPT verb.
The headers you see are not used to deliver the message. Instead, SMTP defines a concept called envelope which has a completely different set of headers. As a common illustration, this is how Bcc: works, too; the recipient address is copied to the envelope, then removed from the message which goes inside the envelope.
When a message is delivered, the envelope sender is usually copied into the Return-Path: header, and you can sometimes see the envelope recipient information copied into the Received: headers.
At the point where a mail server decides to which destinations a message should go, there is commonly one copy of the message (spool file, queue entry, what have you) for each distinct destination. So if you send to from you#company to me#company, you#private, and friend#gmail, the first server might determine that the external addresses should both go to the same outbound relay, while the internal copy is delivered locally. Then the relay similarly decides that it needs to contact two distinct destination, so the single incoming message gets copied to two outbound messages, one to you#private and the other to friend#gmail.