Fetchmail and Office365 and backslashes, oh my!
Recently we were forcibly "migrated" to Office365 for our email services, and with the change numerous things were broken. One of the last things I cannot figure out how to solve is how to pass an alias email address to the MS server. So in .fetchmailrc I have a line like this: user "prof.bozo@utoronto.ca\department.chair@utoronto.ca" there with password "XYZZY" is bozo here But this fails, and in the fetchmail logfiles it represents the username as not having a backslash: Jul 4 14:35:14 fetchmail[13829]: IMAP> A0002 LOGIN "prof.bozo@utoronto.camedieval.philosophy@utoronto.ca" * Jul 4 14:35:14 fetchmail[13829]: IMAP< A0002 NO LOGIN failed. Jul 4 14:35:14 fetchmail[13829]: IMAP> A0003 LOGOUT If I double-backslash it, though, I get a double backslash in the username. Ditto for triple, quadruple, and quintuple backslashes. Nor does it work if I single-quote the backslash. According to the fetchmail manual, the backslash is used to communicate via ASCII sequences and codes with fetchmail, and it's pretty clear that this is part of the problem. Google turned up a few people with the same problem, but they seemed to have their problems (from 7+ years ago) fixed by multiple backslashes, which clearly aren't working for me. Any ideas? Our local people maintaining the MS Office365 nonsense dont' support fetchmail and are not interested in helping ... -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
Peter King via talk wrote on 2018-07-04 2:53 PM:
Recently we were forcibly "migrated" to Office365 for our email services, and with the change numerous things were broken. One of the last things I cannot figure out how to solve is how to pass an alias email address to the MS server. So in .fetchmailrc I have a line like this:
user "prof.bozo@utoronto.ca\department.chair@utoronto.ca" there with password "XYZZY" is bozo here
But this fails, and in the fetchmail logfiles it represents the username as not having a backslash:
Jul 4 14:35:14 fetchmail[13829]: IMAP> A0002 LOGIN "prof.bozo@utoronto.camedieval.philosophy@utoronto.ca" * Jul 4 14:35:14 fetchmail[13829]: IMAP< A0002 NO LOGIN failed. Jul 4 14:35:14 fetchmail[13829]: IMAP> A0003 LOGOUT
If I double-backslash it, though, I get a double backslash in the username. Ditto for triple, quadruple, and quintuple backslashes. Nor does it work if I single-quote the backslash.
According to the fetchmail manual, the backslash is used to communicate via ASCII sequences and codes with fetchmail, and it's pretty clear that this is part of the problem.
Google turned up a few people with the same problem, but they seemed to have their problems (from 7+ years ago) fixed by multiple backslashes, which clearly aren't working for me.
Any ideas? Our local people maintaining the MS Office365 nonsense dont' support fetchmail and are not interested in helping ...
I was having the same issues a year ago and basically gave up. I'm not 100% sure if this will work for you but it magically started working for me. I switched my O365 account two-factor auth (they use SMS to transmit the code, Ew) and had to create a bunch of app passwords[0] to be able to connect with my main email client. Randomly one day decided to see if this would work with fetchmail and lo and behold it did. I don't know why it worked or if something was disabled in that year but it works now. [0]: The location where you generate new passwords is here: <https://account.activedirectory.windowsazure.com/AppPasswords.aspx> Off-Topic: One of the scary things about finding URLs to generate passwords on Super User is the fact the `windowsazure.com` whois lookup responds with MarkMonitor Inc., they help corporations protect their domain names. So you have to check the SSL certificate first to see if it's a valid domain. I had to use this command: echo | openssl s_client -showcerts -servername account.activedirectory.windowsazure.com -connect account.activedirectory.windowsazure.com:443 2>/dev/null | openssl x509 -inform pem -noout -text to review if it's valid.
On Wed, Jul 04, 2018 at 03:08:03PM -0400, Myles Braithwaite 👾 via talk wrote:
I was having the same issues a year ago and basically gave up. I'm not 100% sure if this will work for you but it magically started working for me.
I switched my O365 account two-factor auth (they use SMS to transmit the code, Ew) and had to create a bunch of app passwords[0] to be able to connect with my main email client. Randomly one day decided to see if this would work with fetchmail and lo and behold it did.
I don't know why it worked or if something was disabled in that year but it works now.
[0]: The location where you generate new passwords is here: <https://account.activedirectory.windowsazure.com/AppPasswords.aspx>
Thanks for the suggestion! Unfortunately, it didn't work. I tried representing the backslash in .fetchmailrc as "\92" which is the decimal escape ASCII code for the backslash. It failed, but, oddly, the logfile has the MS Office365 server reading the username with a double backslash. No idea why. (The same results with octal and hex codes for the backslash.) If you put in a single backslash, it vanishes. A double backslash remains a double backslash. A triple backslash becomes a double backslash. Quadruple remains quadruple. Quintuple becomes quadruple. And so on. Either fetchmail or Office365 (or both together) are doing something to that backslash that seems to be causing the problem. I can't figure out what, though. Arrgghh. I use Linux to avoid this sort of idiotic issue, not to have to batter through it. -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
In a previous life, I worked with one of the people MS hired to do the ergonomics of Windows 95. Their task was to provide a lovely user interface that was *as different as possible* from any of Xerox Star, Apple Mac or Unix X11, specifically so that people would find them frustrating if they had ever used Windows. Regrettably, that meant that Windows was frustrating to anyone who had ever used a computer. Net result: they had to compete on quality alone, and Mac won. It looks like history is now repeating itself. I rather hope with the same results. --dave On 04/07/18 06:20 PM, Peter King via talk wrote:
On Wed, Jul 04, 2018 at 03:08:03PM -0400, Myles Braithwaite 👾 via talk wrote:
I was having the same issues a year ago and basically gave up. I'm not 100% sure if this will work for you but it magically started working for me.
I switched my O365 account two-factor auth (they use SMS to transmit the code, Ew) and had to create a bunch of app passwords[0] to be able to connect with my main email client. Randomly one day decided to see if this would work with fetchmail and lo and behold it did.
I don't know why it worked or if something was disabled in that year but it works now.
[0]: The location where you generate new passwords is here: <https://account.activedirectory.windowsazure.com/AppPasswords.aspx> Thanks for the suggestion! Unfortunately, it didn't work.
I tried representing the backslash in .fetchmailrc as "\92" which is the decimal escape ASCII code for the backslash. It failed, but, oddly, the logfile has the MS Office365 server reading the username with a double backslash. No idea why. (The same results with octal and hex codes for the backslash.)
If you put in a single backslash, it vanishes.
A double backslash remains a double backslash.
A triple backslash becomes a double backslash.
Quadruple remains quadruple. Quintuple becomes quadruple. And so on.
Either fetchmail or Office365 (or both together) are doing something to that backslash that seems to be causing the problem. I can't figure out what, though. Arrgghh. I use Linux to avoid this sort of idiotic issue, not to have to batter through it.
--- Talk Mailing List talk@gtalug.org https://gtalug.org/mailman/listinfo/talk
-- David Collier-Brown, | Always do right. This will gratify System Programmer and Author | some people and astonish the rest davecb@spamcop.net | -- Mark Twain
| From: Peter King via talk <talk@gtalug.org> | Recently we were forcibly "migrated" to Office365 for our email services, Interesting. I take it that this is per department. I just checked on email to my CS email account and it is only touched by Postfix and EXIM servers. | One of the last things | I cannot figure out how to solve is how to pass an alias email address to | the MS server. I'm not going to help you, I'm going try to learn from you. I don't understand fetchmail that well -- I set it up many many years ago and haven't looked at it since. What's an alias in this context? Is it like what newaliases(1) maintains for PostFix or Sendmail? | So in .fetchmailrc I have a line like this: | | user "prof.bozo@utoronto.ca\department.chair@utoronto.ca" there with password "XYZZY" is bozo here I don't see \ used in this way in the fetchmail manual. I don't think it is what they call multidrop. Is it part of the IMAP protocol? Where is it documented? How about two different user lines: poll whatever.utoronto.ca user "prof.bozo@utoronto.ca" there with password "XYZZY" is bozo here poll whatever.utoronto.ca user "department.chair@utoronto.ca" there with password "XYZZY" is bozo here or even: poll whatever.utoronto.ca user "prof.bozo@utoronto.ca" there with password "XYZZY" is bozo here user "department.chair@utoronto.ca" there with password "XYZZY" is bozo here PS: I've heard recommentations for getmail over fetchmail. I haven't needed to investigate. | But this fails, and in the fetchmail logfiles it represents the username as | not having a backslash: | | Jul 4 14:35:14 fetchmail[13829]: IMAP> A0002 LOGIN "prof.bozo@utoronto.camedieval.philosophy@utoronto.ca" * | Jul 4 14:35:14 fetchmail[13829]: IMAP< A0002 NO LOGIN failed. | Jul 4 14:35:14 fetchmail[13829]: IMAP> A0003 LOGOUT | | If I double-backslash it, though, I get a double backslash in the username. | Ditto for triple, quadruple, and quintuple backslashes. Nor does it work | if I single-quote the backslash. | | According to the fetchmail manual, the backslash is used to communicate via | ASCII sequences and codes with fetchmail, and it's pretty clear that this | is part of the problem. What piece of the software is supposed to handle \ the way you expect? The sendmail manual seems to say that \ is just a way to denote tricky characters. So I think you expect your meaning of \ to be handled by the IMAP server. And that the old IMAP server did handle it.
On Wed, Jul 04, 2018 at 07:59:02PM -0400, D. Hugh Redelmeier via talk wrote:
| Recently we were forcibly "migrated" to Office365 for our email services,
Interesting.
I take it that this is per department. I just checked on email to my CS email account and it is only touched by Postfix and EXIM servers.
It might be by broader area -- Arts and Science, for instance. It may get you yet. The University has been filled with proud pronouncements about its email+ system (which is code for Office365).
What's an alias in this context? Is it like what newaliases(1) maintains for PostFix or Sendmail?
Near as I can tell, it's what the MS techies call a "shared email account", that is, a non-personal email account that isn't tied to a single person. In my case it's an email account associated with a departmental role (such as chair or undergraduate advisor).
I don't see \ used in this way in the fetchmail manual. I don't think it is what they call multidrop. Is it part of the IMAP protocol? Where is it documented?
It isn't, at least not in the man page. In fact it's worse: fetchmail seems to reserve '\' as a special character, usually to introduce some ASCII code. I've tried replacing the backslash with the ASCII code for the backslash following fetchmail's rules, but no go.
How about two different user lines:
poll whatever.utoronto.ca user "prof.bozo@utoronto.ca" there with password "XYZZY" is bozo here
poll whatever.utoronto.ca user "department.chair@utoronto.ca" there with password "XYZZY" is bozo here
or even:
poll whatever.utoronto.ca user "prof.bozo@utoronto.ca" there with password "XYZZY" is bozo here user "department.chair@utoronto.ca" there with password "XYZZY" is bozo here
Interesting! I'll try it.
PS: I've heard recommentations for getmail over fetchmail. I haven't needed to investigate.
I know nothing about getmail, but I'll look into it. Fetchmail has served me faithfully and well for decades, and I'm used to it; if needed I'll move to something else -- but since it has worked so well for so long, and also continues to work well with all my other email accounts, I'm reluctant to change my habits to accommodate MS Office365. I will if that's what it takes, though.
The sendmail manual seems to say that \ is just a way to denote tricky characters. So I think you expect your meaning of \ to be handled by the IMAP server. And that the old IMAP server did handle it.
Yes to both. It may well be that the problem is not in fetchmail at all but in how Office365 interprets the string. I am told that people who use other clients (Thunderbird etc.) have no trouble with the backslash addressing, though, so it's still unclear where the problem lies. All this would be much easier if the IT staff were interested in helping. As it stands, they intone the mantra that if you use *NIX you're on your own, and clearly don't know or don't care about the technical details. Such is the penalty for not swimming with the rest of the fish. -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
On 2018-07-04 02:53 PM, Peter King via talk wrote:
Recently we were forcibly "migrated" to Office365 for our email services, and with the change numerous things were broken.
You might not be able to. I have a newish O365 account that staunchly refuses to use any protocol but MS Exchange. POP3 or IMAP at best timeout, but feed the same credentials into any exchange/outlook client and I'm in. I've also noticed in the main O365 account I pay for, plus a couple set up for me by past consultancies, was that my login ID was always an @office365.com, @live.ca (!) or some other MS-owned domain. I could merrily send and receive e-mail under the company domain, but the credentials were always at the same domain. Unpopular opinion: Outlook is the best e-mail client. In many technical and UX aspects it may be the worst, but it is how corporate e-mail is done, and that's that. Stewart
On Thu, Jul 05, 2018 at 05:26:44AM -0400, Stewart C. Russell via talk wrote:
You might not be able to. I have a newish O365 account that staunchly refuses to use any protocol but MS Exchange. POP3 or IMAP at best timeout, but feed the same credentials into any exchange/outlook client and I'm in.
Well, fetchmail can access O365 as an IMAP client and pick up my mail reliably (and msmtp deliver it reliably). That took a bit of fussing but nothing special. The problem is in accessing shared mailboxes, where O365 seems to want the username to be of the form: personal.email.account@domainname\shared.email.account[@domainname] I don't know if the problem is in fetchmail or O365 yet.
I've also noticed in the main O365 account I pay for, plus a couple set up for me by past consultancies, was that my login ID was always an @office365.com, @live.ca (!) or some other MS-owned domain. I could merrily send and receive e-mail under the company domain, but the credentials were always at the same domain.
My login ID is @utoronto.ca, though the server is outlook.office365.com. As far as I can tell by looking at the headers, that is the domain name; if I had to venture a guess, I'd say that the University of Toronto pays for running its own domain and having Microsoft manage the email. -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
On 2018-07-05 05:26 AM, Stewart C. Russell via talk wrote:
On 2018-07-04 02:53 PM, Peter King via talk wrote:
Recently we were forcibly "migrated" to Office365 for our email services, and with the change numerous things were broken.
I haven't thought about Outlook and Exchange for quite a while. I remember some years ago when the topic came up for someone wanting a Linux compatible solution. There was a program for Linux that was compatible with Exchange and Outlook. It has been too many years since I last read about that topic. I don't remember the name of the program. There may be more than one solution available nowadays. Just mentioning it in case it offers a way to use some Office stuff while staying Linux. -- Cheers! Kevin. http://www.ve3syb.ca/ | "Nerds make the shiny things that https://www.patreon.com/KevinCozens | distract the mouth-breathers, and | that's why we're powerful" Owner of Elecraft K2 #2172 | #include <disclaimer/favourite> | --Chris Hardwick
Thanks for all the suggestions and advice. I installed getmail, and, with a very simple configuration -- the only kind I can write yet -- it fetches mail from O365 shared mailboxes just fine. So I suppose I'll migrate to getmail from fetchmail, which seems to run more efficiently anyway. Thanks again! -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
For those who might be interested, here's the working getmailrc configuration I'm using successfully with Office365 (suitably anonymized): .getmail/getmailrc: [options] delete = true message_log = ~/.getmail/log message_log_verbose = true [retriever] type = SimpleIMAPSSLRetriever server = outlook.office365.com username = PERSONAL_EMAIL@utoronto.ca\SHARED_MAILBOX@utoronto.ca port = 993 password = PASSWORD [destination] type = MDA_external path = /usr/bin/procmail unixfrom = True That's the complete file. Here's what seems to be going on. The first stanza, [options], gives global instructions to getmail. Here, the first, delete = true, just deletes the messages off the O365 server after it retrieves them. The next two lines specify where getmail should write its log messagees, and that they should contain a certain amount of detail. The second stanza, [retriever], tells getmail which account to try to get the email from. Note that there is only a single account listed. That's a feature, not a bug, of getmail. If you want to get email from several accounts, you need to use several distinct getmailrc configuration files (each uniquely titled), and then run through them all -- a simple shell script will do that. Anyway, here the type variable says that I'm picking email up from an IMAP server using SSL; the server variable gives the address of the server. The form of the username here, with the personal email address + backslash + address of the shared email account, is where getmail succeeds in the case fetchmail failed -- for whatever reason getmail passes that on to the O365 server in a way that it parses correctly. (It seems that all "shared mailboxes" in O365 are in fact something like aliased subirectories under a personal account for reasons I don't understand.) The next two are obvious: the port gives the port, and fill in your password next. The last stanza, [destination], describes what getmail should do with the email. Here I have it set to hand the email off to procmail for sorting: type = MDA_external tells getmail to hand over the email to a distinct mail delivery agent; the value of path tells getmail where to find the MDA. The unixfrom = true tells getmail that it can expect the emails to begin in proper *NIX fashion with a From: line. In case anyone is interested, here's the relevant bit of procmail configuration, from .procmailrc: # Define variables MAILDIR=$HOME/Mail DEFAULT=$MAILDIR/NEWMAIL LOGFILE=$HOME/.procmaillog # :0: * ^To.*SHARED_MAILBOX@utoronto.ca $HOME/Mail/DESTINATION_MAILBOX The variables just configure my local setup; the line ":0:" is a magic incantation that tells procmail not to keep a copy once it carries out the instructions in the next two lines; the first looks for any email where the address (To:) header is the SHARED_MAILBOX, and the last line tells it to write that email into the mailbox DESTINATION_MAILBOX. On my first test pass, I had getmail deliver the mail directly to its destination, so the whole stanza looked like this: [destination] type = Mboxrd path = ~/Mail/DESTINATION_MAILBOX Here the first line declares that the email is heading for a mailbox in mbox format, and the second line that the mailbox is as before the DESTINATION_MAILBOX found at ~/Mail/. So far, Things Just Work. Since getmail gets the mail, I assume it passes the relevant SSL tests. I'll have to learn some more to make sure it is using, or at least is able to use, fingerprint validation. Right now I'm testing and running getmail manually, but I expect to make it just an ordinary cron job once I learn enough to be confident that everything works properly. -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
On Sun, Jul 15, 2018 at 11:42:12PM -0400, Peter King via talk wrote:
username = PERSONAL_EMAIL@utoronto.ca\SHARED_MAILBOX@utoronto.ca
Just for curiosity, can you try "PERSONAL_EMAIL@utoronto.ca"\x5c"SHARED_MAILBOX@utoronto.ca" in you .fetchmailrc ? The manpage is not clear whether you can use \xhh in quoted string (though highly likely). -- William Park <opengeometry@yahoo.ca>
On Mon, Jul 16, 2018 at 12:25:10AM -0400, William Park via talk wrote:
Just for curiosity, can you try "PERSONAL_EMAIL@utoronto.ca"\x5c"SHARED_MAILBOX@utoronto.ca" in you .fetchmailrc ? The manpage is not clear whether you can use \xhh in quoted string (though highly likely).
Tried it, and in fetchmail it still passes the wrong number of backslashes. As far as I can tell, fetchmail and O365 only exchange an even number of backslashes: one gets you 0, two gets 2, three gets 2, four gets 4, and so on. I have no idea why this should be so, apart from the Universal Law of Software Perversity. Doesn't matter if I do straight characters, quoted or escaped characters, decimal, octal, hexadecimal ... -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
On 2018-07-16 11:23 AM, Peter King via talk wrote:
fetchmail it still passes the wrong number of backslashes. As far as I can tell, fetchmail and O365 only exchange an even number of backslashes: one gets you 0, two gets 2, three gets 2, four gets 4, and so on.
This is starting to sound like a bug in fetchmail or it is some sort of compatibility "feature" with Office365. If fetchmail isn't converting two backslashes to one I would talk with the fetchmail people and/or file a bug report. -- Cheers! Kevin. http://www.ve3syb.ca/ | "Nerds make the shiny things that https://www.patreon.com/KevinCozens | distract the mouth-breathers, and | that's why we're powerful" Owner of Elecraft K2 #2172 | #include <disclaimer/favourite> | --Chris Hardwick
On Mon, Jul 16, 2018 at 12:25:34PM -0400, Kevin Cozens via talk wrote:
On 2018-07-16 11:23 AM, Peter King via talk wrote:
fetchmail it still passes the wrong number of backslashes. As far as I can tell, fetchmail and O365 only exchange an even number of backslashes: one gets you 0, two gets 2, three gets 2, four gets 4, and so on.
This is starting to sound like a bug in fetchmail or it is some sort of compatibility "feature" with Office365. If fetchmail isn't converting two backslashes to one I would talk with the fetchmail people and/or file a bug report.
Given that getmail works seamlessly, I'm inclined to think it's a bug in fetchmail (or a bug in the way it interacts with Office365). I'll send a note to ESR about it. -- Peter King peter.king@utoronto.ca Department of Philosophy 170 St. George Street #521 The University of Toronto (416)-978-3311 dept Toronto, ON M5R 2M8 CANADA http://individual.utoronto.ca/pking/ ========================================================================= GPG keyID 0x7587EC42 (2B14 A355 46BC 2A16 D0BC 36F5 1FE6 D32A 7587 EC42) gpg --keyserver pgp.mit.edu --recv-keys 7587EC42
participants (7)
-
D. Hugh Redelmeier -
David Collier-Brown -
Kevin Cozens -
Myles Braithwaite 👾 -
Peter King -
Stewart C. Russell -
William Park