Content tagged email

Install the ssmtp MTA:

sudo apt-get install ssmtp

Then configure it:

]==> cat /etc/ssmtp/ssmtp.conf
mailhub=your.smtp.net:587
AuthUser=your_login
AuthPass=your_password
UseTLS=YES
UseSTARTTLS=YES

Map users:

]==>  cat /etc/ssmtp/revaliases
local_username:username@domain.net:your.smtp.net:587

Make sure it's safe:

chown root:mail /etc/ssmtp/ssmtp.conf
chmod 640 /etc/ssmtp/ssmtp.conf
usermod -a -G mail local_username

Log out and log in again.

Intro

It has been a while since I wrote the post on moving emails out of GMail. I have since managed to experiment a little with various web apps available on the internet and arrived at a solution that is not perfect, but meets my needs well. I will describe what I currently use for dealing with emails, contacts and calendars in this article. It's really quite astonishing how many different pieces are needed to provide the GMail experience.

Mail

The goal is to be able to read, send and consistently browse emails from Linux PCs using a fast and responsive client with offline browsing support, from a Web browser using something that looks nice and supports threading, and from an Android mobile phone. There is plenty IMAP and SMTP clients out there for all of these platforms, but the following three worked best for me:

  • Thunderbird - no fireworks, but it's good enough. The only real issues I have with it are: the message threading being messed up on occasions, ie. it groups messages that don't belong together; and the address book not managing to handle the full vCard information properly, there will be more about this later. I've recently learned about Trojitá and it may actually be exactly what I need - to be evaluated.
  • RoundCube - pretty great, but may get sluggish at times, although this is probably due to a fact that I run it using a shared hosting. I extend it with the two following plugins in order to handle multiple accounts with one "instance":
  • Kaiten Mail - by far the best email client for Android that I have ever used. I like it for good threading support and intuitive GUI.

Contacts and Calendar in the cloud

Another big piece of GMail's added value is the ability to manage contacts and calendar (the calendar actually is a separate product) and keep them synchronized between devices. It does it in a rather messy way for my taste, but it does it well. I have once found an informative how-to article describing a solution to this using ownCloud as a CardDAV and a CalDAV server, but I did not like the web interface ownCloud provides. There's a couple of alternatives doing the same thing, and the one I like is Baïkal.

I install the flat version and switch it to use basic http authentication. Baïkal (actually SabreDAV, which Baïkal uses internally for DAV) uses the digest authentication by default, which is supposed to be more secure, but I found it glitchy with many clients.

Baïkal settings panel
Baïkal settings panel

Baïkal is just a CardDAV and CalDAV server and, as such, just manages the users and resources, it doesn't come with any clients, so you'll need to install them separately. I use:

  • CalDavZAP - a calendar client. In order to configure it, set the href element of the globalNetworkCheckSettings array to 'https://yourdomain.com/baikal/cal.php/principals/'
  • CardDavMATE - an addressbook client. Change the href element of the globalNetworkCheckSettings array to 'https://yourdomain.com/baikal/card.php/principals/'. I also change globalContactStoreFN, this affects how other clients see the contact's Formatted Name - I like to see the first name first.

Note: the two above work reliably only with basic http authentication, so you probably want to use https access to the DAV server.

You can then export all your contacts from GMail to a vCard format and then upload them to Baïkal using another tool of mine.

Calendar and address book clients

Thunderbird will need three extra plug-ins:

  • Lightning - for calendar support:
  • Go to File->New->Calendar..
  • Select On the network
  • Select CalDAV and tick the Offline support
  • Enter address of the calendar, ie: https://yourdomain.com/baikal/cal.php/calendars/user/calendarid
  • SOGo Connector and MoreFunctionsForAddressBook - for decent enough address book functionality:
  • Go to Address Book
  • Go to File->New->Remote Address Book
  • Give a name to the address book
  • Enter the URL of the address book, ie: https://yourdomain.com/baikal/card.php/addressbook/user/bookid/

RoundCube needs two plug-ins:

  • carddav and carddav_plus:
    • Go to Settings
    • Go to CardDAV Settings
    • Add the address books using the same addresses as in the case of Thunderbird
    • Go to Address Book
    • Click Synchronize

Android apps needed to integrate the calendar and the address book:

  • CardDAV-Sync beta
  • CalDAV-Sync beta
  • If you use HTC Sense, you may have some issues integrating the contacts, this should help in solving them
  • The new Android contact editor is also somewhat retarded, but you can still download a port of the old one.

What's next?

The setup described above allows for handy integration of all my devices. Although, I find having to log-in to all the web apps separately somewhat annoying. I will need to find some single-sign-on solution to integrate them. Another big thing missing is GTalk. I know it's just jabber, but I find it handy to have the chat history accessible from one place. I will play with this a bit as well.

Why, oh why?

GMail is pretty great, but I don't think I need to tell you that. I started to use it because of its killer spam filter and search capabilities, then, as new features were introduced, I liked it more and more. The fact that it picked into the mails I send and receive didn't really bother me that much until recently. My view on all this has changed after the new privacy policy was introduced, saying that they will share this sort of private info between all of the Google's services. Many people may be fine with it, but I am not. After reading a couple of blog posts (like this one, and this one), and seeing some lame Microsoft commercials (youtube), Google's reading my emails started to creep me out. And it didn't really matter how much they could possibly improve my world by doing this (fixing the search).

How?

Using IMAP, of course. There's a couple of articles on the web describing how to do that, but they all have one problem. They tell you to copy your emails from the "All Mail" folder in the gmail account, which means that you will lose all your labels, and that's probably not acceptable for someone with thousands of conversation threads all placed in the boxes they belong to. I have written a couple of python scripts that should help solving this problem. You can get them from github. Remember: You use them at your own risk. Read on, if you want to know what they do.

Problem with labels

There are some issues with labels when accessing gmail through IMAP. Since the labels are mapped to the IMAP folders, you might, at first, expect that, when you open the folder and browse it, you will see all the emails in all the conversations bearing the label. This is not true. GMail operates and labels things at the level of conversations, IMAP, however, operates at the level of messages, so you will only see the messages that were part of the conversation when you assigned the label but not the new ones. This makes some sense if you use gmail with a normal IMAP client, ie. you don't see the same new messages arriving in many folders, an you avoid having the archived messages back in the IMAP inbox when somebody responds to a message in an archived conversation.

This is a real problem when you'd like to move your nicely sorted conversations out of gmail to some other service. There is a suggestion at the Google Product Forums on how to deal with it: you should go to the web browser, open the conversation, remove the label, and then re-apply it. It's not really a viable solution if you have more than a couple of conversations, fortunately Google provides an API that will enable us to do that auto-magically.

This is what the gmail_label_remap.py does, it finds every conversation and re-applies the labels. You should see something like this when you run it from your terminal:

]==> ./gmail_label_remap.py
username (won't be echoed):
password (won't be echoed):
[i] Identify the IMAP folder name for "All Mail"...
[i] Done: [Gmail]/Tous les messages
[i] Opening [Gmail]/Tous les messages
[i] [Gmail]/Tous les messages contains 15299 messages
[i] Identify all the threads (may take some time)
[i] Found 4182 threads
[i] Processing thread 4182 of 4182
[i] Orphaned threads:   414
[i] Sent only threads:  255
[i] ALL DONE

It will create two extra labels:

  • orphaned - for the conversations without labels
  • sent_only - for the emails that you have sent but got no answer to them, so they have no assigned label

Moving emails to a different IMAP account

Now that all the label issues are sorted out, you need to copy the emails. You may use an IMAP client of your preference, or the imap_copy.py script. Using the script you may get the list of all your IMAP folders:

]==> ./imap_copy.py --list --source=imap.gmail.com:993
imap.gmail.com's username (won't be echoed):
imap.gmail.com's password (won't be echoed):
Fun (1641)
INBOX (114)
Private (484)
Work (1010)
[Gmail]/Brouillons (0)
[Gmail]/Corbeille (563)
[Gmail]/Important (5204)
[Gmail]/Messages envoy&AOk-s (4440)
[Gmail]/Spam (16)
[Gmail]/Suivis (59)
[Gmail]/Tous les messages (14736)

It will also let you copy the messages from one IMAP account to another:

]==> ./imap_copy.py --copy=INBOX.gmail-inbox,Fun \
                    --source=imap.gmail.com:993  \
                    --destination=imap.somewhere.else:993
imap.gmail.com's username (won't be echoed):
imap.gmail.com's password (won't be echoed):
imap.somewhere.else's username (won't be echoed):
imap.somewhere.else's password (won't be echoed): 
Copying INBOX => gmail-inbox
Copying 114 of 114
Copying Fun => Fun
Copying 1641 of 1641
ALL DONE

What you specify after the 'equal' sign of the --copy parameter is a comma separated list of folders. It may just be a folder name, like Fun in the example above. Or something like: INBOX.gmail-inbox which will take the INBOX folder at the source and copy its contents to the gmail-inbox folder at the destination.

What's next?

For the time being I will use KMail for e-mails and Psi for jabber, they are pretty decent, but I liked having everything (conversation history, settings, contacts) in one place on the web, so I will probably look for, or help developing, a solution that will give me all that and more. RoundCube looks promising.

Edit 28-04-2012: Continuation post has been added.