Working with Exported Messages
One annoying thing that researchers will run into through open records/FOIA request processes that include email messages is how to actually view the data in a useful way. We recently had a scenario where we were delivered over 1700 messages broken out into individual .msg files, which are Outlook specific. Another way that messages may be exported is a .pst file, which you can logically think of as an archive of multiple Outlook messages, again in a proprietary format. The plus being that you can share a bunch of messages in one .pst file instead of thousands of individual .msg files.
Risks of working with .msg files
One of the biggest risks is the accidental manipulation of email metadata, including send/receive dates.
Working with messages locally
If you have a copy of Outlook and don’t need to share the data with multiple people, you can simply create a new folder and drag the .msg files directly into the Outlook folder. However, this does have one big caveat: you must already have a working Outlook profile setup. Considering that Outlook is typically only used in a corporate setting in conjunction with an Exchange Server setup (either hosted on-prem or via Office 365), this means that there’s a high likelihood of accidentally adding personal metadata to items as they’re imported to Outlook, especially when they’re imported into a mailbox on Exchange instead of a local folder. Even if that’s not a concern, from a digital forensics standpoint this is still frowned upon as the number one goal is to leave the original data in an unchanged format.
Problem: Non-Outlook/Non-Exchange Clients
Since .msg/.pst files are proprietary, non-Microsoft products generally do not have native support for importing this data correctly. In my testing I tried multiple solutions and ran into roadblocks such as loss of formatting for messages that use the RTF format or loss of attachments. Both of these are pretty big issues in research.
The Goal
My goal was to be able to provide a webmail interface to the information provided through our Open Records request so researchers on any platform in any location would be able to access the data and sift through it however they see fit. Requiring specific software like Outlook or an Exchange environment is clearly not an option due to the licensing, cost, platform restrictions, and huge infrastructure prerequisites such as Active Directory. Not to mention the abysmal state of on-prem Exchange’s security. Email itself is absolutely a solved problem, so I started testing a few different ideas.
Scenario 1 – Local Outlook, Sync to non-Exchange IMAP Server
As mentioned before, .msg is a proprietary format. Since I did have a personal Outlook install, my first thought was to stand up a non-Exchange email server, import the message files locally to Outlook, and then have Outlook sync them to the server.
This, unfortunately, was a bust. Because Outlook expects to work with Exchange, messages that were created in an Exchange environment are exported in a format that includes additional metadata. This shows up to non-Outlook clients as an attachment named winmail.dat. When I did my initial testing I found that while viewing the messages on the email server through Outlook everything appeared fine, but when accessing through a webmail portal there were issues such as loss of formatting and attachments being missing, instead replaced with that winmail.dat file. This is clearly not ideal.
Scenario 2 – Mozilla Thunderbird
Mozilla Thunderbird is a freeware email client, and as such was my next attempt at viewing the data. It does not support .msg files natively. If you search you will find that there was a .msg import addon but it was unfortunately abandoned and does not work with any recent version of Thunderbird, so sadly that was out. I did test viewing mails that were synced to the server by Outlook within Thunderbird and unfortunately still had the same loss of formatting/attachments issue.
Conversion Utilities
It quickly became clear that I would have to find some way to convert the .msg files to some other more open standard. The obvious target here would be converting from .msg to .eml format, as the latter can be natively imported in many clients (including Thunderbird). It is not an ideal solution as this does mean the data will be manipulated, but it shouldn’t result in loss of metadata or attachments.
However, if you search for tools to convert MSG to EML files you’ll quickly find yourself in SEO (search engine optimization) hell. Because this is a somewhat common situation for people in the end user computing profession, there are predictably hundreds of people that will try to sell you a tool to perform this conversion. They all pretend to be free utilities which sound great until you actually try to convert more than a handful, at which point you’ll be told to buy a license or the product will stop working. This is not ideal. There are also a few Python projects kicking around on github; I tried a few but didn’t have much success as there isn’t generally a big need for Linux admins to manipulate Microsoft-centric files.
The Actual Solution – MailStore Home
After beating my head against the wall, I suddenly remembered that before I switched away from Windows for my day-to-day personal computing, I used a free program called MailStore Home to archive my email locally, both from Exchange as well as other standard mail servers that use IMAP. It is free to use for non-commercial purposes, and can be installed in a portable version that is easily archived and shareable such that anyone on a Windows machine can simply extract, run the program, and be able to interact with the email archive with no other dependencies. I had only used it to archive email sitting on a server so that was my initial thought, but once I spun up a VM to test it out I realized it was even more useful.
Portable Installation
To install MailStore Home to a self-contained folder, just download and launch the installer. Accept the license and hit Next.
Select “Install portable version in other directory…”
Choose/create the directory where you want to store the app and its data, then hit OK.
Hit Install and the process will begin.
Once you’re here just hit Finish and the app will open, ready for use.
MailStore Home – Direct .MSG Import
Much to my surprise, MailStore Home has the ability to directly ingest .msg files without needing a mail server or client at all. The biggest advantage here is that it means that it’s the closest you can likely get to a forensic-style handling of the messages, metadata and attachments without using actual digital forensics software.
Just open the program and select Archive E-Mail.
You can see the versatility here- just choose “EML and MSG files” and it will open up a wizard that lets you point it to a folder containing all of your .MSG files. Uncheck “Read MailStore Headers” since these messages didn’t come from MailStore, then hit “Next.”
On the next screen, you shouldn’t have to change anything. If you wanted to target a specific date/time you can do that here, but typically we want to import everything and sort later. Hit “Finish” and it will create a profile ready to run.
You’ll get this confirmation screen that tells you to double-click the new profile entry to start ingesting the data.
Now just double-click the profile and the messages will be imported into the MailStore Home archive. You should see a progress window come up and once done, your messages are indexed and ready to search.
Now if you expand “My Archive” in the left hand pane, you’ll be able to see all of the messages and browse at will, with easy access to the original headers and source of each message.
With the “Select E-mail Client…” dropdown at the top you can also directly open the message in a local Thunderbird or Outlook install, or even selectively restore messages directly to an IMAP Mailbox (i.e. upload them to your own mailserver). That tipped me off to the real power of the tool for assisting in these records requests.
MailStore Home – .MSG to IMAP Bridge
Once all of the messages are inside MailStore’s archive, you can then archive that folder and move it to any Windows PC to search and review message data. However, since my real goal was to publish the data to a webmail frontend, we now want to export the data to our own mail server directly. MailStore Home makes this a trivial task.
Select the “Export E-mail” tool from the left hand pane and review the options.
Choose “IMAP Mailbox” and a wizard will pop up that’s very similar to the Import wizard. Here you can choose which folder(s) you want to send to the mail server – select the desired items then hit Next.
Fill in the connection information for your mailserver and hit “Test” to ensure it’s configured correctly.
Then just hit “Finish” and the profile will be created.
Once that’s done, just double click the profile and the export process will begin. When the process finishes, check your webmail client and you should be pleased to find all of the messages available with metadata intact, including envelope info, headers, and attachments.
Conclusion
Hopefully this post helps out anyone wishing to help expand their research capabilities, especially when working as a team. Simply creating and passing around the MailStore Home archive as a zip file is an easy way for anyone on Windows to start reviewing data with minimal effort, but if you have the ability to spin up a mail server with webmail access then using MailStore as an IMAP conversion bridge is really a great option. For my purposes, I used docker-mailserver for the mailserver portion and Afterlogic’s docker-webmail-lite for webmail. Both were straightforward to implement as long as you’re familiar with Docker. They both worked extremely well and even provided full text search via solr through a bit of extra tweaking.
The main win by using the above methods is that the barrier to entry for users who simply want to review the data is very low. If they’re familiar with an email client or webmail – as we all are – they won’t need any assistance to start digging through records and identifying anything of interest. I’m hardly a digital forensics/OSINT expert so I would love to hear other approaches or feedback – feel free to either make a comment here or reach out via Twitter at @sally_yachts.