« Prioritize your E-mails with a star rating | Main | What are your Entourage & Exchange pain points? »

Use AppleScript to read ActiveSync Sent Items in Entourage

[Download the Read ActiveSync Sent Items script]

Exchange users with new iPhones are re-discovering an old bug where messages sent through ActiveSync for mobile devices do not appear properly in both Entourage 2004 and Entourage 2008. The messages sometimes look like a mix of HTML, E-mail headers and what's left of the original message or they look like an unintelligible jumble of letters and numbers. The recipient and the subject are also empty.

corruptmessage.jpg

This problem was addressed in the knowledgebase article "The header information is not displayed in Entourage 2004 for an e-mail message that is sent by using Exchange ActiveSync". However, the workaround mentioned in the article, which is to view the message in Outlook Web Access (OWA), does not work in either the full version that Internet Explorer for Windows users would see or the light version that Mac users would see.

What's happening and how can you read these "corrupt" messages?

Who's the culprit?

Exchange users assume this must be a bug in both Entourage 2004 and Entourage 2008 because they do not see this problem in Outlook for Windows. However, if they also try to view the files in OWA they will find the messages missing altogether.

What do Entourage and OWA have in common? They both connect to Exchange via WebDAV. So, this problem isn't necessarily a bug in Entourage. In fact, it's a bug in ActiveSync for Exchange. ActiveySync is the Microsoft technology that allows mobile devices such as Windows Mobile phones and the iPhone connect to Exchange. Furthermore, it's specifically a bug in ActiveSync for Exchange Server 2003. Entourage users connecting to Exchange Server 2007 servers should not see this problem.

What causes the problem?

So, what is ActiveSync doing to mail messages as they pass through it and why doesn't Outlook or any other E-mail client have this problem?

ActiveSync is munging the headers of the mail messages. Notice in the earlier screenshot that the mail message includes X-MimeOLE, Received, MIME-Version and other lines of information you normally wouldn't see. These are the headers, or the envelope of your mail message. They are suppose to be in a specific format so that mail applications can translate them and make the message presentable.

But look at what ActiveSync is doing to the message you send. The headers are appearing in the body of the message, which is not normal:

message.jpg

If you look at the source, or the full message with headers and body combined, by selecting Source from the Message menu (select Source from the View menu in Entourage 2004), you'll see it is identical to the body of the message except for two (sometimes more) additional headers at the top:

messagesource.jpg

Technically, a mail message header is a sequence of non-empty lines at the beginning of the mail message. But if we are to believe that with this example message then the entire header would simply be the first two lines because they are followed by an empty line:

MAIL FROM: <William.Smith@domain.com>
RCPT TO: <talkingmoose@gmail.com>

ActiveSync is appending these bogus headers and causing the real headers to fall into the body of the message. But why isn't Outlook for Windows susceptible to this problem? Most likely it has simply been programmed to realize these two headers are bogus and to continue reading the rest of the headers properly.

Are the messages still readable?

Yes! We simply need to remove the bogus headers and then view the messages in their correct format. To say that these messages are "corrupt" is a misnomer. They are not irreparably damaged. They can be properly viewed with the help of AppleScript. I've written a script to do just that.

Download the script and follow the instructions to install it into the Entourage Script Menu Items folder in your Microsoft User Data folder. Once installed it will be immediately available in the Script menu in Entourage.

To use the script simply select the unreadable message in the message list, or open it, and then select the Read ActiveSync Sent Item script from the Script menu. If the script detects the message has the improper headers then it will open the message in a new window and it should be readable. So the example message above should now display as:

restoredmessage.jpg

The new message is not really in Entourage but just a temporary file. If you wish to save it to your Desktop then you'll have that option:

copymessagedialog.jpg

The new message file will be saved with the Subject as its name.

Due to limits with Entourage's AppleScript capabilities, the message cannot be automatically saved back into Entourage. You can, however, drag and drop the message into the original folder and delete the old message. Sent date and time will be preserved so the message will properly sort within the message list. Only one message can be read and saved at a time.

[Note: The download file contains an Example.eml file, which I used for the examples in this post. I've included it for anyone interested in examining this problem in detail.

Special thanks to Jon Connell in the Microsoft Entourage newsgroup for his insight that this problem could possibly be solved with AppleScript!]

Comments (40)

Jason Berk said:

Would you be able to tweak the script that it will process multiple messages at once. I'd love to be able to batch a group of messages every month until Microsoft fixes this.

I hope to.

The script started simple and then grew more complex as I threw in the ability to run it with the message open or closed, threw in error checking and threw in some cleanup routines in case something went wrong.

I only got it posted just last night so I'll take your feedback and feedback from others for the next tweaks that folks need.

Calbhac O'Carroll said:

Great! The end of my frustration. Batch conversion would be a nice addition.
Thank you!

RageGuy said:

The workaround in the KB article does work, I can view the messages fine (in Sent Items folder, which were sent thru my EAS based device) without any issue in Outlook and Outlook Web Access using any browser on any platform.

John Banks said:

Excellent, thank you William!

For me, my iPhone sent messages do not appear at all in Entourage nor Outlook Web Access (OWA).

Let's talk in a few weeks, RageGuy, and I'll try to show you my examples. ;-)

Michael Antolovich said:

I've noticed that once the emails are converted and everything re-synced, that the converted messages now have the conversion date and time (actually only when viewed on my iPhone. Entourage and Outlook do the right thing). It isn't tragic, just a little weird...and possibly only seen on my setup...

Randy Pinner said:

Hey William, thanks for the script! This was getting frustrating for me since I'm the only iPhone user in my office and was trying to vet the phone for full network integration.

dazza said:

Thanks william, worked for me... will be a big help for others in my organisation. Thanks again.

I would say this is a bug in exchange for not stripping the headers correctly in the protocol before presenting it to users via webdav

is it not possible to make a filter on exchange to correct this ?

regards

John Jones
http://www.johnjones.me.uk


hafstang33 said:

I've experienced this exact problem with messages sent from my iPhone 3G and viewed in Entourage 2004. Thanks for the script which I will try. I'm also wondering if another issue I am having is related. This morning I launched Entourage and got an error message that my database was corrupted and needed verifying. It then asked me if I wanted to rebuild it, which I did. At the end of the process, numerous old e-mails appeared in my outbox and got sent to several recipients confusing them and prompting them to write to me asking what was going on? i never had this problem prior to buying the iPhone a couple weeks ago. Comments welcome.

Peder said:

My sent emails show uo fine in OWA but comes out corrupt in Entourage 2004 and 2008. Doesn't that imply that the bug is in Entourage?


According to one MacBU developer, the workaround in the Microsoft KB article, which is to read the messages in OWA, does work. However, I've seen in two different Exchange Server 2003 systems where the messages simply don't appear in OWA. I'll be visiting MacBU in a little more than a week and hope to discuss this with him more. I'm hoping he can explain why this sometimes works and sometimes doesn't.

However, for reasons I've already explained in this post, the problem is not a bug with Entourage. The problem is with munged headers. Entourage has not been programmed to interpret these munged headers as I suspect Outlook has been.

Norb said:

has anyone seen a fix for this yet?

this is a exchange 2003 problem. i read somewhere where they say in the exchange 2007 this is fixed

brandon said:

Nice work. Thank you very much for this. As some others mentioned, a method to batch process these messages would be fantastic.

Michael Wilmar said:

Thanks so much for this script. But just one further comment: why doesn't MS fix this!!??

Dave said:

Thanks for the background info on how the messages are mishandled. I was not able to get the script to work within entourage (2004) - it did not display the corrected message and when opting to save to the desktop, an error (...can't make desktop type constant) occurs. The script runs fine in the script editor, correcting and displaying the selected message, but when opting to save to the desktop, it errs again. Same behavior on two different machines (10.4.11, ent2004; must be something simple). Was able to edit the messages 'by hand', though, after seeing what was causing all the trouble.

Michael Wilmar said:

This script is no longer working well for me. Worked fine at first, but now takes forever to execute. Entourage also becomes very sluggish in the meantime.

I did try to add a key shortcut through Quickeys so that I didn't have to use the mouse to press the No button on the Save to Desktop screen. Maybe that did it, but getting rid of the shortcut didn't help with performance.

Any ideas?

Hi Michael!

I have no idea why the script would be getting slower. It simply reads what you've selected and acts only on that information.

You can also open the script in the Script Editor found in /Applications/AppleScript and run it directly from there. Saving it again may help.

Hope this helps!

Karl Bundy said:

In case anyone is interested, I have modified this script to be able to be run with many selected items. If the email does not need to be fixed, it is skipped, otherwise it is processed using the logic from William Smith's original AppleScript. After the first message is processed, it prompts you if you would like the entire batch of email messages to be saved to the desktop or not, similar to how the original version did for a single email. I am a bit of a hack with AppleScript, so I apologize in advance for the clumsiness of the code! Here is the URL:


http://www.sopriswest.com/downloads/Read_Selected_ActiveSync_Sent_Items.zip

Very welcome changes indeed, Karl! A lot of folks have been asking for this capability.

Thanks for your work! I'm sure folks will be appreciative. :-)

Gijs said:

Hi,

Inspired by William's research I wrote an Applescript that works better for me. You can run the script in any way you please but putting it in the Entourage script menu is convenient. The script makes a new 'readable' outgoing message for every 'iPhone originated' outgoing message, while deleting the latter.

It works fine for me. But use it at your own risk:

tell application "Microsoft Entourage"
set theExchangeAccount to item 1 of every Exchange account
set theSendbox to sent items folder of theExchangeAccount
set theMessages to every message in theSendbox
repeat with theMessage in theMessages
set theSource to source of contents of theMessage
if theSource begins with "MAIL FROM: set the AppleScript's text item delimiters to "Received: from"
set theNewSource to "Received: from" & text item 2 of theSource
set the AppleScript's text item delimiters to ""
set theExchangeAccount to item 1 of every Exchange account
set theSendbox to folder "Verzonden items" in theExchangeAccount
set theEntourageMessage to make new outgoing message with properties {source:theNewSource}
move theEntourageMessage to theSendbox
delete theMessage
end if
end repeat
end tell


Have Fun,

Gijs

Gijs said:

Hi,

Pasting the script as text in the previous forum message wasn't such a good idea after all, as some characters got lost. I did some more work on the script, removed a bug that prevents the script from running on non-Dutch version of Entourage, and uploaded it to Scriptbuilder.net
You can find MakeSentFromiPhoneReadable 1.0 at

http://scriptbuilders.net/files/makesentfromiphonereadable1.0.html


Hope it works for you too!

Gijs


benny said:

Hey Gijs,

thanx for your investigation in this issue. I tried to use your script, copying the file in the "msuserdata" folder, but when executing the script nothing happens?

I user the german version - does this ma any difference?

Thanx in advance
benny

Miles Flint said:

I've downloaded the script and it works fine on my Mac. However I also use Outlook on a Windows pc (same server) and in the Outlook Sent items folder the item shows up with the date (i.e. today) that it was re-imported into Entourage when the mail item was actually sent six months ago and shows the six months ago date in Entourage

Any advice. Many thanks.

@Mike Flint

Probably like Entourage, Outlook can display two dates: Sent and Received.

The Sent date is part of the original message and is timestamped by the Sender. The Received date is the message creation date timestamped by your own E-mail program.

Make sure you're viewing by Sent and not Received.

Kevin Walsh said:

Bill,

So where do we complain about this. This is nuts to have this broken this long.

@ Kevin Walsh

Because the problem lies with Exchange and it is primarily an enterprise product, the first place you should consider is your company's Microsoft representative if you have one.

I doubt seriously this will get fixed, however, since it was resolved in Exchange Server 2007. Exchange Server 2003 is now in maintenance mode. It'll get patched for security issues but I'd imagine little else.

Thomas said:

I noticed a few things while I tried to use the script provided by Gijs:
1. the initial dialog causes an error (on German Entourage 08). Removed it and it works.
2. the scripts looks for "iPod mail" in text. Needs to be updated to "iPhone mail" to work.
3. "iPhone mail" header was introduced by a newer iPhone firmware. Older mails do not have it
4. script searches for "Received: from" which is wrong for older mails (older firmware). Here you will find two blanks between ":" and "from"!

Otherwise I was able to "recover" all mails ;-)

Hope this helps!

Thomas

Ashok Aiyar said:

Here is my version of this script. This script sequentially does the following:

a) Read each item that is received in my Sent Mail folder,
b) Remove the erroneous headers,
c) Copy the rest of the message to a new file placed in the Sent Mail folder, and
d) Delete the original message

Usage instructions are at the end of the script.

tell application "Microsoft Entourage"
	set myExchangeAccount to item 1 of every Exchange account
	set selectedMessages to (current messages)
	repeat with suspectMessage in selectedMessages
		set theSource to source of suspectMessage's contents
		set lineCount to 1
		set aParagraph to "not empty"
		repeat until aParagraph is ""
			set aParagraph to paragraph lineCount of theSource
			set lineCount to lineCount + 1
		end repeat
		
		set newSource to ""
		repeat with i from lineCount to count of paragraphs of theSource
			set newSource to newSource & paragraph i of theSource & return
		end repeat
		
		set theNewMessage to make new outgoing message with properties {source:newSource}
		move theNewMessage to sent items folder of myExchangeAccount
		set read status of theNewMessage to read
		delete suspectMessage
	end repeat
end tell

How to install -

a) Copy this file (FixActiveSyncSentItems.scpt) to
~/Documents/Microsoft User Data/Entourage Script Menu Items

b) Create a rule in Entourage - called FixBlackBerrySentItems with the following criteria
Folder is "Sent Items"
Message body contains "X-rim-org-msg-ref-id:"
then
Run AppleScript "FixBlackBerrySentItems"

Note: This script searches for a BlackBerry specific header that is stuck in what appears to be the message body. If you use an iPhone, just replace this header with an equivalent iPhone header.

Hope this help!

Ashok

Ujwal Setlur said:

So, the scripts posted here work, but unfortunately they run very slowly on large messages when message sizes get larger than 50k or so. I guess it is a performance issue reading lines in Applescript. Anyway, I reworked the script where the heavy lifting is done by sed since all we need to do is strip off the offensive headers until the real headers start. The modified script runs orders of magnitude faster. Here it is:

tell application "Microsoft Entourage"
set theProcessedItemsCounter to 0
set theFaildItemsCounter to 0
set theExchangeAccount to item 1 of every Exchange account
set theSentbox to sent items folder of theExchangeAccount
set theSentMessages to current messages
set tmpfolderpath to (path to temporary items folder from user domain) as Unicode text

repeat with theMessage in theSentMessages
set theSource to source of theMessage

set theheader to headers of theMessage
if theheader contains "Subject" then
--return -- do not continue if this is a properly formatted message
else
try
set sourceMacPath to (tmpfolderpath & "entourage-message-" & theProcessedItemsCounter)
set sourcePosixPath to POSIX path of sourceMacPath
set fd to open for access sourceMacPath with write permission
set eof of fd to 0
write theSource to fd
close access fd

-- translate mac to unix line endings
set targetPosixPath to (sourcePosixPath & ".unix")
do shell script "tr '\\r' '\\n' " & targetPosixPath

-- get rid of the malformed headers
set fixedMessagePath to (sourcePosixPath & ".fixed")
do shell script "sed -e '1,/^$/d' " & targetPosixPath & " > " & fixedMessagePath

-- read in the corrected message
tell application "Finder"
set fd to POSIX file fixedMessagePath as alias
set newSource to (read fd)
end tell
set theEntourageMessage to make new outgoing message with properties {source:newSource}
move theEntourageMessage to theSentbox
set properties of theEntourageMessage to {color:{0, 0, 0}, priority:normal, read status:read, delivery status:sent, replied to:false, redirected:false, forwarded:false, connection action:keep on server, flagged:false}

-- clean up
do shell script "rm " & sourcePosixPath
do shell script "rm " & targetPosixPath
do shell script "rm " & fixedMessagePath
delete theMessage

set theProcessedItemsCounter to theProcessedItemsCounter + 1

on error
set theFaildItemsCounter to theFaildItemsCounter + 1
end try

end if
end repeat

-- give feedback
set theText to "Microsoft Entourage:" & return & return & theProcessedItemsCounter & " iPhone messages have been made readable" as text
if the theFaildItemsCounter > 0 then
set theText to theText & return & theFaildItemsCounter & " messages failed to process" as text
end if
display dialog theText buttons {"OK"} default button "OK"


end tell


Ujwal Setlur said:

I have uploaded my version of the script to

http://files.macscripter.net/ScriptBuilders/Entourage/fix%20iphone%20sent%20mail.scpt.zip

Basically, it "borrows" from the other scripts posted here, and dramatically improves performance by using "tr" and "sed" to do the actual message fixing.

Thanks to everyone for the discussion here. This thing was driving me crazy.

Mshah said:

The last posted script works great! The previous scripts would not work on all my messages, it looks as though they were looking for a specific start of message, and my iphone doesn't always have that start.

Jason Boehle said:

This is ridiculous. There is no reason a fix for Entourage hasn't been issued to address this. Just another drop in the bucket of "why I hate Entourage" issues.

Mark Norris said:

I am new to applescript...how do I get Ujwal Setlur's script above installed and working within Entourage? I was able to copy the script into a new script and save it to the ~/Documents/Microsoft User Data/Entourage Script Menu Items folder. Then I added a rule like in one of the above examples, but I get zero results when I run the rule.

Help, step by step instructions would be great. I have iPhone, not Blackberry.

Thanks!

Jeff Tarver said:

I set up a rule with the script from Ujwal Setlur by:

If:
Specific Header-MAIL FROM:-Contains-

Then:
Run AppleScript-Script-Fix iPhone Sent Mail.scpt (or whatever you named the script)

That did the trick for me - it is now automatic. I hope this helps the thread. Jeff

Roger said:

The fix iphone sent mail script from Ujwal Setlur works perfect. Thanks a lot!!!

mcstever said:

Thanks guys
Ujwal's script is great and the notion of using the rule makes it better. Great collaborative effort!!

TXJohn said:

Thanks Guys!!

Terry said:

Thank you! Ujwal's script works perfectly for me.

I had a little trouble understanding and implementing Jeff's rule because I understood it to mean leaving the contains field blank, which Entourage won't let me do. But I saw a similar rule on another blog (RickyCampbell.com) that was a bit clearer:

Specific Header > Subject > Does not exist

Does the same thing - I just stupidly hadn't realised there was a 'does not exist' from the dropdown!

Thanks for all your hard work :)

Post a comment

(If you haven't left a comment here before, you may need to be approved by the site owner before your comment will appear. Until then, it won't appear on the entry. Thanks for waiting.)

About

This page contains a single entry from the blog posted on August 31, 2008 8:30 PM.

The previous post in this blog was Prioritize your E-mails with a star rating.

The next post in this blog is What are your Entourage & Exchange pain points?.

Many more can be found on the main index page or by looking through the archives.

Creative Commons License
This weblog is licensed under a Creative Commons License.