What do you mean you don't know what's error 9548? What are you doing on my blog then? :)

OK, let go back to the beginning..... it all began ....never mind when it began. Here's the story.

Prior to Exchange Server 2000, Exchange servers have their own user directories, separate from the Windows users directories. In those days, if you want a Windows user account to have a mailbox, you will have to create a corresponding account in Exchange for it. Life was good, everybody was happy. NOT!, but I digress...

Post Exchange 5.5, with the release of Windows 2000, and Active Directory and LDAP and all, it became unnecessary (inelegant) to be maintaining 2 account databases in the same organization/domain just for the purpose of getting emails. Effeciency and collaboration and TCO became huge buzzwords. So, we started using AD as the account repository for exchange objects. Makes sense. The accounts are already there, single-sign-on is a good thing, so no need to duplicate accounts unnecessarily.

Then came the realization that, sometimes attritions come faster than a broken Vegas marriage. Employees leave employers for various reasons, and, when they do, employers usually need a way to prevent that employee from accessing corporate resources - including emails. In the very old days, the thing to do is just delete that user's corporate account. QED. But in this age of compliance and retention and litigation and discovery, deleting accounts may not be practical. So, IT administrators the world over discovered the next best thing - disable the account until such time it becomes feasible and practical to actually send it to the junk yard. Thus was born the "deprovisioning" practice.

Unfortunately, thus was born the infamous error 9548. Among other quirks of the Exchange server, it happens that Exchange expects to find any account that is associated with a mailbox to be ALWAYS enabled - or deleted. Either enabled/deleted or Exchange will scream and, like a true brat, refuse to play ball with such account UNLESS it finds some whacky attibute called msExchMasterAccountSID.

You see, every account (Security Principal) in Windows has a SID, and, under normal circumstances, access to Windows resources are granted/denied using SID (objectSID, to be exact). However, the Exchange uses this attribute (objectSID) ONLY IF the account in question is ENABLED. Otherwise, it uses the (ahem.....wacky) msExchMasterAccountSID. As far as anyone can tell, this behavior is peculiar to Exchange. IF an account is disabled, and Exchange does NOT find the msExchangeAccountSID attribute for that account populated, then Exchange does the following:

• Throws the event ID 9548 error. Lord have mercy on your soul IF the account in question happens to belong to a lot of Distribution Groups.
• In addition to throwing the said error EVERYTIME an email hits the DL to which the account belongs, the Exchange Store itself goes into panic mode. It hangs and can throws tantrums - like refusing to deliver the email to anyone else in the DL, or delivering it to some and not others, just because there is one disabled user in the list of recipients.
  
• In addition to the above, the Exchange server's performance degrades noticeably over time IF you have a lot of 9548 errors.
  
• OK, maybe the you remove the disabled account from EVERY DL it used to belong to, then 9548 should go away, right? Wrong! You see, there are various other things that Exchange could be doing that will touch that particular account - like run the cleanup Agent periodically, calculate RUS, like you would like to move that mailbox to another server, like that user has access to Public Folders and the Public Folder is requested by someone. Every time Exchange recalculates the permission on that PF, it sees the disabled account and goes into its warp mode.

OK, so, in the old days, we used to prevent all these unpleasantness by simply hacking around the dependency on msExchMasterAccountSID. How? We simply associate the disabled account with a an account called SELF. Yup! You heard it, it's a "trick account", so to speak. You see, SELF equals SELF. Self has the same SID as well, the account itSELF. By granting the account called SELF the "Associated External Account" rights on a disabled account called "deji", we create an ability for Exchange to find a usable objectSID, although the objectSID of SELF is the same as the objectSID of "deji" regardless of whether or not "deji" was disabled.

So, Exchange finds the SELF SID, and then it's happy. No error log, no slow down, no hiccup. It was a hack-a-mole, but it worked and it's documented, and supported. Unfortunately, unless you have a well-defined account provisioning and de-provisioning mechanism in place, you wouldn't usually do this until your Exchange has started running like a WWII-era Beetle. Even then, you would have spent a number of hours or days trouble-fooling the "issue" from various unrelated angles because, come to think of it, Error 9548 is so blah, that you will hardly link it to anything remotely causing your Exchange server to slow down. You read the event log, see error 9548 and the first thing that pops into your mind is "OK, I'll look into that later. As soon as I figure out why my Exchange server is crawling".

This brings us to today. Yesterday, March 21st, 2006 to be exact. Microsoft finally released a fix that makes all this go away - almost. See http://support.microsoft.com/default.aspx?scid=kb;EN-US;903158

I said almost because, as of this writing, the available fix only works on Exchange 2003 SP1. If you have Exchange 2003 SP2, then you have to suffer 9548 some more. ALSO, there is no mention of Exchange Server 2000!!!! What's up with that???? Haven't touched 2000 in a while, but I know quite a few folks who still live in the 2000 world (5.5 world is not a "world", it's prehistoric, so there :)), and AFAIK, 9548 afflicts Exchange 2000 Servers equally. What's the deal-o with that???