Server 2012 Feature: Dynamic Access Control

At last month's CITPG meeting there was a presentation on the Dynamic Access Control feature of Server 2012 and its Microsoft first major push with a GUI as it were in the data governance space.  It's definitely worth a look if your business or client has strict regulatory compliance needs.

Dynamic Access Control: Scenario Overview: http://technet.microsoft.com/en-us/library/hh831717.aspx

If you have not had the privilege of working with Varonis software, this Server 2012 feature will be a direct competitor and without the big ticket licensing and maintenance cost of DatAdvantage.

The scenario overview above is dated 2/2012, but I can't help chuckling a little when I read about the recent VRNS IPO announcement: http://www.forbes.com/sites/eliseackerman/2013/10/23/israeli-software-maker-varonis-systems-files-for-ipo/

Enterprise Vault MAPI troubleshooting

An often overlooked component in Enterprise Vault (EV) Server health is MAPI connectivity.

Main EV Components that use MAPI:
Agent Client Broker
Archive Task
Exchange Provisioning Task
Journal Task
Public Folder Task
Retrieval Task

One of the main culprits to MAPI connectivity issues is the 100 session limit in the Outlook 2007 SP3 client.  When EV was originally designed there was no MAPI session limitation in Outlook 2003, Outlook 2007 first introduced a 32 session limit, but this was hotfixed and later rolled up into SP3.  Latest EV Deployment Scanner checks for the existence of this hotfix or SP3 when you run it.

Outlook 2010 removes this limit, but it is not on the compatibility list (yet).

The first article to troubleshoot possible MAPI issues is a good start: Troubleshooting Exchange connectivity issues with Enterprise Vault (EV)

Common Event IDs indicating MAPI related issues include: 3411, 3431, 3432, 3230, 2263, 2246

Decoding the error codes:

Error Code	Log Information
8004010F	mapi_e_not_found
80040111	mapi_e_login_failed
8004011C	mapi_e_unconfigured
8004011D	mapi_e_failoneprovider
80040700	mapi_e_ambiguous_recip
80004005	mapi_e_call_failed

Warning: Incorrect use of the Windows registry editor may prevent the operating system from functioning properly. Great care should be taken when making changes to a Windows registry. Registry modifications should only be carried-out by persons experienced in the use of the registry editor application. It is recommended that a complete backup of the registry and workstation be made prior to making any registry changes.

Sorry, couldn't help myself.

Here are a few registry keys that I have been implementing in our client environments to ensure MAPI stays healthy:

1) If you are currently archiving/journaling from an Exchange 2010 environment, you should change the following registry key:

Key	Location	Settings
DS Server	HKEY_CURRENT_USER \Software   \Microsoft      \Exchange       \Exchange Provider	Value name: DS Server Data type: REG_SZ (string) Value data: FQDN of the CAS server

Reference: http://www.symantec.com/docs/TECH139751

If you're using a load balancer in front of your CASarray, you do not want to use that FQDN due to MAPI timeout (F5, Barracuda) issues, this does introduce a single point of failure, but if your CAS server is down, you have bigger problems than EV archiving to fix.

Check out the More Information/Exchange 2010 Server section of the following TechNet article for Microsoft's take on the subject: http://support.microsoft.com/kb/319206 - Oh, NSPI...

2) The next three live in the Agents registry key, here's a reference of most of the keys: http://www.symantec.com/docs/HOWTO32040

32-bit: HKLM\Software\KVS\Enterprise Vault\Agents\

64-bit: HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\KVS\Enterprise Vault\Agents

Value Name: ProfileExpire

Value: Change from 3 to 1

Reference: http://www.symantec.com/docs/TECH49069

The default for this setting is 3 . This represents the number of days before systematically created Outlook Profiles are expired.  Changing this value to 1 (day) will expire the systematically created Outlook Profiles faster, improving overall Outlook performance.

Value Name: RestartOnMAPIMutexError

Data Type: DWORD

Value: 1

Reference: http://www.symantec.com/docs/HOWTO61101

Here's another article, which references RestartAllMAPITaskIntervalMins, I tested it, and it was restarting tasks at different times of the day and generating extra archiving reports as a result: http://www.symantec.com/docs/TECH68433

Value Name: RemoveEmbeddedAttachments

Data Type: DWORD

Value: 0

Note: this one isn't documented really well, I came across it from the Symantec Connect forums, and confirmed it's usefulness with Symantec Support

Reference: http://www.symantec.com/connect/forums/archive-task-failing

3) At the end of our maintenance window, I have a scheduled task that restarts the Task Controller service nightly on each EV server, this will clear/reset the MAPI resources in use or resources that could not be cleared successfully.

4) Reduce the total number of threads used by all the tasks on a given Enterprise Vault Server. Each thread of a Mailbox Archiving task can hold two MAPI sessions (one for archiving and the other for synchronization) and hence setting the task to 5 threads/connections, actually means 10 profiles/sessions in the Outlook cache.

5) The Vault Service Account (VSA) temp directory contains many ExchangePerflog*.dat files and is filling up the drive.

HKEY_LOCAL_MACHINE\SOFTWARE\KVS\Enterprise Vault\AdminService
Value Name: DelFileTypes

Data Type: String

Value: ExchangePerf*.dat;EV$*.*;*.tmp,*.dat
Reference: http://www.symantec.com/docs/TECH61099

Cluster nodes behaving badly? (BSOD)

Before applying a hotfix for"0x0000009E" Stop error when you add an extra storage disk to a failover cluster in Windows Server 2008 R2:  http://support.microsoft.com/kb/2520235

You might want to check out the 2008 R2 default HangRecoveryAction setting.  In our case we changed it to 1 until the hotfix could be applied.
http://blogs.technet.com/b/askcore/archive/2009/06/12/why-is-my-2008-failover-clustering-node-blue-screening-with-a-stop-0x0000009e.aspx

tl;dr version:
HangRecoveryAction

This property controls the action to take if the user-mode processes have stopped responding. For the HangRecoveryAction, we actually have 4 different settings with 3 being the default.

0 = Disables the heartbeat and monitoring mechanism.
1 = Logs an event in the system log of the Event Viewer.
2 = Terminates the Cluster Service.
3 = Causes a Stop error (Bugcheck) on the cluster node.  <<-- default for 2008

If you want to change the setting, you would issue the command:
cluster /cluster:clustername /prop HangRecoveryAction=x

With this setting instead of a BSOD, you get Event ID 4869 repeated every 60 seconds:

Event ID: 4869
Source:  Microsoft-Windows-FailoverClustering
Description:  User mode health monitoring has detected that the system is not being responsive. The Failover cluster virtual adapter has lost contact with the 'C:\Windows\Cluster\clussvc.exe' process with a process ID '%1', for '%2' seconds. Please use Performance Monitor to evaluate the health of the system and determine which process may be negatively impacting the system.
* where %2 is the value of ClusSvcHangTimeout
* where %1 is the Process ID you would see in Task Manager

NIC Binding Order in Windows 2008 Server R2

While working on an Enterprise Vault issue, one of the Symantec KB articles cited improper NIC order as one of the possible causes of the Event ID, among a truckload of other optimizations:

http://www.symantec.com/business/support/index?page=content&id=TECH62307

Most of the Symantec docs were written with Windows 2003 in mind, so the instructions listed were ambiguous at best.

Took some digging, but a Technet Microsoft DNS article squared me away:
http://technet.microsoft.com/en-us/library/dd391967(WS.10).aspx

Here's the tl;dr version:

1. Click Start, click Network, click Network and Sharing Center, and then click Manage Network Connections.
2. Press the ALT key, click Advanced, and then click Advanced Settings. If you are prompted for an administrator password or confirmation, type the password or provide confirmation.
3. Click the Adapters and Bindings tab, and then, under Connections, click the connection you want to modify.
4. Under Bindings for <connection name>, select the protocol that you want to move up or down in the list, click the up or down arrow button, and then click OK.