exchange 2010 DAG problem -- Unable to contact the Cluster service on other 1 members (servername) o

  • Thread starter wendy.wu
  • Start date Views 3,088
Status
Not open for further replies.
W

wendy.wu

Hi, I installed Exchange 2010 release version and created DAG but encountered this problem when adding the second member server into exchange 2010 DAG, which puzzled me for a few days. I searched in MS forum and internet but got no answers.

Here are the details of my environment:

1. ESXi4 VM: Windows 2008 x64 SP2 + AD box

2. ESXi4 VM: Windows 2008 x64 SP2 + Exchange 2010 release version + hub transport + client access role

3. ESXi4 VM: Windows 2008 x64 SP2 + Exchange 2010 release version + mailbox role 1

4. ESXi4 VM: Windows 2008 x64 SP2 + Exchange 2010 release version + mailbox role 2

I created DAG and I added one mailbox role into DAG, but when I added the second mailbox role. The issue occured. I always got the error message 'Unable to contact the Cluster service on other 1 members (servername) of the database availability group.' via local powershell. And got the error message 'The Microsoft Exchange Replication service does not appear to be running on " servername" . Make sure that the server is operating, and that the services can be queried remotely.' via remote powershell.

And here is the log via local powershell:

add-databaseavailabiltygroupserver started on machine WU-WE02-E14-2.

[2009-11-18T11:18:30] add-dagserver started

[2009-11-18T11:18:31] commandline: $scriptCmd = {& $wrappedCmd @PSBoundParameters }

[2009-11-18T11:18:31] Option 'Identity' = 'wu-we02-e14-dag'.

[2009-11-18T11:18:31] Option 'MailboxServer' = 'wu-we02-e14-2'.

[2009-11-18T11:18:31] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.

[2009-11-18T11:18:31] Option 'WhatIf' = ''.

[2009-11-18T11:18:31] Process: w3wp w3wp.exe:2076.

[2009-11-18T11:18:31] User context = 'NT AUTHORITY\SYSTEM'.

[2009-11-18T11:18:31] Member of group 'Everyone'.

[2009-11-18T11:18:31] Member of group 'BUILTIN\Users'.

[2009-11-18T11:18:31] Member of group 'NT AUTHORITY\SERVICE'.

[2009-11-18T11:18:31] Member of group 'NT AUTHORITY\Authenticated Users'.

[2009-11-18T11:18:31] Member of group 'NT AUTHORITY\This Organization'.

[2009-11-18T11:18:31] Member of group 'BUILTIN\IIS_IUSRS'.

[2009-11-18T11:18:31] Member of group 'LOCAL'.

[2009-11-18T11:18:31] Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.

[2009-11-18T11:18:31] Member of group 'BUILTIN\Administrators'.

[2009-11-18T11:18:31] Updated Progress 'Validating the parameters.' 2%.

[2009-11-18T11:18:31] Working

[2009-11-18T11:18:31] Mailbox server: value passed in = wu-we02-e14-2, mailboxServer.Name = wu-we02-e14-2, mailboxServer.Fqdn = wu-we02-e14-2.exchange2010.com

[2009-11-18T11:18:31] LogClussvcState: clussvc is Running on wu-we02-e14-2.exchange2010.com. Exception (if any) = none

[2009-11-18T11:18:31] The IP addresses for the DAG are (blank means DHCP): 155.35.82.12

[2009-11-18T11:18:31] Looking up IP addresses for wu-we02-e14-dag.

[2009-11-18T11:18:31] wu-we02-e14-dag = [ 155.35.82.12 ].

[2009-11-18T11:18:31] Looking up IP addresses for wu-we02-e14-2.

[2009-11-18T11:18:31] wu-we02-e14-2 = [ 155.35.82.14, ::1 ].

[2009-11-18T11:18:31] Looking up IP addresses for wu-we02-e14-2.exchange2010.com.

[2009-11-18T11:18:31] wu-we02-e14-2.exchange2010.com = [ 155.35.82.14, ::1 ].

[2009-11-18T11:18:31] DAG wu-we02-e14-dag has 1 servers:

[2009-11-18T11:18:31] DAG wu-we02-e14-dag contains server wu-we02-e14-1.

[2009-11-18T11:18:31] Updated Progress 'Checking if Mailbox server 'wu-we02-e14-2' is in a database availability group.' 4%.

[2009-11-18T11:18:31] Working

[2009-11-18T11:18:31] Mailbox server 'wu-we02-e14-2' is already a member of cluster 'wu-we02-e14-dag'.

[2009-11-18T11:18:31] DumpClusterTopology: Opening remote cluster wu-we02-e14-dag.

[2009-11-18T11:18:31] Dumping the cluster by connecting to: wu-we02-e14-dag.

[2009-11-18T11:18:31] The cluster's name is: wu-we02-e14-dag.

[2009-11-18T11:18:31] Groups

[2009-11-18T11:18:31] group: Available Storage [not a CMS]

[2009-11-18T11:18:31] OwnerNode: wu-we02-e14-2.exchange2010.com

[2009-11-18T11:18:31] State: Offline

[2009-11-18T11:18:31] group: Cluster Group [Cluster Main Group]

[2009-11-18T11:18:31] OwnerNode: wu-we02-e14-2.exchange2010.com

[2009-11-18T11:18:31] State: Online

[2009-11-18T11:18:31] Resource: Cluster Name [Online, type = Network Name, PossibleOwners = WU-WE02-E14-2 ]

[2009-11-18T11:18:31] NetName = [WU-WE02-E14-DAG]

[2009-11-18T11:18:31] RequireKerberos = 1. Computer account created on DC ''.

[2009-11-18T11:18:31] Resource: Cluster IP Address [Online, type = IP Address, PossibleOwners = WU-WE02-E14-2 ]

[2009-11-18T11:18:31] Address = [155.35.82.12]

[2009-11-18T11:18:31] EnableDhcp = [0]

[2009-11-18T11:18:31] Network = [Cluster Network 1]

[2009-11-18T11:18:31] Nodes

[2009-11-18T11:18:31] node: wu-we02-e14-2.exchange2010.com [ state = Up ]

[2009-11-18T11:18:31] Subnets

[2009-11-18T11:18:31] Name(Cluster Network 1), Mask(155.35.82.0/24), Role(ClusterNetworkRoleInternalAndClient)

[2009-11-18T11:18:31] NIC 155.35.82.14 on Node wu-we02-e14-2 in State=Up

[2009-11-18T11:18:31] Opening the cluster on nodes [wu-we02-e14-1].

[2009-11-18T11:18:31] GetRemoteCluster() failed possibly with error 64854444, ex = Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API 'OpenByNames('wu-we02-e14-1.exchange2010.com') failed for each server. Specific exceptions: 'An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" OpenCluster(wu-we02-e14-1.exchange2010.com) failed with 0x5. Error: Access is denied" ' failed.'.' failed. ---> Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" OpenCluster(wu-we02-e14-1.exchange2010.com) failed with 0x5. Error: Access is denied" ' failed. ---> System.ComponentModel.Win32Exception: Access is denied
--- End of inner exception stack trace -
at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByName(AmServerName serverName)
at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByNames(IEnumerable`1 serverNamesToTry)
--- End of inner exception stack trace -
at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByNames(IEnumerable`1 serverNamesToTry)
at Microsoft.Exchange.Management.SystemConfigurationTasks.AddDatabaseAvailabilityGroupServer.FindClusterForDag()

[2009-11-18T11:18:31] The operation wasn't successful because an error was encountered. You may find more details in log file " C:\ExchangeSetupLogs\DagTasks\dagtask_2009-11-18_11-18-30.800_add-databaseavailabiltygroupserver.log" .

[2009-11-18T11:18:31] WriteError! Exception = Microsoft.Exchange.Management.Tasks.DagTaskServerCanNotContactClusterException: Unable to contact the Cluster service on other 1 members (wu-we02-e14-1) of the database availability group.

[2009-11-18T11:18:31] Updated Progress 'Done!' 100%.

[2009-11-18T11:18:31] COMPLETED

add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

And here's the log via remote powershell:

add-databaseavailabiltygroupserver started on machine WU-WE02-E14-1.

[2009-11-18T11:20:12] add-dagserver started

[2009-11-18T11:20:12] commandline: $scriptCmd = {& $wrappedCmd @PSBoundParameters }

[2009-11-18T11:20:12] Option 'Identity' = 'wu-we02-e14-dag'.

[2009-11-18T11:20:12] Option 'MailboxServer' = 'wu-we02-e14-2'.

[2009-11-18T11:20:12] Option 'DatabaseAvailabilityGroupIpAddresses' = ''.

[2009-11-18T11:20:12] Option 'WhatIf' = ''.

[2009-11-18T11:20:12] Process: w3wp w3wp.exe:4492.

[2009-11-18T11:20:12] User context = 'NT AUTHORITY\SYSTEM'.

[2009-11-18T11:20:12] Member of group 'Everyone'.

[2009-11-18T11:20:12] Member of group 'BUILTIN\Users'.

[2009-11-18T11:20:12] Member of group 'NT AUTHORITY\SERVICE'.

[2009-11-18T11:20:12] Member of group 'NT AUTHORITY\Authenticated Users'.

[2009-11-18T11:20:12] Member of group 'NT AUTHORITY\This Organization'.

[2009-11-18T11:20:12] Member of group 'BUILTIN\IIS_IUSRS'.

[2009-11-18T11:20:12] Member of group 'LOCAL'.

[2009-11-18T11:20:12] Member of group 'IIS APPPOOL\MSExchangePowerShellAppPool'.

[2009-11-18T11:20:12] Member of group 'BUILTIN\Administrators'.

[2009-11-18T11:20:12] Updated Progress 'Validating the parameters.' 2%.

[2009-11-18T11:20:12] Working

[2009-11-18T11:20:12] Mailbox server: value passed in = wu-we02-e14-2, mailboxServer.Name = wu-we02-e14-2, mailboxServer.Fqdn = wu-we02-e14-2.exchange2010.com

[2009-11-18T11:20:12] LogClussvcState: Received an InvalidOperationException for server wu-we02-e14-2.

[2009-11-18T11:20:12] LogClussvcState: clussvc is unknown on wu-we02-e14-2.exchange2010.com. Exception (if any) = System.ComponentModel.Win32Exception: Access is denied

[2009-11-18T11:20:12] The IP addresses for the DAG are (blank means DHCP): 155.35.82.12

[2009-11-18T11:20:12] Looking up IP addresses for wu-we02-e14-dag.

[2009-11-18T11:20:12] wu-we02-e14-dag = [ 155.35.82.12 ].

[2009-11-18T11:20:12] Looking up IP addresses for wu-we02-e14-2.

[2009-11-18T11:20:12] wu-we02-e14-2 = [ 155.35.82.14 ].

[2009-11-18T11:20:12] Looking up IP addresses for wu-we02-e14-2.exchange2010.com.

[2009-11-18T11:20:12] wu-we02-e14-2.exchange2010.com = [ 155.35.82.14 ].

[2009-11-18T11:20:12] DAG wu-we02-e14-dag has 1 servers:

[2009-11-18T11:20:12] DAG wu-we02-e14-dag contains server wu-we02-e14-1.

[2009-11-18T11:20:12] Updated Progress 'Checking if Mailbox server 'wu-we02-e14-2' is in a database availability group.' 4%.

[2009-11-18T11:20:12] Working

[2009-11-18T11:20:12] GetRemoteCluster() for the mailbox server failed with exception = An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" OpenCluster(wu-we02-e14-2.exchange2010.com) failed with 0x5. Error: Access is denied" ' failed.. This is OK.

[2009-11-18T11:20:12] Ignoring previous error, as it is acceptable if the cluster does not exist yet.

[2009-11-18T11:20:12] DumpClusterTopology: Opening remote cluster wu-we02-e14-dag.

[2009-11-18T11:20:12] DumpClusterTopology: Failed opening with Microsoft.Exchange.Cluster.Replay.AmClusterApiException: An Active Manager operation failed. Error: An error occurred while attempting a cluster operation. Error: Cluster API '" OpenCluster(WU-WE02-E14-DAG.exchange2010.com) failed with 0x5. Error: Access is denied" ' failed. ---> System.ComponentModel.Win32Exception: Access is denied
--- End of inner exception stack trace -
at Microsoft.Exchange.Cluster.ClusApi.AmCluster.OpenByName(AmServerName serverName)
at Microsoft.Exchange.Management.SystemConfigurationTasks.DumpClusterTopology..ctor(String nameCluster, ITaskOutputHelper output)

[2009-11-18T11:20:12] Dumping the cluster by connecting to: wu-we02-e14-dag.

[2009-11-18T11:20:12] DumpClusterTopology: Not dumping, since the cluster could not be contacted.

[2009-11-18T11:20:12] DumpClusterTopology: Opening local cluster.

[2009-11-18T11:20:12] Dumping the cluster by connecting to: .

[2009-11-18T11:20:12] The cluster's name is: WU-WE02-E14-DAG.

[2009-11-18T11:20:12] Groups

[2009-11-18T11:20:12] group: Available Storage [not a CMS]

[2009-11-18T11:20:12] OwnerNode: wu-we02-e14-1.exchange2010.com

[2009-11-18T11:20:12] State: Offline

[2009-11-18T11:20:12] group: Cluster Group [Cluster Main Group]

[2009-11-18T11:20:12] OwnerNode: wu-we02-e14-1.exchange2010.com

[2009-11-18T11:20:12] State: Failed

[2009-11-18T11:20:12] Resource: Cluster IP Address [Failed, type = IP Address, PossibleOwners = WU-WE02-E14-1 ]

[2009-11-18T11:20:12] Address = [155.35.82.193]

[2009-11-18T11:20:12] EnableDhcp = [1]

[2009-11-18T11:20:12] Network = [Cluster Network 1]

[2009-11-18T11:20:12] Resource: Cluster Name [Failed, type = Network Name, PossibleOwners = WU-WE02-E14-1 ]

[2009-11-18T11:20:12] NetName = [WU-WE02-E14-DAG]

[2009-11-18T11:20:12] RequireKerberos = 1. Computer account created on DC ''.

[2009-11-18T11:20:12] Nodes

[2009-11-18T11:20:12] node: wu-we02-e14-1.exchange2010.com [ state = Up ]

[2009-11-18T11:20:12] Subnets

[2009-11-18T11:20:12] Name(Cluster Network 1), Mask(155.35.82.0/24), Role(ClusterNetworkRoleInternalAndClient)

[2009-11-18T11:20:12] NIC 155.35.82.13 on Node wu-we02-e14-1 in State=Up

[2009-11-18T11:20:12] Opening the cluster on nodes [wu-we02-e14-1].

[2009-11-18T11:20:12] Other mailbox servers in the DAG are already members of cluster 'WU-WE02-E14-DAG'

[2009-11-18T11:20:12] The server wu-we02-e14-2 does not belong to a cluster, and the other servers belong to WU-WE02-E14-DAG.

[2009-11-18T11:20:12] Successfully resolved servers based on the AD configuration.

[2009-11-18T11:20:12] The following servers are in the StartedServers list:

[2009-11-18T11:20:12] The following servers are in the StoppedServers list:

[2009-11-18T11:20:12] Verifiying that the members of database availability group 'wu-we02-e14-dag' are also members of the cluster.

[2009-11-18T11:20:12] Verifying that the members of cluster 'wu-we02-e14-dag' are also members of the database availability group.

[2009-11-18T11:20:12] GetNodeClusterState( wu-we02-e14-2 ) failed with 5. Continuing anyway (assuming clustering is still installed though).

[2009-11-18T11:20:12] Updated Progress 'Done!' 100%.

[2009-11-18T11:20:12] COMPLETED

add-databaseavailabiltygroupserver explicitly called CloseTempLogFile().

Btw, I always got the following warning message:

WARNING: Insufficient permissions to access file shares on witness server 'WU-WE02-E14-CAS.EXCHANGE2010.COM'. Until this problem is corrected, the database availability group may be more vulnerable to failures. You can use the

Set-DatabaseAvailabilityGroup cmdlet to try the operation again. Error: Access is denied

Can someone help me out of this annoying issue?

Many many thanks!
 
S

StMarysSchool

Hi - I have the exact same issue! Did you find the answer?

Thanks!
 
P

PeterPhi

Hi all,
I have the same problem, pls help me
[PS] C:\Windows\system32>Add-DatabaseAvailabilityGroupServer -identity DAG1 -MailboxServer EX2
Unable to contact the Cluster service on other 1 members (ex1) of the database availability group.
I using Hyper -V Server 2008 R2
Thanks
 
D

Dave Kawula - TriCon

Hey Guys,

I just installed 2 x DAG this week with similar configurations to yourself -- One thing that I always do is run the Powershell as Administrator.

In reviewing you log files you are getting Access Denied error messages -- I find this only really happens if you don't run PowerShell As the Administrator or Domain Admin.

Are you Adding the extra nodes to the DAG cluster remotely or from the destination node -- You may want to try from the destination node. I did up a nice video on www.insidetheregistry.com -- But the problem with this video is it that it was pre-RTM Code. So not all the cmdlets will be correct.

What was the cmdlet that you used to build the DAG -- Did you assign a static IP or a DHCP / Have you gone into the Failover Cluster Manager to ensure that the Cluster Services are online?

Do you have separate networks for MAPI / Replication and Heartbeat?

I just setup one of these up this week for the first time using the GUI for adding the last 2 x nodes to a 3 node multi-site DAG configuraiton and it actually worked quite well.

So in closing -- Run As Admin is your friend / Not all of the configruations can be done remotely -- Logon to the destination DAG member and try there / If all else fails -- Just email me. We can setup a remote session and I can quickly walk you through your issue.

THanks

Dave Kawula

Principal Consultant

TriCon Technical Services Inc.

www.triconts.com

dkawula@triconts.com
 
W

wendy.wu

Finally I found the root reason. Originally, I created an ESXI VM and installed windows 2008 for the first OS. And cloned it without running newsid. And I resolved this problem by re-setup environment and running newsid after cloning monthes ago. And Exchange DAG is always running well now.
 
B

Brian Day MCITP [MVP]

Finally I found the root reason. Originally, I created an ESXI VM and installed windows 2008 for the first OS. And cloned it without running newsid. And I resolved this problem by re-setup environment and running newsid after cloning monthes ago. And Exchange DAG is always running well now.

In addition to what Martin mentions, I'm curious if this really was the root cause or not given Mark's research on this issue and nobody could ever find where the machine's SID was used in a way that could cause issues if there were dupes.Brian Day, Overall Exchange & AD Geek
MCSA 2000/2003, CCNA
MCITP: Enterprise Messaging Administrator 2010
 
F

Ferry Kho

I encountered the same error and came to this page on my troubleshooting, but the proposed solution here is not applicable for me. I'm not using VM, all servers involved in my testing was built from scratch, physically.

So, I want to post my solution here, hopefully next time anyone encounter the same error should not get the hard way to find the cure. ;-)

The trick to fix this is to add Exchange Trusted Subsystem group into every Exchange and Witness server local Administrators group. Good luck!
-ferry-
 
Status
Not open for further replies.
Top