RichardLetts, many thanks for your suggestions.
I have checked the UDT polling intervals:
my logs were defaulted on Debug
I checked the udt.buisnesslayer.log as you have suggested - wow there is a lot in there ! ! !
this is what I am seeing:
2014-04-10 07:16:55,788 [ 3] ERROR - Error while processing job '[N:85|Layer2]' (0e235b77-1c1c-4828-806a-1448775d340f) [SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor]
System.InvalidOperationException: The requested operation cannot be completed because the connection has been broken.
at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction(TransactionRequest transactionRequest, String name, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlDelegatedTransaction.Initialize()
at System.Transactions.TransactionStatePSPEOperation.PSPEInitialize(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification)
at System.Transactions.TransactionStateActive.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction)
at System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification)
at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.EnlistTransaction(Transaction transaction)
at System.Data.SqlClient.SqlConnection.EnlistTransaction(Transaction transaction)
at SolarWinds.UDT.BusinessLayer.DAL.TransactionHelper.ExeInTransactionScope(Int32 deadlockPriority, IsolationLevel isolationLevel, Action`1 method)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.UpdateNodeData(UDTNode currNode, UDTPortCollection prevPorts)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.<>c__DisplayClass7.<UpdateNodeData>b__4(UDTNode cachedNode)
at SolarWinds.UDT.BusinessLayer.Cache.UDTNodeCache.UpdateNode[T](Int32 nodeID, Func`2 handler)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.UpdateNodeData(UDTNode currNode)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.DoProcessing(UDTJobResult jobResult)
at SolarWinds.UDT.BusinessLayer.Polling.UDTJobResultProcessorBase.Process(UDTJobResult jobResult)
2014-04-10 07:17:00,780 [ 3] ERROR - Error while processing job '[N:81|Layer2]' (e660ec88-b3af-46c6-a1b5-e83e6b31ebf1) [SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor]
System.InvalidOperationException: The requested operation cannot be completed because the connection has been broken.
at System.Data.SqlClient.SqlInternalConnectionTds.ExecuteTransaction(TransactionRequest transactionRequest, String name, IsolationLevel iso, SqlInternalTransaction internalTransaction, Boolean isDelegateControlRequest)
at System.Data.SqlClient.SqlDelegatedTransaction.Initialize()
at System.Transactions.TransactionStatePSPEOperation.PSPEInitialize(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification)
at System.Transactions.TransactionStateActive.EnlistPromotableSinglePhase(InternalTransaction tx, IPromotableSinglePhaseNotification promotableSinglePhaseNotification, Transaction atomicTransaction)
at System.Transactions.Transaction.EnlistPromotableSinglePhase(IPromotableSinglePhaseNotification promotableSinglePhaseNotification)
at System.Data.SqlClient.SqlInternalConnection.EnlistNonNull(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.Enlist(Transaction tx)
at System.Data.SqlClient.SqlInternalConnection.EnlistTransaction(Transaction transaction)
at System.Data.SqlClient.SqlConnection.EnlistTransaction(Transaction transaction)
at SolarWinds.UDT.BusinessLayer.DAL.TransactionHelper.ExeInTransactionScope(Int32 deadlockPriority, IsolationLevel isolationLevel, Action`1 method)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.UpdateNodeData(UDTNode currNode, UDTPortCollection prevPorts)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.<>c__DisplayClass7.<UpdateNodeData>b__4(UDTNode cachedNode)
at SolarWinds.UDT.BusinessLayer.Cache.UDTNodeCache.UpdateNode[T](Int32 nodeID, Func`2 handler)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.UpdateNodeData(UDTNode currNode)
at SolarWinds.UDT.BusinessLayer.Polling.UDTLayer2ResultProcessor.DoProcessing(UDTJobResult jobResult)
at SolarWinds.UDT.BusinessLayer.Polling.UDTJobResultProcessorBase.Process(UDTJobResult jobResult)
2014-04-10 07:35:52,359 [ 53] ERROR - Job finished with error [N:17|RemoteEventLog] [SolarWinds.UDT.BusinessLayer.Polling.UDTJobResultManager]
2014-04-10 08:05:51,776 [ 45] ERROR - Job finished with error [N:17|RemoteEventLog] [SolarWinds.UDT.BusinessLayer.Polling.UDTJobResultManager]
2014-04-10 08:35:53,584 [ 52] ERROR - Job finished with error [N:17|RemoteEventLog] [SolarWinds.UDT.BusinessLayer.Polling.UDTJobResultManager]
Not sure if all this is normal.
My L3 enviroment hasn't changed - one core switch that is my router (L3) and all other switches work at L2