Review Board 1.7.22

Race condition in LinkRegistry.cpp

Review Request #6396 - Created Aug. 6, 2012 and updated

Kenneth Giusti
aconway, gordon, tross
Occasionally, the cluster tests will fail in the test_federation_multilink_failover test with the following error:

cluster_tests.ShortTests.test_federation_multilink_failover ................................................................................................................... fail
Error during test:  Traceback (most recent call last):
    File "/home/kgiusti/Desktop/work/qpid/trunk/build/qpid/src/tests/python/commands/qpid-python-test", line 340, in run
    File "/home/kgiusti/Desktop/work/qpid/trunk/qpid/cpp/src/tests/", line 992, in test_federation_multilink_failover
      assert self._verify_federation(src_cluster[1], "FedX/two", dst_cluster[1], "destQ2")

The problem is due to a race condition in the LinkRegistry code.  When a new connection event occurs for a federation Link, the LinkRegistry attempts to find a Link instance that is attempting to connect to the remote in order to assign the connection.  The problem is due to the fact that the search for the target link is done under a lock, but the assignment is done outside of the lock (to prevent lock inversion).

The proposed fix has LinkRegistry hold all disconnected Links in a separate container, and perform the search of that container (and the removal on match) while holding a lock.
Federation and cluster unit tests.
Ran test_federation_multilink_failover repeatedly with no crash.
Posted (Aug. 7, 2012, 4:51 p.m.)
Where is "the assignment is done outside of the lock" in the old code? The assignment in notifyConnection seems to have the same locking in both old and new code.
  1. You're correct, the call to link::established() still happens outside the lock (it has to, to avoid lock inversion).
    What the patch does is prevent simultainous calls into LinkRegistry::notifyConnection (from parallel threads) from being able to mistakenly select the _same_ pending Link.  It does this by holding all "pending" Links in a container, and removing them - while locked - as they are selected.  Notice the call to pendingLinks.erase() while the lock is held.
    This prevents two threads from selecting the same pending Link.
  2. So it is possible for multiple connections to match the same link? In which case, how do you know you've selected the correct one?
  3. Yes - it is possible to have multiple links running between the same two brokers.  You can create such links via management (QMF) by assigning unique names (since they will have the same remote address).
    The link really doesn't care which connection it gets as long as the connection is to the right destination.  So if two links to the same destination request connections at the same moment, it really doesn't matter which link gets assigned which connection.
    As an aside: a -much- better solution would be to have the Link that requested the connection get notified of the connection directly, rather having the notification go through the LinkRegistry, which then has to find a Link that is pending a connection to the remote.  The LinkRegistry really shouldn't be in the middle of the connection assignment.  I looked at doing that awhile back, but the implementation started getting complex....  
  4. OK, gotcha. Agreed with your aside. Ship it:)
Ship it!
Posted (Aug. 7, 2012, 9:33 p.m.)
Ship It!
Ship it!
Posted (Aug. 8, 2012, 9:34 a.m.)
Ship It!