jokerconf Joker 2016 (14.10.2016 — 15.10.2016)

Deadlocks in enterprise systems: sources, impact, tackling


Deadlocks in Enterprise solutions 24x7 — theory (a little), examples based on own experience.

Goal: to share an experience of deadlock tackling in enterprise 24x7 solutions.

Problem: multi-component enterprise system owns the bigger risk of deadlock than standalone application. Recommendations that work for simple unitary applications aren’t always efficient for enterprise ones. So complete deadlock protection is not possible in such complex environments.

Solution: we will discuss how to detect deadlocks, minimize impact and prevent outages. Based on examples of own experience in mass service telecom and finance solutions.

  • Synchronization in Java: where deadlock comes from. Types of deadlocks: recursive, mutual, circuitous.

  • Basic rules to avoid deadlock: lock hierarchy, foreign code prevention.

  • Why deadlocks are unavoidable in corporate applications: variety of resources, a lot of foreign code, dynamic scenarios violating lock hierarchy.

  • Avoiding deadlocks by reducing usage of 'synchronized': 'volatile', nonblocking algorithms, 'java.util.concurrent' etc.

  • Detecting deadlocked threads and objects in corporate application: alive requests, update counters, heartbeats.

  • Testing approaches to detect deadlocks: proper automation with realistic/stochastic profile.

  • Conclusion. Examples from telecom and finance mass service platforms with 24x7 SLA and thousands of requests per second. Developed by me and my teams, obviously.