Debugging the Impossible Race Condition

I have a race condition that only happens in production, never in dev. When I add logging to debug it, the bug disappears. Help me think through this systematically.

This is a classic heisenbug! Let's think about what makes production different from dev: 1. Load: Production has real concurrent traffic 2. Timing: Production servers are likely under CPU/memory pressure 3. Network: Real network latency vs localhost 4. Logging effect: Adding console.log introduces timing delays The fact that logging makes it disappear is the CLUE. Your bug is likely timing-sensitive.

Exactly! It's a payment processing system where sometimes double charges occur. But only 0.1% of the time, and never when I'm watching.

Ah! Payment processing plus timing-dependent. You're probably checking for existing payments BEFORE the transaction lock is acquired. Two requests come in simultaneously and both see no existing payment, then both process. That's the race.

That's exactly it! But how do I fix this without killing performance?

You need a database-level constraint or lock. Use a unique constraint on invoice_id and user_id, then use INSERT ... ON CONFLICT to handle duplicates gracefully. Now race conditions are impossible at the database level.

Brilliant! But what about the logging making it disappear? Can you explain that part?

Perfect question! When you add console.log, three things happen: 1. Time delay: Logging takes 1-10ms per call 2. Event loop: Node.js yields to other events during I/O 3. JavaScript is single-threaded: The yield gives the first request time to finish and commit Without logging, both requests race through. With logging, request A gets enough time to complete before request B even starts. The fix: Use database transactions with proper isolation levels, not application-level checks.