Why More Monitoring Locations Can Make Alerts Worse

More monitoring locations should mean better coverage. Check from New York, London, and Tokyo—see your service from every angle. But there's a hidden problem: without proper configuration, each location becomes another source of false alerts.

One flaky network hop, and suddenly your service is "down" according to Singapore while everyone else sees it fine.

The Multi-Location Problem

How Global Monitoring Works

Your Service ← Check from New York
            ← Check from London
            ← Check from Frankfurt
            ← Check from Singapore
            ← Check from Tokyo

Each location checks independently. Each has its own network path.

What Can Go Wrong

Your service: Running fine

Network paths:
├── New York → Service: ✓ Success
├── London → Service: ✓ Success
├── Frankfurt → Service: ✗ Timeout (ISP issue)
├── Singapore → Service: ✓ Success
└── Tokyo → Service: ✓ Success

Result with naive alerting: "Site down!" (1 failure)
Reality: Site is fine, one network path has issues

The False Alert Math

With simple "any failure = alert" logic:

1 location: 1 potential failure point
3 locations: 3 potential failure points
5 locations: 5 potential failure points

More locations = More chances for network issues
= More false alerts (with naive alerting)

Why Network Paths Fail

Monitoring checks traverse the internet:

Monitoring Server (Frankfurt)
        │
        └── Frankfurt ISP
                │
                └── Internet Exchange
                        │
                        └── Transit Provider A
                                │
                                └── Transit Provider B
                                        │
                                        └── Target ISP
                                                │
                                                └── Your Server

Any hop can fail:

• ISP maintenance
• BGP route changes
• Submarine cable issues
• Peering disputes
• DDoS attacks on transit providers

These affect the monitoring path, not your service.

Real-World Scenarios

Scenario 1: Regional ISP Issue

Time: 2:30 AM
Your service: Running normally
Asian ISP: Having routing issues

Checks:
├── US-East: ✓ 120ms
├── US-West: ✓ 140ms
├── EU-West: ✓ 160ms
├── Singapore: ✗ Timeout
└── Sydney: ✗ Timeout

Naive alerting: "Service down! 2 failures!"
Reality: Regional ISP issue, service is fine

Scenario 2: Transit Provider Problem

Time: 11:00 AM
Your service: Running normally
Transit provider: Dropping packets

Checks (every minute):
├── 11:00 - All pass
├── 11:01 - Frankfurt fails
├── 11:02 - All pass
├── 11:03 - Frankfurt fails
├── 11:04 - All pass

Naive alerting: Multiple "down" and "up" alerts
Reality: Intermittent transit issue, service is fine

Scenario 3: Actual Outage

Time: 3:00 PM
Your service: Database connection exhausted

Checks:
├── US-East: ✗ 503 Error
├── US-West: ✗ 503 Error
├── EU-West: ✗ 503 Error
├── Singapore: ✗ 503 Error
└── Sydney: ✗ 503 Error

All locations report failure: Actual outage

The Solution: Consensus-Based Alerting

Don't alert on single location failures. Require consensus.

Basic Consensus

Alert rule: Notify when 2+ locations report failure

Scenario A (network issue):
├── Frankfurt: ✗ Timeout
└── All others: ✓ Success
Result: No alert (1 failure, threshold is 2)

Scenario B (actual outage):
├── All locations: ✗ Error
Result: Alert (5 failures, threshold is 2)

Consecutive Failure Requirement

Add time-based confirmation:

Alert rule: Notify when failure persists for 2 consecutive checks

Scenario A (momentary blip):
├── Check 1: ✗ Timeout
├── Check 2: ✓ Success
Result: No alert (not consecutive)

Scenario B (actual outage):
├── Check 1: ✗ Error
├── Check 2: ✗ Error
Result: Alert (consecutive failures)

Combined Approach (Best Practice)

Alert when: 2+ locations report failure
AND: Failure persists for 2+ consecutive checks

This eliminates:
├── Single location network issues
├── Momentary blips
├── BGP route flapping
└── Most false positives

Configuring for Different Scenarios

Scenario: Global User Base

Monitoring locations: US, EU, Asia
Consensus: 2 of 3 must fail
Consecutive: 2 checks

Why: Covers all major regions, tolerates single-region issues

Scenario: US-Only Service

Monitoring locations: US-East, US-West
Consensus: Both must fail
Consecutive: 2 checks

Why: Both US coasts, eliminates regional network issues

Scenario: Critical Service

Monitoring locations: 5 regions
Consensus: 3 of 5 must fail
Consecutive: 3 checks

Why: Higher threshold for alerting, fewer false positives

Scenario: Regional Service (EU Only)

Monitoring locations: EU-West, EU-Central
Additional: US (for external perspective)
Consensus: 2 of 3 must fail

Why: Focus on user region, external check for complete outage

How Many Locations Do You Need?

More isn't always better. 3 well-chosen locations often beat 10 poorly configured ones.

Location Selection Principles

1. Monitor from where users are — If users are in EU, monitor from EU
2. Geographic diversity — Don't put all locations in the same region
3. Different network paths — Locations should use different providers
4. Relevant to SLA — Monitor from where SLA applies

Wakestack's Approach

Wakestack handles multi-location monitoring with built-in consensus:

Configuration Options

Monitor: api.example.com
├── Locations: US-East, EU-West, Asia
├── Alert when: 2+ locations fail
├── Consecutive failures: 2
└── Check interval: 1 minute

What This Means

Check flow:
├── 10:00:00 - All 3 locations check
│   ├── US-East: ✓
│   ├── EU-West: ✗ (network issue)
│   └── Asia: ✓
│   Status: Passing (only 1 failure)
│
├── 10:01:00 - All 3 locations check
│   ├── US-East: ✓
│   ├── EU-West: ✓ (recovered)
│   └── Asia: ✓
│   Status: Passing
│
└── No alert sent (single transient failure ignored)

Actual Outage Handling

Check flow:
├── 10:00:00 - All 3 locations check
│   ├── US-East: ✗
│   ├── EU-West: ✗
│   └── Asia: ✗
│   Status: Failing (3 failures, waiting for confirmation)
│
├── 10:01:00 - All 3 locations check
│   ├── US-East: ✗
│   ├── EU-West: ✗
│   └── Asia: ✗
│   Status: Confirmed down
│
└── Alert sent (consensus + consecutive failure)

Set up reliable monitoring — Multi-location with smart alerting.

Common Mistakes

1. Alert on Any Single Failure

Configuration: Alert if 1+ locations fail
Result: False alerts from every network hiccup

Fix: Require 2+ location failures.

2. No Consecutive Check Requirement

Configuration: Alert immediately on consensus failure
Result: Brief network issues trigger alerts

Fix: Require failures to persist for 2+ checks.

3. Too Many Locations, Same Region

Locations: New York, Boston, Washington DC
Problem: All use similar network paths
Result: Regional ISP issue affects all three

Fix: Geographic diversity across regions.

4. Ignoring Location-Specific Failures

Pattern: Singapore always fails while others succeed
Assumption: "Must be network issues"
Reality: CDN not configured for Asia

Fix: Investigate persistent single-location failures.

5. Same Threshold for All Services

Critical payment service: 2-of-5 consensus
Marketing website: 2-of-5 consensus

Problem: Same sensitivity for different priorities

Fix: Adjust thresholds based on criticality.

Tuning Your Configuration

Start Conservative

Begin with:

• 3 locations
• 2-of-3 consensus
• 2 consecutive failures

This catches real outages, ignores most network issues.

Monitor Before Alerting

Run for a week with alerts disabled:

• How many "failures" occur?
• Are they real or network issues?
• Which locations are reliable?

Adjust Based on Data

If getting false alerts:

• Increase consecutive failure requirement
• Raise consensus threshold

If missing real outages:

• Decrease consecutive failure requirement
• Add more locations for better coverage

Key Takeaways

• More locations ≠ automatically better monitoring
• Each location is an independent network path that can fail
• Consensus-based alerting prevents false positives
• Consecutive failure requirements filter transient issues
• 3 well-configured locations beat 10 poorly configured ones
• Tune based on actual data, not assumptions

Related Resources

• Global Uptime Monitoring
• What Causes False Downtime Alerts
• Uptime Monitoring Guide