Serving Thousands of Users With One Paid API Call per Location

Costs Scale with Locations × Time, Not Users

When designing client applications like weather forecasts, developers often hit the harsh economic reality of third-party APIs. Weather data providers charge for every single API call. If you design your application naively, your operating costs will scale linearly with the number of active users. For any sustainable project, this is unacceptable.

The typical solution is to implement a cache with a Time-To-Live (TTL). However, under high load, the cache itself fails the moment a record expires. Imagine a storm approaching and five hundred users in the same city opening the app at the exact same second. If the cache entry has just expired, a naive system will let all five hundred requests pass straight through to the paid upstream API. This results in a massive spike in costs and potential rate-limiting from the provider.

The real key to minimizing costs is rigorous API call deduplication. The goal is to ensure that the cost curve does not track the number of users, but instead follows this formula:

Cost = number of tracked locations × time granularity

If you have ten thousand users spread across fifty different cities, the system should send at most fifty requests to the external API per update window. All other queries must be served from the cache, or wait for the in-flight request for that location to complete. This principle is known as single-flighting and is a fundamental building block of an efficient backend architecture.

Rounding Coordinates: Neighbors Share a Cache Entry

The first step to successful deduplication is unifying cache keys. Mobile devices and modern browsers provide GPS coordinates with high precision, often to six or more decimal places. In practice, this means millimeter-to-centimeter accuracy. If you cached raw coordinates, almost every user would have a unique key, rendering the cache useless. Two neighbors in the same apartment building would generate completely different API requests.

The solution is to intentionally reduce coordinate precision by rounding to four decimal places. This creates a virtual geographic grid with a resolution of approximately eleven meters at the equator. For weather forecasting, this level of accuracy is more than sufficient, but for caching, it makes a massive difference. People on the same street, in the same office building, or in the same park will share identical cache keys.

The cache key is constructed as a string containing the schema version and the rounded coordinates:

function generateCacheKey(latitude: number, longitude: number, version: number): string {
  const lat = latitude.toFixed(4);
  const lon = longitude.toFixed(4);
  return `weather:v${version}:${lat}:${lon}`;
}

Including the schema version in the key itself is critical for zero-downtime deployments. If the data bundle structure changes in a new version of the application (for example, adding new metrics or changing the JSON format), simply bumping the version in the configuration immediately invalidates the old cache. This prevents deserialization errors on the client side.

Layer 1: In-Process Single-Flight

The first line of defense is deduplication at the level of individual application server instances (in-process). If multiple requests for the same location arrive at the same time within a single Node.js process, we must not initiate an asynchronous operation to the database or Redis cache for each of them.

To achieve this, we use a shared map of active Promises. When a request arrives, the application first checks if a fetch is already in progress for that cache key. If it is, it returns the existing Promise. All concurrent clients then wait for that single network request to resolve.

The following example illustrates the principle of in-process deduplication:

import { Injectable } from '@nestjs/common';

@Injectable()
export class SingleFlightService {
  private pendingRequests = new Map<string, Promise<unknown>>();

  async execute<T>(key: string, fetchFn: () => Promise<T>): Promise<T> {
    const existingPromise = this.pendingRequests.get(key);
    if (existingPromise) {
      return existingPromise as Promise<T>;
    }

    const promise = fetchFn().finally(() => {
      this.pendingRequests.delete(key);
    });

    this.pendingRequests.set(key, promise);
    return promise;
  }
}

This effectively eliminates concurrent queries within a single replica. However, if you run the application in a multi-replica environment (such as Kubernetes or serverless platforms), the in-process memory of individual instances is isolated. This is where the second layer comes in.

Layer 2: Distributed Locking via Redis

To prevent duplicate requests across different application server replicas, we must coordinate access to the external API using a distributed lock. Redis is the ideal choice for this coordination layer.

When attempting to fetch new data, a replica tries to acquire a lock for the given cache key. It uses the SET command with the NX (set if not exists) and PX (expiration in milliseconds) options. The expiration is a critical safety measure in case the process that acquired the lock crashes unexpectedly. The lock must not hang indefinitely.

const lockAcquired = await redis.set(lockKey, uniqueToken, 'NX', 'PX', lockTimeoutMs);

Releasing the lock is just as important as acquiring it, and it must be done safely. We cannot simply delete the lock key. If process A holds the lock for too long and it expires, process B might acquire it. If process A then finishes its work and deletes the key, it would inadvertently release the lock belonging to process B.

To release the lock safely, we use a Lua script to guarantee atomicity. The script first verifies that the lock's value matches the unique token generated by the process when it acquired the lock. It only deletes the key if they match.

if redis.call("get", KEYS[1]) == ARGV[1] then
    return redis.call("del", KEYS[1])
else
    return 0
end

Lock Losers Don't Wait Idle — They Poll the Cache

What happens to a request on a replica that loses the race for the lock? This process (the lock loser) must not fail, nor should it immediately call the paid API. It knows that another replica is currently fetching the data and will soon save it to the shared cache.

Instead of generating additional load, the lock loser enters a short polling loop, querying Redis at regular intervals to check if the new data has arrived.

async function waitForData(cacheKey: string, maxAttempts = 5, delayMs = 200): Promise<unknown> {
  for (let attempt = 0; attempt < maxAttempts; attempt++) {
    const data = await redis.get(cacheKey);
    if (data) {
      return JSON.parse(data);
    }
    await new Promise(resolve => setTimeout(resolve, delayMs));
  }
  throw new Error('Timeout waiting for data replication');
}

If the data arrives in the cache within the timeout, the process returns it to the client. In the worst-case scenario where polling times out (for example, if the winning replica crashed mid-process and the lock expired), the lock loser will attempt to acquire the lock itself, perform a fallback call to the paid API, and save the result for others.

Stale-While-Revalidate: No One Waits for the Network

Users expect immediate responses. While deduplication saves money, waiting for a network request to an external weather provider increases latency. To solve this, we use the Stale-While-Revalidate (SWR) pattern.

In this mode, the cache distinguishes between two expiration thresholds:

1. Soft expiration (stale limit): The data is considered stale but is still returned to the user immediately.
2. Hard expiration (TTL limit): The data is too old and can no longer be used.

If a user requests data that has passed the soft expiration threshold but is still within the hard expiration limit, the system immediately returns the stale data from the cache. A revalidation process is triggered asynchronously in the background. This process acquires the lock, fetches fresh data from the API, updates the cache, and the client sees the latest information on their next request. In the vast majority of cases, the user never waits for an external network response.

This approach is fully integrated into the architecture of the entire Weather InTouch project. As a result, the user interface is extremely fast and fluid, regardless of the current latency of the upstream weather services.

Without Redis: Graceful Degradation to Memory

A robust architecture must account for failures. If the Redis cluster goes down, or if you run the application in a local development environment without the full infrastructure, the system must not crash. Instead, it gracefully degrades.

If the application detects that the Redis server is unavailable, it automatically falls back to in-process memory. We use a bounded LRU (Least Recently Used) cache to prevent memory leaks or out-of-memory errors when querying a large number of locations.

While we lose distributed coordination and cross-replica deduplication in this mode, the application remains fully functional. In-process single-flighting still works perfectly on each individual replica. Running costs will temporarily increase due to duplicate API calls, but this is an acceptable trade-off for high availability.

For a deeper dive into how this resilient infrastructure is designed and how the individual components communicate, check out the article on how the entire platform is put together.

Conclusion

Managing costs effectively in data-dependent applications requires more than basic caching. Combining coordinate rounding, in-process deduplication, distributed locking, and a Stale-While-Revalidate strategy allows you to scale to thousands of users while keeping infrastructure costs to a minimum. You can explore the complete production implementation of these principles in the Weather InTouch project details.