Skip to content
Hyperwatch

Case study

How Open Collective watches its traffic

Open Collective is an open source platform where communities raise and spend money in full transparency. Its public pages attract people, search engines, AI crawlers and scrapers. Hyperwatch helps the team tell them apart, protect the service, and act at the edge.

Platform
opencollective.com, a Next.js frontend on an Express server
Edge
Cloudflare, with WAF custom rules
In the app
Identity-aware rate limiting and load shedding
Watcher
The hyperwatch CLI with one config file

A simplified version of the production setup, with placeholder values.

The challenge

Not all traffic is equal

  • Crawlers everywhere

    Search engines and AI crawlers read thousands of collective pages. Some are welcome, some are not, and many claim to be someone else.

  • IP limits miss the point

    A crawler spread over hundreds of addresses slips under per-IP limits, while people behind a shared IP get blocked.

  • Protect the service

    When load spikes, people using the platform must come first. Robots can wait.

  • Keep data in house

    Traffic data about a financial platform should stay on infrastructure the team controls.

The setup

Light in the app, heavy in the watcher

The frontend runs Hyperwatch in-process, just enough to know who is calling. A separate watcher subscribes to its raw logs, enriches them, and decides what Cloudflare should block.

  1. 01 · Edge

    Cloudflare

    Every visitor goes through Cloudflare. WAF custom rules block or challenge the clients on Hyperwatch's lists.

  2. 02 · App

    Frontend servers

    An Express middleware logs every request and identifies known clients. Rate limits and load shedding use that identity.

    /_hyperwatch/logs/raw

  3. 03 · Watcher

    hyperwatch frontend.js

    The CLI subscribes over WebSocket, on a laptop or a server. It runs the full enrichment, aggregations and firewall lists.

    firewall sync up → Cloudflare

Step 1

Plug Hyperwatch into the Express server

The frontend is a Next.js app served by a custom Express server. Hyperwatch lives in the same process: one function receives the Express app and wires everything in.

  1. 1Mount it first. Call it before any other middleware or route, so every request is seen, including the ones rejected later by the rate limiter.
  2. 2One log per request. The middleware records the request as it arrives, attaches req.hyperwatch, and completes the log with the status and response time when the response finishes.
  3. 3No second server. Hyperwatch's API and live streams are routes of the app, under /_hyperwatch and behind Basic Auth, ready for the watcher.
  4. 4Stay light. Only the modules needed to identify clients run in the app: real IP from Cloudflare, User-Agent parsing, verified hostnames and identity.
server/index.js
const express = require('express');
const next = require('next');
const hyperwatch = require('./hyperwatch');
const { identify, rateLimiter, serviceLimiter } = require('./limits');

const app = express();

// Behind Cloudflare: trust its addresses so req.ip is the visitor
app.set('trust proxy', cloudflareIps);

const nextApp = next({ dev: process.env.NODE_ENV !== 'production' });

nextApp.prepare().then(() => {
  // 1. Hyperwatch first, so it sees every request
  hyperwatch(app);

  // 2. Then the middlewares that use its identity (step 2)
  app.use(identify, rateLimiter, serviceLimiter);

  // 3. Finally, Next.js renders the pages
  app.all('*', nextApp.getRequestHandler());

  app.listen(process.env.PORT || 3000);
});
server/hyperwatch.js
const hyperwatch = require('@hyperwatch/hyperwatch');
const basicAuth = require('express-basic-auth');
const expressWs = require('express-ws');

module.exports = function (app) {
  const { input, modules, pipeline } = hyperwatch;

  // Keep the servers light: only what the app needs to decide
  hyperwatch.init({
    modules: {
      logs: { active: true }, // stream logs over WebSocket
      cloudflare: { active: true }, // real client IP from Cloudflare
      agent: { active: true },
      hostname: { active: true },
      identity: { active: true }, // "Googlebot", "GPTBot", …
    },
  });

  // Expose the API and live streams, behind Basic Auth
  expressWs(app); // WebSocket support for the live streams
  const auth = basicAuth({ users: { watcher: process.env.HYPERWATCH_SECRET } });
  app.use('/_hyperwatch', auth, hyperwatch.app.api);
  app.use('/_hyperwatch', auth, hyperwatch.app.websocket);

  // Every request goes through Hyperwatch
  const expressInput = input.express.create({ name: 'opencollective.com' });
  app.use(expressInput.middleware());
  pipeline.registerInput(expressInput);

  // No hyperwatch.start(): the app's own server does the serving
  modules.start();
  pipeline.start();
};

Step 2

Put identity to work

For every request, the app asks Hyperwatch who is calling. A verified crawler gets an identity such as Googlebot or GPTBot, confirmed by reverse DNS or published IP ranges. Anyone else is known by IP address.

  • Rate limiting counts a crawler as one client across all its addresses.
  • Load shedding turns away non-essential robots first when the service is under pressure.
  • Fast mode skips slow lookups, so the check never holds a request up.
server/limits.js
// Who is this? A verified identity, or just an IP address
async function identify(req, res, next) {
  const log = await req.hyperwatch.getAugmentedLog({ fast: true });
  req.identity = log.get('identity');
  req.identityOrIp = req.identity || log.getIn(['request', 'address']);
  next();
}

// Rate limit per identity: a crawler is one client, whatever its IPs
const rateLimiter = createRateLimiter({ key: (req) => req.identityOrIp });

// Under load, shed non-essential robots first
function serviceLimiter(req, res, next) {
  if (isUnderLoad() && nonEssentialRobots.includes(req.identity)) {
    return res.status(503).send('Service temporarily limited');
  }
  next();
}

module.exports = { identify, rateLimiter, serviceLimiter };

Step 3

Watch from anywhere with one config file

No extra service to build: the global hyperwatch command runs a single config file. It subscribes to the frontend's raw logs and runs the modules that are too heavy for the app: GeoIP, per-address and per-signature aggregations, and the firewall lists.

Point it at production, or at a frontend running on your machine. The pipeline is the same. With several server instances, register one input per instance.

export HYPERWATCH_SECRET=

# Watch production
export HYPERWATCH_URL=wss://opencollective.com/_hyperwatch/logs/raw

# Or a frontend running on your machine
export HYPERWATCH_URL=ws://localhost:3000/_hyperwatch/logs/raw

# Serve the watcher on another port than the frontend
PORT=4000 hyperwatch frontend.js
frontend.js
// frontend.js: run with `hyperwatch frontend.js`
module.exports = function (hyperwatch) {
  const { pipeline, input } = hyperwatch;

  // The heavy lifting happens here, not on the servers
  hyperwatch.init({
    modules: {
      logs: { active: true },
      cloudflare: { active: true },
      geoip: { active: true },
      agent: { active: true },
      hostname: { active: true },
      address: { active: true },
      signature: { active: true },
      identity: { active: true },
      firewall: { active: true },
    },
    persistence: { enabled: true, namespace: 'frontend' },
  });

  // Subscribe to the raw logs of the frontend
  pipeline.registerInput(
    input.websocket.create({
      name: 'opencollective.com',
      type: 'client',
      address: process.env.HYPERWATCH_URL,
      username: 'watcher',
      password: process.env.HYPERWATCH_SECRET,
      reconnectOnClose: true,
    })
  );

  // Skip static assets, and keep sign-in tokens out of the logs
  pipeline
    .getNode('main')
    .filter((log) => !log.getIn(['request', 'url']).startsWith('/static'))
    .map((log) =>
      log.updateIn(['request', 'url'], (url) =>
        url.replace(/^/signin/[^/?]+/, '/signin/[token]')
      )
    )
    .registerNode('main');

  // One stream for known clients, one for everyone else
  const [identified, unidentified] = pipeline
    .getNode('main')
    .split((log) => log.has('identity'), ['identified', 'unidentified']);
  identified.registerNode('identified');
  unidentified.registerNode('unidentified');

  // Requests slower than a second
  pipeline
    .getNode('main')
    .filter((log) => log.get('executionTime') > 1000)
    .registerNode('slow');
};
$ npm install -g @hyperwatch/hyperwatch

Step 4

Investigate, then act at the edge

The watcher serves everything the team needs to investigate:

  • /logs/unidentified the traffic nobody vouches for
  • /logs/slow requests slower than a second
  • /addresses top clients over 15m and 24h
  • /signatures clients grouped by header fingerprint
  • /identities known crawlers and their volume

When a client needs to be stopped, it goes on a firewall list. Hyperwatch tags its requests right away, and sync up pushes the list to a Cloudflare custom rule. The frontend never sees that traffic again.

# Review a suspicious client, then add it to a list
curl -X POST localhost:4000/firewall/lists/challenge-ips/add \
  -H 'content-type: application/json' \
  -d '{ "value": "203.0.113.7", "reason": "Scraping", "source": "watch" }'

# Push the lists to their Cloudflare custom rules
hyperwatch firewall sync up

Outcome

A clear picture, and the tools to act on it

Fair limits

Rate limits follow real clients. Crawlers can no longer spread over many IPs, and people on a shared network are not punished for each other.

Service first

Under pressure, robots are the first to wait. People keep using the platform.

Edge enforcement

Abusive clients are stopped at Cloudflare, before they cost anything to the servers.

Live visibility

The team can tail unidentified or slow traffic in production, or against a local frontend, in seconds.

One small config

The watcher is the stock CLI and one JavaScript file, versioned with the rest of the code.

Data stays home

Logs flow from the servers to the watcher, and nowhere else.

Start watching your traffic in five minutes

Install it, point it at a log, open your browser.

npm install -g @hyperwatch/hyperwatch && hyperwatch