My own product, May 2026 to present

Verid

My own product, co-founded in May 2026 and built on my own time, outside Jfreaks. I designed and built the backend from the first commit (166 of 265 commits); my co-founder built the marketing pages.

Postgres is the only scheduling source of truth. Redis can be wiped and the scheduler picks up exactly where it left off.

Web change detection for developers. Point it at a URL, pick an extraction rule and a predicate (a rule such as "the price went down"), and get one signed webhook per meaningful change, not one per diff.

TypeScript, Next.js 16, React 19, BullMQ, PostgreSQL, Knex, Prisma, Redis, Cloudflare R2, Creem, Resend, Docker

Problem

Most change-detection tools alert on any diff, so a rotating ad or a timestamp fires the same alert as a price drop. Developers wanted a specific field out of a page (a price, a version string, a stock flag), a rule for when a change matters, and a delivery they could verify and replay. It also had to keep working on pages that block plain HTTP, and it had to be cheap enough to run on two small servers.

Hardest problem

An out-of-order cancellation locked a paying customer's monitors. Creem, our payment provider, delivered the cancellation for an old subscription after the activation of the new one, and the webhook handler applied events in the order they arrived, so the account fell back to the free tier. I changed every billing handler to ignore event order and re-derive the tier from the subscriptions table on each event, so any delivery order ends in the same state.

What I built

A Next.js app (marketing, SSR dashboard, REST API, billing) and a separate BullMQ worker sharing one Postgres, plus an Express admin panel and a Nextra docs site. Shared packages mean the API, worker, pricing page and playground import the same tier, extractor and predicate code, so a pricing change cannot drift from enforcement
Six extractors: CSS, XPath, JSONPath, regex, full page text, and an LLM prompt that returns JSON against a schema
A predicate engine: any change, a named field, percent or absolute thresholds, regex, equality, and AND/OR composites
Deliveries that are rows, not fire-and-forget: HMAC-signed webhooks, email, Slack and Discord, with every attempt recorded, a dead-letter path, and replay for any delivery
A three-tier fetcher: plain HTTP, then a pooled stealth Chromium, then a residential proxy with sticky sessions
Visual monitoring: full-page or region screenshots, pixel diffs with a change threshold, PNGs in Cloudflare R2 behind signed URLs
In-house auth (scrypt, DB-backed sessions, Google OAuth with PKCE) replacing a third-party library, plus scoped API keys with rotation
Billing on Creem (a merchant-of-record payment provider that handles sales tax), and a public playground that runs the real extraction pipeline
A zero-dependency TypeScript SDK on npm (@verid.dev/sdk), a hand-maintained OpenAPI 3.1 spec, a Postman collection, and an llms.txt for agents

Engineering decisions

Scheduler claims due monitors with SELECT FOR UPDATE SKIP LOCKED and bumps next_run_at in the same transaction, so two workers can never double-run one monitor
The API never enqueues scrape jobs. Run now sets next_run_at; pause and delete only remove pending jobs. Redis is treated as disposable and is not backed up
Webhook signing mirrors Stripe: a per-monitor secret encrypted at rest with AES-256-GCM, a t=…,v1=… header, a five-minute replay window, constant-time comparison
Six delivery attempts on a fixed schedule of increasing delays (immediately, then 5m, 15m, 30m, 1h, 2h), so every attempt is its own row and a customer can predict when a retry lands. A delivery already marked success or dead is skipped on redelivery
Static fetches that return a 200 with empty fields on CSS or XPath rules escalate to the browser and evaluate the selectors in the live DOM. JSONPath never escalates, because a browser would wrap the JSON in HTML
For SERP tracking (rank monitoring on Google, Bing and others), a Redis circuit breaker separates refusals from unusable pages, doubles its cooldown per refusal from 15 minutes up to 6 hours, and paces requests through a Lua-reserved slot
LLM extraction is cached in Redis for 30 days by content hash, so an unchanged page never bills the user twice; OpenAI is primary with an Anthropic fallback
Tier caps live in one typed module read by the API, the worker, the pricing page and the dashboard. A new tier fails to compile until it has a rate-limit budget

In production

The worker survives restarts and concurrent instances without duplicate runs, because scheduling state lives in Postgres alone
Runs on two self-hosted VPS that I operate: images built in GitHub Actions, migrations applied before the worker restarts, nightly pg_dump shipped to Backblaze B2, and TLS-expiry alerting added after a real outage

Request flow

1.Client calls POST /v1/monitors with an API key. The route authenticates, applies a per-tier sliding-window rate limit in Redis, validates the body, and checks tier caps
2.The monitor row is inserted with next_run_at set to now. If a webhook target exists, a secret is generated and stored encrypted
3.Every 30 seconds the worker claims due monitors with FOR UPDATE SKIP LOCKED and enqueues one scrape job per monitor
4.The scrape job fetches (static, browser, or proxy), extracts the configured fields, diffs them against the last successful run, and evaluates the predicate
5.If the predicate fires, one delivery row is created per target and a deliver job is queued. Webhooks are signed and POSTed with a 15 second timeout
6.Failures step through the six scheduled attempts. After the sixth, the delivery is marked dead and the user is emailed from the dead-letter queue
7.The receiver verifies the signature with the shared secret and a five-minute timestamp tolerance. Any delivery can be replayed from the API

Architecture

A Next.js app owns the API, dashboard and billing, and never enqueues work. A separate BullMQ worker owns scheduling, fetching, extraction and delivery. Both share one Postgres and one Redis across two VPS. Postgres is the source of truth; Redis holds queues, rate limits and caches and can be lost without data loss.

clientserver or workerdata storeexternalrequest, responsequeued or scheduled

Connections

1.Dashboard, SDK ↔ Next.js app: HTTPS, REST, API key
2.Creem → Next.js app: Signed billing webhooks (queued or scheduled)
3.Next.js app ↔ Postgres: Write monitors, next_run_at
4.Next.js app ↔ Redis: Rate-limit windows
5.Worker ↔ Postgres: Claim due rows (SKIP LOCKED), write runs
6.Worker ↔ Redis: Scrape and deliver jobs (queued or scheduled)
7.Worker → Cloudflare R2: Upload screenshots
8.Worker ↔ Monitored pages: Fetch, escalating tier by tier
9.Worker → Customer endpoints: Signed POST, six scheduled attempts

Components

Dashboard, SDKBrowser, API key: The SSR dashboard in the Next.js app, the published TypeScript SDK, Postman, and any HTTP client with a vrd_ API key. Marketing pages are statically generated; the public playground reuses the production extraction pipeline under an IP rate limit.
Next.js appAPI, auth, billing: Next.js 16 App Router serving the marketing site, the dashboard, /api/v1 REST routes, in-house auth (scrypt passwords, DB sessions, Google OAuth with PKCE), scoped API keys, Creem checkout and webhooks, and the playground. Rate limiting is a Lua sliding window in Redis keyed by user and tier. The app never enqueues scrape jobs; it only writes next_run_at and delivery status to Postgres.
CreemBilling webhooks: The merchant-of-record payment provider. Checkout, renewals and cancellations arrive as HMAC-verified webhooks. Handlers do not trust arrival order: each event re-derives the tier from the subscriptions table.
RedisDisposable: Redis 7 carries the BullMQ queues, rate-limit windows, the 30-day LLM cache, proxy host flags and the SERP circuit breaker. It is not backed up: wiping it loses queued jobs, and the scheduler recreates them from Postgres on its next tick.
PostgresSource of truth: PostgreSQL with Prisma for migrations and Knex for every runtime query. Tables cover users, sessions, API keys, monitors (with next_run_at), runs, snapshots, deliveries, templates, usage and billing.
Cloudflare R2Screenshots: Screenshot PNGs for visual monitoring, served behind signed URLs. Only object keys are stored in Postgres.
Monitored pagesHTTP, browser, proxy: Monitored pages and APIs. Pages that block plain HTTP, or return empty fields to it, are retried through a stealth browser pool, then through a residential proxy with sticky, geo-matched sessions.
WorkerSchedule, fetch, deliver: A single Node process running four BullMQ workers (scrape, deliver, dlq, preview), a 30-second scheduler tick, an hourly lifecycle-email tick and a daily retention sweep. The scheduler claims due monitors atomically with SELECT FOR UPDATE SKIP LOCKED. Scrape jobs run the three-tier fetcher, the extractor, the diff and the predicate, then write the run and queue deliveries.
Customer endpointsWebhook, Slack, email: Webhook endpoints, Slack, Discord and email. Webhooks are POSTed with an HMAC-SHA256 signature header and a 15 second timeout; failures step through six scheduled attempts, then the dead-letter worker emails the user.