Jfreaks Software Solutions, Sep 2024 to present
Commodity Price API
Jfreaks Software Solutions. I took the codebase over from a previous developer in September 2024 and have been its primary maintainer since.
Live prices are answered from a map in the API process. Loaders push changes in; MongoDB is the fallback, not the hot path.
A paid REST API for live and historical commodity prices, from gold and crude to US energy, USDA livestock and EU electricity.
TypeScript, Express, MongoDB, Next.js 16, node-cron, Stripe, MCP, Docker, pm2
Problem
The original API read from the database on every request and covered a narrow set of scraped symbols. Customers asked for three things: more symbols from sources we could legally resell, gold and silver faster than once a minute, and a self-serve trial. All of it had to land without a rewrite, on a codebase inherited from a previous developer.
Hardest problem
Taking reads off the database without a rewrite. Every price request used to query MongoDB. I moved live prices into a map inside the API process: the loaders push the full map in over a JWT-authenticated internal route each time they run, the API answers from memory, and a miss falls through to MongoDB and is written back into the map. Plans with slower updates read a snapshot of the map copied every ten minutes, so the controllers contain no per-tier branching.
What I built
- The loader to API hot path: a one-minute loader pushes the full rate map into the API's in-memory map, with MongoDB as the lazy fallback
- One-second gold and silver bid/ask fetchers with dedicated endpoints that break prices down by gram, tola and carat
- Official-source data: loaders for EIA energy, USDA AMS livestock and grain, and EU electricity (ENTSO-E, energy-charts and Eurostat), plus monthly IMF and World Bank importers
- v2 of the public API: latest, historical, time-series and fluctuation endpoints, quote-currency conversion, and a consistent error format
- Billing and accounts: Stripe Checkout and Billing Portal, idempotent webhooks, seven-day trials with self-service and admin extension, nightly trial-expiry and yearly-plan reset crons, account deletion
- Write-behind API metering with a per-day usage ledger that powers the dashboard heatmap
- @commoditypriceapi/mcp, the official MCP server, published to npm, with eight tools that map one-to-one onto v2 endpoints
- The marketing site, docs and dashboard in Next.js 16, and a separate free-tools site with per-symbol price pages, metal calculators and ratio pages
Engineering decisions
- Loader change detection honours a per-source preference ladder: lower preference wins, equal preference must differ in value. Only changed symbols are written to MongoDB; the full map is pushed to the API
- Usage counting happens after the response finishes, in memory. A five-minute cron flushes deltas with a single $inc bulkWrite and puts them back on failure. The trade-off: a crash can under-count up to five minutes of usage, which only ever favours the customer
- Unknown API keys are cached as null, so a bad key never hits the database twice
- Gold and silver loops use a five-second per-call timeout, exponential backoff from one to thirty seconds, and stop with an email after five consecutive failures
- Official-source loaders poll change signals, not data: EIA's manifest hourly, USDA's publish and correction feeds every fifteen minutes, with a safety valve that forces a full run after six failed checks
- EU electricity data is only stored if the response declares a CC BY licence and the expected zone and unit; anything else fails closed and alerts. Attribution was added to the symbols page and Terms
- Ticks go to a MongoDB time-series collection with a short TTL; official daily prints go to a separate OHLC collection upserted on commodity and date, so history never depends on the scrape feed
- Stripe webhooks store each event ID in a unique collection and check its status before processing, so redeliveries are no-ops
In production
- Coverage grew from scraped exchange quotes to include 103 symbols from official US and EU sources: seventeen from EIA, sixteen from USDA and seventy EU power prices
- Developers can call the API from Claude, Cursor or any MCP client without writing HTTP code
Request flow
- 1.Every minute the loader fetches quotes from commercial market-data feeds in two batched requests and compares each symbol against its in-memory map
- 2.Changed symbols are inserted into MongoDB. The full map is POSTed to the API's internal route with a short-lived service JWT
- 3.A client calls GET /v2/rates/latest?symbols=XAU,BRENT with an API key in the header
- 4.The API validates the key against its user map, rejects ended trials and inactive plans, and checks the monthly quota
- 5.The controller picks the live map or the ten-minute snapshot map from the plan's update frequency, then reads each symbol. Misses fall through to MongoDB and are written back to the map
- 6.Symbols from official sources (EIA, USDA, EU power, IMF, World Bank) are read from the OHLC collection instead
- 7.If a quote currency is requested and the plan allows it, rates are converted. A conversion failure still returns 200 with a warning field
- 8.When the response finishes, usage is incremented in memory and flushed to MongoDB by the five-minute cron
Architecture
Two lanes of loaders feed one store. The market lane runs every second to every minute and pushes the live map into the Express API over a JWT-authenticated internal route; the official lane polls government change feeds and upserts daily prints. The API answers from memory and falls back to MongoDB on a miss. The sites and the MCP server are ordinary API-key clients of v2.
Connections
- 1.Market feeds → Live loaders: Poll every second to every minute (queued or scheduled)
- 2.Live loaders → API server: Push the full map (service JWT) (queued or scheduled)
- 3.Live loaders → MongoDB: Write changed symbols only
- 4.Official sources → Official loaders: Poll change feeds (queued or scheduled)
- 5.Official loaders → MongoDB: Upsert OHLC by commodity and date
- 6.API server ↔ MongoDB: Miss fallback, usage flush
- 7.Web, tools, MCP ↔ API server: REST v2, API key
Components
- Market feedsCommercial quotes
- Commercial market-data feeds for live quotes, plus a separate gold and silver bid/ask feed polled every second. Currency conversion comes from a separate rates provider.
- Live loaders1-min and 1-s loops
- A one-minute cron for live quotes in two batched requests, one-second gold and silver loops, and a scrape loader every ten minutes. Each compares incoming values with its map and writes only what changed.
- API serverLive + snapshot maps
- Express 4 with Mongoose, Passport (local, JWT cookie, Google OAuth) and zod. Holds the live rate map, the ten-minute snapshot map, the gold and silver map, and a user map with cached quota state. Enforces trials, plan quotas, symbols per request and quote conversion per plan. Handles Stripe Checkout, Billing Portal and idempotent webhooks. Runs under pm2 in Docker.
- MongoDBTicks, OHLC, usage
- A time-series collection for ticks with a short TTL, an OHLC collection for official daily and monthly prints with a unique index on commodity and date, and collections for users, plans, Stripe webhook events (unique event ID), per-day API key usage and trials. Read on API boot to seed lookup maps and on cache misses; otherwise off the hot path.
- Official sourcesEIA, USDA, EU, IMF
- EIA Open Data, USDA AMS My Market News, ENTSO-E, energy-charts and Eurostat for EU power, and monthly IMF and World Bank sheets. Government sources are polled through their change signals (manifests, publish feeds) rather than on a blind loop.
- Official loadersChange feeds, licence
- EIA on an hourly manifest check, USDA on a fifteen-minute publish and correction watch, EU electricity on day-ahead and settlement schedules in Berlin and Brussels time, and IMF and World Bank importers run by hand each month. EU data without a CC BY licence or with the wrong zone or unit fails closed and alerts.
- Web, tools, MCPNext.js 16, npm
- The marketing site, docs and dashboard (Next.js 16, App Router), the free tools site with per-symbol pages, metal calculators and ratio pages, and the @commoditypriceapi/mcp server. All of them call v2 with an API key; the web app additionally uses cookie-based JWT auth against v1 for accounts and billing.