Najeeb Said

Bell Partners · 2025 – 2026

Bell Analytics & DistroNation

Multi-tenant analytics platform and automated reporting suite for music distributor and record label teams.

SFTP / Spotify Bulk
BullMQ workers
materialized views
PostgreSQL·DynamoDB·BullMQ·Redis·Node.js

Problem

Historical analytics data (8.4M records across 5.8M tracks and 2.5M assets) sat in DynamoDB, where dashboard filters triggered full table scans. Ingestion pipelines for daily Amazon DSP drops (SFTP) and Spotify Bulk API archives (.json.gz) suffered memory exhaustion and third-party rate-limit bans.

Approach

Engineered a 23-migration pipeline transferring historical DynamoDB data into a PostgreSQL star-schema warehouse. Implemented materialized views (revenue_filters) with scheduled concurrent refreshes (REFRESH MATERIALIZED VIEW CONCURRENTLY) to serve dashboard aggregates.

Built BullMQ worker services utilizing ssh2-sftp-client for SFTP drops and custom Spotify Bulk API ingestion pipelines. Decompressed multi-gigabyte .json.gz archives on the fly in 10,000-line streaming chunks.

Customized BullMQ retry strategies to parse Spotify's HTTP 429 Retry-After headers. Built an automated monthly revenue reporting system scanning DynamoDB users, fetching AppSync creator IDs in controlled batches of 5, and delivering HTML reports via Resend.

Technical Decisions

  1. Materialized views with concurrent refresh

    Replaced SELECT DISTINCT scans with materialized views indexed on composite keys, supporting background refreshes without read-locking dashboard users.

  2. Header-driven rate limit backoff in BullMQ

    Parsed HTTP 429 Retry-After headers directly within BullMQ worker retry algorithms, optimizing throughput while staying within Spotify API rate limits.

  3. Decoupled discovery and ingestion workers

    Separated SFTP/API archive discovery workers from ingestion workers, ensuring file scan failures do not interrupt pipeline processing.

  4. Chunked database batch writes for OOM prevention

    Capped Sequelize bulkCreate transactions at 10,000 rows and tuned connection pools (max 80 connections) to eliminate process out-of-memory crashes.

Key Achievements

  • Migrated 8.4M historical records (5.8M tracks, 2.5M assets) from DynamoDB into a PostgreSQL star-schema warehouse.
  • Engineered BullMQ workers streaming and decompressing multi-gigabyte .json.gz Spotify archives in 10,000-line chunks.
  • Customized BullMQ retry logic to parse HTTP 429 Retry-After headers, preventing API throttling bans.
  • Created revenue_filters materialized view with REFRESH MATERIALIZED VIEW CONCURRENTLY, eliminating read locks during updates.
  • Automated monthly revenue report generation and email fan-out via Resend API across creators.

Outcome

Analytics dashboards render instantly from pre-computed materialized views, multi-gigabyte archives ingest reliably without memory leaks, and monthly revenue reports generate and dispatch automatically.