LeadBenji
Website → normalize → validate → dedupe → export. Automation, not a model.
- Role
- Automation Engineering
- Type
- Data pipeline / crawl-export
- Client
- LeadBenji
- Status
- Live
Outcome
Partial recrawls scale with failure count, not full site size. Junk rows are stripped before CRM import instead of in a spreadsheet afterward.
Sales teams collected prospects by hand from inconsistent sites. Full re-crawls after a few 404s wasted wall time. Raw HTML in a sheet meant cleanup after import. Ban risk from naive parallel fetching.
Staged pipeline with per-site config (start URLs, selectors, rate limits) stored as data. Bounded concurrency, timeouts, backoff. Normalization and dedupe before the download endpoint. Job documents for queued/running/succeeded/failed.
- Frontend
- Small operator UI for triggering runs and fetching results (Vercel).
- Backend
- Node HTTP API hands work to Python via subprocess, queue, or shared DB.
- APIs
- JSON contracts for job status and record batches. Pagination on large sets.
- Databases
- Job and record persistence as implemented for status and idempotent retries.
- Per-site config as data — new sources should not require a deploy
- Normalize (trim, phone/email regex, canonical names) before dedupe keys
- Dedupe on domain + normalized email/phone with configurable strictness
- CSV or NDJSON with stable column order
- Run id, counts, last error; per-URL error log
Challenge
HTML structures differed by source.
Decision
Selector maps plus fallbacks, not one global scrape.
Implementation
Per-site config: start URLs, selector maps, rate limits.
Result
A broken site does not require rewriting the runner — it requires config and selector work.
Challenge
Long jobs were unobservable; retries re-crawled successes.
Decision
Persist job state and per-URL errors; idempotent retries.
Implementation
Job documents with state transitions. Reruns skip succeeded pages where safe.
Result
Recovery wall time tracks failures, not total crawl size.
Python for fetch/parse, Node for the HTTP control plane
Parsing and timeouts are straightforward in Python. Triggering and auth fit a Node API the rest of the stack already used.
Alternative. Single-language (all Python or all Node).
Tradeoff. Two runtimes to deploy. Isolation between crawler crashes and the API is the benefit.
Deterministic validation — not a classifier
Required fields and regexes are explainable to operators. A model would add cost and failure modes this product did not need.
Alternative. LLM extraction / classification.
Tradeoff. Brittle on novel layouts; honest. An AI-enhanced extractor would be a different case study.
- BuiltCrawl/parse pipeline and per-URL failure handling
- BuiltNormalize / validate / dedupe / export stages
- BuiltJob API for trigger, status, and downloads
- Not labeled as AI. If a later version uses a model for classification, that is a separate case study.
Operators retry failed URLs instead of full crawls. Export schema is strict before CRM import. Ban risk is reduced by caps and backoff versus unbounded fan-out. No published lead-conversion metrics.