SiliconX Documentation
SiliconX is a PostgreSQL-first Fair Market Value (FMV) engine for second-hand enterprise compute hardware. The system is a pnpm monorepo with three runtime apps: a Hono REST API deployed as a Lambda ECR container, a Vite+React analyst UI on S3+CloudFront, and a market data ingestion pipeline (ScrapingBee as primary scheduled source, with eBay/CDW/ServerMonkey as manual-only legacy sources). Aurora Serverless v2 is the canonical data store; Kysely provides type-safe access with types generated from the live schema via kysely-codegen. FMV is calculated as baselinePrice × ageMultiplier × configMultiplier × conditionMultiplier × marketabilityFactor, driven by confidence-weighted market comp aggregates (sku_mcmp) and effective-dated rule tables.
Quick Start
git clone git@github.com:SiliconExchange/fmv-data-model.git
cd fmv-data-model
make bootstrap # install → schema → seed → types-gen
make api-dev # Hono API on :3001
make web-dev # Vite UI on :5173Prerequisite: Docker (for local Postgres), Node 20+, pnpm 9+.
After bootstrap, the local Postgres container (PG 16) holds the full schema and prototype seed data. DEV_ROLE=tester_alice is the default analyst role; set DEV_ROLE=tester_bob or tester_carol for alternate roles.
Architecture
Request Lifecycle
Documentation Map
| Section | Description | Audience | Link |
|---|---|---|---|
| Getting Started | Bootstrap, local dev, env vars | All engineers | 01-getting-started/ |
| Architecture | System design, ADRs, data flow | Senior engineers | 02-architecture/ |
| API Reference | All REST routes, auth, middleware | API consumers / FE engineers | 03-api/ |
| Ingestion Pipeline | ScrapingBee orchestration, staging promoter, LLM normalizer | Data engineers | 04-ingestion/ |
| Frontend | Vite+React UI, TanStack Router, Hono client types | Frontend engineers | 05-frontend/ |
| Database | Schema migrations, views, SQL tests, tbls docs | DBA / backend engineers | 06-database/ |
Key Files
| File | Purpose | Notes |
|---|---|---|
apps/api/src/index.ts | API entrypoint; middleware order + route registration; exports AppType | loadSecrets() fetches siliconx/lambda-secrets from Secrets Manager on Lambda cold start |
apps/api/src/middleware/role.ts | JWT validation, role resolution, per-request PG pool checkout, SET LOCAL ROLE | JWKS hardcoded (no remote fetch from Lambda VPC); SingleConnectionProvider pattern |
apps/api/src/routes/assets.ts | Asset CRUD, FMV preview/snapshot, annotations, CSV export | FMV kernel shared between preview (no write) and snapshot (persists) |
apps/api/src/routes/skus.ts | SKU catalog, pricing stats (IQR-fenced 90-day), images, scrape targets, bundle components | Auto-generates GEN-* / CB-* verifiedSku codes |
apps/api/src/routes/snapshots.ts | Audit-lock FMV snapshot detail via v_fmv_snapshot_inputs | Read-only; coerces numeric/Date fields |
apps/api/src/routes/fleet.ts | Fleet summary: totals + by-seller + by-class in one round-trip | Powers dashboard aggregate cards |
apps/api/src/routes/sellers.ts | Seller org management; public list + admin CRUD | adminOnlyMiddleware gates write routes to ops_admin Cognito group |
apps/api/src/routes/transactions.ts | Marketplace transaction record; sets asset.lifecycleStateId='sold' | Admin-only write; validates allowsListing=true before insert |
apps/ingestion/src/pipeline/promoter.ts | Validates staging rows → writes to canonical market_comp / sku | Gatekeeper between raw scrape data and canonical tables |
apps/ingestion/src/pipeline/mcmp.ts | Recomputes sku_mcmp confidence-weighted aggregate after new market_comp rows | Feeds FMV baseline |
apps/ingestion/src/llm/normalizer.ts | LLM-based listing normalization (ScrapingBee raw → structured) | Used in ScrapingBee path only |
packages/shared/src/db.d.ts | Auto-generated Kysely DB types; never hand-edit | Regenerated via make types-gen after every schema change |
db/schema/ | SQL migrations 010–452; append-only | Never edit applied files; always add a new numbered file |
Makefile | All dev targets: bootstrap, deploy, test, docs | Primary dev entry point |
CONTEXT.md | Bounded domain vocabulary | Read before naming anything |
Tech Stack
| Technology | Role | Version / Config |
|---|---|---|
| PostgreSQL | Primary data store | PG 16 (Aurora Serverless v2 in prod; Docker in dev) |
| Hono | REST API framework | Lambda ECR container; exports AppType for client type inference |
| Kysely | Type-safe SQL query builder | Types from kysely-codegen against live schema |
| Aurora Serverless v2 | Managed PostgreSQL | us-east-1; VPC-isolated (no outbound HTTP from Lambda) |
| AWS Lambda | API compute | Container image (ECR); siliconx/lambda-secrets via Secrets Manager |
| AWS S3 + CloudFront | Web app hosting + SKU image CDN | Vite build → S3 sync → invalidation |
| AWS Cognito | Production authentication | Pool us-east-1_cEADa8wlF; groups ops_admin / ops_analyst |
| Vite + React | Analyst UI | TanStack Router (file-based); hc<AppType>() for type-safe API calls |
| ScrapingBee | Primary market data source | EventBridge-scheduled; writes to market_comp_staging |
| pnpm workspaces | Monorepo package management | apps/api, apps/web, apps/ingestion, packages/shared |
| tbls | Schema documentation generator | make schema-docs → docs/schema/ |
| Vitest | SQL behavior tests | tests/sql/*.sql via make test-sql |
| bats | Schema harness tests | make schema-test |
FMV Calculation
The FMV kernel resolves five factors per asset:
fmv = baselinePrice
× ageMultiplier -- from age_curve table (asset class + model version)
× configMultiplier -- class_attributes JSONB vs. base config
× conditionMultiplier -- condition_band (new / refurbished / used)
× marketabilityFactor -- marketability_tier (A / B / C)baselinePrice comes from sku_mcmp (confidence-weighted aggregate of market_comp rows, IQR-fenced per sku_price_bound). Friction costs are applied via friction_cost_rule (effective-dated, no-overlap enforced by EXCLUDE constraint — see ADR 0001).
DB Schema Overview
54 tables across 10 domains. Auto-generated ER diagrams and per-table docs live in docs/schema/.
| Domain | Key Tables |
|---|---|
| Hardware catalog | asset, sku, asset_class, asset_class_schemas |
| Market pricing | market_comp, sku_mcmp, sku_price_bound, market_comp_staging |
| FMV engine | fmv_pricing_snapshot, age_curve, condition_multiplier, friction_cost_rule, marketability_assignment |
| Orgs | seller_org, buyer_org |
| Exchange | marketplace_transaction |
| OEM specs | sku_oem_spec, sku_list_price_history |
| Ingestion | asset_import_staging, scrape_target |
| Images | sku_image |
| Lifecycle | asset_lifecycle_state |
| Geography | data_center, region |
class_attributes JSONB on sku and asset is validated against versioned JSON Schemas in asset_class_schemas (see ADR 0002).
Role Model
| Role | Source | DB Role | Capabilities |
|---|---|---|---|
tester_alice / tester_bob / tester_carol | Dev env (DEV_ROLE) | direct | Full dev access |
ops_readonly | Cognito group ops_analyst | ops_readonly | Read-only analyst queries |
ops_admin | Cognito group ops_admin | ops_readonly + adminOnlyMiddleware | All routes including org management, transactions |
Per-request: BEGIN → SET LOCAL ROLE → SET LOCAL statement_timeout='5s' → query → COMMIT/ROLLBACK.
Ingestion Pipeline
Legacy sources (eBay, CDW, ServerMonkey) follow the same *_staging → promoter path but are triggered manually via pnpm scrape:legacy.