Juicebox Observability

Overview

Juicebox provides AI-powered people search and analysis where query performance, dataset ingestion rates, and quota consumption are the primary observability concerns. Monitor analysis completion times to ensure interactive UX, track ingestion pipeline health for data freshness, and watch quota usage to prevent mid-workflow cutoffs. Slow queries or failed ingestions degrade recruiter productivity and data accuracy.

Key Metrics

Metric	Type	Target	Alert Threshold
Search latency p95	Histogram	< 2s	> 5s
Analysis completion time	Histogram	< 10s	> 30s
Dataset ingestion rate	Gauge	> 100 records/s	< 50 records/s
API error rate	Gauge	< 1%	> 5%
Quota usage (daily)	Gauge	< 70%	> 85%
Query result relevance	Gauge	> 80% precision	< 60%

Instrumentation

async function trackJuiceboxCall(operation: string, fn: () => Promise<any>) {
  const start = Date.now();
  try {
    const result = await fn();
    metrics.histogram('juicebox.api.latency', Date.now() - start, { operation });
    metrics.increment('juicebox.api.calls', { operation, status: 'ok' });
    return result;
  } catch (err) {
    metrics.increment('juicebox.api.errors', { operation, error: err.code });
    throw err;
  }
}

Health Check Dashboard

async function juiceboxHealth(): Promise<Record<string, string>> {
  const searchP95 = await metrics.query('juicebox.api.latency', 'p95', '5m');
  const errorRate = await metrics.query('juicebox.api.error_rate', 'avg', '5m');
  const quota = await juiceboxAdmin.getQuotaUsage();
  return {
    search_latency: searchP95 < 2000 ? 'healthy' : 'slow',
    error_rate: errorRate < 0.01 ? 'healthy' : 'degraded',
    quota: quota.pct < 0.7 ? 'healthy' : 'at_risk',
  };
}

Alerting Rules

const alerts = [
  { metric: 'juicebox.search.latency_p95', condition: '> 5s', window: '10m', severity: 'warning' },
  { metric: 'juicebox.api.error_rate', condition: '> 0.05', window: '5m', severity: 'critical' },
  { metric: 'juicebox.quota.daily_pct', condition: '> 0.85', window: '1h', severity: 'warning' },
  { metric: 'juicebox.ingestion.rate', condition: '< 50/s', window: '15m', severity: 'critical' },
];

Structured Logging

function logJuiceboxEvent(event: string, data: Record<string, any>) {
  console.log(JSON.stringify({
    service: 'juicebox', event,
    operation: data.operation, duration_ms: data.latency,
    result_count: data.resultCount, query_length: data.queryLen,
    // Redact candidate PII — log only aggregate counts
    timestamp: new Date().toISOString(),
  }));
}

Error Handling

Signal	Meaning	Action
429 rate limit	Quota exhausted for period	Pause queries, check daily allocation
Search timeout > 5s	Complex query or service load	Simplify filters, retry with narrower scope
Ingestion stall	Dataset too large or format error	Check upload logs, validate schema
Empty result set	Index gap or query mismatch	Verify dataset freshness, adjust search params

Resources

• Juicebox Dashboard

Next Steps

See juicebox-incident-runbook.