How We Built a Human-in-the-Loop Baseline Verification System for Pricing Data

When you're tracking pricing data for 290+ SaaS tools, accuracy isn't optional—it's existential. A single false positive (claiming Notion raised prices when they didn't) destroys user trust. Here's how we built a verification system that combines AI extraction with human oversight to achieve 95%+ accuracy.

1. The Problem: AI Extraction Isn't Perfect

Our initial approach was straightforward: use Gemini Pro to extract pricing data from HTML, store it, compare against previous extractions, flag changes. Simple, right?

After analyzing 1,900+ snapshots, we discovered several failure modes:

• Billing period confusion: AI extracting "$48/year" as "$48/month" (12x error)
• Promotional pricing: Capturing sale prices instead of regular prices
• Duplicate plans: Same plan extracted multiple times with different names
• Missing context: Enterprise "Contact Sales" interpreted as $0

The core insight: you can't automate what you can't verify. We needed a way to establish a trusted "ground truth" before automation could take over.

2. The Baseline Concept

A baseline is the first verified snapshot for each tool. Think of it as the "anchor point" that all future comparisons reference. The key principle:

Here's the workflow:

1. First crawl: AI extracts pricing data → stored as "candidate baseline"
2. Human review: Admin reviews extraction against actual pricing page
3. Approval/Rejection: If correct, baseline is approved. If wrong, admin corrects it.
4. Automation activates: Future crawls compared against verified baseline
5. Anomaly flagging: Significant deviations trigger new human review

This creates a "trust chain" where automated decisions inherit confidence from human-verified foundations.

3. Anomaly Detection Thresholds

Not every difference is an anomaly. Pricing pages change formatting, add/remove trial badges, update copyright years. We needed thresholds that catch real changes while ignoring noise.

After analyzing historical data, we settled on these thresholds:

Here's the actual implementation:

const ANOMALY_THRESHOLDS = {
  PRICE_CHANGE_PERCENT: 50,  // >50% change is suspicious
  MAX_NEW_PLANS: 3,          // More than 3 new plans is suspicious
  MAX_MISSING_PLANS: 2,      // More than 2 missing plans is suspicious
  MIN_CONFIDENCE: 0.5,       // Below this, flag as anomaly
};

4. Confidence Scoring Algorithm

The confidence score quantifies how much we trust an extraction compared to the baseline. It starts at 1.0 (100%) and gets penalized for various issues:

The algorithm in code:

function calculateConfidence(
  baseline: PlanData[],
  current: PlanData[]
): number {
  let confidence = 1.0;

  // Penalize plan count differences
  const planDiff = Math.abs(baseline.length - current.length);
  confidence -= planDiff * 0.10;

  // Penalize missing plans
  const missingPlans = baseline.filter(
    b => !current.some(c => normalizePlanName(c.name) === normalizePlanName(b.name))
  );
  confidence -= missingPlans.length * 0.15;

  // Penalize anomalous price changes
  for (const basePlan of baseline) {
    const match = findMatchingPlan(current, basePlan.name);
    if (match) {
      const change = calculatePriceChange(basePlan.price, match.price);
      if (Math.abs(change) > ANOMALY_THRESHOLDS.PRICE_CHANGE_PERCENT) {
        confidence -= 0.20;
      }
    }
  }

  return Math.max(0, Math.min(1, confidence));
}

Plan Name Normalization

Comparing plan names requires fuzzy matching. "Professional Plan" and "Professional" should match. We normalize by removing common suffixes:

function normalizePlanName(name: string): string {
  return name
    .toLowerCase()
    .replace(/s*(plan|tier|package|edition)s*/gi, '')
    .replace(/s+/g, ' ')
    .trim();
}

// "Professional Plan" → "professional"
// "Enterprise Tier" → "enterprise"
// "Business Package" → "business"

Price Normalization

Different tools quote prices in different periods. We normalize everything to monthly:

function normalizeToMonthly(price: number, period: string): number {
  const periodLower = period.toLowerCase();

  if (periodLower.includes('year') || periodLower.includes('annual')) {
    return price / 12;
  }
  if (periodLower.includes('quarter')) {
    return price / 3;
  }
  if (periodLower.includes('week')) {
    return price * 4.33; // Average weeks per month
  }

  return price; // Assume monthly
}

5. The Admin Review Interface

Speed matters for human review. With 290 tools, even 2-minute reviews would take 9+ hours. We built an admin interface optimized for rapid decision-making:

The interface uses a card-based design where each monitor is expandable:

// Simplified component structure
function BaselineCard({ monitor }: { monitor: Monitor }) {
  const [expanded, setExpanded] = useState(false);

  return (
    <div className="border rounded-lg p-4">
      <div className="flex justify-between items-center">
        <div>
          <h3>{monitor.name}</h3>
          <span className={statusBadge}>{monitor.baselineStatus}</span>
        </div>
        <button onClick={() => setExpanded(!expanded)}>
          {expanded ? 'Collapse' : 'Expand'}
        </button>
      </div>

      {expanded && (
        <>
          <PlansTable plans={monitor.candidateBaseline.plans} />
          <div className="flex gap-2 mt-4">
            <button onClick={handleApprove}>✓ Approve</button>
            <button onClick={handleReject}>✗ Reject</button>
          </div>
        </>
      )}
    </div>
  );
}

6. Database Schema Design

The audit trail is critical for debugging and compliance. We track every review decision:

// Drizzle ORM schema
export const baselineReviews = pgTable("baseline_reviews", {
  id: uuid("id").defaultRandom().primaryKey(),
  snapshotId: uuid("snapshot_id").notNull(),
  monitorId: uuid("monitor_id").notNull(),
  reviewerEmail: varchar("reviewer_email", { length: 255 }).notNull(),
  status: varchar("status", { length: 50 }).notNull(), // approved | rejected
  originalPriceData: jsonb("original_price_data"),
  correctedPriceData: jsonb("corrected_price_data"), // If admin made corrections
  notes: text("notes"),
  reviewedAt: timestamp("reviewed_at").defaultNow(),
});

// Monitor table additions
export const monitors = pgTable("monitors", {
  // ... existing fields
  baselineSnapshotId: uuid("baseline_snapshot_id"),
  baselineStatus: varchar("baseline_status", { length: 50 })
    .default("pending"), // pending | verified | needs_review
});

The correctedPriceData field is key—when an admin rejects and corrects data, we store their corrections. This creates training data for improving the AI extractor.

7. Lessons Learned

What Worked

• Thresholds calibrated from real data: Analyzing 1,900+ snapshots before setting thresholds prevented both false positives and missed changes.
• Speed-optimized admin UI: Pre-loading all context reduced review time from 2+ minutes to 30-60 seconds per tool.
• Audit trail from day one: Every decision is logged, making debugging and pattern analysis possible.

What We'd Do Differently

• Earlier human review: We initially ran 3 months of automated extraction before realizing accuracy issues. Should have started with human review.
• Tighter price normalization: Our first version didn't handle quarterly/weekly periods, causing false positives.
• Batch review mode: Reviewing 290 tools one-by-one is tedious. A "quick approve similar" feature would help.

Key Takeaways

Try It Yourself

The baseline verification system powers SaaS Price Pulse's pricing intelligence. We track 290+ tools with 95%+ accuracy, catching real price changes while filtering false positives.

Start tracking competitor pricing →