← Back

Fraud Detection V2.0: Industrialization of Deception

·Bryan Lai
TechnicalEngineeringDesignFraud

Modern Fraud Systems Cheatsheet

Note: Refer to the Glossary at the end of the article for understanding unfamiliar terms.

Fraud can now be pumped out like a factory. Attackers have become more sophisticated, deploying agentic swarms to attack companies every millisecond. Fighting Fraud is an Infinite Game.

From 2026 onwards, we will be fighting more Synthetic Identity Fraud that has existed for 30 years, but now with deepfakes that bypass Liveness Detection in real-time. But there is hope, we can improve the system by improving Recall @ 1% FPR (False Positive Rate) and a Custom Financial Reward Function over vanity metrics.

High-Level Overview (Orchestration Layer)

We move from identity verification to behavioral systems. We don't just ask "Who are you?" We need to ask "How do you hold your phone?" and "Is your camera feed actually a virtual driver?"

Layer 1: The Gateway (KYC (Know Your Customer) / KYB (Know Your Business) & Deep Metadata)

Simple KYC (Know Your Customer) and KYB (Know Your Business) are no longer sufficient because GenAI manufactures faces and voices with ease. Visuals are now a liability. Let's look at how we secure this:

  • Deep Metadata Forensics: GenAI can't yet perfectly fake the hardware-level "noise" or EXIF data of a real camera sensor.
    • Example: We use a library like ExifTool to scan a .jpg. If the Software tag says "Snap Camera" or the "Camera Model" is null, we flag the session. It is likely a virtual driver, not a human holding a phone.
  • Synthetic Identity Surveillance: This is the "Long Con." Synthetic Identity Fraud accounts for ~85% of ID fraud. We watch Credit Builder Loans because fraudsters "cook" these IDs for years before the Bust-out Fraud exit scam.
    • Example: An applicant uses a real SSN (Social Security Number) belonging to a deceased child but has a clean 750 credit score built over 5 years. We check for impossible gaps in the "birth-to-credit" timeline.
  • Consortium Data: Attackers hit 10 fintechs in 10 minutes. We can use a shared industry "blacklist."
    • Example: If a device ID was caught doing Card Testing at Stripe 10 minutes ago, we can block their application instantly using shared Consortium Data.

Layer 2: Real-time Orchestration & Plumbing

If the math in the lab isn't the math in the app, you’re essentially funding the attacker’s next vacation. We move from Lambda Architecture (split paths) to Kappa Architecture (unified streams).

  • The Problem: Understanding Training-Serving Skew. If your model was trained on cleaned data but receives raw API feeds, it will misfire.
    • Example: Training uses avg_spend rounded to 2 decimals, but live inference receives 4. That tiny shift can cause a False Positive Ratio storm where we block thousands of good users.
  • The Fix: We use Stateful Stream Processing (like Apache Flink) to maintain Tecton Feature Store Guide or Online/Offline Parity.
    • Example: Logic for Velocity Features like login_attempts_3m is defined once in a feature store. The same code runs on the Kafka stream (online) and the data lake (offline). No re-coding in SQL.
  • The Metric: Data Freshness Metrics: Ensuring Insights. In an ATO (Account Takeover), seconds are the difference between a secure account and an empty bank balance.
    • Example: If failed_login_count has 10-second latency, an attacker scripts 100 attempts before the Orchestration Layer knows to trip the Circuit Breaker.

Layer 3: The Decision Engine (GNN (Graph Neural Network) & Explainability)

We stop looking at users in isolation and move to the Heterogeneous Graph.

  • Inductive over Transductive Learning: We use Inductive Learning. Transductive models only know nodes seen during training. Fraudsters are always "new." Inductive models let us score a brand-new user based on their relationship to existing bad actors via Message Passing.
    • Example: New User A connects to Device X. Device X was used by a Money Mule last week. Risk "leaks" through the graph, flagging A before they finish signing up.
  • Advanced Features: We move beyond basic totals to Behavioral Biometrics and Behavioral Entropy.
    • Example: time_between_keystrokes_variance (Bots type at 0 variance; humans have rhythm) or count_distinct_users_on_device_1h (High count = Device Farm).
  • The Graph UI: AI is great, but analysts need to see the "why." We provide a visual UI of the Heterogeneous Graph connections to show "Guilt by Association."
    • Example: An analyst looks at a high-risk alert and sees a "Spider-Web" cluster: "User 1" and "User 2" have different names but share the same browser_fingerprint and physical_address. That's a fraud ring, not a coincidence.
  • Safety: We use a Circuit Breaker. If the GNN (Graph Neural Network) service lags, the system falls back to simple hard-coded rules to keep the bank running.

Layer 4: Action, Operations & Compliance

Every decision has a legal tail. We use a three-tier logic to manage the Uncertainty Band.

TierRisk ScoreAction
Green0–20Auto-Approve
Yellow21–79Uncertainty Band: Inject Friction (Liveness / SMS / Queue)
Red80+Auto-Block (SAR Filed)
  • Ops: The Manual Review Queue is our Active Learning gold mine. Analyst decisions feed directly back into the training set. We measure Inter-rater Reliability (IRR) to ensure they are consistent.
  • Legal: For rejections, we log a Reasoning Trace to generate the CFPB (Consumer Financial Protection Bureau) Circular 2022-03: Adverse Action Requirements. We never say "Denied due to risk score." We say "Denied due to high application volume on this phone number."
  • Compliance: If we detect laundering, we file a SAR (Suspicious Activity Report) to FinCEN (Financial Crimes Enforcement Network). We follow BSA (Bank Secrecy Act) mandates to avoid a Consent Order—the slow-motion car crash of a company.

Layer 5: The MLOps Feedback Loop

The system stays alive here. We solve Point-in-Time Correctness by joining labels to features as they existed at the microsecond of the event.

  • Drift: We monitor the PSI (Population Stability Index). If the "Actual" scores look nothing like the "Expected" scores, we have Concept Drift.
  • Evaluation: We look at AUPRC (Area Under the Precision-Recall Curve) and Brier Score.
  • Reliability:
    • Calibration (Platt Scaling): Ensures the model's math matches reality. If the model says 80%, and reality shows 20%, the math is lying.
    • Cohen’s Kappa: Ensures the humans provide consistent labels. You use this to make sure the data you use for the calibration is actually high-quality.
  • The "Time" Fix: Real fraud labels take 90 days. We use Proxy Labels (e.g., "Account blocked by manual review") as immediate signals to retrain next week, not next quarter.
  • Testing: We use Shadow Deployment to run new models in "ghost mode" before they go live.

Connecting it All Together

Ultimately, F1-Score doesn't pay the bills. We optimize for a custom Financial Reward Function:

Profit = (Value of Stopped Fraud) - (Cost of False Positives * Customer LTV) - (Compute Cost)

A model with lower technical accuracy but higher financial profit is the winner. Every part of the stack must protect the Chargeback Ratio.

Real-World Example: Instant Cash Product Protection

For a product like Instant Cash, look out for First-Party Fraud (friendly fraud). Users borrow money they never intend to repay. We use Benford’s Law to check if their declared income digits follow natural math patterns. We apply Cost-Weighted Loss to ensure the model cares 10x more about missing a $1,000 Bust-out Fraud than a $100 "oops" default. Every payout uses an Idempotency Key to prevent double-spending, and if a user is being tricked via an APP (Authorized Push Payment) Scam or BEC (Business Email Compromise), our system injects a "Break the Spell" intervention.

The Glossary: Formulas & Definitions

1. Population Stability Index (PSI)

Measures how much the distribution of your risk scores has shifted over time. PSI=∑((Actual%−Expected%)×ln⁡(Actual%Expected%))PSI = \sum ((Actual\% - Expected\%) \times \ln(\frac{Actual\%}{Expected\%}))

  • PSI < 0.1: No shift.
  • PSI > 0.25: Major shift; retrain immediately.

2. Recall @ 1% FPR (False Positive Rate)

Recall=TruePositivesTruePositives+FalseNegativesRecall = \frac{True Positives}{True Positives + False Negatives} FPR=FalsePositivesFalsePositives+TrueNegativesFPR = \frac{False Positives}{False Positives + True Negatives}

3. Brier Score (Calibration / MSE)

Brier=1N∑t=1N(ft−ot)2Brier = \frac{1}{N} \sum_{t=1}^{N} (f_t - o_t)^2 0 = Perfect accuracy; 1 = Perfect wrongness.

4. Cohen’s Kappa (κ\kappa)

Measures agreement between human analysts. κ=po−pe1−pe\kappa = \frac{p_o - p_e}{1 - p_e}

  • pop_o: Observed agreement.
  • pep_e: Expected agreement by chance.
  • Note: If κ<0.6\kappa < 0.6, your manual review is a coin toss.

Definitions

  • Point-in-Time Correctness: Traveling back in time to see exactly what a feature looked like when a transaction happened.
  • Training-Serving Skew: When your model's "lab environment" doesn't match the "real world."
  • Feature Freshness: The time it takes for an action (event) to become a usable feature.
  • Inductive Learning: Predicting risk for entities the model has never seen before by looking at neighborhood relationships.
  • Transductive Learning: Predicting risk only for specific entities seen during training (useless for new attackers).
  • Heterogeneous Graph: A map of different data types (User connected to IP connected to Device).
  • Message Passing: How risk "leaks" from one node to its neighbors in a graph.
  • Exit Scam / Bust-out Fraud: Building trust and high credit over years just to steal the max limit once and disappear.
  • Synthetic Identity Fraud: Frankenstein IDs made of real and fake data.
  • Benford’s Law: The math rule that leading digits in real data follow a specific curve. If they are uniform, they are faked.
  • Idempotency Key: Ensures "Exactly Once" execution. A unique ID that ensures a transaction happens only once, even if the button is clicked twice.
  • APP (Authorized Push Payment) Scam: Tricking a user into sending their own money to a criminal.
  • Adverse Action Reason: The legal reason why you said "No" to a customer.
  • SAR (Suspicious Activity Report): A legal report filed when you detect potential money laundering (AML).
  • Disparate Impact: When an algorithm unintentionally discriminates against a protected group.
  • Reasoning Trace: A detailed log of the logic that led to a specific decision.
  • Account Takeover (ATO): When an attacker hijacks a legitimate user's session or credentials.
  • Card Testing: Automated attempts to check if stolen credit cards are active.
  • First-Party Fraud: When a real customer lies to get a refund or free loan.
  • Money Mule: Someone used to transfer and "wash" stolen funds.
  • Orchestration Layer: The brain that coordinates between vendors, models, and actions.
  • Circuit Breaker: A safety switch that stops a failing ML service to save the whole system.
  • Shadow Deployment: Running a model in the background to test it without risk.
  • AUPRC: The area under the precision-recall curve; the best metric for rare events like fraud.
  • Concept Drift: When fraudsters change their behavior to defeat your current model.
  • Calibration: Forcing model scores to represent actual real-world probability.
  • Platt Scaling: A method to calibrate model outputs into meaningful probabilities.
  • Liveness Detection: Proving a face is a real person, not a mask or recording.
  • Velocity Features: Counts over time (e.g., 'Logins in 5 mins').
  • Consortium Data: Risk intel shared across many companies.
  • KYC (Know Your Customer) / KYB (Know Your Business): Legal identity checks.
  • AML (Anti-Money Laundering): Systems to stop the movement of illegal funds.
  • Manual Review Queue: The desk where humans check "gray area" cases.
  • False Positive Ratio: How many "good guys" you block for every "bad guy" caught.
  • Chargeback Ratio: Your % of disputed sales.
  • Lambda/Kappa Architecture: Frameworks for data processing speeds.
  • Stateful Stream Processing: Computations that "remember" previous events.
  • Online/Offline Parity: Ensuring the data logic is identical in the lab and production.
  • Cost-Weighted Loss: Penalizing expensive errors (missing $1k) more than small ones.
  • Inter-rater Reliability (IRR): The degree of agreement among independent analysts.
  • Behavioral Entropy: The randomness in human interaction. Bots have low entropy; humans have high entropy.
  • Uncertainty Band: The "Yellow Zone" (Risk Score 21–79) where high-friction tools and humans live.
  • FinCEN (Financial Crimes Enforcement Network): The US Treasury bureau that collects and analyzes SARs.
  • BEC (Business Email Compromise): Fraud where an attacker hijacks business email accounts to redirect payments.
  • BSA (Bank Secrecy Act): The US law requiring financial institutions to assist government agencies in detecting money laundering.
  • Consent Order: A legal agreement where a company agrees to stop illegal practices and accept government monitoring to avoid a lawsuit.
References & Further Reading