{"title":"Data Collection Strategies","description":"","content":"Every ML model is only as good as the data it’s trained on.\n\nYou can have a strong architecture, well-tuned hyperparameters, and plenty of compute, but if the training data doesn’t match real-world conditions, performance won’t hold up in production.\n\nData collection isn’t just pulling in logs. It’s a deliberate process that shapes everything downstream, from feature engineering to evaluation.\n\n---\n\n# Types of Training Data\n\nBefore collecting anything, you need to understand what kinds of data exist and what each is good for. Training data for ML systems generally falls into five categories: implicit feedback, explicit feedback, system-generated data, third-party data, and synthetic data.\n\n### Implicit Feedback\n\nImplicit feedback is what users do without being asked. Clicks, page views, scroll depth, dwell time, purchases, add-to-cart actions, search queries, video watch time. This is the bread and butter of most production ML systems because it's abundant and cheap to collect.\n\nYouTube's recommendation system, for example, trains primarily on implicit signals. Watch time is the core signal, but the system also tracks whether a user clicked a video and left within seconds (a negative signal), whether they watched to completion, and whether they shared or added it to a playlist.\n\nThe volume advantage is massive. Netflix collects billions of implicit interaction events per day. Asking users to rate every show they watch would yield maybe a few million ratings per day. Implicit signals outnumber explicit ones by orders of magnitude.\n\nThe downside is noise. A user might click on a video by accident, leave a page open while getting coffee, or scroll past content they actually liked but didn't engage with. Implicit feedback tells you what users did, not what they wanted.\n\n### Explicit Feedback\n\nExplicit feedback is what users tell you directly: star ratings, thumbs up/down, reviews, survey responses, report-as-spam actions. It's higher quality per data point because it captures user intent, not just behavior.\n\nThe problem is volume. Fewer than 5% of users typically leave explicit feedback. This creates a selection bias: users who rate content tend to be either very satisfied or very dissatisfied. The silent majority in the middle is underrepresented. Yelp, for instance, has long dealt with the \"J-curve\" in restaurant ratings, where most reviews cluster at 1 star or 5 stars.\n\nIn practice, production systems combine both. Spotify uses listening history (implicit) alongside thumbs up/down on Discover Weekly (explicit) to train its recommendation models. The implicit data provides coverage and volume, while the explicit data anchors ground truth for evaluation.\n\n### System-Generated Data\n\nNot all training data comes from users. Server logs, latency measurements, error rates, CPU utilization, and network traffic patterns are valuable for models that monitor infrastructure or detect anomalies.\n\nFraud detection systems at companies like Stripe and PayPal rely heavily on system-generated data: transaction metadata, IP addresses, device fingerprints, velocity patterns. These signals are often more predictive than user-provided information because they're harder to fake.\n\n### Third-Party Data\n\nSometimes you need data you can't collect yourself. Weather data for a delivery time prediction model. Demographic data for an ad targeting system. Financial market data for a trading model. Third-party data fills gaps in your own collection.\n\nThe trade-offs are cost, freshness, and control. You're dependent on the provider's quality, update frequency, and continued availability. LinkedIn's economic graph combines internal professional data with third-party company information (funding rounds, headcount) to power features like job recommendations.\n\n### Synthetic Data\n\nWhen real data is scarce, expensive, or privacy-constrained, synthetic data offers an alternative. This is artificially generated data that mimics the statistical properties of real data.\n\nWaymo generates millions of synthetic driving scenarios to train autonomous vehicle models, covering edge cases (pedestrian jaywalking in rain at night) that would take years to encounter in real driving. Tesla uses a similar approach for rare crash scenarios.\n\nSynthetic data works well for augmenting underrepresented classes, pre-training before fine-tuning on real data, and testing system robustness. It works poorly when the synthetic distribution diverges from real-world distribution, which is the fundamental risk.\n\nThe following table summarizes the trade-offs across data types:\n\n\n| Data Type | Volume | Quality per Point | Cost | Bias Risk | Best For |\n|-----------|--------|--------------------|------|-----------|----------|\n| Implicit feedback | Very high | Low-medium | Low | Noisy, position bias | Recommendations, ranking, search |\n| Explicit feedback | Low | High | Low | Selection bias (vocal minorities) | Evaluation, ground truth anchoring |\n| System-generated | High | Medium | Low | Instrumentation bias | Anomaly detection, fraud, monitoring |\n| Third-party | Varies | Varies | Medium-high | Provider's collection bias | Filling data gaps, enrichment |\n| Synthetic | Unlimited | Low-medium | Medium (compute) | Distribution mismatch | Rare events, privacy-sensitive domains, pre-training |\n\n\n\n```mermaid\nflowchart TD\n A[Training Data
Sources]:::primary --> B[User-Generated]:::teal\n A --> C[System-Generated]:::teal\n A --> D[External]:::teal\n\n B --> E[Implicit Feedback
clicks, views,
watch time]:::orange\n B --> F[Explicit Feedback
ratings, reviews,
thumbs up/down]:::orange\n\n C --> G[Server Logs
latency, errors,
throughput]:::orange\n C --> H[Application Events
transactions,
API calls]:::orange\n\n D --> I[Third-Party Data
purchased datasets,
public APIs]:::orange\n D --> J[Synthetic Data
generated,
augmented]:::orange\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n```\n\n\n---\n\n# Designing a Logging Infrastructure\n\nKnowing what data types exist is one thing. Actually capturing them reliably at scale is a different engineering problem entirely. A poorly designed logging system creates gaps, inconsistencies, and schema drift that poison downstream models.\n\n### What to Log\n\nThe instinct is to log everything. Don't. Logging everything is expensive, creates storage headaches, and most of it becomes noise. Instead, start from your ML problem formulation and work backward. If you're building a search ranking model, you need queries, impressions, clicks, and click positions. You don't need every CSS rendering event.\n\nA useful framework is to think about logging at three levels:\n\n\n| Level | Examples | Typical Volume | Use Case |\n|-------|----------|----------------|----------|\n| User actions | Clicks, searches, purchases, page views | 10K-1M events/sec | Training labels, feature engineering |\n| Context | Device type, location, time of day, session ID | Attached to each action | Contextual features, segmentation |\n| System state | Model version, experiment ID, ranked candidates | Attached to each serving request | Offline evaluation, debugging |\n\n\nThe system state level is easy to forget but critical. If you don't log which model version served a prediction and what candidates were ranked, you can't do counterfactual evaluation later. Pinterest logs the full ranked list (not just what the user saw) for every recommendation request, which enables offline experiments comparing new ranking models against the production model's logged decisions.\n\n### Event Schema Design\n\nA well-designed event schema prevents months of data cleaning later. Every event should capture a consistent set of fields that make it joinable with other events and usable for ML.\n\nA typical event schema looks like this:\n\n\n```python\n{\n \"event_id\": \"uuid-v4\",\n \"event_type\": \"video_watch\",\n \"timestamp\": \"2025-03-15T14:23:01.456Z\",\n \"user_id\": \"u_abc123\",\n \"session_id\": \"s_def456\",\n \"item_id\": \"v_789xyz\",\n \"properties\": {\n \"watch_duration_sec\": 145,\n \"video_length_sec\": 600,\n \"completion_rate\": 0.24,\n \"source\": \"home_feed\",\n \"position\": 3\n },\n \"context\": {\n \"device\": \"mobile_ios\",\n \"app_version\": \"4.2.1\",\n \"country\": \"US\",\n \"experiment_ids\": [\"exp_ranking_v2\", \"exp_ui_test_7\"]\n },\n \"model_context\": {\n \"model_version\": \"ranking_v2.3\",\n \"prediction_score\": 0.82,\n \"candidates_shown\": 20\n }\n}\n```\n\n\nA few principles that save pain down the road:\n\n- **Use a schema registry.** Enforce schema validation at write time, not read time. If an event doesn't match the schema, reject it immediately. Catching a missing field six months later when you're trying to train a model is far more expensive.\n- **Version your schemas.** Data changes over time. New features get added, old ones get deprecated. Schema versioning lets you handle migrations gracefully.\n- **Include experiment context.** Attach experiment and model version IDs to every event. Without this, you can't attribute outcomes to specific model versions or A/B test variants.\n- **Use consistent identifiers.** User IDs, session IDs, and item IDs should be consistent across all event types. This sounds obvious, but in practice, different teams often use different ID formats for the same entity.\n\n### Logging Architecture\n\nThe path from user action to training data involves several components, each with its own reliability and latency requirements.\n\n\n```mermaid\nflowchart LR\n A[Client App]:::primary --> B[Event
Collector]:::teal\n B --> C[Message
Queue
e.g. Kafka]:::orange\n C --> D[Stream
Processor
e.g. Flink]:::teal\n C --> E[Batch
Pipeline
e.g. Spark]:::teal\n D --> F[Real-Time
Feature Store]:::green\n E --> G[Data
Warehouse
e.g. BigQuery]:::green\n G --> H[Training
Pipeline]:::primary\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\nThe event collector sits between the client and the rest of the pipeline. Its job is to accept events at high throughput, validate schemas, and forward them. Companies like Uber use a custom event collection service that handles 1M+ events per second.\n\nThe message queue (typically Kafka) decouples producers from consumers and provides durability. If the downstream processors go down temporarily, events are buffered rather than lost. This is important because lost training data can introduce selection bias: if your logging system drops events during peak traffic, your model underrepresents high-traffic scenarios.\n\nFrom there, events flow into either a stream processor for real-time features or a batch pipeline that loads data into a warehouse for training. The training pipeline pulls from the warehouse, not from the raw event stream, because training needs stable, consistent snapshots of data.\n\n### Client-Side vs Server-Side Logging\n\nWhere you place the logging instrumentation matters more than you might expect.\n\n\n| Aspect | Client-Side Logging | Server-Side Logging |\n|--------|---------------------|---------------------|\n| What it captures | UI interactions, viewport, scroll, dwell time | API calls, served predictions, response data |\n| Reliability | Can be lost (app crash, network issues, ad blockers) | Highly reliable |\n| Completeness | Captures what the user actually experienced | Captures what the server sent, not what user saw |\n| Privacy exposure | Collects data on user device first | Data stays within backend |\n| Latency | Batched, delayed delivery | Near-instant |\n\n\nIn practice, you need both. Server-side logging gives you reliable records of what was served and predicted. Client-side logging tells you what actually happened in the user's experience: did they see the recommendation, how long did they look at it, did they scroll past it?\n\nAirbnb logs search results server-side (which listings were ranked and in what order) and engagement client-side (which listings the user viewed, hovered over, or clicked). Joining these two streams creates the full picture needed for training a search ranking model.\n\n---\n\n# The Cold Start Problem\n\nEvery ML system faces the same chicken-and-egg problem at launch: you need data to train a model, but you need a model (or at least a product) to collect data. This isn't just a startup problem. It resurfaces every time you launch a new feature, enter a new market, or onboard a new user segment.\n\n### Bootstrapping Strategies\n\nThere's no single solution to cold start. The right approach depends on how much you can afford to be wrong early on and how quickly you need the model to improve.\n\n\n```mermaid\nflowchart TD\n A[Cold Start:
No Training Data]:::red --> B{Related data
available?}:::primary\n B -->|Yes| C[Transfer Learning
Fine-tune pretrained
model on your domain]:::green\n B -->|No| D{Can you serve
random/diverse
content?}:::primary\n D -->|Yes| E[Exploration Strategy
Bandits, random
serving, epsilon-greedy]:::green\n D -->|No| F{Domain experts
available?}:::primary\n F -->|Yes| G[Rule-Based System
Hand-crafted rules
that generate labels]:::green\n F -->|No| H[Public/Open-Source
Datasets
Pretrain, then adapt]:::green\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n**Transfer learning** works when a related domain has abundant data. A new e-commerce site in Brazil can fine-tune a product recommendation model pretrained on a similar US marketplace. The model won't be perfect out of the gate, but it will outperform random recommendations on day one. GPT-style foundation models have made this even more accessible: fine-tuning a pretrained language model on a small domain-specific dataset is now a standard approach for NLP tasks.\n\n**Rule-based systems as data generators** is an underrated strategy. Before Uber had enough ride data to train an ETA prediction model, they used simple heuristics (distance / average speed for the region) to provide initial estimates. These heuristic predictions, paired with actual trip durations, became the first training set for the ML model. The key insight is that the rules don't need to be accurate. They need to be good enough to keep users engaged while the system collects real data.\n\n**Exploration strategies** like epsilon-greedy or Thompson sampling intentionally show users diverse content rather than always exploiting the current best guess. This sacrifices short-term quality for long-term data diversity. DoorDash uses exploration in new markets to collect data on restaurant preferences across different neighborhoods, even if it means occasionally recommending a restaurant that wouldn't be the top choice.\n\n**Public and open-source datasets** provide a starting point for pre-training. ImageNet for vision tasks, Common Crawl for language tasks, MovieLens for recommendation experiments. The model won't work perfectly on your specific domain, but it gives you a reasonable initialization to fine-tune once your own data starts flowing.\n\n### How Long Does Cold Start Last?\n\nIt depends on traffic volume and the complexity of the task. A simple click-through rate model might need a few hundred thousand impressions to become useful, which a moderate-traffic site can collect in days. A personalized recommendation system that needs to learn individual user preferences might take weeks or months to reach acceptable quality for long-tail users.\n\nThe practical answer: plan for 2-4 weeks of degraded model quality after launch. Set explicit thresholds (e.g., \"switch from rules to ML when we have 500K labeled examples\") and monitor the transition closely.\n\n---\n\n# Privacy and Compliance\n\nData collection doesn't happen in a vacuum. Regulations like GDPR (EU), CCPA (California), and LGPD (Brazil) impose strict rules on what you can collect, how long you can keep it, and what you must do when users request deletion.\n\nThis isn't a legal formality. These regulations have direct engineering consequences for ML systems.\n\n### Key Regulations at a Glance\n\n\n| Aspect | GDPR (EU) | CCPA (California) | LGPD (Brazil) |\n|--------|-----------|-------------------|----------------|\n| Scope | EU residents' data | California residents' data | Brazilian residents' data |\n| Consent required? | Yes, opt-in (explicit) | No, but opt-out right | Yes, opt-in |\n| Right to deletion? | Yes (\"right to be forgotten\") | Yes | Yes |\n| Data portability? | Yes | Yes | Yes |\n| Penalties | Up to 4% of global revenue | $2,500-$7,500 per violation | Up to 2% of revenue in Brazil |\n| Impact on ML | Must delete training data on request | Must disclose data usage | Must have legal basis for processing |\n\n\n### Engineering Implications\n\n**Right to deletion** is the hardest one for ML systems. When a user requests their data be deleted, you must remove their data from your databases, your data warehouse, your feature store, and potentially from models trained on their data. Retraining a model from scratch without one user's data is often impractical at scale.\n\nThe common engineering solution is to design for deletion from the start:\n\n- Store training data with user IDs that can be filtered out during retraining.\n- Schedule regular model retraining (daily or weekly) so that deleted user data naturally cycles out of the model within a bounded time window.\n- Use differential privacy techniques to limit how much any single user's data can influence the model, making individual deletion less critical.\n\n**Consent management** affects what data you can collect in the first place. Under GDPR, you need explicit consent for each category of data processing. A user who consents to their click data being used for \"improving search results\" hasn't consented to that same data being used for \"targeted advertising.\" Your logging infrastructure needs to tag events with the consent categories they fall under.\n\n**Data retention policies** set time limits on how long you can keep data. If your policy says 12 months, your training pipeline must exclude data older than 12 months. This is actually beneficial for model quality in many cases, since very old interaction data often reflects outdated user preferences and product states.\n\n**Anonymization vs pseudonymization.** Anonymization removes all identifying information and is irreversible. Pseudonymization replaces identifiers with tokens that can be reversed with a key. GDPR treats these differently: truly anonymized data falls outside GDPR's scope entirely, while pseudonymized data is still personal data. For ML training, pseudonymization is usually sufficient and more practical, since you still need to join events by user for feature engineering.\n\n---\n\n# Building a Data Flywheel\n\nThe most powerful advantage an ML system can have isn't a better algorithm. It's a self-reinforcing cycle where the product generates the data that makes the model better, which makes the product better, which attracts more users and generates more data.\n\n\n```mermaid\nflowchart LR\n A[More Users]:::primary --> B[More
Interaction
Data]:::teal\n B --> C[Better Model
Training]:::orange\n C --> D[Better
Predictions]:::green\n D --> E[Better User
Experience]:::primary\n E --> A\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef teal fill:#38d9a9,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n```\n\n\n### How Flywheels Work in Practice\n\nGoogle Search is the canonical example. More users means more search queries. More queries means better understanding of what results are relevant (through click patterns). Better relevance means users prefer Google over alternatives, which brings more users. Google processes over 8.5 billion searches per day, and every single one generates training signal.\n\nSpotify's Discover Weekly follows the same pattern. User listening data trains the recommendation model. Better recommendations increase listening time. More listening time generates richer preference signals. This creates a compounding advantage: the model gets better for each user as they listen more, and better across all users as the user base grows.\n\n### Accelerating the Flywheel\n\nNot all data is equally valuable. A few strategies help the flywheel spin faster:\n\n- **Active learning.** Instead of passively waiting for users to interact with whatever the model serves, actively select which items to show based on where the model is most uncertain. This collects maximally informative data points rather than redundant ones. A content moderation model, for example, can prioritize borderline cases for human review rather than cases the model is already confident about.\n- **Implicit feedback loops.** Design the product to naturally generate useful training signals. YouTube's \"Not Interested\" button doesn't just improve the immediate experience. It creates explicit negative labels that are rare and valuable for training.\n- **Multi-task data sharing.** Data collected for one model can often train another. Uber's ride completion data trains both ETA prediction and dynamic pricing models. This multiplies the value of each data point.\n\n### When Flywheels Stall\n\nFlywheels can stall for several reasons:\n\n#### **Feedback loop bias**\n\nThe model only learns from data it generates. If a recommendation model never shows certain types of content, it never learns whether users would like that content. Over time, the model becomes increasingly confident in a narrow slice of the content space and increasingly ignorant about everything else. This is the \"filter bubble\" problem.\n\n#### **Diminishing returns**\n\nAfter a certain scale, more data provides marginal improvement. Going from 1M to 10M training examples might increase accuracy by 5%. Going from 100M to 1B might increase it by 0.1%. The flywheel still turns, but the returns per revolution shrink.\n\n#### **Data quality degradation**\n\nAs a platform grows, the proportion of spam, bot traffic, and adversarial behavior tends to increase. If you don't invest in data quality alongside data quantity, the flywheel can actually hurt model quality. This is why data validation (covered in a later chapter) is essential infrastructure.\n\nThe fix for stalling flywheels is usually deliberate exploration: introducing randomization or diversity into what the model serves, even at the cost of short-term engagement. Twitter/X's \"For You\" feed deliberately mixes in content from outside a user's typical engagement pattern to prevent the recommendation model from collapsing into a narrow echo chamber.\n\n\n```mermaid\nflowchart TD\n A[Healthy Flywheel]:::green --> B[More Data
Collected]:::primary\n B --> C{Data Quality
Maintained?}:::orange\n C -->|Yes| D[Model
Improves]:::green\n C -->|No| E[Model Quality
Degrades]:::red\n D --> F[Better
Predictions]:::green\n E --> G[Worse User
Experience]:::red\n F --> H[More User
Engagement]:::primary\n G --> I[Users Leave
Flywheel Stalls]:::red\n H --> A\n\n classDef primary fill:#00ceff,stroke:#000,color:#000\n classDef orange fill:#ffa94d,stroke:#000,color:#000\n classDef green fill:#69db7c,stroke:#000,color:#000\n classDef red fill:#ff8787,stroke:#000,color:#000\n```\n\n\n---\n\n# Quiz","pageType":"ml-system-design"}

Data Collection Strategies

Get Premium