Data Collection

Data Collection is the first process of Phase 2 and marks the transition from planning to execution. This process involves identifying all potential data sources, securing access, and establishing the mechanisms to gather data for analysis and model development.

The quality and completeness of data collection directly impacts every subsequent phase. Poor data collection leads to poor models — no amount of sophisticated algorithms can compensate for missing or inadequate data.

This process produces a Data Inventory that catalogs all available data sources and a Data Access Plan that ensures the team can reliably retrieve data throughout the project.

ML projects typically draw from multiple data source types. Identify all relevant sources for your project:

• 1
  Data Source Identification
  Map all potential data sources based on the problem framing from Phase 1. Consider both obvious and non-obvious sources that might contain useful signals.
• 2
  Access Request & Provisioning
  Request access to identified data sources. Work with data owners, IT, and security teams to obtain necessary permissions and credentials.
• 3
  Data Extraction Setup
  Establish extraction mechanisms: database connections, API integrations, file transfer protocols, or streaming pipelines as appropriate.
• 4
  Initial Data Pull
  Execute initial data extraction to verify access works and get representative samples. Document any issues or limitations encountered.
• 5
  Data Inventory Creation
  Document all collected data sources with metadata: source, format, volume, refresh frequency, access method, and data owner.
• 6
  Storage & Organization
  Set up appropriate storage infrastructure (data lake, staging area) and organize collected data with clear naming conventions and folder structures.

Different data sources require different collection approaches. Match the method to your source and requirements:

Create a comprehensive inventory for each data source. This becomes a critical reference document throughout the project:

Ensure you have addressed all access requirements before considering data collection complete:

• ✓
  Start Early with Access Requests
  Data access often takes longer than expected. Begin the approval process immediately and in parallel with other activities.
• ✓
  Document Everything
  Future team members will need to understand and replicate your data collection. Document sources, methods, and any quirks discovered.
• ✓
  Validate Early and Often
  Don't wait until you have all data to start validation. Check samples early to catch issues before investing significant effort.
• ✓
  Think About Reproducibility
  Design collection processes that can be repeated reliably. Avoid one-off manual steps that can't be automated.
• ✓
  Consider Production Needs
  Think ahead to deployment. Can the same data be accessed in production? At what latency? Plan for production from the start.