Preparing web data for AI and analytics
Models need data that is clean, consistent and traceable
Most of the effort in a data science project goes into preparing data. Web data is especially messy: duplicates, boilerplate, mixed encodings, inconsistent units, layouts that change. A dataset that is ready to use saves weeks and avoids subtle errors in models and dashboards.
Here is what “ready to use” means in practice.
1. Design the schema first
Decide what one record is: a product, a listing, a review, a page. Then list the fields, their types and units. A fixed schema forces consistency across sources and runs, and it is the basis for documentation and validation.
2. Provenance on every row
Every record should carry its source URL and collection timestamp. With provenance you can trace any output back to its origin, audit a surprising result and remove records from a source if needed. Without it, the dataset is a black box.
3. Deduplicate
Duplicates distort training and counts. Remove exact duplicates, then near-duplicates: the same item on different URLs, the same text with trivial differences. Keep a record of what was merged.
4. Normalise
- Units and currencies to agreed standards
- Dates to one format and time zone
- Text to one encoding, with boilerplate such as navigation and cookie banners removed
- Categories mapped to your taxonomy
5. Tag where it helps
Simple tags make datasets easier to use: language, category, content type. Agree them up front. Many teams prefer to run their own models for richer labels on the clean data.
6. Exclude personal data by default
Public pages can contain names, emails, phone numbers and photos of people. Unless your project has a lawful basis to use them, remove them during collection. This simplifies compliance and reduces risk. See public versus personal data.
7. Think about rights
Facts and attributes, such as prices, specifications and listings, are the safest focus. Text, images and other creative content may be protected. Discuss permitted use before collecting it in bulk. This is not legal advice.
8. Choose the right format
- Parquet for analytics and warehouses
- JSON Lines for text and nested records in ML pipelines
- CSV for simple flat tables
- Database loads where you want it ready to query
See choosing a delivery format for more.
9. Document it
Ship every dataset with:
- A data dictionary: field names, types, units, definitions
- Collection notes: sources, dates, method, known gaps
- A sampling report: a random sample for manual review, with quality metrics
Documentation is what lets someone else use the data correctly a year later.
10. Plan refreshes
Most datasets need updating. Decide between incremental updates (only new and changed records) and full refreshes. Version each release so models can be tied to the data they were trained on.
Common uses
- Domain-specific training sets from the sites that define your field
- Evaluation sets built from categorised public pages
- Retrieval indexes of product, policy or specification pages
- Analytics tables that join cleanly with your own data
Getting started
Describe the records you need, the sources and the use. We will propose a schema and deliver a documented sample. See web data for AI and the AI and machine learning industry page, or send us your requirements. You can also email [email protected].