CSV, JSON or API: choosing how to receive scraped data

· 3 min read · CrawlMine guides

Format follows the person using the data

The right delivery format is the one that lands the data where it gets used, with the least work. An analyst who lives in spreadsheets wants a file. A pricing engine wants an API. A data team wants tables in its warehouse. Start from who opens the data and how often.

CSV and Excel

CSV is the most common choice and for good reason. Every tool opens it, it is easy to inspect, and it loads into almost any database. It suits flat data: one row per product, listing or posting.

Excel suits teams who will read the data directly and want formatting, several sheets or a summary tab. It is less suited to automated loading.

Watch for: commas inside values (they need quoting), number formats (thousands separators can turn numbers into text), and dates. Agree a date format, ideally ISO such as 2026-01-10, and plain numbers without separators for machine use.

JSON and JSON Lines

JSON handles nested data well: a product with several variants, each with its own price and stock, or a listing with a list of amenities. Developers prefer it because it maps directly to objects in code.

JSON Lines puts one JSON record per line. It streams well and is the usual choice for large datasets and machine learning pipelines.

Parquet

Parquet is a column-based file format used by data warehouses and analytics engines. It is compact, typed and fast to query. For large recurring datasets going into a warehouse or a data lake, it is often the best option. See web data for AI for how we deliver datasets to data teams.

Cloud storage drops

Instead of emailing files, many teams receive them in a cloud storage bucket they control. Each run adds a dated file. Their own systems pick it up on a schedule. This is simple, reliable and keeps a history of every delivery.

API

An API lets your systems ask for data when they need it. It suits applications: a repricing service, a product feature, an internal dashboard. Good data APIs support incremental queries, for example “give me rows changed since yesterday”, so loads stay small.

Our custom web data API serves your feed through documented endpoints, with run status so your systems know when fresh data is ready.

Webhooks

A webhook is a message we send to your system when something happens, such as a run completing. It removes the need to poll. Often combined with an API or storage drop.

What to agree before the first run

Whatever format you choose, agree these in writing:

  • Schema. Column names, types and units. Who can change them, and how changes are announced.
  • Keys. Which field identifies a record across runs, so you can track changes.
  • Timestamps. Collection time on every row, with a time zone.
  • Empty values. How missing data is shown: empty, null or a marker.
  • Full or incremental. Does each delivery contain everything, or only what changed?
  • Delivery time. When the file or API update is ready each day.

These small decisions prevent most integration problems. Our guide to data quality checks covers what should be checked before each delivery.

Combining formats

Many teams want more than one: a daily CSV for analysts and an API for the product. Both should come from the same pipeline and schema so numbers always match.

A quick decision guide

Who uses it Good choice
Analysts in spreadsheets CSV or Excel, emailed or in shared storage
BI dashboards CSV or Parquet in storage, or direct database load
Data warehouse Parquet or CSV in storage, incremental
Applications API with incremental queries, plus webhooks
Machine learning JSON Lines or Parquet with a data dictionary

If you are not sure, start with CSV in cloud storage. It is easy to change later. Tell us how your team works in the quote form or at [email protected] and we will suggest a setup.

Talk to us directly

[email protected]

Send the websites and fields you need. A person on our team reads every message.

Email us

Send us the sites tonight. Data by morning.

Tell us which websites and fields you need. We confirm what is practical, how it should be delivered and what it costs.

  • Goes straight to [email protected]
  • Reply within one working day
  • A sample or pilot before any recurring plan
  • NDA available before you share details

[email protected]