Alternative data from the web for investment research
Nothing here is investment or legal advice. It describes data methods only.
What alternative data is
Alternative data is information outside traditional financial sources, such as filings and market prices, that helps explain how a business is doing. Much of it is published on the public web: what a company charges, how many people it is hiring, how many stores it runs, how its products are rated. Collected consistently, these become signals that move before reported results.
Common web signals
- Prices and promotions across a company’s product range, as a read on pricing power and demand
- Stock levels on its online store or retailers’ sites
- Job postings by function and location, as a read on growth plans. See job postings data.
- Store and outlet counts from public store locators
- Listings for property or vehicles, as market activity indicators. See real estate data.
- Ratings and review counts as a read on product reception
- App rankings and release cadence for consumer apps
- Published rates for banks and lenders
Point-in-time is essential
Research data must reflect what was knowable at the time. If a value is corrected later and the history is overwritten, a backtest will use information nobody had, and the signal will look better than it is.
So collect point-in-time: keep every value with its collection timestamp, never overwrite, and record corrections as new rows. Our financial and company data service works this way by design.
Mapping to entities
Web data arrives under brand names, store names and website domains. Research needs it mapped to companies and their identifiers. Keep the mapping as its own documented table, with dates, because brands are bought and sold and the mapping changes over time.
Backtesting pitfalls
- Short history. Many web signals only exist from the day you start collecting. Plan collection early.
- Coverage changes. If you add sources over time, aggregate signals shift. Track coverage alongside values.
- Survivorship. Products, stores and companies that disappear must stay in history.
- Seasonality. Hiring and pricing are seasonal. Compare like periods.
- Overfitting. Many candidate signals, short history: be strict with out-of-sample tests.
Provenance and compliance
Compliance teams will ask where data came from and how it was collected. Keep:
- A list of sources and what was collected from each
- The method: public pages only, no logins, moderate request rates
- Confirmation that personal data is excluded
- Raw snapshots where agreed, to show what a page displayed
We collect publicly available business information only and document each dataset. Firms should apply their own policies on data sourcing.
Building a signal
Start with a hypothesis: for example, that a retailer’s discount depth predicts its margins. Collect the data that tests it, for long enough to cover a few cycles. Clean, map and validate before modelling. Many of the best signals are simple counts done consistently.
Getting started
Tell us the companies you cover and the signals you want to test, and we will propose sources and a schedule. Send your requirements or email [email protected]. The finance and investment industry page lists more examples.