Historical Data Sources
Where Do You Find Reliable Historical Data for Backtesting?
Historical data is the foundation of backtesting. Garbage data produces garbage results, no matter how rigorous your analysis. The challenge isn't scarcity—abundant data sources exist for equities, futures, forex, and cryptocurrencies. The challenge is quality. Many sources contain gaps, errors, survivorship bias, or adjusted prices that distort backtests. Understanding where to source data, how to evaluate its quality, and which pitfalls to avoid is essential for building confidence in your backtest results.
A backtest is only as good as the data it uses. If your historical data is missing transactions on important days, miscalculates corporate actions like splits and dividends, or excludes delisted companies, your backtest results won't represent reality. Professional traders often use multiple data sources to cross-check, automate data quality checks, and maintain detailed audit trails of which data version was used for each backtest.
Quick definition: Market data history is historical prices (open, high, low, close) and volume for financial instruments, typically sourced from exchanges, brokers, or data vendors. High-quality data is accurate, complete (no gaps), adjusted for corporate actions, and representative of actual tradeable prices.
Key takeaways
- Free data sources (Yahoo Finance, Quandl, Alpha Vantage) are convenient but often have gaps, errors, or limited history
- Paid professional data vendors (Bloomberg, FactSet, TradeStation data) cost $500–$3,000+ monthly but offer quality assurance and comprehensive coverage
- Backtesting with insufficient data (fewer than five years) carries high overfitting risk; aim for 10–20 years minimum
- Survivorship bias (excluding delisted companies and bankrupt firms) inflates equity backtest returns by 1–3% annually
- Always verify data quality by checking for gaps, extreme outliers, and comparing prices across independent sources
Free data sources and their trade-offs
Yahoo Finance is the most accessible option for equities and some forex. It offers decades of historical data, splits and dividend adjustments, and zero cost. However, Yahoo Finance sometimes has gaps around corporate actions, occasionally delays data updates, and sometimes backfills historical data when corporate actions are corrected. For casual backtesting on major equities, it's acceptable. For professional trading, it's a starting point, not a final source.
Quandl aggregates data from multiple free and paid sources. It offers equities, futures, forex, and crypto data. The free tier covers major instruments but with limited depth (e.g., only a few years for some crypto data). The paid tiers ($100–$500 monthly) unlock deeper history and proprietary datasets. Quandl's advantage is ease of API access for automated backtesting pipelines.
Alpha Vantage provides free equities and forex data via API, with one-minute to daily bars. The free tier is rate-limited (five calls per minute), making bulk data pulls slow. It's useful for rapid prototyping but impractical for large-scale backtesting.
CBOE and exchange websites directly offer some free data. The CBOE publishes VIX history; futures exchanges (CME, ICE) sometimes offer limited free history. These are reliable but usually require manual downloads and lack standardized formats.
The core limitation of free data: it's often delayed, incomplete, or maintained without quality guarantees. If the data is free, you're not the customer; you're a user of a free service that may change or disappear.
Professional data vendors
Bloomberg Terminal is the gold standard for professional traders. It costs $20,000–$30,000 annually and offers comprehensive, real-time data; historical coverage spanning decades; adjusted prices for splits and dividends; and direct backtesting tools. For institutional traders, the cost is justified.
FactSet similarly costs $10,000–$20,000+ annually and covers equities, fixed income, derivatives, and alternative data. It emphasizes data quality and standardization across asset classes.
TradeStation, Interactive Brokers, and other brokerages include historical data as part of their platform. TradeStation offers clean, broker-grade futures and equities data at no additional cost (to account holders). This is often a practical middle ground: adequate quality for retail trading without expensive standalone data subscriptions.
Pinnacle Data and other specialized vendors focus on specific asset classes (forex, crypto, or specific exchanges) and cost $50–$500 monthly depending on coverage and depth.
Refinitiv (formerly Reuters) and S&P Capital IQ serve institutional clients with premium data and research. Costs are typically negotiated per enterprise.
Evaluating data quality
Before using any historical dataset, perform basic quality checks:
Check for gaps: Are there any missing days? Legitimate gaps exist (market holidays, weekends), but if price data is missing on a trading day, that's a red flag. A simple check: count the number of trading days in your data and compare to the actual market calendar.
Verify split and dividend adjustments: If a stock split 2:1, historical prices before the split should be halved so the price series is continuous. If a company paid a dividend, it should be reflected in adjusted close (not just historical price). Compare your dataset's adjusted close to multiple sources.
Check for obvious errors: Look for prices that seem impossible (e.g., intraday high lower than close, or a single-day price move of >50% without news). Run a simple outlier detection script and manually verify the most extreme moves.
Compare across sources: Pull the same data from two independent sources (e.g., Yahoo Finance and your broker's data) and compare. Discrepancies of >0.1% suggest quality issues.
Verify volume data: Check that volume spikes coincide with price moves and that average volume is reasonable for the instrument. Implausibly high or low volume indicates a data error or a delisted/illiquid period.
Decision tree
Survivorship bias and its impact
Survivorship bias is the tendency of historical returns to overstate long-term equity performance because delisted and bankrupt companies are excluded from the dataset. If you backtest a stock-picking strategy using only companies that survived to the present, you're implicitly excluding all the companies that went bankrupt—which would have hurt your returns if you'd owned them.
Studies show survivorship bias inflates long-term equity returns by 1–3% annually. A strategy backtested on companies that survived to 2025 might appear to return 12% annually, but if you'd actually traded it historically (including bankrupt and delisted names), it would have returned only 9–11% annually.
Free data sources often have strong survivorship bias because they focus on currently active companies. Professional vendors address this by maintaining "delisted history"—data for companies that are no longer trading. If you're backtesting a buy-and-hold equity strategy, account for survivorship bias by either:
- Using a data vendor that includes delisted companies
- Applying a 1.5–2.5% annual drag to simulate realistic losses from bankruptcy/delisting
- Backtesting only on current index constituents and applying historical returns, acknowledging the upward bias
Adjustments for splits, dividends, and corporate actions
Raw price data reflects actual historical prices, but to analyze long-term returns, prices must be adjusted for corporate actions:
Stock splits (e.g., 2:1 split) multiply shares outstanding and divide price. Historical prices before the split must be divided by the split factor to create a continuous price series. Missing this adjustment creates a false gap in your data and ruins backtests.
Dividends are either handled by adjusting the close price downward on the ex-dividend date (reducing the adjusted close) or by tracking them separately. Most sources use adjusted close, which implicitly assumes reinvestment. For dividend-paying stocks, this is more realistic than ignoring dividends.
Mergers and acquisitions create discontinuities. If you own Company A and it merges into Company B, your position continues under the new ticker. But your price series ends, and you must manually map to Company B's history. Professional data vendors handle this; free sources often don't.
Spinoffs and reorganizations similarly require manual mapping. The data might show a gap or discontinuity if you're not careful.
Always confirm that your data source clearly states which adjustments have been applied. Look for terms like "adjusted close" or "total return series." If unclear, assume prices are unadjusted and perform adjustments yourself using published split and dividend schedules.
Real-world examples
A trader backtesting a mean-reversion strategy on the S&P 500 pulls 20 years of daily data from Yahoo Finance. The data is free, readily available, and convenient. However, it excludes delisted companies (survivorship bias) and occasionally shows minor discrepancies with market data during dividend reinvestment dates.
The backtest returns 11% annually with a Sharpe ratio of 1.2. However, if the trader accounted for survivorship bias (subtracting 2% annually), realistic returns drop to 9%. If the trader also accounted for slippage and commissions not reflected in historical data (subtracting another 1–2%), realistic live returns might be 7–8%—still positive but significantly lower than the backtest suggested.
By contrast, a professional trader uses Bloomberg data for the same backtest. Bloomberg includes delisted securities, properly adjusts for all corporate actions, and includes transaction costs if desired. The backtest generates 9% annual returns—already accounting for survivorship. The trader is more confident the live results will approximate the backtest because the data quality is superior.
Another trader backtests a crypto strategy on Binance data sourced from Quandl. The data covers three years of hourly bars. However, Binance's own API shows volume on two particular days significantly higher than Quandl's data, suggesting data quality issues. The trader queries Binance's official API directly, confirms the correct volume, and re-runs the backtest with corrected data. Results improve by 5%, revealing that the initial backtest was distorted by inaccurate volume data.
Common mistakes
Using too little historical data. A five-year backtest is insufficient for stocks because of regime dependence and low sample size. Use 10–20 years minimum. For crypto or recently launched instruments, this may be impossible, in which case acknowledge the limitation and trade smaller positions.
Trusting free data without verification. Free sources are convenient but unvetted. Always spot-check against a second source or your broker's data. A single misplaced decimal point in volume can dramatically skew results.
Ignoring survivorship bias. When backtesting equities, assume your results are inflated by 1–3% annually. Adjust expectations downward accordingly, or use data that includes delisted companies.
Neglecting corporate actions. If your data doesn't properly adjust for splits and dividends, your price series will have false gaps. Verify adjustment methodology with your data source.
Backtesting on live prices instead of tradeable prices. Historical data should reflect prices at which you could actually trade (not theoretical mid-prices). Bid-ask spreads, market impact, and slippage are separate but should be accounted for, not hidden in the price data.
Mixing data from multiple sources without reconciliation. If you use Yahoo Finance for the first five years and your broker's data for the next ten years, ensure no gap or discontinuity exists where the data switches. Different sources may adjust dividends or splits differently, creating artificial jumps.
FAQ
How far back does historical data need to go?
Aim for at least 10 years for equities and 5–10 years for more recent instruments like ETFs or forex pairs. Longer is better (20+ years is ideal) because it captures different market regimes: bull and bear markets, different volatility environments, and structural changes in markets. If data spans only two years, recognize that overfitting risk is high and expect live results to differ significantly from backtest.
Should I use bid-ask spreads in my historical data?
Historical data typically shows actual transaction prices (close) or bid-ask midpoints. For backtesting, use realistic bid-ask spreads as separate transaction costs, not embedded in the price. Most backtesting platforms allow you to specify slippage (bid-ask cost) separately from the price data, which is cleaner.
What data frequency is best for backtesting?
It depends on your strategy. A daily chart strategy uses daily bars. A four-hour strategy uses four-hour bars. A one-minute scalper uses one-minute bars. Use the lowest frequency sufficient to capture your entry and exit logic; don't use one-minute data for a daily strategy (you'll overfit to intraday noise) or daily data for an intraday strategy (you'll miss opportunities). Intraday data is heavier computationally, so balance accuracy with tractability.
How do I handle gaps in historical data?
Minor gaps (e.g., market holidays) are normal and should be ignored. If price data is missing on a legitimate trading day, investigate: Was the market open? Was the instrument tradeable? If yes to both, the data is corrupted. Either source corrected data or exclude that period. Never interpolate missing prices; it's fiction.
Can I backtest strategies on data from different time zones?
Yes, but be careful. Forex pairs trade globally, so data from different zones may reflect different trading sessions. Equities trade primarily during market hours. If your strategy is sensitive to session boundaries, ensure your data aligns with the session you intend to trade.
Should I use split-adjusted or unadjusted prices?
For backtesting, use split-adjusted prices so the price series is continuous across corporate actions. This simplifies the math and prevents false gaps. Your data vendor should handle this; confirm that prices are adjusted for splits and dividends.
Related concepts
- Backtesting Overview — The fundamentals of backtesting and its purpose
- In-Sample vs. Out-of-Sample Testing — How to structure backtests to avoid overfitting on limited data
- Interpreting Backtest Results Correctly — How to evaluate whether backtest results are trustworthy
- Walk-Forward Testing for Realistic Results — Advanced technique using multiple windows of historical data
Summary
Historical data is the foundation of backtesting; quality data matters enormously. Free sources like Yahoo Finance are convenient and adequate for initial prototyping but carry risks of gaps, incomplete adjustments, and survivorship bias. Professional traders use premium vendors (Bloomberg, FactSet, broker data) or carefully validate free data against multiple sources. Always verify data quality by checking for gaps, extreme outliers, proper split and dividend adjustments, and comparing across independent sources. Survivorship bias inflates equity backtest returns by 1–3% annually; account for this when interpreting results. Use at least 10–20 years of historical data to reduce overfitting risk and capture multiple market regimes. The investment in data quality—through premium vendors or rigorous validation of free data—pays dividends in backtest reliability and live trading confidence.