Building a Crypto Trading Bot from Scratch - 6: The Foundation of Backtesting
After building the exchange adapter and data manager, I actually needed to fill my local storage with historical data. Because here’s the thing: you can’t backtest a strategy without data. And you can’t download a year of hourly candles with a single API call.
This is where things get practical—and where you learn a lot about API rate limits, missing data, and the frustrating reality that historical data is never as clean as you’d hope.
Why Historical Data Matters
Before I get into the technical details, let me explain why this matters so much.
When you develop a trading strategy, you have two choices:
Test it live with real money - Learn fast, but expensive
Test it on historical data first - Slower to learn, but free
I’m risk-averse, so I went with option 2. But to backtest effectively, you need comprehensive, accurate historical data. Not just a few days—you need months or years across multiple market conditions (bull markets, bear markets, sideways action, high volatility, low volatility).
That means downloading potentially hundreds of thousands of candles across multiple symbols and timeframes. And doing it without getting banned from the exchange.
The Download Script
I built a command-line script to handle historical data downloads. Here’s what using it looks like:
# Download 30 days of BTC hourly data
python scripts/download_historical_data.py --symbol BTCUSDT --timeframe 1h --days 30
# Download specific date range
python scripts/download_historical_data.py --symbol ETHUSDT --timeframe 15m \
--start 2025-01-01 --end 2025-03-31
# Download multiple symbols at once
python scripts/download_historical_data.py --symbol BTCUSDT,ETHUSDT --timeframe 1h --days 365
The script handles:
Multiple symbols and timeframes
Flexible date ranges (relative like “30 days” or absolute like “2025-01-01 to 2025-03-31”)
Checking for existing data to avoid re-downloads
Progress reporting
Error handling and retries
Here’s the core structure:
