0.2 — Project 1: Backtesting Engine (Foundations) Guide#

This guide walks you through the “Steps to Complete the Project” from the project overview. Each step is framed as a task with hints to point you in the right direction. Full code and “raw” answers are hidden in Show Solution accordions—try the step yourself first, then expand only when you need to.

The goal is not to give you a giant finished script, but to help you build it step‑by‑step with support when you’re stuck.

1. Set up the environment#

Goal: Create a clean Python environment and install the core libraries so you can run Pandas, NumPy, yfinance, and (optionally) matplotlib.

Why it matters: A dedicated virtual environment keeps this project’s dependencies separate from other work and makes it easy to reproduce later (e.g. with a requirements.txt).

Hints:

• Use a virtual environment (e.g. python -m venv .venv) and activate it before installing anything.
• Install at least: numpy, pandas, yfinance. Add matplotlib if you plan to plot the equity curve or drawdown.
• Decide on a simple folder layout: e.g. a project root plus a main script (e.g. run_backtest.py) and optionally a package folder (e.g. backtesting/) for modules.
• Pinning versions in a requirements.txt (e.g. pandas>=2.0,<3.0) helps others (and future you) reinstall the same setup with pip install -r requirements.txt.

Try it: Create and activate a venv, then install the core dependencies. Optionally add a requirements.txt with pinned or minimum versions.

python -m venv .venv
source .venv/bin/activate  # On Windows: .venv\Scripts\activate

pip install numpy pandas yfinance matplotlib

numpy>=1.24,<3.0
pandas>=2.0,<3.0
yfinance>=0.2,<0.3
matplotlib>=3.8,<4.0

2. Load and clean historical data#

Goal: Download daily OHLCV data for at least one symbol and end up with a clean pandas.DataFrame that has a DatetimeIndex, no duplicate dates, and a consistent way of handling missing values.

Why it matters: The rest of the backtest (indicators, signals, P&L) assumes one row per date and aligned columns. Gaps, duplicates, or timezone mix‑ups will cause subtle bugs.

Hints:

• Use yfinance: yf.download(ticker, start=..., end=..., interval="1d", ...) returns a DataFrame. Check the column names (e.g. Open, High, Low, Close, Volume); they may be capitalized.
• Ensure the index is a DatetimeIndex. If it isn’t, convert it with pd.to_datetime(...).
• Sort by date ascending so that “previous bar” always means .shift(1).
• Remove duplicate timestamps (e.g. keep first with ~df.index.duplicated(keep="first")).
• Missing values: Decide on a simple rule—e.g. forward‑fill then back‑fill, or drop rows with NaNs. Document your choice. Avoid filling long stretches without a clear rationale.
• Sanity checks: After cleaning, confirm df.index is a DatetimeIndex, there are no duplicates, and no obvious long gaps (aside from weekends/holidays).

Try it: Write a small script that downloads a symbol for a date range and a helper (e.g. clean_ohlcv(df)) that returns a cleaned DataFrame with the properties above.

import yfinance as yf
import pandas as pd

ticker = "SPY"
start = "2015-01-01"
end = "2025-01-01"

df = yf.download(
    ticker,
    start=start,
    end=end,
    interval="1d",
    auto_adjust=False,
    progress=False,
)
print(df.head())

def clean_ohlcv(df: pd.DataFrame) -> pd.DataFrame:
    if not isinstance(df.index, pd.DatetimeIndex):
        df.index = pd.to_datetime(df.index)
    df = df.sort_index()
    df = df[~df.index.duplicated(keep="first")]
    df = df.ffill().bfill()
    return df

df = clean_ohlcv(df)
print(df.info())

3. Compute indicators (e.g., Simple Moving Average)#

Goal: Add at least one indicator as a new column to your DataFrame. A good first choice is a Simple Moving Average (SMA) so you can later build a moving‑average crossover strategy.

Why it matters: Strategies are driven by indicators. Getting one right (and validated) gives you a template for more.

Hints:

• For an SMA over the close price, you want the average of the last window values at each row. Pandas has a method for “rolling” windows: e.g. df["Close"].rolling(window=..., min_periods=...).
• Use min_periods=window (or similar) so that the first window - 1 rows are NaN—no look‑ahead and no partial windows if you want a strict SMA.
• Add the result as a new column (e.g. df["SMA_50"] = ...) so the index stays aligned.
• Validate: Pick a few rows (e.g. first 10–20), manually compute the average of the last window closes, and compare to your column. They should match up to rounding.
• Consider a small helper like add_sma(df, window=50, price_col="Close") that returns the DataFrame with the new column so you can reuse it for multiple windows (e.g. 50 and 200).

Try it: Implement an SMA and add two columns (e.g. SMA_50 and SMA_200). Check a few values by hand or with a known reference.

import pandas as pd

def add_sma(
    df: pd.DataFrame,
    window: int,
    price_col: str = "Close",
    column_name: str | None = None,
) -> pd.DataFrame:
    if price_col not in df.columns:
        raise KeyError(f"Column {price_col!r} not found in DataFrame")
    if window <= 0:
        raise ValueError("window must be positive")
    col_name = column_name or f"SMA_{window}"
    df[col_name] = (
        df[price_col]
        .rolling(window=window, min_periods=window)
        .mean()
    )
    return df

df = add_sma(df, window=50, price_col="Close")
df = add_sma(df, window=200, price_col="Close")
print(df[["Close", "SMA_50", "SMA_200"]].head(10))

4. Define and implement signal logic#

Goal: Turn your indicators into a signal series (e.g. +1 = long, 0 = flat, −1 = short) aligned with your DataFrame index. For a moving‑average crossover: long when the short MA is above the long MA; flat (or short) when it’s below.

Why it matters: Signals are the bridge between indicators and trading. They must use only information available at each bar (no look‑ahead). The position you use for P&L will be derived from the signal with a one‑bar lag in the next step.

Hints:

• Rule for MA crossover: long when short MA > long MA; otherwise flat (or −1 if you allow shorting). Compare the two SMA columns bar by bar.
• Use vectorized logic (e.g. np.where(short_ma > long_ma, 1, 0)) so the result is a series aligned with the index. Where either MA is NaN (not enough history), set signal to 0 (flat).
• Store the result in a column (e.g. df["signal"]). This is the “decision at the close of this bar”; the position that earns P&L on the next bar’s return should come from shifting this signal by one period—you’ll do that in the next step.
• If your DataFrame has a MultiIndex columns (common with yf.download for one ticker), ensure you’re indexing the SMA columns correctly (e.g. flatten or select the right level).

Try it: Implement a function that takes the DataFrame and the short/long SMA column names and writes a signal column (+1 / 0 / −1). Inspect a few rows where the MAs cross to confirm the signal flips as expected.

import numpy as np
import pandas as pd

def moving_average_crossover_signals(
    df: pd.DataFrame,
    short_col: str = "SMA_50",
    long_col: str = "SMA_200",
    long_only: bool = True,
) -> pd.DataFrame:
    if short_col not in df or long_col not in df:
        raise KeyError("Missing SMA columns in DataFrame")
    short_ma = df[short_col]
    long_ma = df[long_col]
    signal = np.where(short_ma > long_ma, 1, 0 if long_only else -1)
    signal = np.where(short_ma.isna() | long_ma.isna(), 0, signal)
    df["signal"] = signal
    return df

df = moving_average_crossover_signals(df, short_col="SMA_50", long_col="SMA_200")
print(df[["Close", "SMA_50", "SMA_200", "signal"]].head(20))

5. Simulate trades (positions, P&L, equity curve)#

Goal: Convert signals into positions, then into strategy returns and an equity curve. Avoid look‑ahead: the position on day (t+1) should be based on the signal at the close of day (t).

Why it matters: This is where the strategy actually “trades.” Using the signal from the same bar to compute that bar’s return would be look‑ahead bias. Shifting the signal by one period fixes that.

P&L and slippage: We use close‑to‑close returns and no slippage here. For these assumptions and how to add a simple slippage model later (e.g., fixed bps per trade), see P&L and slippage in the project overview.

Hints:

• Positions from signals: The position for bar (t+1) should equal the signal at bar (t). So: position = signal.shift(1). Fill the first row’s NaN (no prior signal) with 0 (flat).
• Returns: Compute the asset’s period return (e.g. close.pct_change()). Strategy return each period = position * asset_return. Use the position series (shifted), not the raw signal.
• Equity curve: Start with initial capital (C_0). Cumulative growth is the cumulative product of ((1 + \text{strategy_return})). So equity at each step is (C_0 \times \prod(1 + r_t)). Pandas: (1 + strategy_return).cumprod() * initial_capital.
• Sanity check: If you force signal = 1 (always long) after the first date, your equity curve should behave like buy‑and‑hold (same as holding the asset).

Try it: Implement helpers that (1) add a position column from signal with a one‑period lag, and (2) add asset_return, strategy_return, and equity_curve columns. Run a quick check with “always long” to compare to buy‑and‑hold.

def compute_positions_from_signals(df: pd.DataFrame) -> pd.DataFrame:
    if "signal" not in df.columns:
        raise KeyError("DataFrame must contain a 'signal' column")
    df["position"] = df["signal"].shift(1).fillna(0)
    return df

def compute_pnl_and_equity(
    df: pd.DataFrame,
    price_col: str = "Close",
    initial_capital: float = 100_000.0,
) -> pd.DataFrame:
    if price_col not in df.columns:
        raise KeyError(f"Column {price_col!r} not found")
    if "position" not in df.columns:
        raise KeyError("DataFrame must contain a 'position' column")
    df["asset_return"] = df[price_col].pct_change().fillna(0.0)
    df["strategy_return"] = df["position"] * df["asset_return"]
    df["equity_curve"] = (1 + df["strategy_return"]).cumprod() * initial_capital
    return df

df = compute_positions_from_signals(df)
df = compute_pnl_and_equity(df, price_col="Close", initial_capital=100_000.0)

6. Compute performance metrics#

Goal: From your strategy return series (and optionally the equity curve), compute at least: total return, annualized return, annualized volatility, and Sharpe ratio. Optionally add max drawdown, win rate, and number of trades.

Why it matters: These metrics let you compare strategies and interpret backtest results. Use the same definitions as in your statistics lessons (e.g. Returns and volatility for backtesting) so they’re comparable.

Hints:

• Total return: ((1+r_1)(1+r_2)\cdots(1+r_n) - 1). In Pandas: (1 + returns).prod() - 1.
• Annualized return: From total return and number of periods, solve for the constant annual rate that would give that cumulative growth. Assume a fixed number of periods per year (e.g. 252 for daily).
• Annualized volatility: Period volatility (e.g. returns.std(ddof=1)) scaled by (\sqrt{\text{periods_per_year}}).
• Sharpe ratio: Excess return (strategy return minus risk‑free per period) divided by period volatility, then scaled by (\sqrt{\text{periods_per_year}}). Define how you treat the risk‑free rate (e.g. 0 for simplicity) and document it.
• Max drawdown: From the equity curve, compute the running maximum; drawdown at each time is (equity / running max) − 1. Max drawdown is the minimum of that (most negative).
• Win rate (optional): Fraction of periods with positive strategy return: (strategy_return > 0).mean(), or count only days when the position was non‑zero. Alternatively, count discrete trades from position changes and then (winning trades / total trades).
• Number of trades (optional): Count how often the position changes (e.g. position.diff().fillna(0).abs() > 0 then sum, or divide by 2 for round‑trip trades).
• Use ddof=1 for sample standard deviation when computing volatility.

Try it: Write small functions for total return, annualized return, annualized volatility, and Sharpe ratio. Then add max drawdown from the equity curve. Call them on df["strategy_return"] and df["equity_curve"] and print the results.

import numpy as np
import pandas as pd

def total_return(returns: pd.Series) -> float:
    return float((1.0 + returns).prod() - 1.0)

def annualized_return(returns: pd.Series, periods_per_year: int = 252) -> float:
    if returns.empty:
        return 0.0
    cumulative = 1.0 + total_return(returns)
    n_periods = returns.shape[0]
    years = n_periods / periods_per_year
    if years <= 0:
        return 0.0
    return float(cumulative ** (1.0 / years) - 1.0)

def annualized_volatility(returns: pd.Series, periods_per_year: int = 252) -> float:
    return float(returns.std(ddof=1) * np.sqrt(periods_per_year))

def sharpe_ratio(
    returns: pd.Series,
    risk_free_rate: float = 0.0,
    periods_per_year: int = 252,
) -> float:
    if returns.empty:
        return 0.0
    rf_per_period = (1 + risk_free_rate) ** (1 / periods_per_year) - 1
    excess = returns - rf_per_period
    vol = excess.std(ddof=1)
    if vol == 0:
        return 0.0
    return float(excess.mean() / vol * np.sqrt(periods_per_year))

def max_drawdown(equity_curve: pd.Series) -> float:
    if equity_curve.empty:
        return 0.0
    running_max = equity_curve.cummax()
    drawdowns = equity_curve / running_max - 1.0
    return float(drawdowns.min())

metrics = {}
metrics["total_return"] = total_return(df["strategy_return"])
metrics["annualized_return"] = annualized_return(df["strategy_return"])
metrics["annualized_volatility"] = annualized_volatility(df["strategy_return"])
metrics["sharpe_ratio"] = sharpe_ratio(df["strategy_return"], risk_free_rate=0.0)
metrics["max_drawdown"] = max_drawdown(df["equity_curve"])
for name, value in metrics.items():
    print(name, ":", value)

def win_rate(returns: pd.Series, position: pd.Series) -> float:
    traded = position != 0
    if traded.sum() == 0:
        return 0.0
    return float((returns[traded] > 0).mean())

7. Produce a report#

Goal: Print or display a concise summary of the backtest: configuration (ticker, date range, strategy name, parameters) and the main performance metrics. Optionally plot the equity curve or drawdown.

Why it matters: A single, readable output makes it easy to rerun the backtest with different parameters and compare results.

Hints:

• Configuration: Include ticker, start/end dates, strategy name, and key parameters (e.g. short/long windows).
• Metrics: Format percentages (e.g. total return, annualized return, volatility, max drawdown) as readable strings (e.g. "12.34%"). Keep Sharpe as a number (e.g. two decimals).
• A simple approach is a function that takes ticker, start_date, end_date, strategy_name, strategy_params, and metrics dict, then prints section headers and each line. You can later replace or complement with a notebook or a small HTML/Markdown report.
• Plotting (optional): Use matplotlib (or plotly): plot df["equity_curve"] with date on the x‑axis. Add a title and axis labels.

Try it: Write a print_summary(...) (or similar) that prints configuration and metrics. Run it after computing metrics. Optionally add one plot of the equity curve.

def format_pct(x: float, decimals: int = 2) -> str:
    return f"{x * 100:.{decimals}f}%"

def print_summary(
    ticker: str,
    start_date: str,
    end_date: str,
    strategy_name: str,
    strategy_params: dict,
    metrics: dict,
) -> None:
    print("=== Backtest Summary ===\n")
    print("Configuration:")
    print(f"  Ticker:       {ticker}")
    print(f"  Date range:   {start_date} -> {end_date}")
    print(f"  Strategy:     {strategy_name}")
    print(f"  Parameters:   {strategy_params}")
    print("\nPerformance:")
    print(f"  Total return:          {format_pct(metrics['total_return'])}")
    print(f"  Annualized return:     {format_pct(metrics['annualized_return'])}")
    print(f"  Annualized volatility: {format_pct(metrics['annualized_volatility'])}")
    print(f"  Sharpe ratio:          {metrics['sharpe_ratio']:.2f}")
    print(f"  Max drawdown:          {format_pct(metrics['max_drawdown'])}")

print_summary(
    ticker="SPY",
    start_date="2015-01-01",
    end_date="2025-01-01",
    strategy_name="Moving Average Crossover",
    strategy_params={"short_window": 50, "long_window": 200},
    metrics=metrics,
)

import matplotlib.pyplot as plt
df["equity_curve"].plot(figsize=(10, 4), title="Equity Curve")
plt.xlabel("Date")
plt.ylabel("Equity")
plt.show()

8. Document and refactor#

Goal: End up with clean, reusable code and a short README so that you (or someone else) can run the backtest and understand what the strategy does and how to interpret the output.

Why it matters: Good structure (e.g. separate modules for data, indicators, strategy, backtester, metrics, report) makes it easier to add a second strategy (e.g. momentum) or reuse the engine for the next project (e.g. paper‑trading bot).

Hints:

• Split logic into modules by responsibility: e.g. data (download, clean), indicators (SMA, etc.), strategy (signal logic), backtester (positions, P&L, equity), metrics (return, vol, Sharpe, drawdown), report (print/plot). A single top-level script (e.g. run_backtest.py) can then import and wire these together.
• README: Describe the project in one or two sentences, list setup steps (venv, pip install -r requirements.txt), and explain how to run the backtest (e.g. python run_backtest.py). Add a short paragraph on what the strategy does and how to read the summary (e.g. what a higher Sharpe means, caveats of backtests).
• You don’t have to match any particular structure exactly; the important part is that the pipeline is clear and easy to extend.

Try it: Organize your code into at least two or three modules and a main script. Write a README that covers setup, run instructions, and a brief strategy + metrics interpretation.

from datetime import date
import yfinance as yf
from backtesting.data import clean_ohlcv
from backtesting.indicators import add_sma
from backtesting.strategy import moving_average_crossover_signals
from backtesting.backtester import compute_positions_from_signals, compute_pnl_and_equity
from backtesting.metrics import total_return, annualized_return, annualized_volatility, sharpe_ratio, max_drawdown
from backtesting.report import print_summary

def main() -> None:
    ticker = "SPY"
    start_date = "2015-01-01"
    end_date = date.today().strftime("%Y-%m-%d")
    raw = yf.download(ticker, start=start_date, end=end_date, interval="1d", progress=False)
    df = clean_ohlcv(raw)
    df = add_sma(df, window=50, price_col="Close")
    df = add_sma(df, window=200, price_col="Close")
    df = moving_average_crossover_signals(df, short_col="SMA_50", long_col="SMA_200")
    df = compute_positions_from_signals(df)
    df = compute_pnl_and_equity(df, price_col="Close", initial_capital=100_000.0)
    metrics = {
        "total_return": total_return(df["strategy_return"]),
        "annualized_return": annualized_return(df["strategy_return"]),
        "annualized_volatility": annualized_volatility(df["strategy_return"]),
        "sharpe_ratio": sharpe_ratio(df["strategy_return"], risk_free_rate=0.0),
        "max_drawdown": max_drawdown(df["equity_curve"]),
    }
    print_summary(ticker=ticker, start_date=start_date, end_date=end_date,
                  strategy_name="Moving Average Crossover",
                  strategy_params={"short_window": 50, "long_window": 200},
                  metrics=metrics)

if __name__ == "__main__":
    main()

9. Final tips for beginners#

• Work step‑by‑step. Get data loading working before worrying about Sharpe ratios.
• Print and inspect. After each step (data, indicators, signals, positions), print the head of your DataFrame and check that it matches your expectations.
• Start with simple assumptions. No transaction costs, one symbol, daily data only. You can always make it more realistic later.
• Use a known period as a sanity check. A buy‑and‑hold SPY strategy over a long bull market should show positive total return and a rising equity curve.

If you can run one script (or one notebook) that completes this full pipeline and prints a sensible summary, you’ve achieved the core objective of this project.