An Engineer’s Guide to Building and Validating Quantitative Trading Strategies

Originally published on Medium

A backtest is not a trading system. It is an estimate built from historical data, modeling assumptions, and an execution simulator. Each can make a weak idea look profitable.

The work is therefore less about finding an impressive equity curve and more about eliminating false positives. That means using point-in-time data, modeling the trades that could actually have filled, separating research from evaluation, and defining risk controls before going live.

Start with the market, not the model

A strategy operates through an order book. Its theoretical return matters only after spread, fees, slippage, market impact, and missed fills.

Before testing a signal, specify:

the instruments and venues it trades;
the information available when each decision is made;
the order type and expected queue position;
the maximum participation rate relative to available liquidity;
the delay between observing a signal and submitting an order; and
the rule for partial fills, rejected orders, and stale orders.

These assumptions should become less favorable as order size increases. A fill model that executes every trade at the last price is usually measuring an opportunity that was never available.

Treat data as part of the strategy

Bad data does not merely add noise. It can create the entire result.

The common failure modes are familiar but easy to miss:

Survivorship bias: testing only assets that still exist today.
Look-ahead bias: using a value before it was published or tradable.
Revised data: using the latest version of a historical observation instead of the version known at the time.
Timestamp mismatch: joining feeds by nominal timestamps even though they arrived at different times.
Corporate-action errors: mishandling splits, dividends, symbol changes, or delistings.
Missing-data bias: silently dropping periods when the strategy would have been unable to trade.

A useful dataset is reproducible and point-in-time correct. Record its source, retrieval time, timezone, adjustment method, missing-value policy, and any transformations. Cache raw inputs separately from derived features so the research dataset can be rebuilt and audited.

Give the signal an economic reason to exist

A factor should have a plausible mechanism, not just a correlation. Value may reflect compensation for distress or investor neglect. Momentum may persist because information diffuses gradually and positioning adjusts slowly. Short-term liquidity signals may arise from inventory pressure or forced execution.

The mechanism suggests where the signal should work, how quickly it should decay, and what could kill it. Those predictions are useful tests. If a supposed microstructure signal survives a one-day execution delay, for example, it may be measuring something other than the stated mechanism.

Keep the first version simple. Every extra feature, threshold, and conditional branch increases the number of strategies implicitly tested. Complexity is justified only when it improves unseen results across several periods and has a reason to remain stable.

Build a backtest that can say no

The simulator should reject trades that violate cash, leverage, liquidity, borrowing, or venue constraints. It should also account for:

bid-ask spread and exchange fees;
latency between observation, decision, and execution;
market impact that rises with order size;
partial fills and unfilled orders;
short availability and borrow cost;
funding, margin, and liquidation rules; and
instrument-specific events such as expiry or settlement.

Run the strategy under several cost and latency assumptions. If a small change turns the result negative, the edge is too thin or the model is too optimistic. Report turnover, capacity, drawdown, exposure, and concentration alongside return and Sharpe ratio; no single metric describes whether a strategy is tradable.

Separate fitting from evaluation

A single train/test split is weak evidence. Once results from the test period influence a decision, that period has become part of the research process.

Walk-forward analysis better matches live use:

Fit the strategy using only data available before a cutoff.
Freeze the parameters and trade the next unseen period.
Move the cutoff forward and repeat.
Combine only the unseen periods when evaluating performance.

This exposes the strategy to different regimes and reveals how often it needs retraining. The choice of windows should follow the signal’s expected lifetime and the amount of data needed for estimation, not whichever setting produces the best chart.

Keep a final holdout that is not inspected during research. Use it once, after the strategy specification and acceptance criteria are fixed. If the holdout causes another round of tuning, a new holdout is required.

Measure the cost of searching

Trying many ideas creates data-mining bias. Even random data will produce a high-Sharpe strategy if enough variants are tested.

A permutation test estimates how surprising the result is under a world with no usable temporal pattern:

Run the complete selection and optimization process on the real data.
Generate many datasets that destroy the relevant signal while preserving appropriate properties of the original series.
Run the same selection and optimization process on every permuted dataset.
Compare the real result with the distribution of the best results found in noise.

The optimization step must be repeated for every permutation. Comparing an optimized real strategy with fixed parameters on shuffled data understates the advantage created by the search process.

The permutation scheme must also fit the hypothesis. Shuffling individual returns destroys volatility clustering and cross-asset relationships. Block permutations, circular shifts, or other constrained resampling methods may provide a more credible null. There is no universally correct shuffle.

For a stronger test, apply the permutation inside the walk-forward procedure and compare aggregate unseen performance. This asks whether the full research and retraining process could have produced similar results without a persistent edge.

A small p-value is evidence against the chosen null model, not proof that the strategy will make money. It says nothing about bad data, omitted costs, a changing market, or a null model that was too easy to beat.

Define risk before sizing positions

Position size converts forecast error into financial loss. Decide the limits before looking at the final performance:

maximum gross and net exposure;
maximum weight per instrument, sector, or correlated group;
volatility or loss target;
liquidity and participation limits;
leverage and margin buffers;
drawdown response; and
conditions that stop trading.

Kelly-style sizing is extremely sensitive to estimated win rates and payoff ratios. Those estimates are noisy and non-stationary, so practical implementations usually use a conservative fraction and impose hard caps. Volatility targeting is also incomplete by itself: correlations tend to rise during stress, exactly when historical estimates are least reliable.

Test risk controls on gaps, feed outages, duplicate messages, rejected orders, stale positions, and exchange downtime. A limit that depends on the same failed data path as the strategy is not an independent safeguard.

Make live trading observable

The live system should preserve enough state to reconstruct every decision. For each order, record:

the input data and model version;
the signal, target position, and risk adjustments;
decision, submission, acknowledgement, and fill timestamps;
requested and filled quantity;
expected and realized execution price; and
rejection, cancellation, or retry reasons.

Start with shadow trading or small capital. Compare live signals with the research implementation, then compare expected fills with actual fills. Differences usually appear in data timing, state handling, fees, or execution—not in the headline model.

Monitor performance by source: signal return, spread, fees, slippage, market impact, financing, and residual error. Attribution turns “the strategy is underperforming” into a diagnosis. It also distinguishes a fading signal from an execution regression.

A practical release gate

Before allocating meaningful capital, require all of the following:

Point-in-time data and transformations can be reproduced.
The strategy has a stated mechanism and falsifiable expectations.
Results remain viable under conservative costs, latency, and fills.
Walk-forward performance is positive across multiple periods rather than dominated by one regime.
The full research process beats an appropriate data-mining null.
Risk limits hold under historical and synthetic stress tests.
Paper or small-capital trading matches the backtest closely enough to explain every material difference.
Monitoring can attribute P&L and halt trading when state becomes uncertain.

None of these gates guarantees profit. Together they make it harder to deploy a statistical accident or an operationally fragile system.

Conclusion

The central question in quantitative research is not “How good is this backtest?” It is “What would have to be false for this result to disappear?”

Challenge the data, the execution model, the search process, the risk assumptions, and the live implementation separately. A strategy is ready only when its edge survives those challenges and its failures are bounded.

For more on quantitative trading and financial engineering, see my work on Medium or connect with me on LinkedIn.

An Engineer's Guide to Building and Validating Quantitative Trading Strategies