Choosing how much money to put into different investments can look simple at first. You might decide on a percentage for stocks, another for bonds, and perhaps smaller allocations to other assets. But once you start asking how those investments move together, how much risk the portfolio can tolerate, and how often the portfolio needs to be rebalanced, the problem becomes considerably more complicated.

This is where portfolio optimization comes in. Instead of evaluating investments one at a time, an optimization model looks at the portfolio as a whole. It can consider expected returns, volatility, correlations between assets, and practical constraints such as minimum or maximum allocations.

Portfolio optimization does not predict the future, and a mathematically optimized portfolio is not automatically a better investment. Its main value is that it provides a structured way to compare different trade-offs and make those assumptions explicit.

1. The Basic Mathematics Behind Portfolio Optimization

One of the best-known approaches to portfolio optimization comes from Modern Portfolio Theory, associated with economist Harry Markowitz. The basic idea is that portfolio risk depends not only on the risk of individual investments but also on how those investments move relative to one another.

For example, two assets may each be relatively volatile on their own, but if they do not tend to move in exactly the same direction, holding both may produce a portfolio with different risk characteristics than holding either asset alone.

A simplified minimum-variance problem can be written as:

The equation is only a simplified illustration. Real-world portfolio models can include many additional constraints. For example, an investor might limit any single holding to 10%, require a minimum allocation to a particular asset class, or restrict the amount of trading allowed during a rebalance.

The important point is that the model is working with assumptions. If expected returns or historical relationships are poor estimates of future conditions, the resulting allocation can also be misleading.

2. Why the Covariance Matrix Matters

The covariance matrix is one of the most important inputs in many portfolio optimization models. It describes how the returns of different assets have moved together over the period being studied.

With only a few assets, estimating these relationships is relatively straightforward. The challenge grows as the number of assets increases. A portfolio containing dozens or hundreds of securities requires a large number of relationships to be estimated from historical data.

Historical estimates can also be unstable. A covariance matrix calculated during one market environment may look very different from one calculated during another period.

One technique used to address this issue is covariance shrinkage. Rather than relying entirely on the historical sample covariance matrix, shrinkage combines it with a more structured estimate:

The goal is not to make the covariance matrix “correct.” Instead, shrinkage can make the estimate more stable and may reduce the effect of estimation noise. The appropriate method depends on the data, portfolio, and assumptions being used.

This is one reason quantitative portfolio construction is as much about data quality and assumptions as it is about optimization algorithms.

3. Building a Simple Portfolio Optimization Model in Python

Once the expected returns and covariance matrix have been estimated, a Python library such as cvxpy can be used to formulate and solve certain portfolio optimization problems.

The following example demonstrates a simple long-only portfolio model. It does not represent a complete investment system. The expected returns and covariance matrix are deliberately supplied as inputs so that the focus remains on the optimization mechanics.

import numpy as np
import cvxpy as cp


class PortfolioOptimizationEngine:
    def __init__(self, expected_returns, covariance_matrix):
        self.mu = np.asarray(expected_returns, dtype=float)
        self.sigma = np.asarray(covariance_matrix, dtype=float)
        self.num_assets = len(self.mu)

    def minimum_variance_portfolio(self, target_return=None):
        """
        Find a long-only portfolio with minimum estimated variance.

        If target_return is supplied, the optimizer also requires
        the portfolio's estimated return to be at least that level.
        """

        weights = cp.Variable(self.num_assets)

        objective = cp.Minimize(
            cp.quad_form(weights, self.sigma)
        )

        constraints = [
            cp.sum(weights) == 1,
            weights >= 0
        ]

        if target_return is not None:
            constraints.append(
                self.mu @ weights >= target_return
            )

        problem = cp.Problem(objective, constraints)
        problem.solve()

        if problem.status not in ["optimal", "optimal_inaccurate"]:
            raise ValueError(
                "The optimization problem could not be solved."
            )

        return np.round(weights.value, 4)


if __name__ == "__main__":

    # Example annualized expected returns.
    expected_returns = np.array([
        0.07,
        0.05,
        0.09,
        0.04
    ])

    # Example covariance matrix.
    covariance_matrix = np.array([
        [0.040, 0.005, 0.010, 0.002],
        [0.005, 0.025, 0.008, 0.001],
        [0.010, 0.008, 0.090, 0.004],
        [0.002, 0.001, 0.004, 0.015]
    ])

    engine = PortfolioOptimizationEngine(
        expected_returns,
        covariance_matrix
    )

    weights = engine.minimum_variance_portfolio()

    print("Estimated portfolio weights:")
    print(weights)

In this example, the optimizer searches for a combination of the four assets that minimizes estimated variance while requiring the portfolio weights to add up to 100% and preventing negative positions.

In practice, the difficult part usually isn’t writing the optimization code. The more important questions are where the inputs came from, whether they are appropriate for the time period being studied, and how sensitive the results are to changes in those assumptions.

4. Why Transaction Costs and Taxes Change the Picture

A portfolio can look attractive on paper and still be difficult or expensive to maintain. Rebalancing may involve brokerage costs, bid-ask spreads, market impact, and, depending on the investor’s location and account type, taxes.

This matters because an optimizer may recommend relatively frequent changes if doing so improves the mathematical objective by a small amount. Those theoretical improvements may not justify the real-world cost of trading.

A practical model can therefore include a turnover constraint or penalty. For example:

Tax considerations are more complicated because the actual treatment depends on factors such as jurisdiction, account type, holding period, cost basis, and the type of investment. For that reason, tax rules should not be reduced to a single universal formula inside an optimization model.

Investors considering tax-sensitive rebalancing may want to evaluate the potential tax consequences separately and, where appropriate, consult a qualified tax professional.

5. The Black-Litterman Model: A Different Way to Think About Expected Returns

One of the biggest challenges in mean-variance optimization is estimating expected returns. Small changes in return assumptions can sometimes produce large changes in the portfolio weights.

The Black-Litterman framework is one approach designed to make expected-return assumptions more structured. Rather than starting entirely from an investor’s forecasts, it begins with market-implied returns and then incorporates additional views.

A simplified expression for the market-implied equilibrium return vector is:

Additional views can then be incorporated into the model to produce a revised set of expected returns. The resulting portfolio can be optimized subject to the same practical constraints discussed earlier.

Black-Litterman is not a guarantee of more accurate forecasts. Its usefulness comes from providing a framework for combining a market-based starting point with explicit assumptions rather than treating a small set of return forecasts as unquestionable facts.

6. The Biggest Limitation: The Model Is Only as Good as Its Inputs

This is perhaps the most important point to understand about portfolio optimization.

An optimizer can solve the mathematical problem correctly while still producing an allocation that performs poorly in the real world.

There are several reasons this can happen:

• Uncertain expected returns:
  Future returns cannot be known with certainty, and historical averages may not represent future conditions.
• Changing correlations:
  Relationships between assets can change during different economic and market environments.
• Estimation error:
  Historical data is only a sample of possible market outcomes.
• Transaction costs:
  Frequent portfolio changes can reduce realized returns.
• Model assumptions:
  Constraints and objectives can strongly influence the final allocation.
• Unexpected events:
  Financial markets can experience conditions that were not represented in the historical data used by the model.

For these reasons, optimization is better viewed as a decision-support framework rather than a machine that discovers the perfect portfolio.

7. A Practical Checklist for Portfolio Optimization

If you are experimenting with portfolio optimization in Python or simply want to understand how these models work, the following checklist provides a useful starting point:

• Start with clear objectives:
  Decide whether the model is intended to minimize volatility, target a return, manage risk, or explore another measurable objective.
• Use appropriate historical data:
  Check for missing observations, inconsistent pricing data, corporate actions, and other data-quality issues.
• Consider covariance estimation carefully:
  Shrinkage and other regularization techniques may be useful when historical estimates are unstable.
• Add realistic constraints:
  Maximum position sizes, minimum allocations, turnover limits, and other restrictions can make the model more representative of an actual portfolio.
• Test sensitivity:
  Change key assumptions and see whether the recommended allocation changes dramatically.
• Account for implementation costs:
  Consider spreads, commissions where applicable, taxes, and other costs rather than evaluating only theoretical returns.
• Review the model periodically:
  Market relationships and assumptions can change, so an optimization model should not be treated as permanently valid once it has been built.

Final Thoughts

Portfolio optimization provides a useful mathematical framework for thinking about diversification, risk, and allocation decisions. Models based on mean-variance optimization, covariance estimation, and related techniques can help investors and researchers understand how different assumptions affect a portfolio.

But there is an important distinction between solving an optimization problem and predicting investment performance. The first is a mathematical exercise. The second involves uncertainty that no optimization algorithm can completely remove.

A sensible approach is therefore to use optimization as one tool within a broader decision-making process. Clear objectives, diversified investments, realistic constraints, reasonable costs, and an understanding of uncertainty are often more important than finding a mathematically perfect set of portfolio weights.