Building a Factor Risk Model: The Data

The Investments 101 series left a problem unsolved. Portfolio optimization requires a covariance matrix over the whole universe, and the covariance matrix of $N$ securities has $N (N + 1) /2$ distinct entries. For 500 stocks that is $500 \times 501/2 = 125, 250$ variances and covariances. Estimating them directly from historical returns is statistically hopeless: a sample needs more periods than parameters, and markets do not stay still long enough to supply them. The factor risk model is how practitioners estimate one anyway.

This series builds such a model from the ground up. Not a toy example with three assets and a correlation matrix that fits on a napkin, and not a production trading model either: the robustness work a vendor like MSCI or Axioma layers on top is not here. The goal is something real enough to be instructive: a working Barra-style model, on real data, running in a browser tab. We put DuckDB on the work where rows number in the millions, write the math in TypeScript, and wire it all up to interactive visualizations. The model comes first, then the machine it runs on, then the data it runs on.

The Fundamental Factor Model

Factor models already appeared in the Investments 101 series, in their time-series form. Recall the CAPM and Fama-French setup, where one stock’s return history is regressed against known factor returns over time:

r_{t} = α + β_{1} \cdot M K T_{t} + β_{2} \cdot SM B_{t} + β_{3} \cdot H M L_{t} + ϵ_{t}

Where $r_{t}$ is the stock’s return in period $t$ , $M K T_{t}$ , $SM B_{t}$ and $H M L_{t}$ are the market, size and value factor returns for that period, $α$ is an intercept, and $ϵ_{t}$ is the portion of the return the factors leave unexplained. The factor returns are published, which makes them known. The betas are not, so we estimate them with a time-series regression: many dates, one stock. Run it once for every stock in the universe and we have everyone’s factor sensitivities.

A Barra-style fundamental factor model turns this around. Instead of using returns to estimate betas, it uses characteristics as the betas and estimates factor returns.

Concretely, at one date we already know each company’s characteristics (its sector, its market cap, its book-to-price ratio), and those characteristics are the factor exposures. We line up every stock in the universe on that date and regress their returns against the known exposures:

r_{i} = X_{i, 1} \cdot f_{1} + X_{i, 2} \cdot f_{2} + \dots + X_{i, K} \cdot f_{K} + ϵ_{i}

Where $X_{i, k}$ is stock $i$ ’s exposure to factor $k$ , $f_{k}$ is that factor’s return over the period, and $ϵ_{i}$ is stock $i$ ’s residual return. We looked the exposures up rather than estimating them, which leaves the factor returns as the unknowns. This is a cross-sectional regression: many stocks, one date. Each run produces one set of factor returns, and repeating it period by period gives us a time series of them.

So the two approaches solve for opposite unknowns. Fama-French: known factor returns, unknown betas, regress over time. Barra: known betas (characteristics), unknown factor returns, regress across stocks. The Barra approach has a practical advantage for risk modeling. Exposures come from characteristics rather than from a regression over each stock’s own return history, so a new listing does not need years of data before the model can say something about it. It still needs some—namely, a year of returns for volatility, beta and momentum, a month for reversal, and three years of fundamentals for growth. A newly IPO’d company gets its size, value, quality and sector immediately.

Once we have a time series of factor returns, we can estimate how those factors move together, which gives us a small $K \times K$ factor covariance matrix $F$ . We also estimate each stock’s own residual volatility, collected into the diagonal matrix $Δ$ . Those two pieces are enough to reconstruct the covariance between any pair of stocks.

The idea is that two stocks co-move because they share factor exposures. Apple and Microsoft are both large-cap tech companies. If the tech factor and the size factor are both volatile and correlated, Apple and Microsoft will have high covariance. That high covariance comes from mapping their characteristics through the factor covariance structure, not from a measured correlation between their returns. The covariance between stock $i$ and stock $j$ is $x_{i}^{⊤} F x_{j}$ : take each stock’s exposure vector, and let $F$ translate shared exposures into shared risk. In matrix form for all stocks at once: $Σ = XF X^{⊤} + Δ$ .

That is a large reduction in dimension. In place of 125,250 distinct covariance entries for 500 stocks, a twenty-factor model estimates the $20 \times 21/2 = 210$ distinct entries of $F$ and 500 specific variances. Everything starts with data: a universe of securities and their financial history. The two parquet files behind this post provide both.

The Client-Side Constraint

Exposures, factor returns and a covariance matrix are what the model computes. Where it computes them is a separate choice, and this series makes an unusual one. Most quantitative finance work happens on servers, where the code runs on a remote machine with direct access to databases and fast hardware. A browser asks for a page, the server runs the computation, and the finished result comes back over the wire. That arrangement is server-side computing.

Client-side computing inverts it. The code runs on the machine that loaded the page, inside the browser tab, and the server merely hands over the program and the data. This used to be impractical for anything beyond form validation and animations. WebAssembly changed that. Browsers now run compiled Rust and C++ at near-native speed, and projects like DuckDB WASM put a full analytical database engine inside a tab.

Building the entire risk model client-side is partly a constraint this series adopts for its own sake, and partly a measure of how far that infrastructure has come. We run one step outside the browser: the initial data pull, handled by the Python script in this post that downloads the raw data, cleans it, and writes the parquet files. Once those files are uploaded to cloud storage, everything downstream happens in the browser: querying the data, running regressions, estimating covariance matrices, decomposing risk. A browser tab and some WebAssembly do all of it, with no server compute and no API calls to a backend model.

The Data Pipeline

Getting the data into that tab means moving it over the network, so format choice matters. The data is stored as parquet files in cloud storage. Parquet is columnar: it compresses well, supports column pruning so a query fetches only the columns it touches, and stores data in chunks (called row groups) that can be skipped when they fall outside the filter.

The pipeline:

The browser requests a presigned URL from the server, a temporary, time-limited link to the file in cloud storage.
DuckDB WASM uses that URL to fetch the parquet file via HTTP range requests, reading the metadata footer first, then pulling only the specific row groups and columns the query needs.
Query results come back as rows that Svelte components can render.

DuckDB is an in-process analytical database, and the WASM build runs inside a web worker on the page. That gives us a full SQL interface over remote parquet files with no server-side query engine and no database to provision. The server’s only job is handing out presigned URLs.

Before the code, here is what comes out of it. The explorer below loads both parquet files from cloud storage into DuckDB’s in-memory tables when the page first opens.

Security Master Financials Loading DuckDB and fetching data...

The security master maps each security to its identifying information: tickers, names, sector classifications, and other static attributes. The financials dataset contains the time-varying data: returns, fundamental metrics, and the raw inputs for factor construction. The column shapes matter in the next post, where they become the exposure matrix.

Preparing the Data

The explorer above reads what one Python script produced. The raw data comes from two open sources: financial ratios from sovai’s research datasets on HuggingFace, and security metadata from the FinanceDatabase project. The script pulls both, joins them, applies quality filters, and writes the two parquet files.

The script uses PEP 723 inline metadata, so uv run build_data.py runs it directly, with no virtual environment to set up:

# /// script
# requires-python = ">=3.14"
# dependencies = [
#     "duckdb",
#     "polars",
#     "pyarrow",
# ]
# ///
"""Build parquet files for the factor risk model series.

Data sources:
- Financial ratios: sovai open research datasets (HuggingFace)
- Security metadata: FinanceDatabase (JerBouma/GitHub)

uv run build_data.py
"""

import polars as pl
import polars.selectors as cs

The security universe comes first. We filter to US-listed equities with a known sector and deduplicate by ticker:

US_EXCHANGES = ["NMS", "NCM", "NYQ", "ASE"]

master_df = (
    pl.scan_csv(
        "https://raw.githubusercontent.com/JerBouma/"
        "FinanceDatabase/main/database/equities.csv"
    )
    .filter(
        pl.col("currency").eq("USD")
        & pl.col("exchange").is_in(US_EXCHANGES)
        & pl.col("sector").is_not_null()
    )
    .rename({"symbol": "ticker"})
    .with_columns(
        pl.col("ticker").str.strip_chars().str.to_uppercase()
    )
    .unique(subset=["ticker"], keep="last")
    .select(
        "ticker", "name", "sector", "country", "exchange",
        "market", "currency", "isin", "cusip", "figi",
        "composite_figi", "shareclass_figi", "summary",
    )
    .collect()
)

Then we pull the financial ratios, compute returns from adjusted closes, and derive a sales estimate from market cap and price-to-sales:

financials_df = (
    pl.scan_parquet(
        "https://huggingface.co/datasets/sovai/financial_ratios"
        "/resolve/main/financial_ratios.parquet"
    )
    .with_columns(
        pl.col("ticker")
        .str.strip_chars()
        .str.to_uppercase()
        .str.replace_all(r"\.", "-")
    )
    .join(master_df.lazy().select("ticker"), on="ticker", how="semi")
    .filter(pl.col("date") >= pl.lit("2010-01-01").str.to_datetime())
    .sort("ticker", "date")
    .with_columns(
        pl.col("closeadj").pct_change().over("ticker").alias("returns")
    )
    .select([
        "ticker", "date", "closeadj", "volume", "market_cap",
        "returns", "greenblatt_earnings_yield", "ebitda_to_mev_ratio",
        "cash_flow_to_price_ratio",
        "book_to_market_enterprise_value_ratio", "dividend_yield",
        "return_on_equity", "return_on_assets", "gross_profit_margin",
        "debt_ratio",
        (
            pl.col("market_cap")
            / pl.when(pl.col("price_to_sales").gt(0))
            .then(pl.col("price_to_sales"))
        ).alias("sales"),
        "eps_usd",
    ])
    .collect()
)

Now for the outlier treatment, which is where most of the care goes. Raw financial ratios are noisy, and one data error or micro-cap stock with a 10,000% earnings yield will dominate any cross-sectional regression. We estimate scale with the median absolute deviation (MAD), a measure outliers barely move, and clip each ratio to $median \pm k \cdot 1.4826 \cdot MAD$ , with $k$ set to 5 for the ratios and 10 for returns. The 1.4826 is $1/ Φ^{- 1} (3/4)$ , the scaling that makes MAD a consistent estimator of $σ$ when the data is normally distributed.

Before applying the clip, we need the mad_clip function:

RATIO_MAD_K = 5.0
RETURNS_MAD_K = 10.0
MIN_MCAP = 100  # $100M (market_cap is in millions)
MIN_DV_20W = 1_000_000  # $1M rolling 20-week median dollar volume
SKIP_COLS = {
    "ticker", "date", "closeadj", "volume",
    "market_cap", "eps_usd", "returns",
}
MAD_SCALE = 1.4826


def mad_clip(
    expr: pl.Expr,
    k: float,
    mask: pl.Expr | None = None,
    *,
    skip_zero: bool = False,
) -> pl.Expr:
    """Clip to median +/- k * scaled MAD per cross-section (date)."""
    effective = mask
    if skip_zero:
        nonzero = expr != 0
        effective = (
            (effective & nonzero) if effective is not None else nonzero
        )
    clean = (
        pl.when(effective).then(expr) if effective is not None else expr
    )
    median = clean.median().over("date")
    mad = MAD_SCALE * (clean - median).abs().median().over("date")
    return expr.clip(median - k * mad, median + k * mad)

The clip has two subtleties. The median and MAD are computed only over the investable universe (market cap of 100M or more, and a rolling 20-week median dollar volume above 1M), but the clip is applied to all rows, which keeps illiquid micro-caps from skewing the statistics. And for columns where more than half the values are zero (dividend yield is the obvious one), zeros are excluded from the MAD calculation. Otherwise $MAD = 0$ collapses the clip range to a point.

Both rules are in the pass that flags the investable universe and applies the clip:

dollar_vol = pl.col("closeadj") * pl.col("volume")
dv_median = dollar_vol.rolling_median(window_size=20).over("ticker")
financials_df = financials_df.with_columns(
    (pl.col("market_cap") >= MIN_MCAP).alias("investable_mcap"),
    (dv_median >= MIN_DV_20W).alias("investable_liq"),
).filter(dv_median.is_not_null())

investable = pl.col("investable_mcap") & pl.col("investable_liq")
ratio_cols = cs.expand_selector(
    financials_df, cs.float() - cs.by_name(*SKIP_COLS)
)

zero_heavy = set()
for c in ratio_cols:
    s = financials_df[c].drop_nulls()
    if len(s) > 0 and (s == 0).sum() / len(s) > 0.5:
        zero_heavy.add(c)

financials_df = financials_df.with_columns(
    *[
        mad_clip(
            pl.col(c), RATIO_MAD_K, investable, skip_zero=c in zero_heavy
        )
        for c in ratio_cols
    ],
    mad_clip(pl.col("returns"), RETURNS_MAD_K, investable),
)

Finally, we filter the security master to only tickers that appear in the financials and write both files:

master_df = master_df.join(
    financials_df.select("ticker").unique(), on="ticker", how="semi"
)

master_df.write_parquet("data/factor_model_security_master.parquet")
financials_df.write_parquet("data/factor_model_financials.parquet")

The script writes two parquet files. One is a security master with static attributes, and the other is a financials table with time-series data ready for factor construction. Both get uploaded to cloud storage where the browser-side pipeline can reach them.

Data quality checks this series skips

Both sources are free and open, and this series takes them largely at face value. That’s fine for learning the mechanics and not fine for anything else. The checks below are the ones worth having before trusting a number that came out of this. None has been run here.

Survivorship. Nothing here measures how many delisted names this universe keeps, so calling it biased would be a guess. The test is cheap: count distinct tickers per date and look at the shape of the line. If it climbs steadily toward the present and almost no ticker’s history ends mid-sample, the failures have been removed. If ticker histories keep ending mid-sample, at least some of the failures are still in there. It’s worth knowing, because survivorship does its worst damage to the factors this model is built on: a highly levered, low-quality company only appears in a surviving universe if it pulled through, which makes quality and leverage look safer than they were.
Point-in-time fundamentals. One snapshot cannot say whether these ratios are as-reported-then or as-restated-now, and the difference decides whether anything downstream means much, since restating history in place is lookahead bias. Two ways to find out. The cheap one is reporting lag. A fiscal-quarter figure that first shows up 40 to 90 days after period end is plausible, and one that shows up on the period end date itself is not. The dates in the file are enough to check. The thorough one is to pull the same dataset again in six months and diff the history, because if old values have moved, the vendor is restating in place. Production work buys the answer instead: LSEG, Bloomberg, and Compustat point-in-time all sell data timestamped on knowledge date rather than effective date.
Identifier hygiene. The join key is a ticker, because ticker is the only key the ratios dataset exposes. The security master includes CUSIP, ISIN, and FIGI, any of which would be more stable, and none of which the other side can match on. Tickers get recycled after a delisting, so one ticker’s history here can in principle splice two unrelated companies together. Look for a long gap in coverage followed by fundamentals that jump.
Industry resolution. A modeling limit rather than a data defect, and the first one worth changing. Eleven sectors is coarse next to the 60 GICS-based industry factors in Barra’s USE4. At sector resolution nothing separates a regional bank from a property insurer, so whatever moves those two apart ends up in the residual instead of in a factor, inflating specific risk and hiding real industry co-movement.

What’s Next

We have two parquet files and a query engine. The next step turns raw financial ratios into factor exposures: standardized, cross-sectionally comparable numbers that can enter a regression. The choice of standardization matters more than it appears to, and it has a measurable effect on the model that comes out. That’s the next post.