← Resources
Guide · Working notes

Analysing large datasets quickly and cheaply with Claude

A workflow I keep coming back to: point Claude at a folder, get it into DuckDB, and let SQL do the heavy lifting instead of the context window.

Caveat first. This is for large datasets I'm working with locally on my own machine. The "proper" way, if you're doing this at scale or for a team, is to load the data into a cloud warehouse like BigQuery, Snowflake or Redshift, model the transformations in something like dbt, and orchestrate the whole pipeline on a schedule with a tool like Airflow — version-controlled, tested, and reproducible for everyone. That's the right answer when reliability and sharing matter more than speed of setup. What follows is deliberately lighter than that: the setup that's fast, cheap, and low-friction enough that I actually reach for it on my own. If it works for you too, great.

The whole thing rests on one idea: don't make Claude read the data. Large datasets and large language models are a bad financial match — every row you pour into the context window costs tokens, and it adds up frighteningly fast. So instead of loading the data into Claude, I load it into a database and let Claude query it. Claude reads the shape of the data and the results of questions, not the millions of rows in between.

Here's the flow.

  1. Download the data into a directory

    Nothing clever here. Get all the files into one folder — CSVs, Parquet, whatever the source hands you. Keep it tidy, because the folder is the thing you're going to point Claude at.

  2. Point Claude at the directory

    Usually I do this on the command line, from inside the folder, so Claude can see the files directly. That way it can list them, peek at headers, and check row counts itself rather than me describing everything.

  3. Ask it to set up a local database and ingest the data

    This is the key step. I tell Claude to spin up a project using DuckDB and load the files into it. DuckDB is built exactly for this — fast analytical queries over big local files, no server to run, reads CSV and Parquet natively.

    It doesn't have to be DuckDB, though. That's just what I've settled on. SQLite works fine, and if you already run Postgres or query Parquet with something else, use that. The point isn't the specific tool — it's getting the data into something Claude can query with SQL instead of reading row by row. DuckDB is simply the one that's given me the least friction for local files.

    Two things it buys me. Queries get much faster than repeatedly parsing raw files. And, more importantly, it kills the need to load the data into Claude at all — Claude writes SQL, the database returns a small result, and only that result comes back into the conversation. That's the difference between a cheap session and an eye-watering one.

    Set up a DuckDB project in this directory and ingest all the CSVs into it. (Use SQLite or whatever you'd normally reach for — I just want it queryable with SQL.) Then show me the tables and row counts so I can sanity-check the load.
  4. Optionally, explain how the data links together

    Claude can often infer the relationships between files on its own. But it's usually safer to spend a few minutes spelling them out — which column joins to which, which files are just lookup/reference tables, where the keys live.

    This matters most with messy real-world data. NHS prescribing data is the example I always hit: the main prescription files reference codes — BNF codes for drugs, practice and organisation codes — that only make sense once you join them to separate reference files. If you leave Claude to guess those joins, it might guess wrong and quietly give you plausible-but-off numbers. Ten minutes describing the schema up front saves an afternoon of doubting your results.

  5. Build dashboards as HTML

    Once the data's queryable, asking for analysis is the easy part. What I've found actually useful is having Claude build the output as a self-contained HTML page — a little dashboard I can open, keep, and come back to, rather than a wall of numbers in a chat that scrolls away.

    This is exactly what FigurePages is for, and honestly why it got built. Claude generates the page, publishes it, and hands back a link. A link I can message to someone, reopen next week, or line up next to last month's version. That's so much faster to manage than regenerating the same view every time I want to look at it.

    Here's a real one built straight off this workflow — an NHS England medicines snapshot for April 2026. The prescribing data went into a local database, Claude queried it, and the whole thing came back as a single page I can just send to someone.

    FigurePages Share & stats NHS England medicines — April 2026 snapshot A single month across primary care (EPD) and secondary care (SCMD), linked to the dm+d dictionary and NHS ODS. Total medicines spend, April 2026 £3.07bn list price · £907M + £2.16bn Primary care £907M 106.7M items · 9,230 GP practices Secondary care £2.16bn 191 NHS trusts · indicative cost Largest single drug £72M tirzepatide (Mounjaro), primary care 1 · Where the money goes (primary care) Spend by therapeutic area (BNF chapter) EPD actual cost, £, April 2026 06 Endocrine system £198.2M 04 Central nervous system £129.2M 02 Cardiovascular system £104.5M 03 Respiratory system £100.5M 21 Appliances £75.8M 09 Nutrition & blood £72.3M 01 Gastro-intestinal £55.9M 23 Stoma appliances £35.4M 13 Skin £24.5M Spend by NHS region EPD actual cost, £, April 2026 Midlands £185.5M North East & Yorkshire £151.7M South East £139.2M North West £126.9M London £113.4M East of England £101.9M South West £87.7M
    The NHS England medicines snapshot as it renders on FigurePages — built from the local database, not from raw rows in the chat.

Why this works

Strip it back and the trick is just moving the data out of the expensive place (the context window) and into the cheap place (a local database), then using Claude for what it's genuinely good at — writing the SQL, spotting the patterns, and turning a result set into something you can actually read. The data stays on your machine, the queries stay fast, and the bill stays small.

It's not the "right" way to run analytics at scale. It's just a setup that's quick enough and cheap enough that I use it constantly — and that counts for a lot.

Turn your next analysis into a link.

Open FigurePages