Even So, I'm Building Alopex DB

What am I even doing, building a new database at this point?

That thought comes to me again and again while I work on Alopex DB. There are already plenty of excellent databases out there. SQLite is widely used as a self-contained, serverless, embedded SQL database. DuckDB is designed as an in-application analytical database that can also handle surrounding technologies like Parquet and the S3 API. FoundationDB is a key-value store that provides ACID transactions in distributed environments. There are distributed SQL databases like CockroachDB, built around horizontal scaling and fault tolerance. It isn’t that existing databases are immature and that’s why I want to build my own. If anything, each of these products is extremely well made for its own place.

If you want something lightweight running locally, SQLite is strong. If you want to analyze data at hand, DuckDB is convenient. If you want to run robustly in the cloud, you can pick PostgreSQL or a distributed SQL engine. If you need vector search, there are dedicated vector databases. If you deal with graphs, there are graph databases and graph libraries. Each individual choice is perfectly natural.

The problem is that every time I start a project, change the approach, or scale out, I have to redo that choice — and the data conversion — over and over.

It’s Not Just Swapping Out a Library

Over the years I’ve built things like GraphRAG, AI agents, time-series analysis, experiment management, training-data creation, and distributed execution platforms. At the start it’s a small experiment, so a local file or SQLite is plenty. Once I want to analyze a bit, I reach for DuckDB. Once I want to touch it from multiple processes or containers, PostgreSQL becomes a candidate. Take it to the cloud and managed DBs, object storage, queues, and job platforms enter the picture. Think about scaling out and yet another distributed DB becomes a candidate. Add RAG and you need a vector DB; build GraphRAG and you also have to think about where to store the graph structure.

Just carrying a single project through to the end, I end up rewriting the same data many times. At launch I sketch a schema in SQLite. When analysis requirements come up, I dump it into DuckDB. When it becomes collaborative work, I move it to PostgreSQL. When I push it to the cloud, I trim the schema again to fit the managed DB’s constraints. Each time, an ID that was AUTOINCREMENT becomes a UUID; attributes I had stashed in a JSON column get folded back into normalized tables; and things that wouldn’t fit in a table get pushed out into yet another JSON blob. The same “customer,” “document,” and “execution log” take on a slightly different shape every time they move environments.

What happens here is not a mere library swap.

When I was handling a few thousand records locally, running a script once was enough. But once it becomes millions, then tens of millions, and starts to include vectors, graphs, and evaluation results, the word “migration” no longer covers it. You have to read the data out, transform it, load it back, rebuild indexes, regenerate embeddings, reconstruct the graph’s nodes and edges, preserve the correspondence with evaluation results, and make it resumable if it fails partway. The migration script is no longer a few-line converter — it’s a small batch-processing system in its own right.

On the design diagram, this all looks clean. Draw an arrow from SQLite to PostgreSQL, another from PostgreSQL to the vector DB, and another from there to GraphRAG, and you can explain the system’s flow. Abstract the interface, put in a Repository, write an adapter, and it looks tidy as code too. While you’re drawing the arrows and the class diagrams, it really does feel like progress.

But the things that actually flow along those arrows have volume.

The Data Flowing Along Those Arrows Has Volume

Converting a few megabytes of JSON is not the same as converting tens of gigabytes of history. Regenerating a few thousand embeddings is not the same as regenerating tens of millions. The former finishes in an instant; the latter ties up a GPU for hours and hits your wallet through electricity and cloud bills. DuckDB even has disk spill for handling large data, but even so, which operations sit in memory, which spill to disk, and where you make it re-runnable are real design problems. With the very same SQL, whether it finishes overnight or is still going in the morning — having eaten all the memory and died — is decided somewhere between the volume of the data and the limits of the machine.

What makes this even more troublesome is that these days the migration and conversion scripts are often written by a coding agent. Agents are good at writing the transformation logic, the type definitions, and the API boundaries themselves. But whether the data is tens of gigabytes and reading it all into memory takes down the entire WSL environment, whether a failure partway means starting over from scratch, whether the regenerated embeddings and node IDs no longer line up with past evaluation results — that runtime heaviness rarely gets their attention. Unless you point it out, the agent reads everything with read_csv, builds a giant array, exhausts the memory, tweaks it a little after it crashes, and crashes again. That kind of annoying trial and error happens over and over.

Concretely, here’s what happens. Ask it to convert a tens-of-gigabytes table and the agent will blithely write code that loads every row into memory first. It works on a small local sample, so it believes it wrote correct code. Run that at production scale and it uses up WSL’s memory, and either the Linux OOM killer kills the process or the whole of Windows gets dragged down and grinds to a halt. I look at the logs and say, “Do this in chunks.” The agent chunks it. Now the ID numbering drifts at the chunk boundaries. I say something again. Next, it can’t resume from a mid-run failure, so a crash means starting over. The care that batch processing obviously needs — streaming reads, idempotent re-runs, checkpoints — just doesn’t come out on the first try.

This is not a story about the agent being lazy. It’s because today’s AI-assisted development sees the world centered on the shapes of flows, models, APIs, interfaces, and functions. The boundaries of processing are highly visible. But the volume of data passing through those boundaries, the conversion time, the regeneration cost, and the recovery point on failure tend to be treated as background. A human knows in their body that carrying ten cardboard boxes and emptying an entire warehouse are different jobs. That “sense of weight” falls out when you look only at the code’s abstractions.

This is where I feel the precariousness of development in the age of AI.

It’s Not Migration — It’s Rebuilding Knowledge

Say you build nodes from documents in GraphRAG, generate embeddings, draw edges, and evaluate the search results. When you later have to change the DB, the problem is not that “the storage location changes.” The node IDs might change. The unit of embedding generation might change. The boundaries of the split chunks might change. The past search results that the evaluation data referenced might no longer correspond to the new search results.

When that happens, the evaluation results where you once confirmed “this evidence node worked for this question” are left hanging. The node ID they pointed to no longer exists. Change the way chunks are cut by a single step and the graph you thought you built from the same documents becomes a different thing, and the very ground for comparing accuracy disappears. What you took to be just moving the storage location has thrown away the verification you built up.

It’s the same with training data. Which source data a dataset was built from, which script transformed it, which model labeled it, and where a human checked it. Without that history preserved, even if you later think you made the same thing, you can’t really say it’s the same. Bump the version of the model used for labeling and the judgment on borderline cases quietly changes. Mix that into retraining without knowing, and it comes back as an unexplained wobble in your scores. In the Data-centric AI discussion, it’s argued that we need to think about AI performance not just from the model but from data quality, quantity, and maintenance. Surveys of Data-centric AI likewise lay out the need to cover the whole lifecycle, including training data, inference data, and data maintenance.

I strongly agree with this view. But the problem I feel on the ground is at a slightly lower level.

The Data Quietly Becomes Something Else

Even as we say we put data at the center, once it comes to implementation the data quickly becomes subordinate to the convenience of processing. This format locally, that format for analysis, this DB in the cloud, this index for search, this vector for RAG, this CSV for evaluation. While you pick the optimal tool for each purpose, the data slowly turns into something else. Every single move is reasonable and, in the moment, the right call. That’s exactly why there’s no stopping point, and before you know it slightly different versions of the data are scattered all over.

For example, take a table’s primary key. At first a sequential id was enough. Once multiple processes start writing, you switch to a composite (tenant_id, id) to avoid collisions. You add created_at and updated_at for auditing, and you add deleted_at when you move from physical to logical deletes. For capacity or schema reasons you sometimes drop columns that ended up unused later. Each one makes sense as a storage or consistency concern at that point in time. But the coding agent writing this doesn’t know what a column was originally there to express. See a name like deleted_at and it infers a logical-delete flag; see tenant_id and it infers tenant isolation. Within what the name lets it guess, it’s correct. But in reality deleted_at might be operated as “the time it was hidden from view” rather than “the time it was deleted,” and tenant_id might be assigned per contract rather than per tenant. Such original intent — the kind that doesn’t show up in the column name — gets replaced, absent someone to carry it forward, by whatever interpretation happens to be convenient at the time. This isn’t because a batch missed an update. It’s because every time you add or reorganize a column, you’re implicitly re-deciding the very interpretation of what that column represents. And that re-deciding doesn’t happen just once. Every time you have an agent write a pipeline or migration code, another layer of name-inferred interpretation stacks up, and the distance from the original intent widens little by little.

And you notice it has become something else, usually, only after the fact.

The search results are subtly different from before. The evaluation scores don’t reproduce. You can’t tell which version of the data you used for retraining. Take an experiment you tried locally to the cloud and the ID correspondence breaks. It works in a container, but the moment you go to a distributed setup the assumptions about processing order and consistency change. Individually, each is a small discrepancy. But pile them up and you end up spending a whole day chasing “why don’t last week’s numbers come out,” and in most cases the culprit isn’t the model — it’s the data that quietly got swapped out along the way. Problems like these never appear on the clean diagram at the design stage.

What made me want to build Alopex DB is that these experiences kept accumulating.

I Want to Grow Knowledge, Not Repack It

I’m not trying to deny the way of choosing SQLite locally, DuckDB for analysis, PostgreSQL in the cloud, another DB when distributed, yet another for vectors, and a separate library for graphs. That’s realistic, powerful, and still the right choice today.

But if it means rebuilding the data itself each time, the story changes.

Change the DB whenever the development stage changes. Change the schema whenever the DB changes. Write a conversion whenever the schema changes. Regenerate embeddings, rebuild the graph, reorganize the training data, and reconnect the evaluation results with every conversion. This is less like growing the application and more like repacking the same knowledge into a different container over and over. And every time you repack, something spills. Provenance is severed, ID correspondence fades, and past evaluations become unusable.

Concretely, here’s what changes. An object you could nest in Parquet has to be flattened into columns or pushed into a JSON string once you move it to a SQLite or DuckDB table. Move it to a vector DB and you have to recompute the embedding every time you fix a single character of text, and then you’re stuck deciding whether to delete the old vector or leave it in place while past evaluations still reference it. Add sparse search on PostgreSQL and you build yet more columns and indexes for that. Every move changes the very shape of the data.

On top of that, whenever I want to try some feature, the agent adds a column on the spot, picks a name off the cuff, and shows me a result right away. Having trial and error spin around several times a day is genuinely welcome. But when it doesn’t work out, it just adds the next column. And so, ever-so-slightly different data gets duplicated and piles up, and the amount of storage swells by double, then double again.

At first that’s fine. A few thousand records, you can redo. A few tens of thousands, you can wait a night. The trouble is the way it grows. Columns pile up like this, and you rebuild every time you change the DB. You can’t call this knowledge growing. The amount of data increases, but the amount of “information” doesn’t — you’re just changing the format and repeating scrap-and-build every time.

What I Want Alopex DB to Do

This is where what I want to do with Alopex DB lies.

Data born locally keeps being treated as the same data even as it moves to a container, spreads to the cloud, and scales to a distributed setup. The execution form changes, but you don’t rebuild the data model every time. The storage location changes, but the ties to IDs, provenance, generation conditions, and evaluation results don’t break. Embeddings, graphs, and training data are treated as things you’ve built up so far, rather than being remade from zero each time the environment changes. Scaling out should be an operation that makes the same knowledge bigger, not an occasion to throw knowledge away and rebuild it.

This is not simply a wish for “a DB you can use through the same API.”

Even if only the API is the same, it means nothing if the meaning of the data changes underneath. It may look like it works when you swap the connection string, but if the ID numbering or the way types are rounded has changed, it isn’t the same data. What I want is not a thin interface that looks the same from local to distributed, but a database built on the premise that data grows, gets heavier, takes time to move, and costs money to regenerate.

Alopex DB aims to handle structured data, time series, vectors, graphs, metadata, provenance, and audit information within the same body of knowledge from the start. I’m not trying to build a giant DB that does everything omnipotently. If anything, the starting point can be small. Running locally and being embeddable in an application matters. On top of that, when the data grows and the project reaches its next phase, I want to be able to keep the knowledge I built at the start rather than throw it away.

This is the reason I’m building Alopex DB.

It’s not that existing DBs lack features. It’s that simply combining existing DBs makes data’s identity fragile every time the project grows, and enormous cost and time go into guaranteeing that identity. In systems like AI agents and GraphRAG, repairing the broken parts also ends up being handed to the agent in the end. Have it write conversion scripts, build correspondence tables, hunt for missing metadata, redo evaluations. But the agent doesn’t know what the original intent was. So the one spot it thought it fixed spawns another mismatch, and you have it write yet another script to fix that mismatch. Into that endless work, time gets sucked in without limit. Time that could have gone to improving the model or the experience vanishes into reconnecting data’s identity by hand. I want that time back.

Thinking Again, Starting from the Data Itself

I want to think about databases once more, starting from the data itself.

Projects grow and AI advances. To keep up with that pace, switching DBs and storage one after another has become the norm. The shape of the processing and the models we use — the upper layers — shift constantly. The problem is that every switch changes the data model too. Growth means more data, and all of that increase has to be moved over, whole, to the next environment. Moving it over costs enormously in conversion and regeneration. And that cost grows the more the data piles up. The more you grow, the heavier the next step becomes.

That’s exactly why I want the data model and storage to be, not disposable parts you throw away as the stages advance, but a backbone you carry through the whole growth of the project and the AI. Whatever shifts in the upper layers, the data beneath stays the same. I want to reset data into that position.

What I want is not a place to repack data, but a place where you can grow it. A place where trial and error, failures, and evaluations all pile up without being thrown away. That place is what I want to build with Alopex DB.