ENG
Engineering
The Art of Structuring, Part 3 — Choosing and Transforming Representations
Are you comparing hashes, trees, and graphs in the abstract? Choose representations for the query, then design for loss and reconstruction.
Posts tagged#Data
View
ENG
Engineering
The Art of Structuring, Part 2 — Structuring Meaning, Relationships, and Execution
How do schemas, graphs, tasks, and packets connect? This article traces structure from meaning through execution, communication, and storage.
ENG
Engineering
The Art of Structuring, Part 1 — From Messy Information to Multiple Projections
Why are records for the same company totalled separately? Starting with record linkage, this article turns familiar data into searchable, aggregatable forms.
BUILD
In-house Development
What Is the Best Database for a Startup? — A Stage-by-Stage Use-Case Analysis and the 'Scaling Cliff'
An analysis of how database requirements change across the MVP, PMF, growth, and scale stages, comparing the trade-offs of SQLite, Postgres, managed databases, and dedicated vector stores — and where Alopex DB's 'erase the migration cliff' approach fits.
BUILD
In-house Development
What Alopex Chirps Is — A Library for Forming Clusters, and the Consensus It Adopted
Building a distributed system means building node discovery, liveness detection, messaging, and agreement before getting to the actual work. Alopex Chirps takes on exactly those four. This covers what it handles and what it does not, why QUIC was chosen, and how openraft came to be the consensus layer.
ENG
Engineering
Consensus Algorithms, Underground (Part 4) — A World Where Identities Cannot Be Counted: PoW and PoS
In a network where neither the identity nor the number of participants is known, one node one vote does not hold. Probabilistic finality in PoW, the structure by which PoS relies on economic penalties rather than impossibility, Solana and Avalanche, and the option of avoiding agreement — with the criteria for choosing among them. Part 4 of a 4-part series.
ENG
Engineering
Consensus Algorithms, Underground (Part 3) — Lying Nodes, from PBFT to DAG-BFT
The world in which nodes send signed, contradictory information. The meaning of the constant n=3f+1, PBFT's three-phase voting, Tendermint's locking rules, HotStuff's quorum certificates, and what the DAG-BFT designs since Narwhal separated and what they removed. Part 3 of a 4-part series.
ENG
Engineering
Consensus Algorithms, Underground (Part 2) — The Crash Failure Family: Paxos and Raft
The family of algorithms for the crash failure model, where nodes stop but do not lie. From the single tool Paxos relies on — quorum intersection — through Raft's decomposition, EPaxos, Multi-Raft, and the parts outside the papers: persistence ordering and linearizable reads, with minimal code. Part 2 of a 4-part series.
ENG
Engineering
Consensus Algorithms, Underground (Part 1) — Crashes and Delays Are Indistinguishable
A node that stops responding may have crashed, or it may merely be slow; the two cannot be told apart. This part separates leader election, ordering, and commitment into three distinct problems, and covers the practical meaning of FLP impossibility, split brain and fencing tokens, and the boundary between the replicated log and the applied state. Part 1 of a 4-part series.
BUILD
In-house Development
Even So, I'm Building Alopex DB
SQLite, DuckDB, PostgreSQL, distributed DBs, vector DBs, graph DBs — there is no shortage of great products. I'm building a new database anyway because every time the project moves forward, I re-pick the DB and rebuild the data itself. Alopex DB aims to be a foundation that keeps data's volume, conversion time, regeneration cost, and provenance intact as you scale from local to cloud, from single node to distributed.
AI
GenAI
Part 1: What Does Generative AI Learn as the "Answer"? — Self-Supervised Learning and LLMs
Neither LLMs nor image generators learn from training data that humans have labeled item by item. The key is self-supervised learning: mechanically constructing inputs and answers from raw data. Part 1 clarifies the difference between raw data and training data, and shows how an LLM turns text itself into a next-token prediction problem, with minimal code. Part 1 of a 5-part series.
DATA
Data & DB
The Skill AI Can't Beat: Data Engineering
Coding agents are strong at logic and tests. But they can't tell what a piece of data means, who owns it, how fresh it flows, or which copy is authoritative. It follows from the fact that today's AI has no embodiment: the people who rise in value are those who can design the meaning, quality, lineage, and responsibility of data. A look at the trend with the latest data-engineering discussion.
ENGENGENGBUILDBUILDENGENGENGENGBUILDAIDATA
The Art of Structuring, Part 3 — Choosing and Transforming Representations
Are you comparing hashes, trees, and graphs in the abstract? Choose representations for the query, then design for loss and reconstruction.
The Art of Structuring, Part 2 — Structuring Meaning, Relationships, and Execution
How do schemas, graphs, tasks, and packets connect? This article traces structure from meaning through execution, communication, and storage.
The Art of Structuring, Part 1 — From Messy Information to Multiple Projections
Why are records for the same company totalled separately? Starting with record linkage, this article turns familiar data into searchable, aggregatable forms.
What Is the Best Database for a Startup? — A Stage-by-Stage Use-Case Analysis and the 'Scaling Cliff'
An analysis of how database requirements change across the MVP, PMF, growth, and scale stages, comparing the trade-offs of SQLite, Postgres, managed databases, and dedicated vector stores — and where Alopex DB's 'erase the migration cliff' approach fits.
What Alopex Chirps Is — A Library for Forming Clusters, and the Consensus It Adopted
Building a distributed system means building node discovery, liveness detection, messaging, and agreement before getting to the actual work. Alopex Chirps takes on exactly those four. This covers what it handles and what it does not, why QUIC was chosen, and how openraft came to be the consensus layer.
Consensus Algorithms, Underground (Part 4) — A World Where Identities Cannot Be Counted: PoW and PoS
In a network where neither the identity nor the number of participants is known, one node one vote does not hold. Probabilistic finality in PoW, the structure by which PoS relies on economic penalties rather than impossibility, Solana and Avalanche, and the option of avoiding agreement — with the criteria for choosing among them. Part 4 of a 4-part series.
Consensus Algorithms, Underground (Part 3) — Lying Nodes, from PBFT to DAG-BFT
The world in which nodes send signed, contradictory information. The meaning of the constant n=3f+1, PBFT's three-phase voting, Tendermint's locking rules, HotStuff's quorum certificates, and what the DAG-BFT designs since Narwhal separated and what they removed. Part 3 of a 4-part series.
Consensus Algorithms, Underground (Part 2) — The Crash Failure Family: Paxos and Raft
The family of algorithms for the crash failure model, where nodes stop but do not lie. From the single tool Paxos relies on — quorum intersection — through Raft's decomposition, EPaxos, Multi-Raft, and the parts outside the papers: persistence ordering and linearizable reads, with minimal code. Part 2 of a 4-part series.
Consensus Algorithms, Underground (Part 1) — Crashes and Delays Are Indistinguishable
A node that stops responding may have crashed, or it may merely be slow; the two cannot be told apart. This part separates leader election, ordering, and commitment into three distinct problems, and covers the practical meaning of FLP impossibility, split brain and fencing tokens, and the boundary between the replicated log and the applied state. Part 1 of a 4-part series.
Even So, I'm Building Alopex DB
SQLite, DuckDB, PostgreSQL, distributed DBs, vector DBs, graph DBs — there is no shortage of great products. I'm building a new database anyway because every time the project moves forward, I re-pick the DB and rebuild the data itself. Alopex DB aims to be a foundation that keeps data's volume, conversion time, regeneration cost, and provenance intact as you scale from local to cloud, from single node to distributed.
Part 1: What Does Generative AI Learn as the "Answer"? — Self-Supervised Learning and LLMs
Neither LLMs nor image generators learn from training data that humans have labeled item by item. The key is self-supervised learning: mechanically constructing inputs and answers from raw data. Part 1 clarifies the difference between raw data and training data, and shows how an LLM turns text itself into a next-token prediction problem, with minimal code. Part 1 of a 5-part series.
The Skill AI Can't Beat: Data Engineering
Coding agents are strong at logic and tests. But they can't tell what a piece of data means, who owns it, how fresh it flows, or which copy is authoritative. It follows from the fact that today's AI has no embodiment: the people who rise in value are those who can design the meaning, quality, lineage, and responsibility of data. A look at the trend with the latest data-engineering discussion.