BUILD
In-house Development
Alopex DB v0.8.16 Tutorial, Part 5: Opening the Database with a Server
Open an embedded data directory with alopex-server, run the same SQL over HTTP, and read the server-written row through the file API.
Posts tagged#Data
View
BUILD
In-house Development
Alopex DB v0.8.16 Tutorial, Part 4: Vector Search and DataFrames
Store embedding vectors with articles, combine similarity search with SQL filters, and reshape the results with DataFrames in Python and Rust.
BUILD
In-house Development
Alopex DB v0.8.16 Tutorial, Part 3: CASE, Set Operations, CTEs, and Window Functions
Use CASE expressions, UNION and other set operations, common table expressions, rankings, and running totals in Alopex DB.
BUILD
In-house Development
Alopex DB v0.8.16 Tutorial, Part 2: Querying Articles with SQL
Create a table and query it with SELECT, JOIN, aggregation, and subqueries, then combine SQL and key-value data in one transaction.
BUILD
In-house Development
Alopex DB v0.8.16 Tutorial, Part 1: Storing Keys and Values
Install Alopex DB v0.8.16 and learn its key-value API, transactions, deletion, and persistent storage in Python and Rust.
DATA
Data & DB
Indexes and the Art of Searching, Part 8: Choosing Methods and a Decision Tree
Published methods are tuned to the data they were evaluated on. How should you budget and plan the work of tuning them to your own data instead?
DATA
Data & DB
Indexes and the Art of Searching, Part 7: Evaluation Functions That Drive Search
Search only moves once something tells it how good a candidate is. How are heuristics, game evaluations, and verifiers built, and where do they break?
DATA
Data & DB
Indexes and the Art of Searching, Part 6: Local Search and Nature-Inspired Methods
Sometimes getting a usable solution quickly matters more than finding the optimal one. Which components do local search and animal-inspired methods let you tune?
DATA
Data & DB
Indexes and the Art of Searching, Part 5: State-Space Search
Car navigation routes and next moves in shogi have too many candidates to survey. What does a search that alternates local evaluation and global pruning prepare in advance?
DATA
Data & DB
Indexes and the Art of Searching, Part 4: Similarity Search and Recommendation
When neither the keys nor the words match, how do you find things that are close? This part sorts out how search and recommendation pull candidates by distance and order them by behavior.
DATA
Data & DB
Indexes and the Art of Searching, Part 3: Searching by Words and Ranking Results
Once you find the documents that contain a word, you still have to decide which to read first. After an inverted index narrows the candidates, what should the ranking rest on?
DATA
Data & DB
Indexes and the Art of Searching, Part 2: Key Lookup, Absence Checks, and Identity
What should you compute in advance to check a key match with minimal reads? Covers hash tables, six kinds of indexes, Bloom filters, and content identity.
DATA
Data & DB
Indexes and the Art of Searching, Part 1: What to Prepare Before Reading
Reading every row always finds the answer. To avoid that full scan wherever possible, what work do sorting and indexes finish before a query ever arrives?
ENG
Engineering
The Art of Structuring, Part 3 — Choosing and Transforming Representations
Are you comparing hashes, trees, and graphs in the abstract? Choose representations for the query, then design for loss and reconstruction.
ENG
Engineering
The Art of Structuring, Part 2 — Structuring Meaning, Relationships, and Execution
How do schemas, graphs, tasks, and packets connect? This article traces structure from meaning through execution, communication, and storage.
ENG
Engineering
The Art of Structuring, Part 1 — From Messy Information to Multiple Projections
Why are records for the same company totalled separately? Starting with record linkage, this article turns familiar data into searchable, aggregatable forms.
BUILD
In-house Development
What Is the Best Database for a Startup? — A Stage-by-Stage Use-Case Analysis and the 'Scaling Cliff'
An analysis of how database requirements change across the MVP, PMF, growth, and scale stages, comparing the trade-offs of SQLite, Postgres, managed databases, and dedicated vector stores — and where Alopex DB's 'erase the migration cliff' approach fits.
BUILD
In-house Development
What Alopex Chirps Is — A Library for Forming Clusters, and the Consensus It Adopted
Building a distributed system means building node discovery, liveness detection, messaging, and agreement before getting to the actual work. Alopex Chirps takes on exactly those four. This covers what it handles and what it does not, why QUIC was chosen, and how openraft came to be the consensus layer.
ENG
Engineering
Consensus Algorithms, Underground (Part 4) — A World Where Identities Cannot Be Counted: PoW and PoS
In a network where neither the identity nor the number of participants is known, one node one vote does not hold. Probabilistic finality in PoW, the structure by which PoS relies on economic penalties rather than impossibility, Solana and Avalanche, and the option of avoiding agreement — with the criteria for choosing among them. Part 4 of a 4-part series.
ENG
Engineering
Consensus Algorithms, Underground (Part 3) — Lying Nodes, from PBFT to DAG-BFT
The world in which nodes send signed, contradictory information. The meaning of the constant n=3f+1, PBFT's three-phase voting, Tendermint's locking rules, HotStuff's quorum certificates, and what the DAG-BFT designs since Narwhal separated and what they removed. Part 3 of a 4-part series.
ENG
Engineering
Consensus Algorithms, Underground (Part 2) — The Crash Failure Family: Paxos and Raft
The family of algorithms for the crash failure model, where nodes stop but do not lie. From the single tool Paxos relies on — quorum intersection — through Raft's decomposition, EPaxos, Multi-Raft, and the parts outside the papers: persistence ordering and linearizable reads, with minimal code. Part 2 of a 4-part series.
ENG
Engineering
Consensus Algorithms, Underground (Part 1) — Crashes and Delays Are Indistinguishable
A node that stops responding may have crashed, or it may merely be slow; the two cannot be told apart. This part separates leader election, ordering, and commitment into three distinct problems, and covers the practical meaning of FLP impossibility, split brain and fencing tokens, and the boundary between the replicated log and the applied state. Part 1 of a 4-part series.
BUILD
In-house Development
Even So, I'm Building Alopex DB
SQLite, DuckDB, PostgreSQL, distributed DBs, vector DBs, graph DBs — there is no shortage of great products. I'm building a new database anyway because every time the project moves forward, I re-pick the DB and rebuild the data itself. Alopex DB aims to be a foundation that keeps data's volume, conversion time, regeneration cost, and provenance intact as you scale from local to cloud, from single node to distributed.
AI
GenAI
Part 1: What Does Generative AI Learn as the "Answer"? — Self-Supervised Learning and LLMs
Neither LLMs nor image generators learn from training data that humans have labeled item by item. The key is self-supervised learning: mechanically constructing inputs and answers from raw data. Part 1 clarifies the difference between raw data and training data, and shows how an LLM turns text itself into a next-token prediction problem, with minimal code. Part 1 of a 5-part series.
DATA
Data & DB
The Skill AI Can't Beat: Data Engineering
Coding agents are strong at logic and tests. But they can't tell what a piece of data means, who owns it, how fresh it flows, or which copy is authoritative. It follows from the fact that today's AI has no embodiment: the people who rise in value are those who can design the meaning, quality, lineage, and responsibility of data. A look at the trend with the latest data-engineering discussion.
BUILDBUILDBUILDBUILDBUILDDATADATADATADATADATADATADATADATAENGENGENGBUILDBUILDENGENGENGENGBUILDAIDATA
Alopex DB v0.8.16 Tutorial, Part 5: Opening the Database with a Server
Open an embedded data directory with alopex-server, run the same SQL over HTTP, and read the server-written row through the file API.
Alopex DB v0.8.16 Tutorial, Part 4: Vector Search and DataFrames
Store embedding vectors with articles, combine similarity search with SQL filters, and reshape the results with DataFrames in Python and Rust.
Alopex DB v0.8.16 Tutorial, Part 3: CASE, Set Operations, CTEs, and Window Functions
Use CASE expressions, UNION and other set operations, common table expressions, rankings, and running totals in Alopex DB.
Alopex DB v0.8.16 Tutorial, Part 2: Querying Articles with SQL
Create a table and query it with SELECT, JOIN, aggregation, and subqueries, then combine SQL and key-value data in one transaction.
Alopex DB v0.8.16 Tutorial, Part 1: Storing Keys and Values
Install Alopex DB v0.8.16 and learn its key-value API, transactions, deletion, and persistent storage in Python and Rust.
Indexes and the Art of Searching, Part 8: Choosing Methods and a Decision Tree
Published methods are tuned to the data they were evaluated on. How should you budget and plan the work of tuning them to your own data instead?
Indexes and the Art of Searching, Part 7: Evaluation Functions That Drive Search
Search only moves once something tells it how good a candidate is. How are heuristics, game evaluations, and verifiers built, and where do they break?
Indexes and the Art of Searching, Part 6: Local Search and Nature-Inspired Methods
Sometimes getting a usable solution quickly matters more than finding the optimal one. Which components do local search and animal-inspired methods let you tune?
Indexes and the Art of Searching, Part 5: State-Space Search
Car navigation routes and next moves in shogi have too many candidates to survey. What does a search that alternates local evaluation and global pruning prepare in advance?
Indexes and the Art of Searching, Part 4: Similarity Search and Recommendation
When neither the keys nor the words match, how do you find things that are close? This part sorts out how search and recommendation pull candidates by distance and order them by behavior.
Indexes and the Art of Searching, Part 3: Searching by Words and Ranking Results
Once you find the documents that contain a word, you still have to decide which to read first. After an inverted index narrows the candidates, what should the ranking rest on?
Indexes and the Art of Searching, Part 2: Key Lookup, Absence Checks, and Identity
What should you compute in advance to check a key match with minimal reads? Covers hash tables, six kinds of indexes, Bloom filters, and content identity.
Indexes and the Art of Searching, Part 1: What to Prepare Before Reading
Reading every row always finds the answer. To avoid that full scan wherever possible, what work do sorting and indexes finish before a query ever arrives?
The Art of Structuring, Part 3 — Choosing and Transforming Representations
Are you comparing hashes, trees, and graphs in the abstract? Choose representations for the query, then design for loss and reconstruction.
The Art of Structuring, Part 2 — Structuring Meaning, Relationships, and Execution
How do schemas, graphs, tasks, and packets connect? This article traces structure from meaning through execution, communication, and storage.
The Art of Structuring, Part 1 — From Messy Information to Multiple Projections
Why are records for the same company totalled separately? Starting with record linkage, this article turns familiar data into searchable, aggregatable forms.
What Is the Best Database for a Startup? — A Stage-by-Stage Use-Case Analysis and the 'Scaling Cliff'
An analysis of how database requirements change across the MVP, PMF, growth, and scale stages, comparing the trade-offs of SQLite, Postgres, managed databases, and dedicated vector stores — and where Alopex DB's 'erase the migration cliff' approach fits.
What Alopex Chirps Is — A Library for Forming Clusters, and the Consensus It Adopted
Building a distributed system means building node discovery, liveness detection, messaging, and agreement before getting to the actual work. Alopex Chirps takes on exactly those four. This covers what it handles and what it does not, why QUIC was chosen, and how openraft came to be the consensus layer.
Consensus Algorithms, Underground (Part 4) — A World Where Identities Cannot Be Counted: PoW and PoS
In a network where neither the identity nor the number of participants is known, one node one vote does not hold. Probabilistic finality in PoW, the structure by which PoS relies on economic penalties rather than impossibility, Solana and Avalanche, and the option of avoiding agreement — with the criteria for choosing among them. Part 4 of a 4-part series.
Consensus Algorithms, Underground (Part 3) — Lying Nodes, from PBFT to DAG-BFT
The world in which nodes send signed, contradictory information. The meaning of the constant n=3f+1, PBFT's three-phase voting, Tendermint's locking rules, HotStuff's quorum certificates, and what the DAG-BFT designs since Narwhal separated and what they removed. Part 3 of a 4-part series.
Consensus Algorithms, Underground (Part 2) — The Crash Failure Family: Paxos and Raft
The family of algorithms for the crash failure model, where nodes stop but do not lie. From the single tool Paxos relies on — quorum intersection — through Raft's decomposition, EPaxos, Multi-Raft, and the parts outside the papers: persistence ordering and linearizable reads, with minimal code. Part 2 of a 4-part series.
Consensus Algorithms, Underground (Part 1) — Crashes and Delays Are Indistinguishable
A node that stops responding may have crashed, or it may merely be slow; the two cannot be told apart. This part separates leader election, ordering, and commitment into three distinct problems, and covers the practical meaning of FLP impossibility, split brain and fencing tokens, and the boundary between the replicated log and the applied state. Part 1 of a 4-part series.
Even So, I'm Building Alopex DB
SQLite, DuckDB, PostgreSQL, distributed DBs, vector DBs, graph DBs — there is no shortage of great products. I'm building a new database anyway because every time the project moves forward, I re-pick the DB and rebuild the data itself. Alopex DB aims to be a foundation that keeps data's volume, conversion time, regeneration cost, and provenance intact as you scale from local to cloud, from single node to distributed.
Part 1: What Does Generative AI Learn as the "Answer"? — Self-Supervised Learning and LLMs
Neither LLMs nor image generators learn from training data that humans have labeled item by item. The key is self-supervised learning: mechanically constructing inputs and answers from raw data. Part 1 clarifies the difference between raw data and training data, and shows how an LLM turns text itself into a next-token prediction problem, with minimal code. Part 1 of a 5-part series.
The Skill AI Can't Beat: Data Engineering
Coding agents are strong at logic and tests. But they can't tell what a piece of data means, who owns it, how fresh it flows, or which copy is authoritative. It follows from the fact that today's AI has no embodiment: the people who rise in value are those who can design the meaning, quality, lineage, and responsibility of data. A look at the trend with the latest data-engineering discussion.