Alopex DB v0.8.8 Tutorial, Part 1: Storing Keys and Values

Alopex DB is a database engine that handles key-value storage, SQL, and vector search. You can keep the same data files as your application grows from an embedded setup to a server and then a cluster.

The design aims to avoid combining SQLite with Faiss just to add vector search, or migrating from an embedded database to a server database when the application grows.

Part 1 covers the key-value API. We will store a value under a key and retrieve it with the same key.

The examples use Python. Alopex DB itself is written in Rust, and the Python bindings wrap the Rust implementation. Both languages use the same storage format, so a data directory created by either language can be opened by the other.

1. Environment and installation

The results in this article were verified in the following environment.

Install the Python package.

pip install --no-cache-dir "alopex==0.8.8"

Pinning the version lets you compare the results shown here with the results on your own machine.

These examples use the 0.8.8 package published on PyPI. The v0.8.8 release verification installed only the published packages, without mixing in source builds, and confirmed that the library, embedded, server, and cluster modes could operate on the same data.

2. put and get

Use put to write a value and get to read it. Save the following as hello.py.

from alopex import Database, TxnMode

db = Database.new()
with db.begin(TxnMode.READ_WRITE) as txn:
    txn.put(b"greeting", b"hello")
    txn.commit()

The program produces no output yet, so add a read for the value you stored.

with db.begin(TxnMode.READ_ONLY) as txn:
    print(txn.get(b"greeting"))

b'hello'

get() returns the stored value unchanged. The b"..." prefix indicates that both keys and values are byte strings. Decode the value to read it as text.

print(txn.get(b"greeting").decode())

hello

Database.new() creates an in-memory database. It disappears when the process exits, which makes it useful for disposable experiments.

The same result is recorded with VHS. The recording installs alopex==0.8.8 from PyPI inside a container and runs the same put, get, and missing-key operations.

3. Calling get for a missing key

Read a key that has not been stored.

print(txn.get(b"missing"))

None

The call returns None instead of raising an exception. Use this return value to determine whether a key exists.

4. Converting between byte strings and text

Keys and values are byte strings, so encode Japanese text before passing it to the API.

The sample data represents technical articles. A prefix such as article:1 keeps article keys distinguishable if the database later stores other kinds of data.

from alopex import Database, TxnMode

db = Database.new()

with db.begin(TxnMode.READ_WRITE) as txn:
    txn.put(b"article:1", "Rustで書くLSMツリー".encode())
    txn.put(b"article:2", "SQLパーサをNimで実装する".encode())
    txn.put(b"article:3", "ベクトル検索の基礎".encode())
    txn.commit()

with db.begin(TxnMode.READ_ONLY) as txn:
    print(txn.get(b"article:1").decode())
    print(txn.get(b"article:3").decode())

Rustで書くLSMツリー
ベクトル検索の基礎

Convert Japanese text to bytes with .encode() before writing it, then restore the text with .decode() after reading it.

5. Overwriting and deleting values

Writing the same key twice leaves the later value in the database.

with db.begin(TxnMode.READ_WRITE) as txn:
    txn.put(b"article:1", "旧タイトル".encode())
    txn.put(b"article:1", "新タイトル".encode())
    txn.commit()

with db.begin(TxnMode.READ_ONLY) as txn:
    print(txn.get(b"article:1").decode())

新タイトル

Use delete() to remove a value. Reading the deleted key returns None.

with db.begin(TxnMode.READ_WRITE) as txn:
    txn.delete(b"article:1")
    txn.commit()

with db.begin(TxnMode.READ_ONLY) as txn:
    print(txn.get(b"article:1"))

None

Deleting a missing key does not raise an error. The operation establishes that the key is absent, so repeating it leaves the same result.

6. Calling put without commit

Every write so far has called txn.commit(). Remove that call and inspect the result.

db = Database.new()

with db.begin(TxnMode.READ_WRITE) as txn:
    txn.put(b"article:9", "コミットを忘れた記事".encode())
    # commit を書かない

with db.begin(TxnMode.READ_ONLY) as txn:
    print(txn.get(b"article:9"))

None

The value you wrote returns None. put() accumulates changes inside the transaction instead of applying them to the database immediately. commit() applies the accumulated changes together.

7. commit and rollback

This behavior lets you discard a partially completed operation. If a problem is found after a change begins, call rollback() instead of commit().

db = Database.new()

with db.begin(TxnMode.READ_WRITE) as txn:
    txn.put(b"article:1", "1件目".encode())
    txn.commit()

with db.begin(TxnMode.READ_WRITE) as txn:
    txn.put(b"article:1", "書き換えたが取り消す".encode())
    txn.rollback()

with db.begin(TxnMode.READ_ONLY) as txn:
    print(txn.get(b"article:1").decode())

1件目

rollback() prevents changes from that transaction from being applied. The state from before commit() remains.

The three operations work as follows.

1. put() and delete() accumulate changes inside a transaction.
2. commit() applies the accumulated changes together.
3. rollback() discards the accumulated changes.

Leaving a with block without calling commit() also discards the changes, even if you do not explicitly call rollback(). This is why the earlier example returned no value after omitting commit().

8. The two TxnMode values

TxnMode has two values.

• READ_WRITE: allows reads and writes
• READ_ONLY: allows reads only

Writing through a read-only transaction stops at runtime.

with db.begin(TxnMode.READ_ONLY) as txn:
    txn.put(b"article:1", b"x")

crate::error.PyAlopexError: transaction is read-only

Using READ_ONLY for operations that only read prevents code containing an accidental write from proceeding.

9. Database.new and Database.open

The previous examples used Database.new(), so their data disappears when the process exits.

Pass a directory path to Database.open() to retain data in files.

from alopex import Database, TxnMode

db = Database.open("./notes-db")
with db.begin(TxnMode.READ_WRITE) as txn:
    txn.put(b"greeting", b"hello")
    txn.commit()

Running the program creates ./notes-db. The directory contains the lsm.wal write-ahead log and an sst directory for SSTables. This directory is the database. Open it again from another process.

from alopex import Database, TxnMode

db = Database.open("./notes-db")
with db.begin(TxnMode.READ_ONLY) as txn:
    print(txn.get(b"greeting").decode())

hello

The second process reads the value written by the first. Only one line changed from new() to open("./notes-db"); put, get, and commit remain the same.

Shut down the writing process normally before reopening a data directory. A data directory is opened by one process at a time.

10. Using the key-value API from Rust

Rust applications use the alopex-embedded crate. The earlier key-value operations are written as follows.

Add the dependency to Cargo.toml.

[dependencies]
alopex-embedded = "=0.8.8"

Use the following src/main.rs.

use alopex_embedded::{Database, TxnMode};

fn main() -> Result<(), Box<dyn std::error::Error>> {
    let db = Database::new();

    let mut txn = db.begin(TxnMode::ReadWrite)?;
    txn.put("挨拶".as_bytes(), "こんにちは".as_bytes())?;
    txn.commit()?;

    let mut txn = db.begin(TxnMode::ReadOnly)?;
    let value = txn.get("挨拶".as_bytes())?;
    println!("{:?}", value.map(|b| String::from_utf8(b).unwrap()));
    Ok(())
}

Some("こんにちは")

The Python and Rust APIs share the names Database::new(), begin, put, get, and commit. Three details differ.

1. The transaction modes are TxnMode::ReadWrite and TxnMode::ReadOnly, corresponding to Python’s TxnMode.READ_WRITE and TxnMode.READ_ONLY.
2. Keys and values use &[u8]. Where Python creates bytes with .encode(), Rust passes them with .as_bytes().
3. get returns Option<Vec<u8>>. A missing value is None, as it is in Python.

In v0.8.8, a key-value program using alopex-embedded from crates.io runs without additional shared-library configuration. The published runtime dependencies were also covered by the release verification.

Convert a returned byte string with String::from_utf8 to display it as text, corresponding to Python’s .decode().

Use Database::open() for persistent storage. Its argument is a &Path, not a &str, so wrap a string with Path::new().

use std::path::Path;

let db = Database::open(Path::new("./notes-db"))?;

Rust can open a data directory created by Python, and Python can open one created by Rust. The storage format is shared between the languages.

11. Data continuity verified in v0.8.8

This part opened ./notes-db, created by Python, from Rust. The v0.8.8 release verification extended the same process to server and cluster modes and confirmed that the library, embedded, CLI, HTTP, gRPC, and cluster-aware paths could read and write the same data.

Part 1 starts with two embedded paths, Python and Rust. A later part will connect to a server while continuing to use the same notes-db data.

12. API summary

Keys and values are byte strings in both languages. Python uses .encode() and .decode() around text, while Rust uses .as_bytes() and String::from_utf8.

Part 2 creates a table and queries it with SQL.

References

• v0.8.8 release verification report