The Art of Structuring, Part 3 — Choosing and Transforming Representations

Part 2 separated concerns such as meaning, relationships, behaviour, access, communication, and storage for the same entity. It also organised the business, data, application, and technology domains and the context that crosses them.

Part 3 moves from choosing data structures for a query through information loss, reconstruction, and operation.

1. Choosing a logical representation from the query

Start by identifying the relationships and operations that a query requires, then choose a logical representation and access path capable of handling them.

1.1 The relationships a representation preserves

Different representations can be made from the same set of entities according to the relationships that must remain. Put order events in a list and their order of occurrence can be followed. Project only each event’s event_type into a set and it yields distinct event types, but preserves neither the original order nor the number of occurrences. Project an organisation into a tree and direct managers and subtrees become easy to compute, while concurrent assignments and project-participation edges must be placed in a separate graph.

A list preserves order, a set membership, a tree hierarchy, a graph arbitrary relationships, and a map key-value correspondence. These formats are not merely different shapes for the same information: they preserve different relationships and make different queries easy to answer.

Choose first which properties of the entities must survive in the representation. Whether a query uses order, membership, parent-child relationships, dependency, proximity, or similarity determines the logical representation—a list, tree, graph, or another form. Implementation complexity and memory layout belong to the next decision: which physical placement will realise that logical representation.

1.2 Search conditions and access paths

Even when the source product records are identical, the access path suited to a query changes. Searching ten million products by product ID, price range, description text, location, and image similarity requires a different access path for each operation.

Choose an index type according to the operators used for search. PostgreSQL’s documentation on index types explains that B-tree indexes handle equality and range queries on orderable values, while hash indexes handle simple equality comparisons. GiST, SP-GiST, GIN, and BRIN offer other search strategies according to operator classes and the nature of the data.

Full-text search avoids rereading every document body by creating an inverted index that maps terms to document IDs. The Apache Lucene index package documentation describes an index in terms of documents, fields, and terms, with an internal representation that reaches documents containing a term through a postings list.

Choose a nearest-neighbour index according to what its distance means. Spatial search may use a spatial index to handle containment, intersection, and proximity between points and regions. Vector search may use an index that finds neighbours by the distance between embedded points, but geographic distance and distance between semantic vectors must remain distinct.

Indexes impose an update cost. Every change to a source-of-truth record must also update its access paths; asynchronous updates require management of the resulting freshness gap. Editing an index as though it were the source of truth loses the change when the index is rebuilt. Separating what is stored from how it is reached permits several indexes for different uses.

1.3 Similarity in vector search

Embeddings can project documents, images, or products into vectors and retrieve similar candidates by distance or inner product. Continuous proximity can become an access path even without explicit categories or relationships.

Similarity means proximity in the vector space. A separate decision rule is still required before two companies with nearby name embeddings can be judged the same corporation; other data is required before two similarly described services can be judged dependent. A vector index is a purpose-specific projection for retrieving similarity, not a direct representation of identity or relationships.

Once data is projected into vectors, it becomes difficult to explain directly which source words or attributes contributed to a distance. A use case that must validate search results can record the model version and input, distance metric, and an identifier leading back to the source document or entity. Updating the model changes the vector space itself, so the design must also check whether old and new vectors can be compared under the same distance measure.

2. Carrying a logical representation into physical placement

Once the logical representation is chosen, design its placement and transformation path in memory or storage according to read and write frequency, processing time, and data volume.

2.1 Hashing for lookup and hashing for placement

A hash maps an input to a fixed-length value for lookup, placement, or content identification. When customer_id = C-001 is looked up in a hash table, the hash selects a bucket. When the same customer ID is stored in distributed storage, the hash can select a partition. Content-addressed storage derives an address that identifies data from the file contents.

Even with the same hash function, collision handling and invariants differ by use. A hash table compares colliding keys to distinguish them; partitioning assigns every key to one shard. If a content hash determines identity, changing the contents also changes the identifier. The input, purpose of the hash, collision handling, and invariants must therefore be defined separately for each use.

Placement adds the effect of repartitioning. If a change in node count moves many keys to other partitions, data movement and cache misses increase. This placement problem may be inconspicuous in a lookup hash but becomes central in the technology domain.

2.2 I/O changes between row-oriented and column-oriented placement

Place a transaction table containing customer ID, time, product ID, and amount by row, and all fields of one transaction are easy to read and write together. Place it by column, and amounts can be read sequentially for aggregation and compression. The logical schema is unchanged, but physical placement changes I/O and cache locality.

Columnar layout is a physical projection that places values of the same attribute near one another. The Apache Arrow columnar format defines an in-memory layout with buffers for each type, including validity bitmaps and offset buffers. Different implementations can share the same columnar data and pass it to vectorised processing.

A design that produces columnar analytical files from row-oriented transactional storage has two physical projections of the same logical data. Rows and columns can coexist for their respective uses when update frequency, freshness, and reconstruction paths are managed.

The same separation applies to graphs. Even when relationships are understood as a graph in the conceptual model and represented as nodes and edges in the logical model, physical placement can use an adjacency list, adjacency matrix, CSR, or another format suited to the use case. Sparse-graph traversal and dense-graph matrix operations favour different layouts.

2.3 Multi-stage transformations of order data and source code

Consider order acceptance. First identify the order as an entity in the business domain and carry the acceptance operation into the application specification as a command. Carrying input data into the communication specification requires units such as messages, segments, and packets. On the storage side, design the relation, tuple, page, and block for order data. On the analytical side, design order records as events, partitions, row groups, columns, and vectors. The business entity called an order is transformed in stages into management units, messages, and records suited to the operations at each layer.

Source code is also transformed in stages according to the needs of analysis and execution. A compiler, for example, may transform source code into tokens, an abstract syntax tree, an intermediate representation, and machine instructions. The stages and representations differ by implementation. LLVM IR, defined by the LLVM Language Reference, consists of typed instructions in SSA form, basic blocks, functions, modules, and other elements. It is an intermediate representation of properties needed for analysis and optimisation.

Each transformation in this example enables a new operation. An AST represents syntactic parent-child relationships, IR supports control-flow and data-flow analysis, and machine instructions can be executed by a processor. Each representation retains the information needed for the purpose of the transformation.

If source locations are not retained as debugging information, however, a problem in a machine instruction cannot be mapped back to its source line. When a use case must move back and forth across abstraction levels, the transformed instruction or record must inherit the identity and location of its source.

3. Designing reconstruction and operation after transformation

When purpose-specific representations are produced, identify the information lost in transformation and design the reconstruction of split data, updates to derived data, and recovery from failures.

3.1 How split data is put back together

Parallel processing of a large log file divides its input range into several partitions. Jobs, files, messages, streams, and images likewise have partitioning units suited to processing, storage, or transfer.

dataset → partitions
job → tasks
file → blocks
message → fragments
stream → segments
image → tiles

Choose metadata for a partitioned unit from the requirements for reconstruction and recovery. A sequence number can restore order; a total count or checksum can reveal missing pieces; an input range and attempt number can support partial retries. If correspondence with source input is required, also consider a parent ID, offset, length, and schema version.

Reconstruction requirements vary by use. A packet must allow the higher-level payload to be recovered, while an analytical aggregate need not reconstruct every individual event. Deciding how far reconstruction must go determines the metadata and the retention period of the source of truth.

Partition boundaries also determine the scope of failure. Validate one enormous file as a single unit and local corruption requires the whole file to be retrieved again. Per-block checksums can identify the damaged range. Finer tasks narrow the scope of retries but increase the number of scheduling and state-management records.

3.2 Information lost in aggregation

A projection loses some attributes or relationships from its source representation. Converting an ordered list into a set loses order and duplicate counts. Constructing current state from an event history loses intermediate states. Converting Money(100, JPY) into the integer 100 loses the currency. Aggregating a graph into counts per entity loses individual edges.

Loss is acceptable when the omitted information is unnecessary for the projection’s query and can be regenerated from the source of truth when needed. A view optimised for current-state lookup can omit intermediate history; discard that history, however, and it can no longer answer a past state or the reason for a change.

When specifying a transformation, consider its input, output, retained and omitted attributes and relationships, and invariants. Record them at the level of detail required to validate the transformed data and regenerate it when necessary. For a monetary aggregate, invariants include currency alignment, treatment of cancelled transactions, rounding, and the covered period. Re-running the same SQL does not produce an equivalent result if the versions of the rules and inputs differ.

Validating information loss that cannot be seen in the result requires management of transformation rules and provenance. The schema of a sales-by-company table shows that contract details are absent, but the result alone cannot reveal the conditions used to exclude transactions or the point in time used for currency conversion.

3.3 Operating indexes and aggregate tables

For a projection’s update path, consider the operationally necessary parts of source-of-truth synchronisation, invalidation of stale projections, reconstruction, schema changes, and failure recovery. Without a defined synchronisation method, a product-price update may leave search results stale, or a corrected contract may never reach the monthly aggregate.

Recovery conditions for a projection depend on permitted downtime and the reconstruction method. If rebuilding an index or aggregate table takes days, the recovery conditions must include build time, input retention, and an alternative during the outage.

Idempotency—the ability to repeat processing without corrupting its result—is also the transformation’s responsibility. To stop a task retry from creating the same order twice, one design separates business identity from the attempt ID and performs output deduplication at the transformation boundary. Whether a rerun replaces a partially built projection or continues from a delta depends on the output format and recovery method.

Comparing record counts alone cannot detect missing attributes or relationships. Candidate reconciliation measures include monetary totals, foreign-key correspondence, graph reachability, time ranges, and counts by source; choose the ones the projection is intended to preserve. Retaining representative queries as regression tests shows whether a change to a transformation rule has broken its use case.

4. Operating multiple representations in one system

Finally, choose storage and search mechanisms from query and update conditions, then organise how multiple representations of the same entity can coexist without being confused.

4.1 Choosing storage and search mechanisms from query and update conditions

To choose a storage and search mechanism, describe what users need to know, do, and decide as queries and operations. “Retrieve one customer by ID and update a contract,” “traverse several levels of related services to assess incident impact,” and “aggregate a large volume of logs by day” demand different access and update methods.

Next determine entities and granularity. Identity, updating, transfer, storage, consistency, and retry can each use a different unit. An order can remain one business entity while inventory is reserved per item, payment is retried per payment, and data is transferred per packet.

Choose storage and search mechanisms after checking query type, update frequency, transaction scope, response time, data volume, and whether the representation can be rebuilt from its source of truth.

A single system can combine several mechanisms according to their roles as source of truth, search representation, and analytical representation. For example, it might keep order records in an RDBMS, use a search engine for full-text product-description search, and use object storage for log retention and aggregate input.

Comparing MySQL, Oracle Database, and PostgreSQL comes after RDBMS has been selected as a candidate, and depends on required features, operational capacity, and the existing environment. Defining the role of a storage and search mechanism first separates comparison between products of one type from the allocation of responsibilities between different mechanisms.

4.2 Combining RDB, JSON, graph, and vector representations

When one entity is handled in several formats, separate the roles of the source of truth and purpose-specific representations, then link them with IDs, transformation rules, and update paths. A system may manage customer information as RDB records, return JSON from an API, use a graph for relationship traversal, and use vectors for similarity search. For each representation, define its queries, users, attributes and relationships, transformation source, and regeneration method.

With those roles separated, search indexes, caches, materialised views, and aggregates can be regenerated from the source of truth. Changing API JSON need not change the definition of the business entity, and a separate identity decision can be applied to results from embedding-based similarity search.

Reusing a physical identifier as business identity can make it impossible to connect the history of the same entity. Database sequence numbers, process IDs, IP addresses, and file paths can change when an object is recreated or moved and therefore cannot automatically serve as conceptual identity.

A common vocabulary, metadata set, or index without a defined use only adds update paths and version management. Define the queries and processing that will use it at creation time; during operation, review usage and regeneration paths and retire representations that are no longer needed.

RDB, JSON, graphs, events, and vectors address different concerns: relations, messages, relationships, change, and similarity. Separating their roles and lifecycles allows them to coexist in one system according to the use case instead of forcing everything into one format.

4.3 Validating and regenerating representations

The validity of a representation is not determined by whether it is called a list, tree, or graph. A design becomes assessable when it states which attributes and relationships of the entity remain, which queries will use them, what is lost, and from where the representation can be regenerated. In implementation, design the data representations required by use cases as multiple projections and verify that they can be regenerated from the source of truth. Carry each projection and regeneration path into the implementation through repeated examination, design, and validation.

5. Conclusion

Structuring begins by asking what should be treated as one managed subject. Its boundary, identity, attributes, relationships, and constraints are examined while the subject is carried into purpose-specific logical specifications and physical placements and tested for feasibility. To interpret each representation later, retain as context the managed subject it refers to, the time, the information source, and the rules used.

Besides business units such as customers and orders, managed subjects can include files, messages, records, tasks, and log events. The unit of one item and its ID may change across business, data, application, and technology domains even for the same managed subject. The attributes, relationships, and constraints retained are also selected for the use case.

Record linkage, for example, links records from different systems to a common customer ID. In order processing, the same order is handled as a command, message, record, and page. The artefacts and levels of abstraction differ, and each creates design questions: what counts as one managed subject, which attributes and relationships are used, and how the transformed representation is validated.

The art of structuring applies these decisions across business, data, application, and technology, designing the whole system so that it can be changed, validated, and reconstructed.