Metadata Sync

Concepts Covered

• File metadata
• Folder trees
• Rename and move events
• Tombstones
• Metadata journals
• Device sync cursors
• Permission state
• Derived local indexes

Definition

Metadata sync is the process of keeping file system state coherent across devices without treating the file bytes as the only thing that matters.

File bytes are only one part of a cloud drive product. Users also expect names, folders, moves, deletes, sharing rules, version pointers, and local availability state to converge across devices.

A sync system usually separates:

metadata plane: names, folders, versions, permissions, tombstones
blob plane: chunks, bytes, manifests, storage lifecycle

The metadata plane tells clients what exists and what changed. The blob plane moves the bytes.

The Pain That Forces This Concept

Suppose a user renames design.pdf on a laptop while a phone is offline.

If the phone only scans storage for file bytes later, it may not know whether the file was renamed, moved, deleted, replaced, or duplicated. If a desktop client deletes a folder, other devices need to learn about the delete even though no new bytes were uploaded.

Without metadata sync, products drift in strange ways:

• files reappear after delete
• folders contain stale children
• renamed files duplicate instead of moving
• permissions lag behind shared links
• clients download bytes they no longer need
• offline devices overwrite newer server state

The system needs a durable history of metadata changes.

Mental Model

Think of metadata as a synchronized file tree backed by a journal.

event 101: create file f1 under folder root
event 102: upload version v1 for f1
event 103: rename f1 from draft.md to design.md
event 104: move f1 into folder docs
event 105: delete f1

Each device stores a cursor into that journal. When it reconnects, it asks for metadata events after its last cursor and applies them locally.

How It Works

A metadata sync service typically stores source-of-truth records:

file_node
- file_id
- parent_id
- name
- type
- owner_id
- current_version_id
- deleted_at
- metadata_version

sync_event
- sequence
- file_id
- event_type
- metadata_version
- payload
- created_at

Clients sync in two layers:

1. Pull metadata changes after cursor.
2. Decide which file chunks or versions need download.

This prevents a client from downloading large bytes before it knows whether the file is still visible, renamed, or allowed.

Tradeoffs

Metadata sync has to choose product semantics:

The hardest part is not storing metadata. The hard part is defining what each metadata event means when clients are offline and changes overlap.

Operational Reality

Operators should monitor:

• metadata sync lag per device
• event journal retention windows
• snapshot fallback rate
• tombstone count
• metadata conflict rate
• permission propagation latency
• client apply failures
• reconciliation drift between file tree and version records

Failure modes:

• A delete event expires before an offline device syncs.
• A folder move creates a cycle because validation is weak.
• Permission changes lag behind file visibility.
• Local search indexes drift from server metadata.
• A client applies events out of order.
• Metadata commit succeeds but blob upload never finishes.

Related Topics

• Google Drive / Dropbox File Sync System
• Cursor-Based Sync
• Eventual Consistency
• Derived Projections
• Reconciliation Job