feat(sync): chunked/resumable upload for large recordings (T9.2, M4.1)

0 Fork 0

Recordings are now native-quality (~50-100 MB) and put() used to buffer
the whole file into memory with a single PUT/upload call; OneDrive/Graph
also caps a single PUT at 250 MB. This replaces that with disk-streamed,
chunked uploads on both sync backends (FR-SYNC-2/5):

- Both WebDavTarget::put and OneDriveTarget::put now stream the file off
  disk in fixed-size buffers (tokio::fs::File + BufReader) instead of
  tokio::fs::read()'ing it whole — memory use is O(chunk), not O(file
  size), for every upload, large or small.
- Files at or below LARGE_FILE_THRESHOLD (8 MiB) still take a single
  streamed PUT; only large artifacts (in practice, .wav recordings) take
  the chunked path, so small transcript/notes/summary uploads are
  unaffected.
- WebDAV (Nextcloud/ownCloud): implements the chunking-v2 protocol —
  MKCOL an upload collection keyed deterministically off the remote
  path's sha256 (stable across retries), PUT lexically-sortable parts,
  MOVE the virtual `.file` to assemble server-side. A retry PROPFINDs
  the collection first and skips any part already landed, so a
  partially-uploaded large file resumes instead of restarting at byte 0
  — genuine mid-file resume, not just chunk-level retry. Generic WebDAV
  targets (Seafile, Synology, unbranded servers) have no equivalent
  server-side assembly endpoint, so they fall back to one streamed PUT
  above the threshold too — memory-safe, just not chunk-resumable
  (documented ponytail simplification).
- OneDrive/Graph: implements the createUploadSession + Content-Range
  byte-range PUT flow (chunks capped at 10 MiB, a multiple of Graph's
  required 320 KiB granularity). A failed chunk is retried a few times
  within the same put() call. ponytail-documented ceiling: the session
  URL isn't persisted on the sync_jobs row, so a retry from a *later*
  pump cycle (e.g. after an app restart) starts a fresh session and
  re-uploads from byte 0 rather than resuming — true cross-attempt
  resume would need a sync_jobs column, left as a follow-up.
- Progress is reported via the existing ProgressSink after each chunk,
  not just once at the end, so the UI sees steady movement on a large
  upload (FR-SYNC-11).
- OneDriveTarget's Graph URLs are now built from a `graph_base()` helper
  (overridable via WA_GRAPH_BASE_URL, unset in production) so the
  chunked session flow can be exercised against a local mock in tests.

Tests (all against small, in-process, dependency-free mock servers —
std::net only, no new crates, same pattern as oauth::LoopbackRedirect):
- nextcloud_chunked_upload_assembles_and_resumes: multi-chunk upload
  assembles byte-for-byte, then a resumed upload with one chunk
  pre-seeded skips exactly that chunk.
- small_file_skips_chunking_even_on_a_chunking_capable_target: a small
  file never takes the chunked path.
- onedrive_chunked_upload_via_graph_session_mock: multi-chunk Graph
  upload session assembles byte-for-byte with per-chunk progress.
- parse_chunk_sizes_extracts_href_and_length_ignoring_namespace_prefix
  and chunk_name_is_zero_padded_and_lexically_sortable: pure unit tests
  for the PROPFIND-response scraper and chunk-naming scheme.

Skip-if-unchanged (SHA-256, in storage::upsert_sync_job) and the
exponential-backoff retry queue (storage::claim_due_sync_jobs) are
untouched — this only changes how SyncTarget::put moves bytes.

Verified: cargo fmt clean; cargo clippy --features sync and --features
mcp both clean (-D warnings); cargo test --lib (default features:
audio, cpu-transcription, diarization, pst, sync, npu) 165 passed, 0
failed, 7 ignored (unrelated hardware-gated tests); cargo test --features
sync --lib: 18 passed, 0 failed, 1 ignored (webdav_round_trip, opt-in
live-server test per existing convention).

This commit is contained in:

iamdoubz committed 2026-07-07 13:32:09 -05:00

1 parent be025bbd6c

commit 2cc48c9d4d

1 file changed

+952 -35

src-tauri/src/sync/mod.rs

+952 -35

View File

File diff suppressed because it is too large. Load diff