Vitastor 3.2.3 released

Another bugfix release. The main innovation is the new PG chaos test, complemetary to the previous blockstore chaos test - it checks OSD logic on a single PG. The test checks data consistency using a blockstore-specific consistency model, because standard transaction isolation models are not always applicable to them. The PG chaos test logic is described in detail in the documentation.

All that’s left is to add chaos tests for the client library and the network layer, and Vitastor will be fully covered by chaos tests!

Besides the new chaos test and the problems it found, the release also contains fixes from Changwei Ye, Yury Luneff and Alexey Shabalin - thanks for contributing to the project.

• Data correctness
  • Fixed repetitions of commits in the client library in rare cases of OSD crashes with simultaneous PG reconfiguration (applies to configurations with explicit fsync)
  • Fixed possible incorrect marking of replicated objects as HAS_INCOMPLETE during a partially successful write while moving the object to another OSD set
  • Fixed what appears to be a very old and annoying PG hang in EC pools in the STOPPING/REPEERING state: a PG hung when the queue simultaneously contained an active write operation and an initial commit/rollback of the object’s previous unfinished write
  • Fixed incorrect computation of EC object states during PG peering, which in rare situations led to HAS_INCOMPLETE or to leaving garbage object fragments on OSDs
  • Fixed possible read errors during concurrent writes or deletions of the same object that looked like object corruption
  • Fixed possible very rare failures to report PG states to etcd when PG start outran the OSD acquiring its lease in etcd
  • Fixed theoretically incorrect marking of replicated objects as DEGRADED on every object state change - it didn’t cause real problems though, because the state was only used in scrub and in marking partially successful writes, where the object was usually actually DEGRADED
  • Fixed read errors with connection drops from images with more than 255 snapshots
  • Parallel removal of multiple snapshots of the same image from two processes now fails with an error instead of corrupting the snapshot chain
• New and old stores
  • New store: fixed a 3.2.0 regression where big writes hung after a single redirect_intent write waiting for fsync in configurations without immediate_commit
  • New store: fixed several potential data loss errors in rare cases found by chaos tests: incorrect checksum handling for empty writes, incorrect object write after its rollback to version zero, reuse after restart of sequence numbers below the completed_lsn in the header
  • Old store: fixed a rare OSD hang when deleting an object
• Network & security
  • Fixed transport encryption compatibility with some OpenSSL versions containing the unfixed bug 27302: the salt could not be reset, so new connections broke with “Handshake decryption failed” until an OSD restart
  • TCP sockets are now closed using SO_LINGER(1,0)+shutdown()+close() to guarantee real cleanup because the previous approach with only close() turned out to be unstable again - outstanding io_uring requests aren’t cleared on close().
  • Added options for QoS in RDMA networks with PFC: rdma_traffic_class, rdma_sl and rdma_set_flow_label
  • Vault: the client now automatically retries loading keys from Vault after a timeout if the previous attempt failed with an error, not only on pool reconfiguration
  • Vault: fixed read errors from clones of encrypted images caused by the parent image key not being loaded
  • With access control enabled, creating a clone now checks read access to the parent image
  • The client now retries etcd address resolution after DNS errors within the general retry limits instead of killing the client process
• VitastorFS
  • Deleting a directory with many previously deleted files is now faster because fewer redundant keys are scanned in the key-value database
  • Fixed an error when moving a directory over another empty directory
• vitastor-cli
  • fix and describe no longer hang when one of the OSDs is unavailable: such OSDs are skipped with a warning, and the exit code indicates an incomplete result
  • describe --inode shows all objects of an image, not just one
  • osd-tree crashed on etcd errors
  • ls --tree hid images with a filtered-out parent
  • ls-pgs printed an empty table instead of the real error and showed misplaced data as degraded
  • dd: copying a pipe without count= copied only 1 MB, oflag=append didn’t work, status=none and status=noxfer were ignored, devices like /dev/mem couldn’t be read
  • create-pool/modify-pool now refuse pg_size > 64 and renaming a pool to an occupied name; level_placement rules can be reset with an empty value
  • modify --resize silently accepted a broken size like “1.5G”
  • rm-data didn’t validate offset boundaries
  • Fixed minor differences between help, docs and the actual behaviour
• Deployment
  • Partitions with the Vitastor partition type are now owned by the vitastor user even when they don’t yet contain a valid superblock
  • Fixed the QEMU driver build on QEMU versions patched by RHEL 9.1
  • Fixed a small memory leak when destroying the QEMU driver
  • The client now has a mode that handles fatal errors instead of silently killing the whole client process via exit(1). New QEMU driver versions use this mode and should no longer take the QEMU process down when a disk fails to connect; old versions keep the previous behaviour

2026-11-01 Continue reading →

Vitastor 3.2.2 released

Small bugfix release.

• New store (thanks Changwei Ye)
  • Fix rejected compaction when an object is already overwritten again
  • Reduce superblock I/O amplification introduced after previous correctness fixes
• Fix possible vitastor-cli rm crash with assert(rwo->todo == 1) introduced in 3.2.1
• Fix replicated objects possibly incorrectly transitioning to the INCOMPLETE state after multiple failed partial writes
• Fix possible “operation sequencing broken … expected num 2, got 3, stopping client” possible when a cluster is overloaded by peering/OSD flaps

2026-09-27 Continue reading →

Vitastor 3.2.1 released

• Blockstores
  • Add more asynchrony to the chaos test to uncover more issues…
  • …and fix them (~6 in the old store, ~9 in the new store)
  • Fix size check for journal_offset when sharing meta device (thanks Changwei Ye)
• Network layer
  • Correctly handle out-of-order zero-copy notifications (rare but possible in theory)
  • Fix double checksumming of header in proto_checksums=full mode
  • Add rxbounce option for Windows guests with TCP and enabled checksums
  • Close stopped peer sockets early to prevent hung zerocopy notifications and slow failover (~15 min instead of ~10 sec)
• OSD
  • Do not die on local blockstore modification errors (irrelevant since added write error handling)
• CSI
  • Fix VitastorFS NFS daemon lifecycle (thanks Dmitry Naumov)
  • Use a newer qemu-storage-daemon in CSI
• vitastor-cli
  • Add dd opool option
  • Fix rm with “inverse” optimisation incorrectly removing layer data before rename possibly leading to incorrect reads from child images during removal of an older-than-latest snapshot
  • Fix a possible hang in merge and derived commands (rm, flatten)
  • Fix mistakes in the serve OpenAPI spec (thanks Stepan Rabotkin)
  • Fix possible segfaults in serve due to invalid request parsing (thanks Vincent Schweiger)
  • Fix serve no-HTTPS mode (thanks both :))
• Build
  • Fix non-x86 builds failing due to unsupported xxhash x86dispatch
  • Forcibly enable assertions in all builds to remove build profile dependency
  • Fix QEMU driver build with QEMU < 10.2

2026-09-13 Continue reading →

Vitastor 3.2.0 released

This is a bugfix release, and most of the fixes came from a new approach to testing: a chaos test for the block store, which now runs in CI on every change.

Also thanks to Yury Luneff from MIND Software for another part of the fixes.

2026-08-31 Continue reading →

Vitastor 3.1.0 released

Finally, the security release!

New features:

• Fully secure cluster setup support:
  • etcd transport encryption (TLS/mTLS)
  • OSD transport encryption (AES-GCM), header-only by default, thus very fast
  • End-to-end image data encryption (AES-XTS)
  • Optional external Vault AES-XTS key storage
  • Certificate-based authentication
  • Optional user privilege support with etcd- and OSD-based access control
  • xxhash3 transport checksums (data-only by default)
• New on-disk checksum algorithm option: xxh3_32 - xxhash3 truncated to 32 bits
• DNS name support for etcd and Vault URLs
• HTTP API server mode: vitastor-cli serve
• Mixing CAS and non-CAS writes within a single client instance is now allowed and works correctly
• make-etcd can now initialize the whole cluster by copying configs and starting other nodes through ssh
• Debian 10 (Buster) and CentOS 7 support is completed, packages won’t be updated anymore

Read detailed security feature documentation here: https://vitastor.io/docs/usage/security.html

2026-07-24 Continue reading →