Vitastor 3.2.3 released

2026-11-01

Another bugfix release. The main innovation is the new PG chaos test, complemetary to the previous blockstore chaos test - it checks OSD logic on a single PG. The test checks data consistency using a blockstore-specific consistency model, because standard transaction isolation models are not always applicable to them. The PG chaos test logic is described in detail in the documentation.

All that’s left is to add chaos tests for the client library and the network layer, and Vitastor will be fully covered by chaos tests!

Besides the new chaos test and the problems it found, the release also contains fixes from Changwei Ye, Yury Luneff and Alexey Shabalin - thanks for contributing to the project.

  • Data correctness
    • Fixed repetitions of commits in the client library in rare cases of OSD crashes with simultaneous PG reconfiguration (applies to configurations with explicit fsync)
    • Fixed possible incorrect marking of replicated objects as HAS_INCOMPLETE during a partially successful write while moving the object to another OSD set
    • Fixed what appears to be a very old and annoying PG hang in EC pools in the STOPPING/REPEERING state: a PG hung when the queue simultaneously contained an active write operation and an initial commit/rollback of the object’s previous unfinished write
    • Fixed incorrect computation of EC object states during PG peering, which in rare situations led to HAS_INCOMPLETE or to leaving garbage object fragments on OSDs
    • Fixed possible read errors during concurrent writes or deletions of the same object that looked like object corruption
    • Fixed possible very rare failures to report PG states to etcd when PG start outran the OSD acquiring its lease in etcd
    • Fixed theoretically incorrect marking of replicated objects as DEGRADED on every object state change - it didn’t cause real problems though, because the state was only used in scrub and in marking partially successful writes, where the object was usually actually DEGRADED
    • Fixed read errors with connection drops from images with more than 255 snapshots
    • Parallel removal of multiple snapshots of the same image from two processes now fails with an error instead of corrupting the snapshot chain
  • New and old stores
    • New store: fixed a 3.2.0 regression where big writes hung after a single redirect_intent write waiting for fsync in configurations without immediate_commit
    • New store: fixed several potential data loss errors in rare cases found by chaos tests: incorrect checksum handling for empty writes, incorrect object write after its rollback to version zero, reuse after restart of sequence numbers below the completed_lsn in the header
    • Old store: fixed a rare OSD hang when deleting an object
  • Network & security
    • Fixed transport encryption compatibility with some OpenSSL versions containing the unfixed bug 27302: the salt could not be reset, so new connections broke with “Handshake decryption failed” until an OSD restart
    • TCP sockets are now closed using SO_LINGER(1,0)+shutdown()+close() to guarantee real cleanup because the previous approach with only close() turned out to be unstable again - outstanding io_uring requests aren’t cleared on close().
    • Added options for QoS in RDMA networks with PFC: rdma_traffic_class, rdma_sl and rdma_set_flow_label
    • Vault: the client now automatically retries loading keys from Vault after a timeout if the previous attempt failed with an error, not only on pool reconfiguration
    • Vault: fixed read errors from clones of encrypted images caused by the parent image key not being loaded
    • With access control enabled, creating a clone now checks read access to the parent image
    • The client now retries etcd address resolution after DNS errors within the general retry limits instead of killing the client process
  • VitastorFS
    • Deleting a directory with many previously deleted files is now faster because fewer redundant keys are scanned in the key-value database
    • Fixed an error when moving a directory over another empty directory
  • vitastor-cli
    • fix and describe no longer hang when one of the OSDs is unavailable: such OSDs are skipped with a warning, and the exit code indicates an incomplete result
    • describe --inode shows all objects of an image, not just one
    • osd-tree crashed on etcd errors
    • ls --tree hid images with a filtered-out parent
    • ls-pgs printed an empty table instead of the real error and showed misplaced data as degraded
    • dd: copying a pipe without count= copied only 1 MB, oflag=append didn’t work, status=none and status=noxfer were ignored, devices like /dev/mem couldn’t be read
    • create-pool/modify-pool now refuse pg_size > 64 and renaming a pool to an occupied name; level_placement rules can be reset with an empty value
    • modify --resize silently accepted a broken size like “1.5G”
    • rm-data didn’t validate offset boundaries
    • Fixed minor differences between help, docs and the actual behaviour
  • Deployment
    • Partitions with the Vitastor partition type are now owned by the vitastor user even when they don’t yet contain a valid superblock
    • Fixed the QEMU driver build on QEMU versions patched by RHEL 9.1
    • Fixed a small memory leak when destroying the QEMU driver
    • The client now has a mode that handles fatal errors instead of silently killing the whole client process via exit(1). New QEMU driver versions use this mode and should no longer take the QEMU process down when a disk fails to connect; old versions keep the previous behaviour