47 posts tagged with "fusion"

Fusion v2.6.10 fixes a regression introduced in v2.6.9 that failed successful snapshot tasks with exit code 174. If you run Fusion Snapshots on v2.6.9, upgrade.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.10-amd64.json'
}

Bug fixes​

Fusion Snapshots​

• Fixed successful snapshot tasks failing with exit code 174 when the task deleted a temporary file it still had open.

Upgrade notes​

• Drop-in replacement for v2.6.9. No configuration changes required.
• If you pinned snapshot tasks to v2.6.8 to avoid the exit code 174 failures, you can unpin them.

Fusion v2.6.9 fixes three ways a healthy task could be lost in NFS service mode or on EC2: failed output uploads reported as success, a short EC2 instance metadata outage blocking S3 access, and remote I/O errors on a task relaunched from a clean start.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.9-amd64.json'
}

Bug fixes​

NFS service mode​

• Fixed a task exiting with 0 when the final upload of its work directory failed. Fusion now retries the upload for about five minutes, then exits with 174. See Fusion exit codes.
• Fixed a Remote I/O error when a task relaunched from a clean start recreated a path the clean had removed.

AWS​

• Fixed a short gap in the EC2 instance metadata service leaving Fusion with anonymous S3 access for the rest of the task. If the work directory is still unreachable, Fusion exits with 173, which Intelligent Compute relocates to another host.

Dependencies​

• Updated dependencies and CI.

Upgrade notes​

• Drop-in replacement for v2.6.8. No configuration changes required.
• In NFS service mode, a task whose command succeeded but whose final work directory upload failed now exits with 174 instead of 0. A task whose command failed keeps its original exit code.
• On EC2, a host with no instance profile keeps using anonymous S3 access. If a host has an instance profile but no reachable instance metadata service and should use anonymous access, set AWS_EC2_METADATA_DISABLED=true.

Fusion v2.6.8 lets Intelligent Compute relaunch a task from a clean start after a failed snapshot restore, and fixes stale file handle errors on idle files in NFS service mode.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.8-amd64.json'
}

Feature updates and improvements​

Fusion Snapshots​

• Added FUSION_SNAPSHOT_CLEAN_START. When it is set, Fusion cleans the work directory and removes leftover snapshot data before the task starts, so a task whose snapshot restore failed can be relaunched from scratch in the same work directory. Intelligent Compute sets it when it relaunches such a task.

Bug fixes​

NFS service mode​

• Fixed files and directories left idle during a task failing with a stale file handle error.

General​

• Fixed a spurious Check available filesystem space error printed every time a mount shut down.

Dependencies​

• Updated test and CI tooling and dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.7. No configuration changes required.
• FUSION_SNAPSHOT_CLEAN_START applies to NFS service mode only, and you do not need to set it yourself. Without it, Fusion behaves as in v2.6.7.
• File handles in NFS service mode stay valid while the Fusion mount keeps running. They do not survive a restart of the mount.

Fusion v2.6.7 fixes a regression introduced in v2.6.6 that stopped retried tasks from starting when Fusion serves a work directory over NFS. If you run v2.6.6 in that configuration, upgrade.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.7-amd64.json'
}

Bug fixes​

NFS service mode​

• Fixed retried tasks timing out with exit code 173 on every host after an interruption.
• Fixed a work directory emptied by an earlier attempt failing to prepare.
• Halved the storage listings needed to prepare a work directory.

Dependencies​

• Updated build tooling and dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.6. No configuration changes required.
• The regression affects Fusion in NFS mode on v2.6.6 only. Earlier v2.6 releases are unaffected.
• Two limitations remain. A work directory reused without being cleaned is not swept, and leftover names can persist there. Snapshot restore in NFS mode is still under active work. Do not assume every restore path is covered.

Fusion v2.6.6 fixes two Fusion Snapshots restore failures and extends Spot reclamation protection for NFS service mode to Google Cloud and Azure.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.6-amd64.json'
}

Bug fixes​

Fusion Snapshots​

• Fixed snapshot restore failing for a task relaunched after its instance was reclaimed.
• Fixed Fusion Snapshots never writing a checkpoint on Google Cloud VMs launched by Intelligent Compute.

NFS service mode​

• Fixed tasks losing their NFS export when a Google Cloud or Azure host was reclaimed.

Dependencies​

• Fixed CI and updated dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.5. No configuration changes required.
• The permissions fix resolves restore failures for a task relaunched after a reclaim. Snapshot restore in NFS mode is still under active work. If you rely on snapshots, do not assume every restore path is covered.

Fusion v2.6.5 fixes Fusion Snapshots restore for a task that is preempted and relaunched, and fixes renamed directories, the storage region lookup, and cache cleanup on unmount.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.5-amd64.json'
}

Bug fixes​

Fusion Snapshots​

• Fixed a relaunched task not finding the checkpoint its previous attempt took.
• Fixed a snapshot missing files from the task work directory in NFS mode.

General​

• Fixed a renamed directory stranding files that were still uploading.
• Added a timeout to the first lookup of a bucket region.
• Fixed mounts never removing their working folder under the cache location.

Dependencies​

• Updated the Go toolchain to 1.27.1.
• Removed dead code and updated CI and dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.4. No configuration changes required.
• The snapshot restore fix covers a task relaunched on a different instance, which is what happens when an instance is reclaimed. A task relaunched against the same still-running Fusion daemon that drained the previous attempt still fails to restore.

Fusion v2.6.4 adds retryable and non-retryable exit codes for startup failures in NFS service mode, and new health and liveness metrics. It also fixes renamed directories and very large files on Google Cloud Storage.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.4-amd64.json'
}

Feature updates and improvements​

NFS service mode​

• Added distinct exit codes for startup failures when Fusion runs as the NFS daemon and cannot complete its startup handshake with the control node. Fusion exits with 172 when the control node rejects the request, which is a configuration or permission problem that a retry cannot fix. Fusion exits with 173 for anything else, such as an unreachable or slow control node, and for ambiguous failures. Intelligent Compute relocates the task on 173 instead of aborting the run. See Exit codes 172 and 173.
• Added FUSION_NFS_READINESS_TIMEOUT to set how long Fusion waits for its work directory to become ready before it classifies the failure and exits. The default is 60 seconds.

Metrics​

• Added the fusion_health_* metric family to the per-task metrics file. It reports uptime, memory use split into non-overlapping categories, open file handles, cache device disk usage, mount initialization time, and shutdown flush time. Resource exhaustion is now visible after the fact.
• Added two liveness gauges that report how many file system requests are outstanding and the age of the oldest one. The age stays near zero when nothing is calling Fusion and climbs when Fusion has taken a request and not answered it. Use them to tell a stalled mount from an idle one.
• Published a schema and triage runbook for the metrics file. See Fusion task metrics.

Bug fixes​

General​

• Fixed a renamed directory dropping files and symlinks that were not yet uploaded.
• Fixed a "no such file" error on NFS daemon clusters for an input produced by another task.
• Fixed a retried task starting on leftovers from the previous attempt.
• Fixed .tbi, .csi, and .bai index files appearing older than the data they index.
• Fixed symlink targets longer than 2048 bytes being truncated on read.
• Fixed declared outputs with adjacent brace patterns being evicted at shutdown instead of persisted.
• Fixed a truncate or seek-write to an implausible offset exhausting memory and killing the mount, which is now rejected with a clear error.
• Fixed temporary files being uploaded while the task was still writing them, except on runs with snapshots enabled.

Google Cloud​

• Fixed corrupt results for very large files on Google Cloud Storage, roughly 128 GiB and above at the default chunk size.
• Added a check that rejects an object over the 5 TiB Google Cloud Storage per-object limit before uploading.

Dependencies​

• Reorganized the metrics collector, added a streaming upload endpoint on the control interface, and moved fusion-tracer onto it.
• Updated CI, build pipeline, and dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.3. No configuration changes required.
• In NFS mode, fusion-tracer now sends its trace through the control interface and no longer falls back to writing it through the mount. A trace produced while the control node is unreachable is dropped. Tracing is unchanged when the control node is reachable.
• FUSION_NFS_READINESS_TIMEOUT is new and defaults to 60 seconds.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.14-amd64.json'
}

Bug fixes​

General​

• Fixed data loss when a file move is interrupted.
• Fixed declared outputs with brace patterns such as *.{bcf,vcf}{,.gz} not being saved.
• Fixed symbolic links not persisting their backing data.
• Fixed long symbolic link targets being truncated on read.
• Fixed temporary task files being uploaded while still being written.
• Fixed temporary working files being left behind after a retried operation.
• Fixed the advertised file system block size, which is now 1 MiB instead of 512 bytes.
• Fixed an internal cache maintenance task continuing to run after shutdown.
• Fixed an internal inode allocation check returning an identifier that was not free.

Google Cloud​

• Fixed corrupted results when assembling very large files on Google Cloud Storage.

Dependencies​

• Hardened internal tests against data races and a timing-related flake.
• Updated dependencies and GitHub Actions.

Upgrade notes​

• Drop-in replacement for v2.5.13. No configuration changes required.

Fusion v2.6.3 fixes a mount failure for task images with a non-root USER that mount the Fusion NFS export through the setuid helper.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.3-amd64.json'
}

Bug fixes​

NFS service mode​

• Fixed mounts failing with exit code 174 when invoked through the setuid fusion-suid helper.

Upgrade notes​

• Drop-in replacement for v2.6.2. No configuration changes required.

Fusion v2.6.1 adds per-backend request metrics for AWS S3, Azure Blob Storage, and Google Cloud Storage, and fixes uploads, mounts, reads, symlink metadata, and the local file system backend.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.1-amd64.json'
}

Feature updates and improvements​

Metrics​

• Added request metrics for every cloud backend. Fusion records latency, retries, error codes, and outcomes for each request to AWS S3, Azure Blob Storage, and Google Cloud Storage. New metric families cover file system operations, the content cache, and chunked I/O, with standardized names and a per-routine label. Fusion writes the metrics as a compact multi-member gzip file by default. See Fusion task metrics.

General​

• Added a control interface for direct uploads. Studios and fusion-tracer push data to cloud storage without going through the mount, and a task can upload while it is still running. fusion-tracer now uploads trace.json through this path.

Google Cloud​

• Added environment variables to tune the retry backoff timing for Google Cloud Storage.

Bug fixes​

General​

• Fixed a moved file removing its source before the copy completed.
• Fixed the inode allocator handing out an inode that was not verified free.
• Fixed a misleading "is a directory" error, now "permission denied", when opening a bucket you cannot read.
• Fixed a "no such file" error when a task reads a symlink staged by a previous task on Nextflow 24.04 and later.
• Fixed very large writes failing on oversized upload parts on AWS S3 and Azure Blob Storage.
• Fixed throttled mounts timing out during initialization.
• Fixed read operations not releasing their network connections.
• Fixed truncation and full-content writes on the POSIX backend.
• Fixed chunk cleanup failures being swallowed.
• Changed the sync task queue to drop from the tail and skip cancelled tasks.
• Fixed the TTL cache janitor not stopping on shutdown.
• Fixed the reported st_blksize hint, which now advertises a 1 MiB preferred I/O block size.

Google Cloud​

• Added automatic retries for transient Google Cloud Storage errors.

Dependencies​

• Updated the Go toolchain to 1.26.5, fusion-sdk to v0.2.1, and go-fuse to v2.10.1.
• Updated CI and build dependencies.

Upgrade notes​

• Drop-in replacement for v2.6.0. No configuration changes required.
• Fusion now stores the metrics file as gzip by default. If you read the metrics file directly, decompress it first.
• If you rely on fusion-tracer output, skip this release and use v2.6.2.