47 posts tagged with "fusion"

Fusion v2.6.2 fixes a regression introduced in v2.6.1 where fusion-tracer could stop producing output.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.2-amd64.json'
}

Bug fixes​

fusion-tracer​

• Fixed fusion-tracer producing no trace.json when the control node is unreachable by falling back to writing the trace through the mount.

Upgrade notes​

• Drop-in replacement for v2.6.1. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.13-amd64.json'
}

Feature updates and improvements​

General​

• Added a control interface for direct uploads. Studios and fusion-tracer can push data to cloud storage without going through the mounted file system. A task can now upload data reliably while it is still running.
• Improved failure reporting when the Fusion file system process exits before the mount completes. Fusion now reports the failure immediately instead of entering a retry loop.

Google Cloud​

• Added environment variables to tune the retry backoff timing for Google Cloud Storage.

Bug fixes​

General​

• Fixed silent write loss on mounts without write permission by verifying write access at startup.
• Fixed a misleading "is a directory" error, now "permission denied", when opening a bucket you do not have permission to read.
• Fixed large file uploads failing when a write exceeded the cloud provider maximum upload part size on AWS S3 and Azure Blob Storage.
• Fixed interrupted uploads leaking temporary staging data into workflow output folders.
• Fixed mounts timing out under heavy rate limiting at startup.
• Fixed read operations not releasing their network connections.
• Fixed errors during file data cleanup being hidden.

Google Cloud​

• Added automatic retries for transient Google Cloud Storage errors.

Dependencies​

• Updated dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.12. No configuration changes required.

Fusion v2.6.0 adds NFS service mode, the Fusion Doctor diagnostics tool, new metrics, and cloud storage reliability improvements.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.6.0-amd64.json'
}

Feature updates and improvements​

NFS service mode​

• Added NFS service mode. Fusion runs as a long-lived daemon and serves the mount over NFS, instead of one FUSE mount per process. A single Fusion instance backs many tasks and isolates metadata per task. An HTTP control service registers, flushes, and cleans up each work directory. Snapshots and work directory cleanup now survive Spot reclamation.

Fusion Doctor​

• Added the fusion doctor subcommand. It validates the runtime environment and reports common mount, configuration, and connectivity problems with remediation steps. It supports custom thresholds and profile validation, and an opt-in --redact flag strips personal data from reports. Fusion Doctor also ships as a standalone binary and container image for use outside a running mount. See Fusion Doctor.

Storage​

• Added support for prefix-restricted S3 bucket policies, and a fallback chain for S3 bucket region detection.
• Changed archived Azure blobs to map to a dedicated "resource archived" error, and fixed the Azure CopyFromURL size threshold.
• Improved the classification and messages for non-recoverable I/O errors.
• Improved chunk download resilience on unreliable networks. Reads no longer fail with EIO when an object store stream stalls or truncates mid-transfer. Fusion watches each chunk download for progress, cancels stalled streams, resumes from where the transfer left off, and caps downloads with a configurable timeout.

Spot instances​

• Removed the dependency on fusermount. Fusion exits cleanly on Spot reclamation instead of failing with EINVAL, which reduces orphaned mounts and lost work.

Metrics​

• Added OpenTelemetry metrics for read and write operations, with file output and JSON formatting.
• Added process metrics collection to fusion-tracer, written to trace.json. The GPU tracer is enabled by default.

Fusion Snapshots​

• Enabled incremental snapshot dumps by default.
• Updated fusion-snapshot to 1.9.0, with symbolic link remapping in NFS mode.

Dependencies​

• Updated dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.
• NFS service mode applies only to Intelligent Compute. Per-task FUSE mounts on other compute environments are unaffected.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.12-amd64.json'
}

Feature updates and improvements​

General​

• Improved chunk download resilience on unreliable networks. Reads no longer fail with EIO when an object store stream stalls or truncates mid-transfer. Fusion watches each chunk download for progress, cancels streams that stop, and resumes from where the transfer left off with exponential backoff. This removes the multi-minute hangs and read failures seen in environments with intermittent object store connectivity, typically missing VPC endpoints or congested egress paths. Healthy slow links are unaffected.
• Added a log entry when an application tries to create a hard link inside a Fusion mount, for example certain bioinformatics tools and snapshot workflows. The entry names the operation and the calling process and points to the available workarounds, a symbolic link or a copy. Hard links remain unsupported and the error returned to the application is unchanged.

Dependencies​

• Updated dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.4.22-amd64.json'
}

Bug fixes​

Azure​

• Fixed symlink compaction failing with HTTP 409 CannotVerifyCopySource errors for files between 256 MiB and 4 GiB.
• Changed reads of Azure Archive tier blobs to fail immediately instead of being retried.

General​

• Fixed stale symlink content being served after a failed compaction.
• Changed cloud error handling to retry only transient errors and fail fast on permanent errors such as forbidden access and invalid credentials.
• Added a structured warning when an application calls the unsupported link(2) inside a Fusion mount.

Dependencies​

• Updated build dependencies.

Upgrade notes​

• Drop-in replacement for v2.4.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.11-amd64.json'
}

Bug fixes​

AWS​

• Added support for prefix-restricted S3 bucket policies.
• Improved S3 bucket region detection with a fallback chain instead of defaulting to eu-west-1.

Azure​

• Changed reads of an Azure Archive tier blob to return a "resource archived" error instead of a generic cloud provider error.
• Fixed symlink compaction failing for files between 256 MiB and 4 GiB.

General​

• Removed an unnecessary truncate before each append-mode write.
• Fixed a NoSuchKey crash when an application renames a file and immediately creates a new one at the same path.
• Fixed partial chunk uploads producing incomplete or corrupted objects.
• Fixed symlinks being served as a short path string instead of the file content when symlink compaction failed.

fusion-tracer​

• Removed spurious Failed to fetch entry size and Failed to load file metadata errors.
• Built fusion-tracer against glibc 2.17 so it runs on container images with glibc 2.17 or later and loads the NVIDIA management library on GPU nodes.

Dependencies​

• Updated CI and build dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.4.21-amd64.json'
}

Bug fixes​

General​

• Fixed "entry not found" errors when reading a file after a rename where the new name is a prefix of the old name.
• Fixed fchmod() failures on temporary files that were unlinked while still open, such as bash heredoc files.
• Removed the dependency on the fusermount binary for unmounting, and fixed Fusion exiting with EINVAL during Spot instance reclamation.
• Fixed a crash when a root CA certificate bundle contained invalid PEM data.

Logging​

• Demoted license-related error messages to debug level.

Upgrade notes​

• Drop-in replacement for v2.4.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.10-amd64.json'
}

Feature updates and improvements​

Fusion Snapshots​

• Added Spot interruption handling on Amazon ECS. On ECS, Spot interruptions arrive as signals rather than through a metadata endpoint. Fusion intercepts the signal and stops it from reaching child processes, which gives the snapshot time to complete.

fusion-tracer​

• Added process metrics collection. The tracer records CPU, memory, I/O, context switches, cgroup statistics, and GPU metrics, and writes a .fusion/trace.json file when FUSION_WORK is set. Polling intervals adapt to load and metrics are aggregated across the process tree.

Logging​

• Added the available disk space range to the task completion output. Use it to check cache storage capacity.

Bug fixes​

General​

• Fixed incorrect byte ranges during multipart compose and copy operations on AWS S3, Azure Blob Storage, and Google Cloud Storage.
• Fixed fchmod() failures on temporary files that were unlinked while still open, such as bash heredoc files.
• Removed the dependency on fusermount for unmounting, and fixed Fusion exiting with an error during Spot instance reclamation.
• Fixed a crash when a root CA certificate contained invalid PEM data.

Logging​

• Demoted license-related messages to debug level.

Dependencies​

• Bumped the Go toolchain from 1.24.8 to 1.24.13.
• Updated CI and build dependencies.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.4.20-amd64.json'
}

Feature updates and improvements​

Fusion Snapshots​

• Added support for Fusion Snapshots in Google Batch. See Fusion Snapshots for more information.
• Added cleanup for dump folder following Spot termination to prevent unwanted storage costs.

Logging​

• Better log readability: Added [seqera-fusion] prefix to console logs for easier identification in mixed output environments.
• Improved error messages: Added more context and clearer wording to cloud provider errors, including better messages for archived/cold storage objects.

Troubleshooting​

• Expanded Fusion Troubleshooting documentation to include error codes and exit messages.

Bug fixes​

AWS​

• Fixed misleading "file not found" errors: Changed permission errors (403 Forbidden) from cloud providers to display as "Permission denied" instead of incorrectly showing "No such file or directory" (#1068).

General​

• Cloud errors now log properly: Fixed an issue where cloud service errors were silently treated as simple "file not found" cases instead of logging and reporting properly (#1077).

Upgrade notes​

• Drop-in replacement for v2.4.x. No configuration changes required.

How to pin this version​

fusion {
  enabled = true
  containerConfigUrl = 'https://fusionfs.seqera.io/releases/v2.5.9-amd64.json'
}

Feature updates and improvements​

Fusion Snapshots​

• Added support for Fusion Snapshots in Google Batch. See Fusion Snapshots for more information.
• Added cleanup for dump folder following Spot termination to prevent unwanted storage costs.

Logging​

• Branded console logs: Added [seqera-fusion] prefix to all console logs for better identification in mixed output environments.
• Real-time terminal output: Changed command output to display immediately in the terminal instead of buffering, improving visibility during execution.
• Clearer error messages: Improved error messaging for non-recoverable I/O errors and archived objects.

Troubleshooting​

• Expanded Fusion Troubleshooting documentation to include error codes and exit messages.

Bug fixes​

AWS​

• Fixed AWS S3 Forbidden errors not mapping properly to permission denied.

General​

• Fixed cloud storage errors not logging properly in certain scenarios.

Upgrade notes​

• Drop-in replacement for v2.5.x. No configuration changes required.