IT Press Tour 70: Clockwork

Clockwork joined The IT Press Tour for the first time and it gave us the opportunity to understand the mission, vision and solutions developed for GPU clusters, key for AI oriented workloads, by a team of innovators.Clockwork.io describes itself as building “Software-Driven AI Fabrics”. This is a programmable layer between hardware and AI workloads that makes GPU clusters observable, fault-tolerant and fully used, on any accelerator, network or cloud. We met several times its CEO, Suresh Vasudevan, who previously held leadership roles at NetApp, Nimble Storage, acquired by HPE, Sysdig and Omneon, acquired by Harmonic. The other founders and leaders come from Stanford: Balaji Prabhakar, co-founder, Yilong Geng, co-founder and CTO, and Mendel Rosenblum, chief scientist and VMware co-founder.

Click to enlarge

The firm just raised $31M round co-led by Premji Invest, Wing Venture Capital and Seligman Ventures, with NEA and e& Capital also taking part. It brings total funding to $73M. The money will go toward rolling out fault tolerance across training, inference and reinforcement learning, growing enterprise adoption, and selling through cloud partners. Vasudevan called fault tolerance “a goodput multiplier”.

The company’s roots are in global clock synchronization. Its software-only sync reaches accuracy of 10 to 100 nanoseconds across thousands of nodes. NTP and Chrony are less accurate, and PTP needs dedicated hardware. Between 2018 and 2022 Clockwork applied this to fintech, for regulatory timestamping. From 2023 it turned to GPU clusters, first with fleet and workload monitoring and then with workload fault tolerance. The resulting FleetIQ platform became generally available in September 2025.

Click to enlarge

Clockwork identified a serious problem: one failure stalls thousands of GPUs. AI jobs run thousands of iterations in lock-step across thousands of GPUs. The network is part of the compute, one straggler slows the whole job, and a single failure can stall the entire cluster. The deck cites three studies:

• Meta (HPCA 2025): mean time to failure is 7.9 hours for 1,024-GPU jobs
• ByteDance (2025): 42.5% of jobs ran at least 10% slower because of stragglers
• ClusterMAX and Microsoft Research: detection and repair take 1 to 3 hours when someone has to intervene manually

We also learned that Meta saw an unexpected interruption roughly every three hours during 54 days of Llama 3 training on 16,384 GPUs. The standard response is checkpoint-restart. It can take up to 90 minutes, leaves healthy GPUs idle, and forces the job to redo the work done since the last checkpoint, and the cost grows with cluster size. So it appears that some alternative options exist.

The first answer from Clockwork is represented by FleetIQ. FleetIQ runs as a single loop from detection to action, covering training, inference and reinforcement learning. Clockwork claims 30–50% higher GPU utilization and 90% fewer disruptive checkpoint restarts. On the detect angle, two tools cover this side:

• FleetAudit validates clusters before go-live. It checks the frontend and backend networks, NCCL, NVSwitch, GPU health and PCIe topology, and has an API for scheduled and post-maintenance checks
• FleetLens finds faults while clusters run in production. It combines ClockSync (sub-microsecond synchronization) with an out-of-band “probemesh” that measures one-way delay between NIC pairs every 10 ms. Path decomposition and triangulation then isolate the exact link, switch or NIC at fault, or show that the network is not to blame

Clockwork argues that today’s tools, which sample every 10 to 60 seconds, miss microsecond-scale events. That matters in fabrics such as a 9,216-GPU DGX GB300 SuperPOD, which has about 25,000 links and 50,000 transceivers.

Click to enlarge

To address failures, the company develops a comprehensive solution:

• LinkPass is an NCCL plugin. It reroutes traffic to healthy NICs when a link fails or flaps, then fails back automatically, with no change to the model. It is embedded in OCI
• TorchPass live migration moves work from a failing GPU or node to a spare without restarting the job. A central orchestrator, running alongside Kubernetes/Kubeflow, Slurm, Slinky or Soperator, quiesces the job, captures its state, swaps in a replacement and restores the state. Migration can be proactive, for predicted problems such as temperature, ECC errors, recoverable Xid errors, maintenance, patches or stragglers. It can also respond to hard failures such as GPU, HBM, node or kernel faults
• TorchPass snapshots are new and billed as an industry first. They capture a running job across all its nodes, at set intervals or on demand, writing either directly to storage through GPUDirect Storage or to DRAM with an asynchronous drain to storage. Platform snapshots need no code changes, take under 20 seconds, and add about 1% overhead when taken every 25 minutes. Model checkpoints need an opt-in integration, with patches available for TorchTitan, DeepSpeed and Megatron-LM. They take under 5 seconds and add about 1% overhead every 5 minutes

Click to enlarge

To illustrate market traction, the team shared several names as real case studies:

• LinkedIn runs LinkPass in production across its fleet and prevents tens of thousands of GPU-hours of downtime each month. Before LinkPass, one InfiniBand NIC flap could take 8 to 16 GPUs offline. Raghu Hiremagalur, SVP and CTO, infrastructure, said disruptive incidents have become “manageable maintenance events”
• Together AI is bringing TorchPass and LinkPass to market as a service on its GPU Clusters. It will show live failover at the PyTorch Conference. “Our customers grade us on goodput”, said Pavneet Ahluwalia, product lead
• WhiteFiber is expanding its use of FleetAudit across its GPU-as-a-service clusters. The tool has found ConnectX-7 and BlueField NIC issues, marginal optics, MPO cable faults and ECN storms before handover
• A large GPU cloud operator, not shared publicly, is piloting FleetLens to prove or clear the network when jobs slow down
• A global tech company, also hidden, is a design partner using platform snapshots on-premises. Its platform team cannot require hundreds of teams to add checkpointing to their code, so snapshots make every job recoverable

Other customers include Wells Fargo, Nebius, Nscale and DCAI.