SC26: Exclusive Interview with Gary Grider, Leader, HPC Division, Los Alamos National Laboratory
A fascinating discussion with a true pioneer and leading authority in high-performance data storage at scale
By Philippe Nicolas | November 17, 2026 at 2:02 pm
Gary Grider is the Senior Director of the HPC Division at Los Alamos National Laboratory (LANL). He spent so far almost 38 years at LANL at various positions covering the advanced computing environment. As division leader at Los Alamos, Gary oversees all aspects of HPC technologies and their deployment. He also manages the R&D portfolio, driving innovation and maintaining a strong pipeline of emerging technologies to address the laboratory’s evolving HPC challenges through strategic collaborations with academic and industry partners. He started his LANL era with 10 years as computing infrastructure consultant from IBM and previously spent close to 5 years at Sandia National Laboratories as computer systems analyst. With 30 granted patents in high-performance data management, Gary brings more than four decades of expertise in HPC and HPC-related storage, having worked in the field since 1984. He studied at Oklahoma State University from 1977 to 1983 before beginning his career in the fields of wind energy and magnetics, experiences that eventually led him into the data storage industry.
StorageNewsletter: Gary, we have known each other for many years and have met regularly at various industry conferences. SC26 in Chicago this week represents an excellent opportunity to exchange perspectives on the latest trends and evolving dynamics shaping the high-performance data storage landscape. Given your extensive experience with high-performance file systems, how do you compare the storage requirements of traditional HPC environments with those driven by emerging AI workloads? What are the most significant differences, and how are they influencing the evolution of storage architectures?
Gary Grider: HPC workloads were dominated by mass checkpoint/restart that was tightly synchronized across as many as tens of thousands of client nodes (millions of cores) using either a file per core (millions) in a single directory at exactly the same time or one shared file being opened and written to at poorly aligned offsets all at the same time. Other HPC workloads like job launch where the job is driven by a python script and that python script has then includes and each of those have ten includes, etc. This happens for every core all at the same time. You can get to a billion open read closes on a pretty small set of information in a lot of little files all at once.
AI workloads don’t really have many writers into one file at least not yet so far, but the workloads could have a lot of files being written and even more read at the same time, from many (millions) of small files. This part isn’t terribly different than the HPC launch except its less synchronized and predictable. There is a lot of use of local storage on compute nodes in AI that didn’t materialize in HPC because HPC grew up in the disk era and disk introduced failure to compute nodes. The other main difference is that you are likely to be doing inference, tuning, millions of agents doing I/O and they all need to have access to state, some shared state, for solving a problem. The workload is less bulk synchronous checkpoint followed by silence like in HPC/simulation and AI is more concurrent, different workloads hitting the environment in unpredictable ways. Another other part that is different is the KV cache concept. It’s interesting that HPC simulation used representation techniques to make memory seem larger than it is (think adaptive mesh refinement) where AI uses KV cache. Some people conjecture that as the number of experts rises from tens to hundreds or more, AI may also have to go to representations as well as KV cache (more or less smart paging) because memory is limited in capacity.
So there are differences: less predictable, more different things concurrently, etc.
What are the major challenges facing file systems in HPC and AI environments today? Which architectural limitations are you trying to overcome, particularly regarding metadata performance, scalability, and data access?
Metadata access/update, QOS, and efficient orchestration are of the top things in my mind. QOS because of the varied concurrent work. QOS at very large scale in a varied concurrent workload environment has never been conquered completely by object or file systems. Orchestration because tiering and locality problems are getting worse because it’s not just memory to flash anymore. Metadata because metadata is the part that has never scaled well and it stands in the way of getting the most bandwidth and IOPS applied to the problems. Object systems can provide bandwidth as can file systems but if the objects are small, you can’t keep the pipe full. People are packaging files and sending them in streams using G-RPC avoiding the per file protocol and putting them on local to compute flash. Object systems are often built on massively scalable KV’s designed for individual records but to get that scale, working with grouping of objects to efficiently retrieve as a batch got lost in the planet sized scaling thinking in object systems.
In many ways all three of these things are state management. Once you remove the locks and existence checks and similar state, inserts can become in essence log structured and arbitrarily fast. With a good, persistent metadata layout, queries that do both single entries (like stat) and organized like (readdir/stat) queries can be fast with good caching but caching requires state. Orchestration is large limited by state management too. You can only insert as fast as you can existence check and in a distributed metadata system, that’s hard. QOS is about understanding running state as well. So, if you heard me say metadata is the problem – in so many ways it is, but largely it boils down to distributed state management which is in so many ways a lost art. We lived in the embarrassingly parallel world for far too long.
I would say the overwhelming thing is distributed state management or avoiding it through age old techniques like one-sided operations and better living though data structure magic, both of which are somewhat lost art, but with scale up networks the need for these types of solutions is finally growing again.
How would you define a parallel file system, and what fundamental architectural characteristics distinguish it from a traditional distributed file system?
Classically, traditional distributed file systems don’t stripe across multiple data servers but they can stripe across devices. Multi-writers into one file is clearly a differentiator as well. Scalable distributed metadata service is also not typical for traditional distributed file systems and parallel file systems really pushed on all three of these fronts.
To continue on this, is file striping across multiple storage nodes an essential requirement for a file system to qualify as parallel, or can parallelism be achieved through other architectural approaches?
Well, bandwidth is a function of how many storage devices you can move data to/from in parallel. Storage “Nodes” can only have some number of devices and then you have to add storage nodes. So, I think fundamentally parallelism is required. Compression and other things can be done to get bandwidth but there is no substitute for parallelism. That is largely true for both bandwidth and IOPS both to data files and metadata systems. Scale up multi-node systems represent a much larger building block data servers perhaps, but those are limited in how big they can be as well. Now who does the parallelism and how its done can vary widely of course.
Open source has historically been a fundamental requirement for national laboratories and HPC environments. Does this preference extend to AI infrastructure, or is there greater acceptance of proprietary and commercially developed solutions?
I would say that some National Labs have the ability to consider fully self-integrated AI compute and infrastructure (at least integration of pre integrated pod/rack scale systems). Not all labs will invest in that, but there is a double-digit cost penalty for white glove fully integrated at scale approaches. With soaring pricing, that double digit cost is a lot. Open-source system management and cloud style scheduling/etc. exists and some labs will join those communities or even lead them. Networking is now pretty much Ethernet and while it’s not simple, it’s not beyond a lab’s capability to self-integrate rack to rack or pod to pod. Storage systems will be similar. Some labs will go commercial at different levels, and some will invest in/join community efforts. That could be at the file system/erasure level and/or indexing and/or data lakehouse/analytics levels. There are open efforts in all three areas. The more of the solution that is provided commercially, especially if you pay by the byte or seat or something, end up affecting costs in big ways.
I personally prefer open and joining communities, if possible, otherwise there is inferior solutions over time and no cost control basis. Paying by the byte or byte moved is clearly what industry wants, but that clearly will produce rich people and not necessarily the best solutions and can be anywhere from 2x the cost (of base hardware plus open investments) to 30x depending on how many levels you go with commercial solutions on. I wish more would join community efforts, but I am suffering from no illusion that all national labs will want to be active in open community efforts for all parts of the infrastructure ecosystem.
You have selected Lustre-based solutions alongside alternative file system architectures. What technical, operational, and economic criteria guide your selection of a parallel file system for a particular workload?
Lustre was invented for synchronized checkpoint and restore with multi-writers into one file and one file per process/thread in one directory. It came from the disk era as well. It does a lot more than bulk sync checkpointing, but it is good at that. It’s not known for being good at billions of small files and good at distance support/orchestration, and other things. It also is not really a large community, with few companies selling solutions now. It’s a workhorse and it has smart people engaged but its market share is changing because the overall market has gone non-linear.
Other solutions are good at other things, like easy to deploy fast, IOPS to file data, metadata ops, etc.
The one exception in my opinion is pNFS. With the recent achievement of multi-writers into a single striped file with respectable performance and the open-source Lattice/Novus announcement beginning to attack scalable metadata in new ways including state management, the playing field is now more level than ever. pNFS has distance capability and orchestration (due to recallable maps). It has re-export which tames the read storms and directIO all the way from the client to the server-side component file system. It has localIO which lets you leverage data locality and manage compute side flash resources well. It is extremely well suited for pushdown analytics. With RDMA and nConnect it has no issues with bandwidth and with directIO data paths its IOPS are reasonable as well.
Make no mistake, I don’t think one file system is better than all of them added together. What I would say is pNFS has the mechanisms to be near state of the art at least near that for those areas, many even simultaneously. It’s also standard and uses the Linux client and Linux data server (just nfsd). With Lattice there is at least one example of a scalable, open source, user space metadata server organized to scale on many dimensions. Further, pNFS is getting network erasure as an option as well in the coming year or so. The separation of the persistent catalog and the protocol services in the Lattice metadata system, allows bulk metadata activity, direct object access, etc., without going through NFS. Its a compelling story but of course we started that work in 2001.
Beyond open source, open standards and interoperability are increasingly important in file storage. Why are these principles critical for the long-term evolution of HPC and AI infrastructure?
There will be organizations that will get less science/engineering done by using enterprise class commercial offerings only, while other organizations will need white glove service for good reasons, this is a very important consideration. Further if you want to provide data services over decades, you need infrastructure that has extremely long legs and that means large communities for the parts. Without at least a good open alternative, everyone loses, even the providers frankly.
Vast Data has challenged conventional assumptions about NFS performance, demonstrating that the protocol itself is not necessarily the bottleneck many users once believed it to be. How do you interpret these results, and what do they reveal about modern file system architectures?
Well, I must disagree that we all the suddenly know that NFS bandwidth performance and even metadata performance to a way lesser degree is not new. EMC demonstrated multiple NFS servers on top of a block hash systems similar to 3PAR long ago. pNFS was demonstrated on PVFS 20 plus years ago. Having an east west network on the server side and a north south network on the client side to gain parallelism is not new. What is new is an emerging market that causes people to start listening. Timing is everything they say. More people need a solution now for a much broader workloads set and Lustre is too narrow for the broad workloads of today and hard to manage, and pNFS based solutions until the last very few years were still being finished up. My interpretation is the market is finally needing standards-based solutions that scale. Some solutions were poised to exploit that early and others are trying to exploit it now. We have known NFS could be made to scale and pNFS is the richest available mechanism for that pursuit, it’s just nice to think that the massively growing market will pull things forward finally.
However, Vast Data’s approach is not based on pNFS. What makes pNFS particularly relevant for high-performance environments? Is its appeal primarily related to open standards, native client support with its ability to avoid proprietary client software?
Well, it’s not native client or standards totally, although the fact that everyone needs a scalable solution means standards and software that ships in distros has huge appeal. Native data servers, lack of fully open standard solutions, those are important too, but what about pNFS really matter architecturally over just NFS? It’s simple, the client in NFS3 is not scalable aware. It’s really good at lots of things but being parallel aware is not its thing. Using just NFS means you have to hide the parallelism and the client can participate. The pNFS client is the NFS client but with parallel awareness. It also has map recall capability that enables orchestration and with localIO all the way to data servers running on clients using local flash. Fail over is now end-to-end in a completely NFS controlled way, not terminated and regenerated. Re-export is possible without pNFS, but orchestrated re-export is not. Pushdown analytics (compute near storage) is a natural for file based pNFS servers and with energy being the primary concern in computing now, compute near data’s time has potentially come. It’s not that you can’t do compute near storage with other approaches, there is a standards-based way to do it all in pNFS. And there is now, with Lattice/Novus, a fully open source, user space, scalable along many dimensions, community metadata server, so there is at least one path to an entirely open stack. This enables many communities to pitch in. Distributed memory with scale up networks and file systems state management is highly related and a different community than storage. Array/grouping organized key value stores is not file storage community thing. Even scalable protocol service is not necessarily owned by NFS people. It’s the fact we can leverage truly deep intellectual communities that is a huge part of the power of open/community and pNFS is poised to leverage those communities now with Lattice.
The alternative is object, but with the separation of metadata catalog and protocol/state service in Lattice, the best-in-class object systems can be built beside pNFS instead of on top of it, providing full pNFS and object on the same data without the inefficient stacking that many solutions do today.
It’s just a pretty compelling thing for the community. Its way more than just not wanting proprietary, it’s the power of community along all the dimensions that is good.
Several established storage vendors, including Dell, Everpure, Hitachi Vantara and NetApp, historically relied on third-party software to address HPC requirements. IBM promotes Storage Scale, HPE acquired Cray and others and Huawei develops its own. Hammerspace and Peak:AIO have chosen pNFS, I have to mention that NetApp just acquired Peak that confirms a strong decision. With the emergence of AI infrastructure, they are increasingly embracing pNFS-based architectures. How do you interpret this strategic shift?
Yes, Dell has a product that is based partially on pNFS tech, Everpure and Hammerspace have full pNFS solutions and now with multi-writers into one striped file, the HPC simulation community is approachable. NetApp Novus will be another solution with open client and a metadata server that is lattice based, and the NetApp data servers which are enterprise class. So, yes there is industry uptake, in fact there are more companies that are planning to contribute to Lattice as well.
How do I interpret this strategic shift? Well, my interpretation is that every engineer in the world that runs Matlab and CAD on workstations and maybe a server or a few and uses a an NFS filer just has had their work pattern change. Every engineer has 100 agent buddies helping do the engineering, looking wider, writing test code, coming up with hypothesis, etc. So, take your 100 engineer company, it went from 100 engineers computing load to 10,000 engineers computing if they are going to stay in business and keep up. Presto, the NFS filer is no longer sufficient. AI comes to the Engineering enterprise, and presto agents present a large varied concurrent workload beyond most distributed solutions. Lustre by itself could help but it’s not built for such a varied workload. These companies you mention are big in the enterprise and now the enterprise either goes to the cloud (which given costs is not aways a perfect win) or they are suddenly in need of a much more scalable compute and data infrastructure. Either cloud or on prem, scalable file solutions are needs where that wasn’t true not that long ago. The enterprise AI market needs Enterprise thinking but at much larger scale than normal. Each enterprise has different needs, distance and heuristics and orchestration and indexing/rag and … but at larger scales than enterprise class solutions of today can provide. Why did these companies jump in, because that Enterprise AI market while not as big as self-integrated custom AI factories, is far more profitable, or will be at some point because enterprises aren’t national labs, they need enterprise class help. So, the volume market in the cloud/factories need scalable solutions and now the enterprise does as well.
As I just said, NetApp has gone a step further with its acquisition of Peak, strengthening its position in high-performance storage. What impact could this acquisition have on the competitive landscape, particularly in HPC and AI storage?
Well, referring to what I just said, enterprise class AI infrastructure is clearly a good play. Cloud based or on prem. For the community, now there are open-source parts (client, metadata service, data service) for all the parts and different solutions use different open-source parts. NetApp engaging in the open-source Lattice is good for the community and the community is good for Lattice/Novus/NetApp. All consumers want good providers competing and the more enlightened consumers understand that an open option makes everyone better. In a few years, I expect the bulk of the parallel/scalable file system tech to have pNFS tech involved somehow, it’s too compelling and ubiquitous to not leverage. That’s the real goal – we need couple of decades of community driven solutions. Other solutions that compete is good as well, but without at least an open community solution, everyone is worse off.
At extreme scale, how do you assess the role of object storage and S3-based access alongside traditional parallel file systems? Are these technologies complementary, or could object storage eventually replace certain file system workloads?
I think as machines/models start doing all the data access, the need for folders and permissions, etc. could change. Object systems have their uses. It’s sad that the big object system examples are pretty inefficient, using heavy protocols, not dealing with small objects well, not having much win in locality of any kind, just flat to scale to the planet in total size but not great at some organized accesses. I guess I would say that there is a need for both. If you need scale to millions of users and each is not that demanding, object is a good choice. With pNFS and with network erasure being added, pNFS encroaches on Object and it already has RDMA and organized access, etc. File has made quite the come back because of AI writing code. Its coding is very much file based and when it’s not file based, it’s very fast local kv based. The AI community wanting to work in tiny units (word sizes, file sizes, etc.) but having massive bandwidths, capacities, and IOPS is a new demand that is harder for object to service. Using object to get a tar file of a million files to put on your local compute server flash is being done, but other approaches are just running compute near the storage and having the storage build streams of small files, sending them through things like gRPC streams is also being done. The separation of persistent metadata catalog, protocol service, and state service in Lattice allows you to have optimized object service and optimized open standard scalable file service simultaneously. All the orchestration could be leveraged by both object and file along with rexport/caching/etc. I can see a ton of overlap in mechanisms between object and pNFS but to optimize, you want to avoid putting one on top of the other. That has worked out to be workable but slow. The Lattice design addresses that fundamentally.
It would be nice if POSIX could evolve to have array-based opens and reads but across multiple files, not just across different parts of the same file. POSIX could do this pretty easily and it would kill off all this need to tar up files and then ship them. I pushed this idea a little about a decade ago, but it went nowhere. Maybe it’s time has come?
Several initiatives are developing POSIX-compatible file system interfaces on top of object storage. Do you consider this a credible long-term architectural direction, and what technical obstacles must still be overcome?
I think object is good at things and files/file systems are going to be good at other things with, some overlap. The two are just not great stackable mates, but they could share catalog service, state mgmt. service, caching services, orchestration services…. Lattice tried to fix this stacking of one on the other mistake by breaking up the metadata service into parts that scale differently and employ different community expertise. Those services should be able to be used by object and file and KV, etc. as it makes sense. It’s a deeper co-designed way to share between the different access methods than just stacking.
Looking ahead, what are the next major scalability and performance barriers for parallel file systems? Which technological breakthroughs do you anticipate, and what do you expect from the leading vendors and open-source communities addressing these challenges?
I think the lack of memory and the lack of watts may finally push the idea of compute near data (be it near data at rest or data in motion). It’s cheaper to put a processor near storage than it is to put memory near storage. At this point we need to push hard on compute near wherever the data is or where the data will be. Common data in motion formats, allowing remote common addressing of memories, etc. seems like interesting work. Working at a place Von Neumann once worked, saying load/store is dead is not something I will say. A mix of streaming/load store/other one-sided operations seems necessary to save power and memory, and leverage extremely cheap low end processing capability seems like it’s time may be coming. pNFS is somewhat of an enabler for this because maps can be changed on the fly, where data is can change and every distributed part understands all that. This idea of world view for experts and agents to use to solve problems is an opportunity to minimize where the cost is. Energy and compute being more abundant than memory seem like first class cost concerns, and that was never true in the past.
Comments
To further explore these topics, we encourage our readers to follow the latest developments in file systems and object storage, particularly the announcements emerging from this week's SC26 conference, as well as other major industry events such as ISC, MSST, and beyond.
Today's news also highlights several new developments from some of the companies mentioned above, reflecting the rapid evolution of the high-performance storage landscape.
With the AI tsunami reshaping infrastructure requirements, a new competitive race is underway. The growing adoption and validation of pNFS are accelerating much-needed changes across the vendor ecosystem, driving innovation, architectural shifts, and market repositioning. Against this backdrop, we also anticipate further industry consolidation, with several potential acquisitions in the coming months.













