The Effects of AI Expansion on Testing Models and Infrastructure

If you’ve worked in networking long enough, you’ll remember when bandwidth shaped nearly every decision. Dial-up connections had images appear line by line and you were scheduling large downloads to run overnight, and Voice Over IP sounded impractical because the available pipe could barely carry the audio, much less the protocol overhead surrounding it. Engineers had no choice but to think carefully about efficiency because every byte mattered, since every byte had somewhere very narrow to go.

When the pipes grew, 10-gigabit and 40-gigabit connections changed the economics of optimization. With more capacity absorbing an inefficient protocol or an unexpected burst, there just wasn’t enough of an incentive to examine every layer of the stack faded. A protocol worked, the application performed, and the network had room, so the engineering attention moved elsewhere.

Old questions, new conditions

During a recent NANOG session on resource constraints, there was the sense that AI may be bringing some of those older questions back, though under very different conditions.

AI workloads are developing faster than the planning cycles network operators have traditionally relied on, like holiday traffic, scheduled broadcasts, product launches, and other familiar events that usually provide some warning. Even when the exact demand is uncertain, the organization generally knows why traffic may increase and when to expect it.

AI, however, is appearing in too many places for that model to hold. Some deployments are directed by IT, but others arrive through applications, development teams, cloud services, embedded product features, or individual users. Network behavior changes before the operator has a clear picture of what changed, which workloads are responsible, or where the resulting bottleneck is forming.

There was a comment on one of our recent Linkedin posts that mentioned how network hardware appears comfortable at 30 or 40 percent utilization in the context of using more of it before spinning up more, but when a collection of AI workloads begin operating concurrently, that utilization disappears fast. The resulting problem extends beyond raw volume and into training jobs, model queries, customer workloads, and internal services that create traffic patterns that look very different from the applications around which the network was designed.

From the user’s side, an AI interaction can look remarkably ordinary: when a request goes into a chatbot, a loading indicator spins, and a response comes back, that fits neatly into the familiar shape of a web application.

But behind that loading indicator the request may be moving among several nodes in a cluster, where other models may be training at the same time and/or customer jobs may be competing for the same resources. The network may be carrying exchanges among GPUs, storage systems, orchestration components, and application services, all of which contribute to the delay experienced by the person waiting for an answer. So while the visible transaction is simple, the machinery supporting it is not.

That distinction matters because many of the tools operators have used to manage priority were developed around traffic with more predictable characteristics. For example, voice traffic is sensitive to latency, so it naturally gets processing priority, while a business-critical application receives another policy. Eventually, every application owner argues that their traffic is equally critical, and the network arrives back at the familiar problem: when everything has priority, priority stops providing much guidance.

AI makes that problem harder because the workloads are dynamic. A static VLAN or a conventional quality-of-service rule may provide some separation, but those approaches assume the operator can identify the traffic cleanly and place it into a stable category. AI workloads can move among nodes, change behavior according to the job being performed, and share infrastructure with services that have different requirements. As an example, we know that tokenization of jobs is a huge issue. As new efficiency and project metrics get integrated, we fully expect this shift further based on projected token effectiveness per job or other KPIs that organizations would rely on to ensure more efficient use of AI resources, both internal and external.

The NANOG discussion raised the possibility of using metadata to classify that traffic more meaningfully. The goal would be to understand the purpose and behavior of a flow well enough to apply policies that reflect the workload, rather than relying only on ports, addresses, or broad application categories. But that leads to a more difficult operational question: where should the optimization happen?

Some improvements may belong at Layer 2, where performance between nodes can become critical, while others may require routing changes at Layer 3. In some cases, the application itself may be the only place with enough context to make a good decision – adding bandwidth can relieve pressure, but capacity alone does not determine whether the right traffic reaches the right destination at the right time. For network operators, this creates a visibility problem before it becomes a capacity problem.

Like we alluded to earlier, organizations regularly have infrastructure operating well below its theoretical limit while continuing to buy more, which may look irrational from the outside, but it’s often done to account for sudden spikes in utilization to adhere to the 80/20 rule. It could, however, reflect uncertainty – the team may not have a reliable inventory of its assets, a clear understanding of the workloads using them, or enough information to predict the effect of the next traffic spike. Buying more capacity feels safer than changing a system whose behavior is only partially understood and always evolving in some interesting way.

But AI will make that habit increasingly expensive. Additional capacity can conceal inefficient paths and poorly classified traffic for a while, but it does not explain why one workload suddenly consumes the available headroom or why another service begins suffering at the same time. Operators need to know what is present, how it communicates, which dependencies exist between services, and how utilization changes when several workloads overlap.

The practical lesson we carried away from NANOG was that resource pressure creates a reason to revisit technologies that have worked so reliably that engineers have stopped examining them.

SSH and HTTPS leads to better outcomes

Another session discussed SSH, a protocol so familiar that most operators rarely question its basic operating model. Establishing a secure session requires negotiation and authentication, and for one administrator connecting to one device, that overhead is usually insignificant. The calculation changes when an update must reach tens, hundreds, or thousands of nodes, where even a few seconds of session setup repeated across a large environment can become a serious delay, particularly when the update is responding to a zero-day vulnerability or maintenance window deadline. At that point, deployment speed is part of the security response.

The presentation explored using SSH together with HTTPS so that sessions could be maintained more efficiently, reducing the repeated cost of opening and closing connections without discarding SSH’s security role. The reported improvement was actually substantial enough to make us think immediately about our own engineering environment.

The ProVision team operates a much smaller number of nodes than the large AI environments discussed at NANOG, but it regularly distributes DNS updates, zone files, and security patches across those systems. Even with roughly 50 nodes rather than thousands, cutting the time required for an urgent update by a factor of two or three could have real operational value.

The value of connection

This probably goes without saying, but it’s what makes operational meetings such as NANOG so useful: a presentation does not need to describe your exact network to change how you think about it. Problems encountered by organizations operating thousands of GPUs can expose assumptions that also exist in an environment with dozens of DNS nodes, and while scale changes the severity, many of the underlying questions remain recognizable: Where is time being lost? Which overheads have become invisible through familiarity? Which protocols still fit the jobs they are being asked to perform? Which parts of the design are simple because they are well understood, and which are merely old?

In our subsequent conversations about the topic, an interesting design analogy came up about windshield wipers. A conventional design places the wiper somewhere visible and accessible, but a more visually refined design may hide it inside a compartment, preserving the clean line of the vehicle. The hidden design looks better, but it also creates a place where dirt and water can collect, a drainage system that can clog, and a component that may be harder to service several years later.

Engineers can become so absorbed in solving the immediate design problem that they lose sight of the maintenance problem being created alongside it. The elegant solution becomes less elegant when someone must patch it under pressure, diagnose it during an outage, or clean out the engineering equivalent of a blocked drain.

The phrase that surfaced in that same conversation was, “In stupidity, there is strength.” It was said with some humor, but the underlying point was true. Simple systems are often resilient because their behavior is visible. Their failure modes are easier to understand. Their components can be reached, replaced, or repaired without dismantling everything around them.

That principle should remain present as operators adapt networks for AI, where more advanced traffic classification may be necessary, policies will need to become more responsive to workload behavior, and older protocols deserve another look, particularly where their overhead compounds at scale. None of that removes the responsibility to keep the resulting systems understandable and maintainable.

AI is renewing the case for efficiency after years in which capacity often allowed it to be deferred. It is also forcing operators to examine the network as a connected system rather than a collection of links, devices, and applications that can each be expanded independently.

The most useful response begins with visibility: know what infrastructure exists, what is running across it, and how those workloads interact. From there, operators can classify traffic with more context, measure where delays occur, decide which layer is best suited to address them, and revisit familiar technologies when their hidden costs begin multiplying.

The answer may involve more capacity, but it may also involve a better policy, a more efficient update mechanism, an application change, or a protocol that has finally received the second look it deserves.

That’s what made that NANOG session so valuable – it took a problem associated with enormous AI clusters and turned it into a practical question for our own engineering team: where have we accepted unnecessary overhead simply because the system still works? It’s a question worth asking before the next workload arrives.