For AT&T, keeping its 145 million wireless and 16 million broadband customers online is a top priority. A 20-minute outage can be costly for the telecommunications giant, which credits customers’ bills when they temporarily lose service.
Typically, outages are highly localized and don’t generate such national attention, but in rare instances, they can affect millions. In February 2024, a major wireless outage at AT&T blocked more than 92 million voice calls and over 25,000 attempted 911 calls, according to a report published by the US Federal Communications Commission. Other major telecom providers, including Verizon and T-Mobile, have also experienced broad service disruptions in recent years.
To respond more effectively to outages and improve communication with customers, AT&T set out to centralize its system for identifying network interruptions.
“We have to know of any network interruptions immediately and solve them, ideally before any customer feels it,” Andy Markus, AT&T’s chief data and AI officer, told Business Insider.
In 2017, the company began creating its end-to-end incident management system, or EEIM, built on technologies from firms including MongoDB and Snowflake. Initially grounded in traditional machine learning, AT&T added generative and agentic artificial intelligence capabilities in subsequent years, expanding the tool’s functionality, Markus said. The tool has been deployed to address service disruptions for individual customers and small businesses.
Today, the EEIM leverages a mix of AI and machine-learning capabilities that can identify the root cause of a disruption, triage issues by suggesting fixes that can be handled remotely or by sending human technicians into the field, and proactively alert customers when an outage occurs, Markus said.
The tech
Telecommunications outages can stem from a wide variety of events, including inclement weather, system failures, technical glitches, cyberattacks, and infrastructure damage. As customers become aware of connection issues with their smartphones, landlines, or internet connections, they can inundate AT&T’s customer service centers with requests to get their service back online.
Markus said that in the first quarter of 2017, AT&T sought to develop a system that would be more responsive to outages. When the initiative started, AT&T gathered a cross-functional team to encourage broad input from employees across IT, data, and AI. The goal was to develop an EEIM around AT&T’s existing network technology while developing new AI-enabled tools to integrate into workflows.
Feedback was also sought from the frontline field technicians responsible for maintaining and repairing the company’s network, and from the network team that monitors traffic and quickly restores services when interruptions occur, Markus said.
Markus said AT&T also used its EEIM to reorganize 10 petabytes of data — the equivalent of 5,000 billion pages of printed text. This included pulling in network logs, network alarms, incident dispatch tickets for tracking network disruptions, and outage details. He added that the data needed to be consolidated and reorganized to allow AT&T to identify issues and predict when they may occur, direct team members to take action, and support new features and functionality.
MongoDB, a document database vendor, played a central role at this step. Founded in 2007, MongoDB’s platform is designed to be elastic, meaning that as data and traffic needs grow, AT&T only needs to add more capacity, not re-architect the entire platform as new data sources are added, said Ben Cefalo, MongoDB’s chief product officer. Cefalo calls this method “sharding” and said it allows data to be distributed across multiple machines and handle large data sets.
AT&T’s EEIM also relies on the cloud computing platform Microsoft Azure, data analytics via Databricks, and incident reporting via Snowflake, Markus said.
An AT&T spokesperson said the EEIM was launched for broadband fiber in the first quarter of 2018. In the first quarter of the following year, it expanded to cover digital subscriber lines, or DSLs.
In June 2018, AT&T launched a separate application called Atlas, which is used by field technicians and relies on a blend of machine learning and AI models to understand the root cause of an outage and recommend a plan to fix it.
AT&T had embraced these traditional forms of AI before Markus joined the company in July 2020, but his arrival spurred broader companywide AI adoption. Markus said 100,000 employees have access to AT&T’s generative AI tools and that the company consumes over 27 billion tokens per day, while also fine-tuning small language models to help manage costs.
By the first quarter of 2021, the company had also launched its proactive customer notification system, Markus said, which is enabled through its EEIM. “That solves a lot of customer tension and frustration,” he said.
The outcome
Generative AI features were added to the EEIM in the first quarter of 2022, enabling the system to identify the root causes of outages based on historical data from similar situations in AT&T’s past, Markus said.
During the first quarter of 2025, AI agents were added to the platform, Markus said. They can interact with customers to gather information about outages, take action to resolve them, and provide case details to technicians if they need to go into the field to solve problems.
Markus said AT&T has also built more than 30 AI models to help predict when configuration issues, weather, system failures, and other problems may occur, enabling the system to be more proactive.
“It allows us to really drill down into the signals and trends that help us identify either an active issue or an issue that may be coming,” Markus said.
In total, the AI-enabled EEIM has helped AT&T prevent 3.1 million unnecessary field dispatches and reduced customer downtime by more than 12 million hours over the last year, Markus said.
