Fatal Clock Defect Hits ‘Virtually Every’ IT Hardware Vendor

Advisories about a catastrophic clock signal flaw that permanently destroys IT gear have expanded beyond Cisco, and now impact products from Hewlett Packard Enterprise (HPE), Juniper Systems, Dell, Synology and numerous other manufacturers.

HPE in recent days joined a parade of vendors confirming that an outsourced faulty clock signal component in some of its hardware would begin to fail after 18 months, preventing the equipment from booting up and rendering it unrecoverable.

In addition, HPE’s announcement confirmed for the first time that the problem stems from Intel’s Atom C2000-series processors, which also go by the names Avoton and Rangeley, and are the subject of a separate advisory from Intel.

That means the problem is much broader than initially thought, since the chipset is used by a wide range of manufacturers in networking devices, storage systems and for microserver workloads.

“Many…technology vendors make products with Intel Atom C2000 processors, including Dell and Synology,” according to a report in U.K. technology publication The Register. “Other vendors using Atom C2000 chips include Asrock, Aaeon, HP, Infortrend, Lanner, NEC, Newisys, Netgate, Netgear, Quanta, Supermicro, and ZNYX Networks.”

The clock signal component functions as a sort of metronome to synchronize the operation of digital circuits.

The Register reached out to a number of manufacturers to inquire about the problem, though several declined to identify the cause of the clock signal flaw, citing nondisclosure agreements with Intel.

Dell did not immediately respond to that tech publication’s request for comment, while officials as Synology said they are working with Intel to address the problem and that the vendor was not yet seeing increased failure rates for affected products.

That was at odds with accounts from some users of Synology products that contain Atom C2000 chips, including reports of completely fried storage boxes.

That mirrors a description issued by Cisco in an advisory earlier this month.

“Once the component has failed, the system will stop functioning, will not boot, and is not recoverable,” officials at the vendor said in a statement.

The problem affects some of Cisco’s most popular products, including ASA security devices, Nexus 9000 series switches and series 4000 integrated services routers.

Needless to say the defect is creating huge headaches for IT services providers and administrators, who must identify affected hardware in their on-premise environments and arrange for repairs or replacements.

The flaw also poses a huge logistical and financial burden for the manufacturers, which are employing a variety of mitigation strategies, from extending warranties, to proactively contacting affected customers about swapping out hardware.

Cisco announced it set aside $125 million for costs related to the clock signal flaw.

Intel, too, said during a recent earnings call that “a product issue” limited profitability in Q4 and that the company has created a reserve fund for associated expenses.

In a recent article, server specialty publication Serve the Home offered this assessment about the prevalence of the faulty chips:

“Virtually every vendor uses Avoton/Rangeley in some capacity,” the article states.

“There is a good chance that if you are running a SDN switch with something like Cumulus Linux on it, you have a Rangeley chip inside,” it continued. “These things are everywhere.”

Send tips and news to [email protected].