How Spain May Have Ignored Root Cause Failure Analysis in Its Blackout

by | Articles, Energy Management

The Event: What Happened on April 28, 2025

On April 28, 2025 at 12:33 local time, the electrical systems of Spain and Portugal collapsed resulting in a blackout condition on the entire Iberian Peninsula when it was disconnected from the rest of Continental Europe.

Spain’s Ministry for the Ecological Transition and the Demographic Challenge commissioned a ‘Comite 28-A’ to analyze the event, while Red Electrica de Espana (REE), the Spanish transmission system operator, issued an incident report pursuant to its operating rules.

The European Network of Transmission System Operators for Electricity (ENTSO-E) and the North American Electric Reliability Corporation (NERC) and Federal Energy Regulatory Commission (FERC) also published technical notes and lessons learned.

Investigations, Reports, and the Role of Bias

The two front-line organizations that had the raw data and input from the locations provided reports (Spain and REE).  As noted in a presentation at the IEEE PES General Meeting on July 29, 2025, by one of the investigators, some of the conclusions are both politically and legally biased as the >3B Euros worth of losses are being sorted out.

A review of the reports suggest that the times and records were correct but there are implied differences between reports that software modeling (also confirmed at the IEEE PES meeting) was relied on to develop some of the conclusions.  There does not appear to be, nor are there any charts that I’ve seen, that indicate an actual and formal root cause analysis process was performed.

When modeling replaces investigation, bias becomes the foundation for conclusions.

The NERC/FERC report was produced based upon the Spanish and REE reports with a bias towards suggested shortfalls of alternative energy, in particular PV (photo-voltaic).

Several reports, stories, and IEEE presentations provided a variety of similar views.  At the time of this article, the ENTSO-E preliminary report has been released. Still, it clearly states that continued work is in progress, which we hope is an RCFA, as the methodology has not been provided or suggested.

Overall, the event and how a variety of biases are introduced in presentations, discussions, news reports, and professional papers can shape world-changing results, such as impacts to standards, policy, regulations, public opinion, potential elections, power availability, and even quality of life.

Of the reports, the ENTSO-E report appears to be closest to RCFA methodology, but we will not be able to identify that until the final report is provided.

For instance, the impression given based upon one engineering working group discussion was that because PV made up over 40% of the energy production that day, it was unable to manage grid inertia.  However, every report clearly identifies that due to the lighter loads, inertia was not an issue.

Instead, according to the reports, it was a cascading overvoltage condition.  The initial event appears to have been a transformer tripping offline in southern Spain, which connected a mix of PV, wind, and thermal-solar power generation, triggering the event. Incorrect settings and controls followed this.

Spain’s generation and storage mix at 03:00 on April 28, 2025. Source: REE, Generation Dashboard

Spain’s generation and storage mix at 03:00 on April 28, 2025. Source: REE, Generation Dashboard

Each of the reports that we have identified has bias towards it’s particular area of responsibility. 

Installed power generation capacity in Spain as of January 31, 2025. Source: Red Electrica

Installed power generation capacity in Spain as of January 31, 2025. Source: Red Electrica

For instance, Spain avoided discussions related to alternative energy based upon their clean energy policies, the operator groups (REE and ENTSO-E) had a bias in support of the operators with emphasis on other areas than incorrect decisions and controls, NERC/FERC had a bias against alternative energy, and those involved in standards appear to have a clear bias (stated in most meetings and presentations) against inverter based resources (alternative energy).

This author has a clear bias in favor of IBR based on experience related to grid resilience, with the intent in the engineering working groups to provide a balanced approach.

The following events involved the use of two separate AI tools (ChatGPT Agent and Microsoft Copilot), along with confirmation from the Spanish investigator during the engineering meeting, to ensure that we accurately captured the events as presented in each report.

This is in addition to thoroughly reading and taking notes ahead of time before using AI to flesh out any additional points.  For the timeline, we are referring to a 24-hour clock local time.

Technical Breakdown: Oscillations, Overvoltages, and Missteps

The operators indicate that there were small oscillations in frequency at 0.2 Hz during the morning, which appear to result in a local forced oscillation of 0.6 Hz at 12:03 local time across the Iberian Peninsula.

The operators then suggest that the oscillation may have originated at a large PV plant in Badajoz, Spain, which resulted in voltage swings of about 30 kV peak to peak across Southern Spain and Portugal.  REE mitigated the oscillation by changing the 400 kV transmission network (grid) and reducing exports of power to France.  The Iberian Peninsula has limited cross-connects with Europe, making it an effective electric power island with only a few bridges.

In trying to stop the oscillation, REE may have unknowingly primed the system for collapse.

The measures used to dampen (reduce) the 0.6 Hz oscillation increased network impedance and resulted in higher system voltages.  At 12:19 local time, a second, larger 0.2 Hz inter-area oscillation occurred.  REE’s response appears to have further limited voltage control and resulted in raised voltages to the upper operational limits of about 410-420 kV of the 435 kV limit after 12:30 local time.

At this point, 355 MW of generation tripped at a collector transformer in Granada because of an equipment fault on the low-voltage side.  This disconnection removed reactive power absorption, which is used to control voltage, and the voltage increased further, and additional transformers began to trip.

AT 12:33:16 and 12:33:17, two more events occurred, which removed over 1 GW of PV, wind, and solar-thermal generation across several locations.

Timeline of cascading voltage and generation losses on April 28, 2025. Source: REE event sequence

Timeline of cascading voltage and generation losses on April 28, 2025. Source: REE event sequence

Many of the plants appear to have disconnected at voltages below the 435 kV ride-through threshold, indicating mis-adjusted protection settings.  This caused voltages to exceed 440 kV and resulted in the power frequency dropping.

Automatic under-frequency load shedding operated as designed, but load shedding (which reduces line loading) further increased reactive power and voltages.  As frequency and voltage diverged, the Spain-France AC tie lines tripped on loss of synchronism at 12:33:21.  By 12:33:24, the Iberian system collapsed to zero voltage, and other lines shut down.

Most reporting indicates that the issue had been building for weeks leading up to the event.  There were no specific weather events, and no cyber-attack was identified.  Synchronous machines (traditional generation, including gas and nuclear) may have also tripped, and reserves are not available, with several reports emphasizing that the lack of proper setup by operators, as well as decisions, led to the issue – basically, a comedy of errors.

Weeks of overlooked setup errors and misaligned controls turned into seconds of cascading failure.

Some notes were made in several reports related to electrical systems not being set to allow absorption of reactive power that should have been set that way (Voltage control).

Within the engineering community at the PES meeting (Austin, TX, 7/26-7/31/2025), a lack of understanding of the capabilities of IBR technologies and bias towards specific solutions resulted in additional confusion.  A number of the engineers responsible for setting standards for grid regulation identified IBR as ‘black boxes.’

Lessons for the U.S. and the Path Forward

The general interest is to ensure that a similar event does not occur in other countries, such as the USA.  For the USA, the NERC/FERC report identifies that all generation units, including IBRs capable of voltage regulation, perform such control.  It is known that the USA already does this under NERC VAR-002-4 and FERC Order No. 827.

During discussions on standards and in CIGRE and IEEE meetings and presentations at the power engineering society meeting, it became apparent that the average engineers involved were unaware of IBR’s voltage control capabilities.

Of the very large international body present, very few IBR field-experienced OEMs, consultants, users, or SMEs were in attendance.  Most of the meetings attended by the author had only the author as a level of authority on IBR capability.

Other areas identified for the USA included reviewing overvoltage protection level settings and enhancing voltage control resources, such as SVC/STATCOMs (capacitance) and synchronous condensers (synchronous motors and generators turning without a load with special controls).

The conclusions reached by NERC/FERC based on the Spanish Government report were as follows:

  • Strengthen oversight and verification of regulatory compliance.
  • Improve voltage control capabilities and oscillation damping.
  • Increase interconnection with the European system.
  • Review of services framework.
  • Update system restoration procedures.
  • Analyze and update operating procedures.
  • Boost demand.
  • Increase the strength of the grid and flexibility.

The NERC plan, which may include CIGRE/IEEE input from the working groups, will prioritize developing a comprehensive framework to identify risks resulting from the transforming grid (inverter-based resources, large loads, essential reliability services, distributed energy resources, etc.).  Relevant essential reliability services include, for resilience:

  • System inertia.
  • Rate of change of frequency.
  • Frequency response measurement.
  • Reactive capability.
  • Voltage Performance. And,
  • System strength (sensitivity to voltage magnitude and phase).

What the Reports Missed: The Absence of Root Cause Failure Analysis

What is particularly disturbing about events such as these is that most of the work is usually a time narrative and a statement of events without any formal root cause analysis being described.

The approaches described in MotorDoc Vlog 80 Electrical Machinery Forensics An Intro – MotorDocAI, which involve a basic 5-why and more advanced PROACT methodology, and comments associated with who should be involved, were not followed.  Each group brought in their own biases and biased specialists, and while many conclusions were similar, the reports focused on protecting different groups.

For instance, while most organizations, working groups, and uninvolved academic and engineering groups have focused on the impact of IBR in these reports, the front-line data-driven reports identify the shortcomings of synchronous (traditional) generation.

It is this author’s opinion that we can only draw specific conclusions from the report. Still, without a complete RCFA, most of the work is opinion, which is dangerous to use to make major policy and engineering standards decisions.

Author

  • Howard Penrose

    Howard W. Penrose, Ph.D., CMRP, CEM, CMVP, is president of MotorDoc® LLC, a Veteran-Owned Small Business. He chairs standards at American Clean Power (2022-25), previously led SMRP (2018), and has been active with IEEE since 1993. He represents the USA for CIGRE machine standards (2024-28) and serves on NEMA rail electrification standards (2024+). A former Senior Research Engineer at the University of Chicago, he’s a 5-time UAW-GM Quality Award winner. His work spans GM and John Deere hybrids, Navy machine repair, and high-temperature motors. He holds certifications in reliability, energy, M&V, and data science from Kennedy-Western, Stanford, Michigan, AWS, and IBM.

    View all posts
SHARE

You May Also Like