RAS specifications

CPUs should be able to detect and optionally correct errors that occur in the CPU. Today, many of these events are highlighted through specific error architectures. On Intel x86 CPUs this is the Machine Check Architecture ([glos-MCA]). Examples of these events include, but are far from limited to:

All CPUs should support the current x86 MCA framework and emit events to the error banks as configured. The current MCA framework and events is documented in the Intel 64 and Intel IA-32 Architectures Software Developer’s Manual [sdm].

On some systems there is the ability for systems firmware to be notified by the EMCAv2 architecture through an SMM trap. Firmware is allowed to configure and manage this; however, in all such cases it must not consume the event. The event must be delivered to the operating system.

The systems firmware should not make the determination that a given MCA event can or cannot be handled by the operating system. For example, if an uncorrectable memory error is delivered to the operating system, then the systems firmware must not intervene and assert that this cannot be handled. It is up to the operating system to determine what page that memory error occurred on and be able to take the appropriate action. For example, the operating system may decide, based on policy, that if the memory error occurred in a user process that it should kill that process. Otherwise, if it occurred in a page owned by the kernel, the operating system may opt to take down the system.

For any level of error that occurred in this way, the service processor (SP) must log such information in the system event log with enough details to localize the error to a specific device and cause. It should be possible for the operating system to sync up with the SP and determine events that were generated or occurred while it was not running, because, for example, the system was being booted by systems firmware.

If an error is considered fatal to the platform in such a way that it cannot be handled, then the error must be delivered to the SP. The SP should determine whether or not it is possible to deliver an NMI to the operating system to get a crash dump. If it does, then it must provide some means of the operating system being able to get access to this information. The SP or some other part of the system, such as the management engine restarting the system should always be the last resort.

There exist classes of CPU errors which cannot be delivered by the MCA architecture because the CPU is in such a bad state that it is not possible for it to operate. While it may be possible for such an error to be delivered to another socket, given that the system is already in an undefined state, it is not recommended that this occur. Instead, the system should deliver it to the SP which should take care of resetting the system and ensure that the information is recorded and in an easy to obtain way from the operating system after the fact.

When such an error occurs it is tantamount that the SP gather as much information about the system as it can and store it as a form of crash dump.

Today, CPUs are identified in two different ways. They’re identified by computers through the values returned in CPUID which on current generation systems are based on the xapic and x2apic identifier. When working in the context of the operating system, humans use these logical IDs as well.

Technicians on the other hand look at CPUs in terms of FRUs, which is a socket. Technicians utilize the silk screen labeling. It is expected that the system board provides silkscreens for the CPUs. If the CPUs are being described in any kind of out of band management ala Redfish or IPMI, it is expected that it will use identical labels.

Both of these means of identifying a CPU are important; however, we need a programmatic way to map between these identifiers from the running operating system. This is important, because CPU MCA events will come in and refer to a specific logical (CPUID based ID) for a CPU. If a CPU needs to be replaced in the field, the technician needs to understand what the corresponding socket is.

Today there does not exist a standardized way to do this. To facilitate this, we need to introduce a new means of accessing this information. Basic CPU information is already available in the SMBIOS specification ^[1]. We propose to extend this information to provide a means for mapping between the silkscreen socket information on the system board and the information that is provided by the processor via CPUID. For more information, see Proposal 1: CPU to Socket Silkscreen Mapping.

CPUs have firmware that comes in the form of microcode. It is expected that all CPUs that are in use will support some form of online microcode update. While it is nice for BIOS and other firmware updates to start the CPUs with a given microcode, that cannot be relied upon and existing OS based microcode updating tools must be able to function long after the operating system and processor have been started.

There are many different sensors that exist for the CPU. Some of them, such as temperature sensors, are exposed as PCI devices that exist on a CPU socket, core, or logical processor basis. Others are available as MSRs. Sensors which exist through IPMI must be correctly attributed to the CPUs. They should not be attributed to the system board or other peripherals.

One of the primary concerns around memory devices is error detection. These memory devices are capable of generating correctable and uncorrectable errors. Today the errors on all such DIMMs are delivered through the memory controller to the processor as an MCA event.

All such generated events must be delivered to the processor such that the operating system can record them and take action on them. This should be the case for both correctable and uncorrectable errors on DIMMs. In no circumstances should a correctable or uncorrectable error be transformed into an NMI or another level of error, absent some subsequent failure in the system.

Systems firmware should record these events in the system event log in the SP; however, they should not interpose on them or modify them in such a way that the operating system cannot notice them. If the error rate reaches a certain threshold, then systems firmware may perform hysteresis on writing to its system event log; however, if this is performed, the amount recorded must be noted. When reading these events, both the physical address as well as the DIMM and bank should be recorded.

The aforementioned hysteresis should only apply to the act of systems firmware writing in its log. Systems firmware must not interpose or apply this hysteresis for events delivered to the operating system.

Some platforms will opt to disable a DIMM channel for a particular reason during the boot process. This may be because an error occurred during the discovery phase, an error occurred during the training, or errors occurred during memory test. In all such cases, the platform must log a message to the system event log to notify the operator that such an event occurred with detailed information about both the physical address as well as the channel, DIMM, and bank information.

Secondly, the running operating system must be able to discover and distinguish the case where a DIMM channel was disabled by the platform from a case where the DIMM is not plugged in. While it may be possible to rely on the memory controller for the basic state information, that is insufficient. The operating system must be able to determine why the channel was disabled.

Similarly, if a subset of a DIMM, for example, a given rank, was disabled during training or some other part of the process, that information should be recorded and observable.

For more details on the challenges with this, please see Issue 1: DIMM Disablement.

On the whole, it is preferable that the system be able to boot up with a reduced DIMM count than the system refuse to start until a faulty DIMM has been removed or replaced.

The next critical problem that we face with DIMMs is being able to get information about the DIMM. This ranges from the serial number to its capacity.

The system should provide information about DIMMs in SMBIOS by creating type 17 entries in SMBIOS. This captures a reasonable amount of data about the DIMM. In addition, JEDEC has standardized various SPD (Serial Presence Detect) data for DIMMs. This is generally the source that much of the data from SMBIOS comes from.

There should be one SMBIOS type 17 record for every DIMM slot on the system. Note that the number of DIMM slots on the system may be less than the upper bound on implemented channels on the system. There must not be an SMBIOS type 17 entry for such a DIMM slot when there is no corresponding physical slot.

SMBIOS provides a Location Tag which is useful for being able to determine where a given DIMM physically is. However, this is not sufficient. Critically we are missing a programmatic way of mapping between an SMBIOS type 17 entry or SPD data for a DIMM and the corresponding information from the memory controller.

SPD data today provides no form of location information — which makes sense as it is coming from the EEPROM of the DIMM. Dynamic SPD data (such as sensors) can be correlated to an SMBIOS type 17 entry through a DIMM’s serial number.

This lack of mapping between the memory controller level data and the SMBIOS/SPD level data is problematic. For a tentative solution to this, please see Proposal 2: DIMM Identification.

It is not expected that any DIMMs today have firmware that is required to be upgraded. However, for upcoming upcoming non-volatile DIMMs, a means of identifying and performing firmware upgrade on the devices from the running operating system is required. It is more important that in-band firmware information can be communicated, with the eventual expectation that full in-band upload and download of firmware can be performed.

DIMMs expose sensors through their SPD data in the temperature page. In addition, the memory controller may have additional sensors that exist in the system. The SPD sensors should be accessible by the system over SMBus. If sensors are being added at an IPMI layer, then the entity IDs and other information must make it clear which DIMM it corresponds to and this should be able to be correlated to SMBIOS, memory controller, and SPD data. Further, in IPMI sensors for DIMMs must exist under the DIMM records themselves.

All connectors and slots on the system board must be labelled with a silkscreen. The labelling should be clear and it should be obvious as to which component it is referring to.

In particular, when referring to expansion slots such as PCI express, it should be designed in such a way as a full card does not obstruct the silkscreen, if possible.

It should be possible to determine which silkscreen refers to which components programatically from the running system. This may occur through SMBIOS, IPMI, ACPI, or some other systems management interface.

An important thing that the system needs to be able to determine is when slots or various headers are in use and if so, what they are. This covers the following components:

In all cases, systems software should be able to determine which slots are in use and which slots are not. CPUs, DIMMs, and PCIe devices and their slots are all covered in their own respective sections

For the remaining headers, while there is information that the system can transmit methods things such as the SMBIOS type 8 port connector information, it can be hard to make use of those records to understand what has and hasn’t been wired up. For example, a USB header may be in use to connect to a set of auxiliary USB ports that are not a part of the system board. For example, front USB ports on the chassis. An open question is how do we map those potentially in-use headers. See [issue-port-header] for more information.

Another challenge that we have is the ability to map several of the built-in ports to labels that exist. For example, systems today have a combination of USB, Ethernet, VGA, Serial, and more ports sitting on them. These ports are built-in to the system board and exposed, generally, through the rear of the chassis or on the system board inside of the chassis. Importantly, unlike other ports, these are not built into the chassis and connected to a header. Instead, they are part of the system board FRU.

These ports provide a variant of Issue 3: Header/Port mapping where by we need to have a well defined way to understand which ports correspond to which logical devices. While SMBIOS provides type 41 Onboard Devices Extended Information ^[2], it does not provide a good way for us to think of those items which are ports. In addition, for those items which do not employ a PCI-like bus, device, and function, there is no way to map those items to operating systems visible devices.

An important goal for this is to be able to determine whether a given USB port is internal to the chassis or external. This will be used by operators to create different policies around trusting devices plugged into different ports.

The system board is often home to the system BIOS or UEFI implementation. Traditionally, the only way to manage these interfaces and observe them has been through being able to enter the BIOS while the system is booting. This is a major serviceability impediment.

Instead the following information needs to be available through at least out of band management and preferably through in band management as well:

All of these equally apply to UEFI based systems as well as BIOS based systems.

The system board should have the ability to enumerate all of the sensors that are unique to it. These should generally be exposed through IPMI. However, if they are not exposed via IPMI, then the platform should provide some means of enumerating these sensors and how to find them in some dynamic fashion, whether that’s through ACPI, discovery through SMBus, or some other system specific method. These must be publicly documented.

It is important that such sensors be enumerated correctly. Sensors that are not a part of the system board itself or refer to it, should not be enumerated for it. For example, a DIMM temperature sensor should not be enumerated under the system board, but logically must refer to its corresponding DIMM such that we can tell which DIMM slot this refers to.

There are two different SMBIOS information types that are important to be able to program on the running system. These are the System Information (type 1) ^[3] and Base Board (type 2) ^[4]. It should be possible to overwrite the various string fields. The serial number information should be accurate.

It is possible that the system board has other devices that have firmware on them. For example, many system boards may deploy CPLDs or other devices to help drive and manage them. As with other firmware, the revisions of all such components should be made consumable to the operating system and if possible the firmware should be modifiable by the operating system. It is acceptable that this work goes through another component on the system to manage this. For example, if it has to be communicated to the Intel ME, service processor, or some other controller such as the innovation engine.

While we’ve made a focus about silkscreens from the perspective of a system board and mapping CPUs and DIMMs, the chassis has its own labeling that we need to consider.

In particular the following components should be labelled:

Importantly, this labeling shares many aspects of the same set of problems as Issue 3: Header/Port mapping and this labeling will likely end up being a part of that.

The interior lid of the chassis should have diagrams that explain how to perform common service operations. This is useful as a way to remind data center technicians of the steps that need to be taken when operating on a system. However, this should not be construed as a replacement for a full service manual for the system.

A technician should be able to service a majority of the chassis without requiring tools. This implies the following:

The technician should be able to complete all work with either a #1 or #2 Phillip’s head screwdriver. If another type of screw is required, such as a small hex wrench, then the chassis must have a predefined place such that a small tool can be stored and provided with the chassis.

A system should have the ability to have an identification LED that is controlled through IPMI as the chassis indicator. At a minimum the identifier should have two modes:

Ideally the identification LED should support additional modes and be able to operate in the following fashion:

If the chassis has any firmware that belongs to its FRU then that must be reported through to the platform in some way. It should be clear that said firmware belongs to the chassis FRU.

One challenge with the PCH is that it houses two different additional processors which each house their own firmware: the Management Engine (ME) and the Innovation Engine (IE).

The system must provide methods such that systems software can determine the firmware revisions of these devices. In addition, while there are security and complexity concerns for in-band firmware upgrades of these components, some form of firmware upgrade must be provided. It is required that such an upgrade be delivered while the system is in service and deferred until the next system reset.

All PCIe devices must be identifiable as existing in a given location in the system. For devices that are part of the CPU and PCH this is optional, though it is recommended that it is made clear in some form that this is where they reside.

For devices which do not support hotplug (usually found in traditional expansion slots), this information should be obtainable through a SMBIOS type 9, System Slot ^[5], entry. Specific PCI devices should be mapped to the location string through the bus, device, and function of the device.

For U.2, NGSFF, and Intel ruler form factor PCIe devices or other form factors that are being used for hot-pluggable NVMe devices, then the platform must provide a way to map this information to the bay on the chassis that these are accessible on. It is recommended that each bay have an SMBIOS entry so that the platform can enumerate how many hot pluggable PCIe bays exist. For more information on such requirements, please see the Backplanes section.

PCIe devices may contain their own firmware. If so, these devices should conform to the notes in the general discussion on Firmware. A means of native firmware upgrade is required.

Devices that are not standard expansion cards should support hotplug and surprise removal. This generally means all U.2, NGSFF, and other more recent NVMe based devices. However, the rules that follow apply to any device. Here, when we say hotplug, we refer to the idea of a surprise removal. That is, a device which does not follow the requirements for orderly removal as discussed in [pci-hp].

When such a device is placed in the proper slot, the system must report this by setting the 'Hot-plug surprise' and the 'Hot-Plug Capable' bit of the Slot Capabilities Register ^[8] to one.

PCIe hotplug should follow the normal specification and allow an interrupt to be posted to the operating system when such events occur. Such a notification must not be hidden from the operating system by the platform firmware.

Devices should ensure that any acknowledged activities are still completed, even in the face of surprise removal. For example, a NVMe device that has acknowledged a write and then been removed from the system, must still ensure that it is flushed to stable storage.

Devices should provide access to sensors to gauge device temperature. If active cooling is employed, the device must provide a way to understand the health of said cooling devices. Further, if there are failures or any thresholds triggered, those must be presented in some form to the device driver as discussed in the following sections.

This is not limited to just temperature and cooling. This should also cover electrical health (such as voltage, current, wattage, etc.) and other internal aspects of the device.

While the PCIe AER capability is optional, it is required for all discrete expansion cards. Systems software will configure all devices to generate AERs.

Where possible, devices should support a per-function reset capability. If a device only supports a single function and does not support virtual functions, then this capability is not required and power cycling should be able to occur at a device level.

The backplane should support multiple LEDs on a per-bay basis which include:

The service LED should be able to indicate several different states:

Importantly all LEDs must function regardless of physical presence of the device in a backplane slot.

The current LED state should be retrievable from the system. LED settings do not have to persist across power cycles. However, non-activity LED settings should not change because of device insertion or removal.

Each bay and slot must have independent power control. This power control should be available through the basic protocol methods wherever possible and also available through IPMI or similar management.

Identification of devices can be tricky due to the use of both passive and active backplane technologies which indicate how much is known or not by the platform. Regardless, a system should be devised such that it is possible to determine the following information about a given bay:

The operating system must be able to query this information. There are several different schemes and ways that this is done today:

SES provides a programmatic way to describe the bays in a given backplane or chassis and gives basic information about them such as the SAS WWN.

One unfortunate complication with SES is that it is usually only placed on systems when a SAS expander is also used. While sometimes a SAS expander is a necessary evil, it can also often get in the way.

Another way that this is done is by knowing which phy lanes are wired into which slots on the backplane. While this is an option, it is not the preferred method due to the fact that a wiring mistake between the HBA and the backplane or having the HBA placed in a different PCIe expansion slot can end up causing issues with this enumeration.

Whether the backplane is active or passive, there will likely be firmware involved. Even passive backplanes may have a CPLD to help drive the LEDs on the system.

As with other firmware modules, whether upgradeable or not, the system should expose what the revision of the firmware is and the specific FRU that it belongs to.

Regardless of medium, the drives should report standard SATA and SAS errors in response to commands as per the corresponding command sets.

When internal events and errors occur, they should be logged in a device-specific manner. These event and error logs should be exposed through either a standard log page or through a vendor-specific interface.

When vendor specific interfaces are used, it is required that one be able to write open source tools to collect said log information, even if the tooling to interpret it is not open source (though it is recommended that vendors pursue that path whenever possible).

All transceivers should have the manufacturing information filled in. This includes the vendor and serial number information.

If the transceiver possesses a UFM of some form, then it is required that the UFM information be exposed in some way by the transceiver that the operating system can obtain. In addition, the operating system should be able to upload and download new firmware to devices.

Modern power supplies are increasing in complexity. If a power supply has firmware, regardless of its upgradability, then that information must be exposed through either PMBus or an IPMI platform record.

A plan for supporting the firmware upgrade that is executable from a native environment is required.

Some power supplies have LEDs that are present on the units. When such LEDs exist, the ability to toggle them and control the LEDs should be provided over some interface to the operating system. The LEDs should consist of the following different modes:

Ideally these should be exposed over IPMI or a similar systems management interface that can be used by the operating system in conjunction with the systems firmware such that the operating system can override or change the LEDs with firmware’s knowledge.

Unlike other components of the chassis, a power supply exists as part of a broader fabric. A power supply may be plugged into a wall outlet or extension cord, but it is more commonly plugged into a power distribution unit (PDU) in a data center. A server should be able to determine which PDU a given power supply is plugged into and vice-versa. For more information on this, see Issue 2: PSU/PDU Discovery.

Every fan in the system should have a corresponding tachometer sensor provided by IPMI. Conventionally, these sensors should have entity IDs that try to relate it to the sets of fans that are expected to exist.

To break this into the two different phases we propose the following high level principles:

To make this more concrete, SMBIOS would be used to answer the question of how many fan slots exist in the chassis and label them. As well as indicate whether the fan slots are hot-pluggable or not. SMBIOS should not populate any of the information about RPM or other aspects of the system. There may be fewer SMBIOS slots than headers on the system board.

Instead, IPMI should be populated with sensor records that correspond to the different physical fan headers. In addition, the IPMI records should provide information to map the fan sensors back to a corresponding labeled, SMBIOS entry.

Through IPMI senors, it should be possible to distinguish for each fan, whether or not the fan is plugged in and whether or not it is spinning in any way.

Finally, to cover cases like option 3, IPMI should make it clear what entity the fan belongs to. For example, a fan on a power supply or a fan that is part of an active CPU heat sink should make it clear that they map directly to those devices.

The service processor must support the IPMI 2.0 specification and should support a revision of the Redfish specification. The SP should support running IPMI services through the KCS bus. If that is not provided, then another form of in-band management must be documented by the vendor and implemented in the system.

If there are vendor-specific commands that are required for correct operation, those must be properly documented.

The service processor must support being attached to the network through a dedicated Ethernet port. The device should support either static or dynamic configuration through something such as DHCP. If DHCP is used, the SP should support announcing itself through a DHCP client identifier in a way that makes it clear who the manufacturer of the service processor is and the class of system it is a part of.

Historically many dedicated and shared ports have had issues with negotiating speeds while the chassis was powered off or in the middle of a power cycle. The dedicated management port must always support negotiating full-duplex gigabit support through auto-negotiation as per IEEE 802.3 [ieee] regardless of the chassis’s power status.

As per the general [firmware] section, the service processor should support some way of performing firmware upgrade while the main system is operable. This should preferably be driven through native code mechanisms.

There are conditions where the service processor may need to be restarted due to conditions such as memory leaks, firmware upgrades, or as part of other troubleshooting steps. Such a power control should be available to the operating system.

One of the particular challenges with firmware upgrades is the fact that interruptions in the upgrade can cause problems with devices. With that in mind, what we’d like to see is that all devices adopt something similar in spirit to a hybrid of the NVMe and SAS specifications.

Particularly, the model that we’d like to see is that for any given piece of firmware on the device, there exist at least two distinct slots which can hold independent copies or versions of a given piece of firmware. Finally, the device should support the ability to record a firmware revision to take effect on the next power on or reset.

These two features combined make it very easy for systems software (or any other service) to upgrade the firmware without interfering with any currently running service and account for failures that occur due to bugs with the systems software or other incidents such as a power loss. The ability to download and then set firmware to be updated at the next power cycle, provides a much more flexible and powerful system for administrators. This allows firmware upgrades to be rolled out at the same time; however, be applied to the running systems based on operator-defined timing. Further, it means that the heavy lift and time involved in validating the firmware or performing other device-specific work has already been done before the system cycles.

While it is recognized that this process will ultimately be specific to individual devices. The more that this different logic can be made similar to one another, the better. If at a minimum vendors can have the same logical method of flashing the firmware, even if it involves device specific methods, then that will be a net improvement on the serviceability of the system.

When a new firmware image is available, a detailed change log of what has changed between versions is required. If a given image contains sub-components, for example, a BIOS update that embeds a CPU microcode update, then the details of what has changed in those versions is also required.

Traditionally, firmware has been delivered as closed source binary blobs. Where possible, the source code for said firmware should be made available to allow operators to try and better understand how the firmware is operating. This will greatly aide in the debugging process between the operator and the vendor. It is not expected that an environment for building this will necessarily be provided nor is it expected that the operators will want to create their own firmware revisions.

Where possible, firmware should provide means for understanding what is happening in production. It is recommended that some set of statistics or a circular buffer of data be possible to retrieve by the operating system or service processor.

If possible, the firmware should keep detailed statistics about the device to help us understand and explore any notions of why a failure has occurred, various internal error rates, or other aspects that describe the health of the device that might impact it reliability, availability, and should be used to help indicate impending service needs.

While interpreting said data may need vendor-specific, proprietary tools, the ability to capture the data must be able to be implemented in a separate and open-source fashion that can be incorporated as part of the broader platform. For example, for a SAS based hard drive, the data should be consumable via some kind of standard or vendor-specific SCSI command. When debugging production issues, all components are presumed guilty until proven innocent. This includes firmware. The goal of this debugging information should be to exonerate (or indict) the device.