Anomalies in the operation of the solid-state mass memory system on board Mars Express have caused science observations to be temporarily halted. A technical work-around is being investigated that will enable the resumption of a number of observations and should evolve into a long-term solution.

In mid-August, Mars Express autonomously entered safe mode, an operational mode designed to safeguard both the spacecraft itself and its instrument payload in the event of faults or errors.

The cause of entering the safe mode was a complex combination of events relating to reading from and writing to memory modules in the Solid-State Mass Memory (SSMM) system.

This is used to store data acquired by the instruments and housekeeping data from the spacecraft's subsystems, prior to its transmission to Earth, and is also used to store commands for the spacecraft that have been received from the ground stations, while awaiting execution.

Switch to spare memory controller
As the previous safe mode was three years ago and the current event looked like a 'normal' transition to safe mode, the flight control team executed the standard recovery procedure and restarted observations.

A few days later, another fairly similar set of SSMM problems occurred. The decision was then taken to switch over to the cold-redundant spare, or 'B-side', SSMM controller, as this was virtually the only subsystem common to the two events.

The decision to act was triggered by both the need to achieve stable science performance and the need to reduce the consumption of fuel caused by transitions to safe mode.

An important element in trying to reduce the number of safe mode transitions, besides the science observing time being lost, is the amount of fuel used when entering safe mode, which is approximately equal to the amount of fuel required for six months of nominal operations.

The reason for this fuel consumption when entering safe mode is the 'Sun acquisition' process, which requires a significant amount of spacecraft manoeuvring.

Full science observations were resumed on 15 September, using the B-side SSMM controller. Eight days later, the spacecraft again entered safe mode.

The cause was a checksum error during communication between two subsystems within the SSMM. As this event exhibited no similarities with the two events in August, it was decided to treat this as an isolated incident and return to nominal operations; this took place on 29 September.

On 11 October, a similar problem occurred, but no safe mode transition was triggered. As all operations continued normally, it was decided to take no action, but to find a slot to execute a warm reset of the SSMM at the earliest opportunity.

Observations halted
On Sunday, 16 October, Mars Express again entered safe mode. The cause was a sequence of events very similar to the one that triggered the safe mode transition on 23 September, following an internal communication error.

Resuming use of the A-side SSMM controller is not considered a viable option; it would most probably lead to further safe mode transitions caused by memory module read/write errors and the additional fuel consumption caused by these events cannot be sustained. Faced with these circumstances, the Mission Manager decided to halt science observations.

Possible work-around
The SSMM is a critical subsystem on board Mars Express, central to all platform and instrument operations. The ESOC flight control team is preparing a work-around to address the current problems and allow at least partial resumption of science observations.

During normal operations, telecommands for the platform and instrument payload that have been received from the ground stations are stored in the Long Mission TimeLine (L-MTL), a special file in the SSMM that is used to fill the platform's command cache.

In order to cope with anomalous operation scenarios, a hardware-based timeline store - the Short Mission TimeLine (S-MTL) - is available outside the SSMM subsystem.

The work-around now being investigated uses the S-MTL to store telecommands before they are executed.

Good progress is being made with the implementation of this alternative approach to commanding the Mars Express platform and its payload. A full test on the spacecraft will be executed in the near future.

ESA's mission control team and other experts are working closely with the designers and manufacturers of the SSMM subsystem to investigate the cause of the read/write anomalies and to attempt to find a solution that will enable a return to normal operations.