Failure modes

Index, feed.

I have a small bunch of little network-connected projects at home. They were all fun to set up, but having software and hardware in "production" means handling failures in a way or another. One way is to tolerate flakiness, when bearable, another is to increase robustness.

Network usage bargraph

Setup: Adafruit 24-segment bi-color bargraph display, I²C on Raspberry Pi, display-side REST server a simple Python program waiting for numbers getting pushed to it by machine that has network interface to monitor. Number-pushing service is handled by supervisord.

Failure modes:

• Sometimes the display freezes. Restarting the service fixes that, for a while.

I discarded the device after perhaps four years of use, because the LEDs had become dim by then, and replacing it did not seem worth the price of more than USD10 when larger, 32x8 matrix displays are readily available for about CAD10 all-included.

Hygrometer & thermometer

Setup: ESP8266, DHT22 temperature and humidity sensor, optional I²C OLED display.

Failure modes:

• Datapoints are sometimes lost. At the moment, the device builds five-minute summaries of measurements, with attempts to retrieve the latest one every minute. This should be made more robust against extended network flakiness by retaining more than one chunk of data, and letting the client specify how far back it wants to go.

• Sometimes, devices get off the Wi-Fi network, and don't automatically reconnect, needing a reset, upon which they immediately reconnect successfully.

  I have not yet investigated the reasons, and whether the sotware could be improved to automatically recover from this condition. In particular, the device is aware that it is off Wi-Fi; I can tell because those that have a display attached mention it. It is surprising that the built-in Wi-Fi library apparently does not try reconnecting eternally; reading its source code could help understanding what happens.

"Smart" light scheduling

Setup: off-the-shelf smart plugs and smart switches, cron jobs on a local machine.

Failure modes:

• Rarely, the UDP message from the cron job doesn't make it and the light retains its previous state instead of switching.

Light switch repeater

Setup: ESP8266, relay module controlled by separate power supply, off-the-shelf smart plug.

Failure modes:

• Exceptionally, the UDP command doesn't make it from the ESP8266 to the smart plug, requiring two togglings to fix;

• Sometimes the light turns on for a split second and then off again. I can't tell for sure whether the repeater is to blame, since it doesn't emit any kind of logs.

Motion detector

Setup: ESP8266, PIR motion sensor.

Failure modes:

• The light being controlled turns on for no good reason from time to time, perhaps because of vibration or temperature changes in the nearby furnace;

• rarely, the light does not get turned off after the on period has elapsed, probably because of a lost UDP message.

ESP8266-based clock

This has worked fine for months, requiring no maintenance; the only glitch:

• During local network flakiness, the display freezes for a few seconds as UDP updates stop arriving. I only witnessed this once, and the failure is immediately obvious as the normal mode of operation updates every second.

Later:

• 2021-10: dead pixels. One display I got with blue LEDs (four 8x8 modules) had some leds that quickly stopped lighting up. Since I assumed this was a problem with that one specific 8x8 module, I replaced it with a spare one, made of red LEDs. Sadly, other blue modules started losing pixels as well.

ESP8266-based Kodi time remaining display

• Same failure modes as for the clock; plus

• sometimes responses from Kodi come slowly, resulting in minor flashing of the display (as, for example, time displayed goes backward for a fraction of a second).

www.kurokatta.org

David Haguenauer <www@kurokatta.org>
$Id: esp8266-failure-modes.malblog,v 1.8 2021/10/15 15:48:34 homer Exp $