JSI · Physical Security Engineering · Project EGA
The gateway was not running for eight of the last nine days. This is what happened, how we know, what has already been changed so it cannot happen the same way again, and what is still outstanding.
The gateway stopped on 1 September at 06:12 and did not run again until 9 September at 06:49.
For those eight days no detection boxes were drawn and no crossing alerts were sent. The cameras were fine, the recorder was fine, the machine was fine. The gateway itself was dead, and nothing was watching to notice.
That last part — that nothing was watching — is the real defect, and it is the one we have spent today fixing.
Each time the gateway died, it died the same way: two seconds after starting, before it had published anything.
web interface on http://172.16.11.49:8080 The requested address is not valid in its context
The gateway's web interface binds to a fixed network address read from its configuration file. This machine takes its address automatically, and it has moved four times in two weeks:
| Address | What ended it | Gateway |
|---|---|---|
| 172.16.11.238 | renumbered | died |
| 172.16.11.49 | renumbered | died |
| 172.16.1.3 | a scheduled task on the machine rewrote it | died |
| 172.16.11.2 | current | running |
Every move left the configuration pointing at an address the machine no longer held. The gateway started its cameras, tried to open its interface, and exited.
The streaming server keeps its own log, and it records whether anything published video on each day. It is unambiguous.
| Day | Log lines | Anything publishing? |
|---|---|---|
| 1 September | 148,570 | none |
| 2 September | 184,572 | none |
| 3 September | 164,964 | none |
| 4 September | 202,338 | none |
| 5 September | 200,890 | none |
| 6 September | 202,320 | none |
| 7 September | 201,928 | none |
| 8 September | 194,846 | none |
| 9 September | 53,063 | yes, from 06:49 |
Those two hundred thousand daily lines are the recorder asking for the nine camera streams and being told they do not exist — about two requests a second, for eight days. They grew that single log file from 5 MB to 131 MB.
The recorder never gave up, which is to its credit. It simply had nothing to record.
All of the following is done and running as of this morning. Each one closes a way the system could fail without telling anyone.
| Change | What it prevents | Status |
|---|---|---|
| The interface now binds to the machine's own loopback | The address can never take it down again, because loopback does not change. This was the direct cause of all three outages. | done |
| It starts automatically with the machine | A restart no longer leaves it stopped. A scheduled task launches it at boot and relaunches it if it exits. This morning that task fired correctly — what failed was the address, not the automation. | done |
| A supervisor watches every camera | Any camera that goes ninety seconds without delivering a picture is restarted on its own, and says so. One camera sat unable to connect for forty minutes this morning; with this it would have returned in ninety seconds. | done |
| A camera that cannot be opened now says so | That path was completely silent and abandoned a video handle every five seconds. It is very likely why the camera above never recovered by itself. | done |
| A decoding error no longer kills a camera | It is caught, the handle is released, and the camera reconnects. This is what stopped six cameras at once during last week's memory incident, while the interface still showed nine healthy ones. | done |
| The log is written immediately | It was written in blocks, so the minutes before any crash were lost — three days of record disappeared that way. What the log says is now what is happening. | done |
| An 8 GB paging file | Closed the memory exhaustion that stopped six cameras last week. Spare capacity went from 4.2 GB to 11 GB. | done |
Every failure this project has hit presented the same way: the interface showing nine healthy cameras while something was broken. Threads died unnoticed, handles leaked, a camera retried for forty minutes without one line of log. In most cases the information needed to detect the fault was already being recorded — nothing was reading it. That is what changed today.
| Finding | What it means | Next |
|---|---|---|
| Two cameras never reach the monitoring centre | CAM-41 and CAM-43 were added to the gateway but never added to the recorder. The recorder is retrying seven camera paths, not nine. They are being processed and going nowhere. | add them |
| A camera that does not exist is being retried | A source named Cam1 in the recorder points at a stream the gateway
does not produce. About 29,000 failed requests. |
delete it |
| The machine crashed once, on 31 August | A memory fault at kernel level — nothing an application can cause. It has not recurred in the twelve days of records we hold, but the memory has never been tested. | run the test |
| Recording is switched off in the recorder | Turned off during a performance test. Alarms still reach the monitoring centre — that path does not depend on recording — but no archive is kept on the machine. Each camera holds about five days of full-resolution video on its own card, and we have verified we can retrieve it. | decide |
The recording question is the one that is not technical. Right now the archive policy is a side effect of a performance test rather than a decision, and it should be the other way round.
All nine, at 11 pictures per second, since 06:49 this morning.
Across roughly 86,000 pictures processed per camera.
13.5 of 23.75 GB, flat. No growth over the run.
A camera quiet for longer than this is restarted automatically.
The next useful measurement is a full day left untouched. That was not worth doing before today, because the system could not recover from anything on its own. Now it can.