02 · Open source · DevOps
InfraScope: open-source infrastructure monitoring and incident detection
InfraScope checks HTTP(S) endpoints on a schedule, stores every real measurement in PostgreSQL, and opens and resolves incidents automatically. It ships as a Docker Compose stack with a FastAPI backend, a React dashboard and CI that tests the whole stack end to end.
The problem
Monitoring is usually the first thing that's missing in a small IT environment. People find out a service is down when users call. I wanted a monitor I fully understand, one that records real response times, raises incidents without false alarms, and can feed a ticketing system or internal portal.
I also built it to show, in public, how I work: tested, documented, reproducible with one command, and safe by default.
What I'm responsible for
- Backend: FastAPI REST API, background monitor engine, incident rules, API-key authentication.
- Database: PostgreSQL 16 schema and Alembic migrations.
- Frontend: React + TypeScript dashboard with charts, light and dark mode, phone-width layout.
- Delivery: Docker Compose stack, GitHub Actions CI, Dependabot, PowerShell smoke test.
- Documentation: README, architecture, API and integration docs, daily progress logs.
Architecture
Everything runs with one command, `docker compose up`. The database and test target sit on an internal network with no internet or LAN access.
- Edge network
- web: nginx + React build
- api: FastAPI + monitor thread
- Bound to 127.0.0.1 by default
- Monitor engine
- Scheduler loop (1 s tick)
- Thread pool (8 workers)
- httpx checks: no redirects, 5 s timeout
- Backend network (internal)
- PostgreSQL 16 (pgdata volume)
- testtarget: /healthy /unhealthy /slow
- Integrations
- Signed incident webhooks
- Transactional outbox with retries
- VIEWER API keys for portals
Engineering decisions
Incidents need two failures, not one
A target is marked DOWN after 2 consecutive failed checks and resolved after 2 consecutive successes. One dropped packet doesn't page anyone. A database index guarantees there's never more than one open incident per target.
SSRF protection by default
A monitor that fetches URLs is an easy way to probe internal networks. Only allowlisted hostnames are checked: no raw IPs, credentials or redirects. IP monitoring is opt-in per network range, and loopback, link-local, metadata and reserved addresses are always refused.
API keys stored as hashes, shown once
Keys are 32 random bytes; only their SHA-256 hash is stored, and the secret is printed once. VIEWER and ADMIN roles are enforced on every route, and a test checks that every non-public route requires a key.
Webhooks that survive failures
Incident notifications are written to a transactional outbox, signed with HMAC-SHA256 with a 5-minute replay window, and retried with exponential backoff. A receiver being down doesn't lose an incident.
Technical challenges
Measuring real response times
The checker streams the request and times only until response headers arrive, without downloading bodies. Timeouts and connection errors are recorded as results rather than dropped.
Background work without blocking the API
The monitor runs in its own thread, started and stopped by the FastAPI lifespan, so checks never block API requests. Allowlist changes take effect on the next check.
Testing against a real stack
CI runs lint and 141 pytest tests against a real PostgreSQL service, typechecks and builds the frontend, then brings up the full Docker Compose stack for an end-to-end check.
Current status
Built
- Target management REST API and background HTTP monitor
- Incident detection, history and summary API
- API-key auth with VIEWER / ADMIN roles
- React dashboard and Docker Compose stack
- Signed webhooks with outbox and retries
- CI: lint, tests, frontend build, Compose end-to-end
Scope today
- HTTP(S) checks only
- Single API instance, designed to run on one host
Outcomes so far
- pytest tests, incl. integration tests
- 141
- CI stages: backend, frontend, Compose end-to-end
- 3
- failures to open, successes to resolve an incident
- 2 / 2
- license
- MIT
- Public, documented repository with a passing CI pipeline.
- A recorded demo: a target fails, an incident opens after two failed checks and resolves after two successes, all on the real stack.
- Git history audited for secrets before the repository was made public.
