# Hosts and access

## The droplet

Everything runs here. DigitalOcean, hostname `kookwatch`.

```
host   159.223.88.168
user   root
key    ~/kookwatch/id_kookwatch     (on Ben's Mac, mode 600)
ssh -i ~/kookwatch/id_kookwatch root@159.223.88.168
```

Bots live in `/opt/<name>/`. Units are in `/etc/systemd/system/`, all
`enabled` so they survive reboot.

```
systemctl status  <unit>
systemctl restart <unit>
journalctl -u <unit> -n 50 --no-pager
```

Unit names do not always match directory names:

| Bot | Unit | Directory |
|---|---|---|
| crashbot | `crashbot.service` | `/opt/crash-bot` |
| tonalert | `tonalert.service` | `/opt/ton-alert` |
| gramwatch | `gramwatch.service` | `/opt/gramwatch` |
| xalert | `xalert.service` | `/opt/x-alert` |
| rhtracker | `rhtracker.service` | `/opt/rh-tracker` |
| ansemtracker | `ansemtracker.service` | `/opt/ansem-tracker` |
| basewatch | `basewatch.service` | `/opt/basewatch` |
| bwbot | `bwbot.service` | `/opt/basewatch` (`bwbot.py`) |
| kookwatch | `kookwatch.service` | `/opt/kookwatch` |
| watchdog | **`watchdog-bots.service`** | `/opt/watchdog` |

Each bot keeps its own `<name>.log` and `state.json` in its directory. The
logs are what `botctl log` reads.

## The Mac

Runs nothing. Its role is to hold `botctl`, the SSH key, and the parked
plists.

```
~/botctl/            botctl, botctl.py, registry.json
~/kookwatch/         id_kookwatch (SSH key only, no code)
~/bots-mac-disabled/ four parked launchd plists - leave them parked
```

Five stale bot directories also sit in the Mac home (`~/crash-bot`,
`~/gramwatch`, `~/ton-alert`, `~/x-alert`, `~/rh-tracker`). They are dead
twins of the droplet copies. Reading them to understand code is fine.
Editing them accomplishes nothing. Starting them causes duplicate alerts.

The Mac's local network cannot reach `t.me` or `telegram.org` reliably - the
old `~/gramwatch` log is full of SSL handshake timeouts. `api.telegram.org`
works, which is why bots could send but not scrape. Another reason the fleet
belongs on the droplet.

## The watchdog

`watchdog-bots.service` checks all nine other units every 300s, restarts
genuine failures, and posts a daily roster heartbeat at 01:00 UTC.

It judges each unit by the right signal rather than by log chatter. That
distinction is load-bearing: an earlier version read `xalert`'s normal
websocket silence as death and restarted it 51 times in one night, and each
restart cost ~95 seconds of real blindness because twitterapi.io holds the
slot after a dropped socket. Units with `max_silence 0` are never judged by
log age; `xalert` is probed for a live `:443` socket instead.

After 3 restarts in an hour it stops and says so rather than flapping.

Because the watchdog posts as `@crash_accum_bot`, its messages land in the
same thread as Crash wallet alerts.

## Secrets

Tokens live in each bot's `config.json` under `telegram`, except `kookwatch`,
which uses `/opt/kookwatch/.env`.

Never print a token. `botctl config` fingerprints them, `botctl ping`
resolves them to `@usernames` without exposing them. Follow that pattern.

`kookwatch`'s `.env` holds a full Telegram **user session string**, not a bot
token - it authenticates as Ben's own account, because a BotFather bot cannot
read someone else's paid group. Treat that file as the most sensitive thing
on the droplet.