ADN Systems DMR Peer Server

# ADN Systems DMR Peer Server

## Overview

ADN Systems DMR Peer Server is a fork of FreeDMR, implementing a Digital Mobile Radio (DMR) network server. Launched in April 2024 by international amateur radio enthusiasts, it operates on an Open Bridge Protocol (OBP) fostering a decentralized network architecture. The system handles DMR voice and data communication, acting as a conference bridge/reflector that routes traffic between connected systems (repeaters, hotspots, peers) based on configurable bridge rules.

## User Preferences

Preferred communication style: Simple, everyday language.

## System Architecture

### Core Protocol Implementation

**Problem**: Need to implement HomeBrew Repeater Protocol (HBP) and Open Bridge Protocol for DMR communication
**Solution**: Python-based protocol handlers using Twisted framework for asynchronous networking
- `hblink.py` serves as the core protocol engine implementing HBSYSTEM and OPENBRIDGE classes
- Supports both peer and master/server modes for network topology flexibility
- Uses DatagramProtocol for UDP-based DMR packet handling
- Implements custom authentication using SHA256/BLAKE2b hashing with HMAC for security

**Rationale**: Twisted provides battle-tested async I/O suitable for handling multiple simultaneous connections with low latency requirements critical for voice communication.

### Bridge/Routing Architecture

**Problem**: Route voice traffic between multiple DMR systems based on talkgroups and timeslots
**Solution**: Conference bridge pattern implemented in `bridge_master.py` and `bridge.py`
- Traffic routing based on rules defined in `rules.py` configuration
- Systems can be dynamically activated/deactivated on conference bridges
- Supports timeout-based automatic disconnection
- Static routing option available for permanent connections

**Alternatives Considered**: End-to-end routing was rejected in favor of conference bridge approach for better scalability and simpler management.

**Pros**: Flexible routing, easy management, scalable
**Cons**: Systems must individually join bridges (not transparent end-to-end)

### Proxy Architecture

**Problem**: Allow multiple hotspots/repeaters to connect through a single public IP address
**Solution**: UDP proxy implementation in `hotspot_proxy_v2.py` and `hotspot_proxy_self_service.py`
- Dynamic port allocation for incoming connections
- Connection tracking with timeout-based cleanup
- Blacklist support for access control
- Self-service variant includes database-backed client management

**Rationale**: Many amateur radio operators are behind NAT/firewalls; proxy enables connectivity without port forwarding.

### Voice Synthesis System

**Problem**: Generate voice announcements for system events (linking, unlinking, status)
**Solution**: AMBE (Advanced Multi-Band Excitation) voice codec integration
- Pre-recorded indexed voice files in multiple languages (en_GB, es_ES, fr_FR, etc.)
- `read_ambe.py` handles reading/parsing of AMBE audio files
- `mk_voice.py` generates HBP-compliant voice packets from AMBE data
- `playback.py` and `play_ambe.py` handle voice injection into streams

**Rationale**: Provides accessible feedback to users without requiring external TTS systems, maintains compatibility with DMR audio codecs.

### Scheduled Announcements (Locuciones Programadas)

**Problem**: Need to broadcast pre-recorded AMBE voice announcements on a schedule to specific talkgroups
**Solution**: Configurable scheduled announcement system in `bridge_master.py` with up to 4 independent announcement slots

**Configuration in config/voice.cfg [VOICE] section** (separated from adn.cfg):
- Slot 1: `ANNOUNCEMENT_*` (ENABLED, FILE, TG, TIMESLOT, MODE, INTERVAL, LANGUAGE)
- Slot 2: `ANNOUNCEMENT2_*` (same parameters with prefix ANNOUNCEMENT2_)
- Slot 3: `ANNOUNCEMENT3_*` (same parameters with prefix ANNOUNCEMENT3_)
- Slot 4: `ANNOUNCEMENT4_*` (same parameters with prefix ANNOUNCEMENT4_)

**Note**: voice.cfg is loaded automatically from the same directory as adn.cfg. If voice.cfg does not exist, the server starts normally with all announcements and recording disabled (safe defaults).

Each slot supports:
- `*_ENABLED`: True/False to enable/disable
- `*_FILE`: Name of the .ambe file (without extension, located in Audio/<lang>/ondemand/)
- `*_TG`: Talkgroup number where the announcement is broadcast
- `*_TIMESLOT`: Timeslot 1 or 2 (default: 2)
- `*_MODE`: `interval` (every X seconds) or `hourly` (at the top of each hour)
- `*_INTERVAL`: Interval in seconds (only used when mode is `interval`)
- `*_LANGUAGE`: Language folder for the AMBE file (e.g., es_ES, en_GB)

**Behavior**:
- Each slot runs independently with its own LoopingCall timer
- In `hourly` mode: checks every 30 seconds, plays only when minute == 0
- In `interval` mode: plays at the configured interval
- Broadcasts to ALL MASTER systems simultaneously (like bridge routing)
- Uses adaptive 60ms frame timing for clean audio
- Only plays on MASTER systems (not OPENBRIDGE), excludes ECHO/D-APRS
- Skips systems that are busy (RX or TX active)
- Audio source ID: 5000, uses SERVER_ID as peer ID
- Log labels: LOCUCION (slot 1), LOCUCION-2, LOCUCION-3, LOCUCION-4

### Voice Recording System (Grabaciones Locuciones)

**Problem**: Need to record voice announcements directly from radio traffic for later playback as scheduled announcements
**Solution**: AMBE voice recorder that captures traffic on a configured TG/TS

**Configuration in config/voice.cfg [VOICE] section** (same file as announcements):
- `RECORDING_ENABLED`: True/False to enable/disable recording
- `RECORDING_TG`: Talkgroup to monitor for recording
- `RECORDING_TIMESLOT`: Timeslot to monitor (1 or 2)
- `RECORDING_FILE`: Output filename (without .ambe extension)
- `RECORDING_LANGUAGE`: Language folder (determines save path: Audio/<lang>/ondemand/<file>.ambe)

**Behavior**:
- Records any voice transmission on the configured TG/TS
- Extracts raw AMBE bursts (108-bit pairs) from DMRD voice frames
- Maximum recording duration: 2 minutes 45 seconds (2750 frames)
- Automatically saves when transmission ends (voice terminator) or max duration reached
- Saved file is directly compatible with the announcement playback system
- Recording runs in the reactor thread (no blocking)
- Only processes validated frames (after duplicate/rate/loop checks)
- Log label: GRABACION

**Integration with Announcements**:
- Record with `RECORDING_FILE: mi_locucion` and `RECORDING_LANGUAGE: es_ES`
- Playback with `ANNOUNCEMENT_FILE: mi_locucion` and `ANNOUNCEMENT_LANGUAGE: es_ES`

### Individual Password Authentication

**Problem**: Need individual password authentication per Radio ID (indicativo) for enhanced security
**Solution**: JSON-based password storage with automatic reload
- Password file: `data/user_passwords.json` stores individual passwords by Radio ID
- Web dashboard: `dashboard.py` provides admin interface for password management
- Auto-reload: Passwords are reloaded every 10 seconds without server restart
- Fallback: Global passphrase used if no individual password is configured
- Mandatory authentication: Individual or global password required (no null passphrase allowed)

**Suffix Support for Radio IDs**:
- Radio IDs of 9 digits (ej: 214501601-214501699) can use the password of their 7-digit base ID (ej: 2145016)
- Useful for users with multiple hotspots/devices using suffix extensions
- Priority: Exact ID match first, then base ID match

**Authentication Priority**:
1. Individual password for exact Radio ID (if configured)
2. Individual password for base ID (7 digits, for 9-digit IDs with suffix)
3. Global passphrase (if configured)
4. Reject connection (no valid credentials)

**Dashboard Credentials**: Stored in `config/dashboard_credentials.json` file (not uploaded to GitHub)
- Supports multiple administrators with the following format:
```json
{
  "admins": [
    {"user": "admin", "password": "admin123"},
    {"user": "operador1", "password": "clave456"}
  ]
}
```
- Also supports legacy single-admin format for backwards compatibility

**Dashboard Web con Acceso Dual**:
- Pagina principal (`/`): Selector entre acceso usuario y administrador
- Acceso Usuario (`/user/login`): Login con Radio ID y contrasena para cambiar su propia contrasena
- Acceso Administrador (`/admin/login`): Login para gestionar todas las contrasenas
- Los usuarios pueden cambiar su contrasena sin necesitar al administrador

**Busqueda por Indicativo (Callsign)**:
- Descarga automatica diaria de la base de datos de RadioID.net (`data/user.csv`)
- Formato CSV con campos: RADIO_ID, CALLSIGN, FIRST_NAME, LAST_NAME, CITY, STATE, COUNTRY
- Funcion de busqueda en el panel de administracion para encontrar Radio IDs por indicativo
- Muestra toda la informacion del registro: nombre, ciudad, estado, pais
- Soporta busqueda parcial (ej: "EA4" muestra todos los indicativos que empiezan por EA4)
- Limita resultados a 20 para mejor rendimiento
- Boton "Usar ID" para autorellenar el formulario de agregar contrasena

### Servidor de Seguridad Centralizado (Rama: descentralizada)

**Problema**: Gestionar credenciales de forma centralizada desde un servidor remoto
**Solucion**: Descarga automatica de archivos de seguridad desde servidor central

**Configuracion en adn.cfg seccion [GLOBAL]**:
- `URL_SECURITY`: IP o DNS del servidor central de seguridad
- `PORT_SECURITY`: Puerto del servidor central
- `PASS_SECURITY`: Contrasena de autenticacion del administrador
- `USERS_PASS`: Nombre del archivo de contrasenas (default: user_passwords.json)
- `HASH_ENCRYPT`: Nombre del archivo de clave de encriptacion (default: encryption_key.secret)

**Comportamiento de Descarga**:
- `encryption_key.secret`: Se descarga SOLO al arrancar el servidor, sobrescribe el existente
- `user_passwords.json`: Se descarga cada 5 minutos, compara contenido antes de actualizar

**Sintaxis de Descarga (curl)**:
```
curl -L "http://URL_SECURITY:PORT_SECURITY/descargar?pass=PASS_SECURITY&encryption_key.secret"
curl -L "http://URL_SECURITY:PORT_SECURITY/descargar?pass=PASS_SECURITY&user_passwords.json"
```

**Seguridad y Tolerancia a Fallos**:
- Si la descarga falla, se conservan los archivos locales existentes
- No se permite contrasena nula (ALLOW_NULL_PASSPHRASE eliminado)
- Autenticacion obligatoria: contrasena individual o global requerida

**Archivos Descargados**:
- `config/encryption_key.secret`: Clave de encriptacion para descifrar contrasenas
- `data/user_passwords.json`: Archivo JSON con contrasenas encriptadas por Radio ID

### Configuration Management

**Problem**: Complex multi-system configuration with ACLs, bridges, and network parameters
**Solution**: INI-based configuration parsed by `config.py`
- Centralized configuration in `config/adn.cfg`
- ACL (Access Control List) processing for registration and talkgroup filtering
- Support for both whitelist and blacklist approaches
- Language preferences and multi-language support via `languages.py`

### Monitoring and Reporting

**Problem**: Need visibility into system operation, connections, and traffic
**Solution**: Network-based reporting protocol
- `report_receiver.py` and `report_sql.py` consume events from the core server
- Pickle-based serialization for config/bridge state transmission
- Opcode-based protocol defined in `reporting_const.py`
- SQL integration option for persistent event logging

**Rationale**: Separates monitoring concerns from core routing logic, enables multiple monitoring tools.

### Asterisk Integration

**Problem**: Integration with Asterisk PBX for advanced features (app_rpt)
**Solution**: Asterisk Manager Interface (AMI) client in `AMI.py`
- Sends RPT (repeater) commands to Asterisk
- Uses Twisted LineReceiver for line-based protocol handling
- Supports node-specific command routing

### API Layer

**Problem**: External systems need programmatic access to server functions
**Solution**: XML-RPC and SOAP API implementation
- `API.py` implements Spyne-based SOAP service (FD_API)
- Peer validation and authentication endpoints
- `api_client.py` provides example XML-RPC client
- Key-based authentication for peer systems

## External Dependencies

### Network Protocols
- **Twisted** (>= 16.3.0) - Asynchronous networking framework for all protocol handling
- **dmr_utils3** (>= 0.1.19) - DMR protocol utilities for packet encoding/decoding, BPTC, Golay error correction

### Database Systems
- **MySQL/MariaDB** - Via twisted.enterprise.adbapi and MySQLdb
  - Used by `proxy_db.py` for self-service proxy client management
  - Stores client registrations, authentication, and connection tracking
  - Optional for report_sql.py event logging

### Remote Object Communication
- **Pyro5** - Python Remote Objects for inter-process communication
  - Used by proxy services for distributed architecture
  - Enables separation of proxy components across processes/hosts

### Third-Party Services
- **radioid.net API** - DMR ID database lookups
  - `peer_ids.json`, `subscriber_ids.json`, `talkgroup_ids.json` downloaded from external sources
  - Cached locally with staleness checking via `utils.py::try_download()`
  - Provides callsign/name resolution for DMR IDs

### Supporting Libraries
- **bitarray/bitstring** - Binary data manipulation for DMR packet construction
- **configparser** - INI file parsing for hblink.cfg
- **setproctitle** - Process naming for easier system monitoring
- **resettabletimer** - Timeout management for connection tracking
- **hashlib/hmac** - Cryptographic functions for authentication

### Language/Voice Assets
- Pre-recorded AMBE voice files in Audio/ directory
- Multiple language support (en_GB, es_ES, fr_FR, de_DE, dk_DK, it_IT, no_NO, pl_PL, se_SE, pt_PT, cy_GB, el_GR, th_TH, CW)
- Voice file indexing via i8n_voice_map.py