observability-guidelines

Observability Guidelines

Apply these observability principles to ensure comprehensive visibility into distributed systems and microservices.

Core Observability Principles

• Guide the development of idiomatic, maintainable, and high-performance code with built-in observability
• Enforce modular design and separation of concerns through Clean Architecture
• Promote test-driven development and robust observability from the start

OpenTelemetry Integration

• Use OpenTelemetry for distributed tracing, metrics, and structured logging
• Start and propagate tracing spans across all service boundaries
• Use otel.Tracer for creating spans and otel.Meter for collecting metrics
• Export data to OpenTelemetry Collector, Jaeger, or Prometheus
• Configure appropriate sampling rates for production environments

Distributed Tracing

• Trace all incoming requests and propagate context through internal calls
• Use middleware to instrument HTTP and gRPC endpoints automatically
• Include trace context in all downstream service calls
• Create child spans for significant operations within a service
• Add relevant attributes to spans for debugging and analysis

Metrics Collection

Monitor these key metrics across all services:

• Request latency: Track p50, p90, p95, and p99 percentiles
• Throughput: Measure requests per second by endpoint
• Error rate: Track 4xx and 5xx responses separately
• Resource usage: Monitor CPU, memory, disk, and network utilization
• Custom business metrics: Track domain-specific KPIs

Structured Logging

• Include unique request IDs and trace context in all logs for correlation
• Use structured logging formats (JSON) for machine parseability
• Include relevant context: timestamp, service name, trace ID, span ID
• Log at appropriate levels: DEBUG, INFO, WARN, ERROR
• Avoid logging sensitive information (PII, credentials)

Architecture Patterns

• Apply Clean Architecture with handlers, services, repositories, and domain models
• Use domain-driven design principles for clear boundaries
• Prioritize interface-driven development with explicit dependency injection
• Prefer composition over inheritance; favor small, purpose-specific interfaces

Correlation and Context

• Propagate context through the entire request lifecycle
• Use correlation IDs for request tracking across services
• Include service version and deployment information in telemetry
• Tag traces with relevant business context for filtering
• Enable trace-to-log and log-to-trace correlation

Alerting and Dashboards

• Create dashboards for service health and business metrics
• Set up alerts based on SLOs and error budgets
• Use anomaly detection for proactive issue identification
• Document runbooks for common alert scenarios
• Review and tune alerts regularly to reduce noise

Instrumentation Best Practices

• Instrument at service boundaries (entry/exit points)
• Add custom spans for database operations and external calls
• Include relevant attributes (user ID, request type, etc.)
• Avoid over-instrumentation that creates noise
• Use semantic conventions for consistent attribute naming

Production Considerations

• Configure appropriate sampling rates to balance visibility and cost
• Use head-based sampling for consistent trace capture
• Implement tail-based sampling for capturing errors
• Set retention policies based on debugging needs
• Monitor observability infrastructure health