Datadog with Alopex Skulk

What the service does

Datadog monitors cloud and on-premises infrastructure. It collects metrics, turns them into graphs and alerts, and slices them by tag. The data is metric time series tagged with hosts and services.

Where the data lives

Skulk time series: CPU, memory, request counts and latency are appended to Skulk, tagged by host and service.
PromQL: dashboard graphs and alert conditions are PromQL queries.
Time buckets: hourly averages and maximums come from time-bucketed aggregation in SQL.

Collecting and writing the metrics and sending alerts are app code.

Deployment

Skulk is embedded in the application being monitored. The application keeps its own metrics and serves its own dashboards. Metrics from many applications are gathered by a Skulk server or cluster on the same engine.

Why Alopex Skulk fits

Monitoring comes down to tagged time series and the queries over them. Skulk handles ingestion, PromQL and time buckets inside the application, and grows from inside the application to a monitoring platform on the same engine.

See it in code

Metrics are ingested into Skulk tagged by host and read back. Just the core (Rust).

use alopex_skulk::ingest::line_protocol::LineProtocolDecoder;
use alopex_skulk::ingest::{IngestLimits, Ingestor};
use alopex_skulk::model::FieldValue;
use alopex_skulk::store::recovery::{RecoveryConfig, RecoveryStore};

fn main() -> alopex_skulk::Result<()> {
    std::fs::create_dir_all("datadog-data").unwrap();
    let store = RecoveryStore::open("datadog-data", RecoveryConfig::default())?;
    let limits = IngestLimits::default();
    let mut ingestor = Ingestor::new(store, limits);

    // CPU 使用率を、ホストのタグ付きで取り込む（measurement,tag field timestamp[ns]）
    let base: i64 = 7_200_000_000_000;
    let now = base + 300_000_000_000;
    for (i, usage) in [12.5, 40.0, 78.0].iter().enumerate() {
        let ts = base + (i as i64) * 60_000_000_000;
        let line = format!("cpu,host=web1 usage={usage} {ts}");
        let batch = LineProtocolDecoder::new(limits).decode(line.as_bytes(), now)?;
        assert_eq!(ingestor.ingest(batch, now)?.accepted_count(), 1);
    }
    ingestor.sink_mut().flush_all()?;

    // 取り込んだメトリクスを読み出す
    for row in &ingestor.sink().read_measurement("cpu")? {
        if let Some(FieldValue::Float(usage)) = row.row().field("usage") {
            println!("t={} usage={usage}", row.row().timestamp());
        }
    }
    Ok(())
}

Ingest is durable when ingest() returns. Dashboard graphs and alert conditions are queried with Skulk’s PromQL / time-bucket aggregation.

Embed Alopex Skulk, which answers PromQL over tagged time series, and your app carries its own dashboards. Give it a try in a small app.