The challenge: live monitoring safety-critical assets

K2, a startup redefining what’s possible in space, operates at the bleeding edge of satellite innovation. Organizations typically design satellites by delicately balancing tradeoffs between size, weight, power, and cost. Industry-wide, as it has been for many decades, there is a large correlation between a satellite’s mass and its cost (and thus, its power, bandwidth, and capabilities). K2’s mission is to crack this correlation to instead carry high power at low cost. K2 would enable an entire class of missions not previously possible and thereby deliver previously unthinkable value to customers. 

To realize this vision for the next generation of satellite constellations, K2 needs infrastructure designed for the challenge. Furthermore, K2’s testing workflow requires engineers to monitor live telemetry streams to react to anomalies as they occur on the test stand in real time. Without live monitoring, a single missed threshold could result in the loss of expensive hardware and put the team at risk. Maintaining sufficient levels of precision and quality control to build one satellite is already a difficult challenge. But K2’s mission isn’t to build a single satellite—it’s to build a thousand. Scaling to those ambitions demands a telemetry architecture that can ingest tens of millions of data points per second while discerning signal from noise in real-time. The challenge they faced: How can any telemetry stack handle this scale without collapsing under its own weight?
‍

‍

The constraints vs the alternatives: open source, build, or buy?

Most time-series stacks encounter fundamental limitations when faced with large-scale, high-fidelity hardware test data. These limitations can be traced to three interrelated constraints: latency, cardinality, and dynamic schema.

Latency becomes a bottleneck as databases become unable to write and read data fast enough. In a traditional monolithic database, this is typically due to CPU contention, as the same CPU is used to write and then read back all data. While compression can reduce payload size and improve efficiency, its resulting decompression can further exacerbate CPU contention. Latency can also be an issue with traditional databases that try to keep their entire index in memory for fast lookups. During heavy ingestion or replay, this indexing strategy causes memory to balloon uncontrollably, often leading to instability or outright crashes. While adding more RAM can address index-related latency, it is not sustainable in most cases and is outright impossible in the case of on-prem deployments where resources are fixed for hardware-in-the-loop environments. As a result, engineers can’t rely on these systems during critical testing operations where uptime and responsiveness are non-negotiable.

Cardinality presents an equally severe challenge. In hardware testing, machines frequently have many sensor channels with even more channels derived from that sensor. Keeping up with these channels becomes increasingly complex as machines iterate over time, thus changing (and frequently adding to) their respective sensors and channels. This “cardinality bloat” cripples query performance as the database must traverse an ever-growing set of unique keys just to filter or aggregate a simple query. In practice, this means dashboards become sluggish or fail entirely as the number of sensors or test runs increases. This might be an acceptable risk if you’re building something simple, but K2 is building satellites with hundreds of thousands, and eventually millions of sensors across their constellation.

Dynamic schemas complicate matters further. Hardware test environments evolve rapidly. As the build evolves, new sensors, firmware revisions, and measurement types are introduced almost daily. An effective data warehouse must allow the structure of its data to evolve over time without forced migrations or downtime. However, most open source solutions are not built for this kind of flexibility. Each schema change can generate thousands of new index entries, amplifying cardinality bloat and degrading query performance. As a result, teams are often forced to choose between agility and stability. This trade-off is simply unacceptable for mission-critical testing workflows.

‍

‍

The solution: Sift’s live, continuous streaming

To address this challenge and overcome these limitations, K2 partnered with Sift. The platform’s vertically integrated data store balances latency, cardinality, and dynamic schemas rather than optimizing for one at the expense of the others. To handle historical (sometimes called “cold”) data, Sift leverages Parquet as a columnar storage format to decouple storage from compute. While traditional cold storage sometimes takes hours or days to access, Sift’s cold storage is available for query in mere minutes or less. To support live (“hot”) data streaming, we built a custom, proprietary two-fault-tolerant cache file for optimized performance. This is especially valuable during mission-critical operations like launch, where hundreds of people are opening the same links, which can slow performance. Sift’s hot cache prevents this. With this approach, engineers can run queries that span across evolving schemas without the need for migrations or schema rewrites, regardless if it's a legacy sensor format or a new data model. This design ensures consistent performance, even as K2’s data grows in scale and complexity.
‍

‍

The result is Sift: a telemetry solution capable of supporting hundreds of millions of channels across complex machines, enabling real-time visibility without compromising performance. This means that as K2 generates data during tests, Sift can detect anomalies as they happen, empowering K2’s engineers to make crucial decisions on expensive, safety-critical assets in real-time. Where open source architectures buckle under memory pressure or slow queries, Sift maintains low-latency lookups and rapid aggregation on high-cardinality datasets. Write latency (p99) is several hundred milliseconds and remains under one second under typical conditions. This is especially important, as K2’s current hardware generates nearly 20 million data points per second. This adds up to terabytes of total volume on a test day, with expectations that this will grow as the hardware matures.

Sift unites the best of existing solutions: the write efficiency of time-series systems, the query speed of columnar engines, and the schema flexibility of modern data lakes. All in one unified observability platform, Sift is the ideal solution to scale and accelerate K2’s production.

‍

‍

The impact: tens-of-millions-point problem, solved

By integrating Sift into K2’s test and production workflow, K2 has achieved significant improvements in its development cycle:

‍

‍

In the past few months, Sift has enabled K2 to:

• Monitor safety critical test stands: Because Sift can detect anomalies as they happen, K2’s engineers can review data and make crucial decisions on expensive, safety-critical assets in real-time.
• Scale testing without scaling engineering hours: With Sift’s capacity for large-scale data ingestion and analysis, K2 can parse signals from noise for not just one test stand but the entire production line. Over an entire campaign terabytes of data can be leveraged, whether it’s live or historical data, with ease. 
• Collaborate seamlessly: Sift’s collaborative links let engineers share context instantly. No more screenshots, no more custom Python scripts, or no more getting lost on one-off web servers.
• Move faster: Scaled data ingestion, faster time-to-root-cause, and easy collaboration lets K2 accelerate with peace of mind. 

Looking ahead

K2 continues to utilize Sift for full-vehicle lifecycle testing and production. And as K2’s satellite constellation grows, so too will their data needs. Sift’s vertically integrated infrastructure guarantees that our platform will be able to scale alongside our customers’ ambitions. As early adopters of Sift, their team is a trusted partner that provides crucial feedback to ensure we support the workflows and features engineers need to build the next generation of satellites and space vehicles.