Dynamically switching execution platforms with Rust

KISTERS provides software for collecting and analyzing environmental data.
In particular, they provide solutions to warn people of upcoming floods using data obtained from numerous sensor devices deployed throughout the globe.

KISTERS needed the ability to dynamically move computation between traditional servers and edge functions, or offloading to end-user devices. That way, they would be able to handle large traffic spikes efficiently without risking astronomic cloud expenses

The large amounts of data KISTERS obtains from sensor devices need processing to become useful for analysis. Filtering, grouping, and post-processing, depend on dynamic inputs and need to be carried out on demand.

That in and of itself warrants a well-optimized design and use of performant technologies, but in times of calamity, the amount of users requesting information may rise quickly: residents of endangered areas can use KISTERS’ software to obtain real-time information regarding their situation.

Mainmatter proposed and implemented an architecture that isolates the data processing implementation in a platform-independent package written in Rust.

That package can be thinly wrapped for use in a web server, run as an edge function, or compiled to Web Assembly (WASM) for execution in browsers. All of these different environments run the same efficient code with close to no duplication.

Rust, with its strong performance and stability characteristics and solid WASM support, was a natural choice as a language with which to implement the design. What’s more, Polars, the famous high-performance open-source data frame manipulation library, is implemented in Rust itself and easily usable in Rust projects including those targeting WASM.

Upon finishing and delivering the proof-of-concept, Mainmatter provided guidance to KISTERS' team on integrating the code into their platform.

Mainmatter has reviewed and optimized the integration code during several pairing sessions with KISTERS’ engineering team, ensuring the delivery of both the product and the technical knowledge required to maintain and expand it.