Show HN: Open-source serverless security lake powered by Rust + Apache Iceberg

Hacker News - Sat Aug 6 13:31

The open-source security lake platform for AWS.


Twitter Follow

What is Matano?

Matano is an open source security lake platform for AWS. It lets you ingest petabytes of security and log data from various sources, store and query them in a data lake, and create Python detections as code for realtime alerting. Matano is fully serverless and designed specifically for AWS and focuses on enabling high scale, low cost, and zero-ops. Matano deploys fully into your AWS account.

Features

Collect data from all your sources

Matano lets you collect log data from sources using S3 or Kafka based ingestion.

Ingest, transform, normalize log data

Matano normalizes and transforms your data using Vector Remap Language (VRL). Matano works with the Elastic Common Schema (ECS) by default and you can define your own schema.

Store data in S3 object storage

Log data is always stored in S3 object storage, for cost effective, long term, durable storage.

Apache Iceberg Data lake

All data is ingested into an Apache Iceberg based data lake, allowing you to perform ACID transactions, time travel, and more on all your log data.

Serverless

Matano is a fully serverless platform, designed for zero-ops and unlimited elastic horizontal scaling.

Detections as code

Write Python detections to implement realtime alerting on your log data.

Installing

You can install the matano CLI to deploy Matano into your AWS account, and manage your Matano deployment.

Requirements

  • node>=12 and npm
  • Docker

From source

You can manually install from source.

git clone https://github.com/matanolabs/matano.git
make install

Getting started

Read the complete docs on getting started.

Create a Matano Directory

A Matano directory lets you specify data and configuration values for Matano. Run the following to generate a sample directory to get started:

matano generate:matano-dir

This will create a directory with sample detections and log sources.

Fill in your AWS account ID and desired AWS region in the generated matano.config.yml file.

# replace these values
aws_account_id: "012345678901"
aws_region: "us-east-1"

The following commands require valid AWS credentials, either from the environment or the --profile flag.

Bootstrap your AWS account

Initialize your AWS environment before deployment by running matano bootstrap. This will create the necessary resources in your AWS account before deploying Matano.

matano bootstrap [--profile AWS_CLI_PROFILE]

Follow the CLI prompts to ensure your AWS account is ready for deployment.

Deploy Matano

You can now deploy Matano to your AWS account. Run the following command from your Matano directory:

matano deploy [--profile AWS_CLI_PROFILE]

The first deployment can take up to 15 minutes.

Documentation

View our complete documentation.

License