Spear AI

Artificial Intelligence & Machine Learning for National Security

Data Engineer

Data EngineerData EngineerFull TimeRemoteTeam 11-50Since 2020Company SiteLinkedIn

Location

United States

Posted

56 days ago

Salary

Not specified

PythonRustPostgre SQLApache IcebergApache ParquetAmazon S3MQTTRedpandaApache KafkaDagsterApache AirflowProtocol BuffersTime Series Data ProcessingBinary Message ParsingOLTPOLAPDistributed SystemsStreaming ArchitecturesBatch Processing

Job Description

This description is a summary of our understanding of the job description. Click on 'Apply' button to find out more.

Role Description

We’re seeking a skilled Data Engineer to build the next-generation data management and artificial intelligence platform for maritime domain awareness.

What you’ll do:

  • Implement real-time data pipelines with MQTT and Redpanda for stream processing.
  • Implement offline data pipelines using Dagster for batch processing.
  • Parse and process binary message formats from various data sources.
  • Build data warehouses using Postgres, Apache Iceberg, Parquet, and S3.
  • Design data models that allow for high-performance queries.
  • Validate and normalize data sources.
  • Improve local development and CI/CD using modern tooling and GitHub Actions.

Qualifications

  • Expertise in time-series data processing and analysis (windowing, resampling, interpolation, etc.)
  • Proficiency in Python and Rust for data engineering workflows.
  • Experience with binary message parsing.
  • Experience with row-based & columnar-based data formats.
  • Experience with OLTP & OLAP databases.
  • Knowledge of distributed systems, streaming architectures, and batch processing patterns.
  • Hands-on experience with a batch orchestrator such as Dagster/Airflow.
  • Hands-on experience with a streaming platform such as Redpanda/Kafka.
  • Hands-on experience with binary message formats such as Protobuf.

Requirements

  • Experience with IoT devices and sensors (nice to have).
  • Digital signal processing experience (nice to have).
  • Geospatial analysis and GIS experience (nice to have).
  • Familiar with working in monorepos (nice to have).

Benefits

  • Unlimited PTO — Take the time you need to recharge and maintain work-life balance.
  • Dedicated Sick Time — Your health and well-being come first.
  • Comprehensive Health & Benefits – Medical, dental, and vision coverage to keep you and your family protected.
  • 11 Paid Holidays — Enjoy time off throughout the year to celebrate and spend time with loved ones.
  • Professional Development — Educational opportunities and resources to help you grow your skills and advance your career.
  • Collaborative Environment — Work directly with leadership in our flat organizational structure, where your ideas and contributions matter.
  • Mission-Driven Work — Contribute to projects that directly support national security and make a real-world impact.
  • Growth Opportunities — Join us during an exciting expansion phase where you can help shape our future.
  • 401(k) with company match.
  • Onsite / Remote / Flexible work arrangements or hybrid options (position dependent).
  • Relocation assistance (position dependent).
  • Referral bonuses.
  • Performance bonuses.
  • Life insurance and disability coverage.
  • Technology home office setup stipend.
  • Professional certification reimbursement (position dependent).

Job Requirements

  • Expertise in time-series data processing and analysis (windowing, resampling, interpolation, etc.)
  • Proficiency in Python and Rust for data engineering workflows.
  • Experience with binary message parsing.
  • Experience with row-based & columnar-based data formats.
  • Experience with OLTP & OLAP databases.
  • Knowledge of distributed systems, streaming architectures, and batch processing patterns.
  • Hands-on experience with a batch orchestrator such as Dagster/Airflow.
  • Hands-on experience with a streaming platform such as Redpanda/Kafka.
  • Hands-on experience with binary message formats such as Protobuf.
  • Experience with IoT devices and sensors (nice to have).
  • Digital signal processing experience (nice to have).
  • Geospatial analysis and GIS experience (nice to have).
  • Familiar with working in monorepos (nice to have).

Benefits

  • Unlimited PTO — Take the time you need to recharge and maintain work-life balance.
  • Dedicated Sick Time — Your health and well-being come first.
  • Comprehensive Health & Benefits – Medical, dental, and vision coverage to keep you and your family protected.
  • 11 Paid Holidays — Enjoy time off throughout the year to celebrate and spend time with loved ones.
  • Professional Development — Educational opportunities and resources to help you grow your skills and advance your career.
  • Collaborative Environment — Work directly with leadership in our flat organizational structure, where your ideas and contributions matter.
  • Mission-Driven Work — Contribute to projects that directly support national security and make a real-world impact.
  • Growth Opportunities — Join us during an exciting expansion phase where you can help shape our future.
  • 401(k) with company match.
  • Onsite / Remote / Flexible work arrangements or hybrid options (position dependent).
  • Relocation assistance (position dependent).
  • Referral bonuses.
  • Performance bonuses.
  • Life insurance and disability coverage.
  • Technology home office setup stipend.
  • Professional certification reimbursement (position dependent).

Related Categories

Related Job Pages

More Data Engineer Jobs

Senior Data Engineer

Quanata

Quanata is on a mission to help ensure a better world through context-based insurance solutions. With the full backing of State Farm, we’re powering the insurance industry of tomorrow and helping enable better driving behaviors. Our top tier team of tech-minded professionals comprises data scientists, actuaries, engineers, designers and marketers—many from the best companies in Silicon Valley—and we’re inspired to create the insurance products and experiences of the future. Learn more about us and our work at http://www.quanata.com.

Data Engineer56 days ago
Full TimeRemoteTeam 201-500H1B No Sponsor

Senior Data Engineer delivering data science services and streaming data pipelines

AirflowAWSCloudKafkaPythonSQLTerraform
United States
$215K - $300K / year

Senior Data Engineer

Tekmetric

Simplify Your Life. Supercharge Your Shop.

Data Engineer56 days ago
Full TimeRemoteTeam 51-200H1B No Sponsor

Senior Data Engineer responsible for designing data infrastructure for Tekmetric

AirflowApacheETLJavaPythonScalaSparkSQLTableau
United States

Senior Data Engineer – Analytics

Machinify, Inc.

Bending the healthcare cost curve with AI.

Data Engineer56 days ago
Full TimeRemoteTeam 51-200H1B No Sponsor

Data Engineer transforming raw external data into powerful datasets at Machinify.

AirflowAWSCloudKafkaPythonSparkSQL
California

Manager, Ads Data Engineering

Netflix

Where you come to do the best work of your life. Follow @WeAreNetflix on Twitter, IG, Facebook, & Youtube for more

Data Engineer56 days ago
Full TimeRemoteTeam 10,001+Since 1997H1B Sponsor

Data Engineering Manager leading ads data initiatives at Netflix

HadoopSparkSQL
United States
$360K - $920K / year