Diaconia

Creative IT Solutions | An Officially Great Place To Work!

Senior AWS Data Engineer

Data EngineerData EngineerFull TimeRemoteTeam 51-200Since 2020H1B No SponsorCompany SiteLinkedIn

Location

United States

Posted

6 days ago

Salary

$145K - $160K / year

Bachelor Degree8 yrs expEnglishAWSCloudETLJenkinsSparkSQLTerraform

Job Description

• Build and maintain scalable data pipelines in AWS to support ingestion, transformation, and enrichment of structured and semi-structured data • Design and implement Delta Lake tables optimized for ACID compliance, partition pruning, schema enforcement, and query performance across large datasets • Develop ETL and ELT workflows that integrate multiple source systems into a centralized, query-optimized data warehouse architecture • Leverage AWS tools to implement business rules, dimensional joins, and aggregation logic aligned to warehouse modeling best practices • Collaborate with data architects and engineers to implement cloud-native data solutions on AWS using S3, Glue, RDS, and IAM for secure, scalable storage and access control • Optimize pipeline performance through intelligent partitioning, caching, broadcast joins, and adaptive query tuning • Deploy and version data engineering assets using Git-integrated development workflows and automate deployment with CI/CD tools such as GitLab or Jenkins • Monitor pipeline health, job execution, and cluster utilization using AWS CloudWatch, identifying bottlenecks and optimizing cost-performance tradeoffs • Conduct technical discovery and mapping of legacy source systems, identifying required transformations and designing end-to-end data flows • Implement governance practices including metadata tagging, data quality validation, audit logging, and lineage tracking using platform-native features and custom logic • Support ad hoc data access requests, develop reusable data assets, and maintain shared notebooks that meet operational reporting and analytics needs across teams

Job Requirements

  • This role requires knowledge and/or experience with Spark, Delta Lake, and distributed data pipelines
  • The ideal candidate brings both engineering and strategic insight into enterprise data modernization
  • 8+ years of experience in data engineering and Agile analytics
  • 5 years of experience building scalable ETL and ELT workflows for reporting and analytics
  • 3+ years of experience building enterprise data engineering solutions in the cloud, with preferred experience with cloud native technologies from AWS
  • Hands-on experience in the following: Glue / Spark SQL–based data transformations
  • S3 partitioning strategy
  • Step Functions–based orchestration
  • Infrastructure as Code (Terraform and/or CloudFormation)
  • Deployment automation for data pipelines
  • AWS services integration
  • Experience with data quality, validation frameworks, and storage optimization strategies
  • BA or BS degree
  • Excellent communication and organizational skills with the ability to manage multiple priorities
  • U.S. Citizenship is required by the Federal Client
  • Must have or able to obtain DoD Public Trust Clearance

Related Categories

Related Job Pages