Sales enablement platforms customized for media, and ad tech companies that help you close more deals.
Data Engineering Lead
Location
United States
Posted
2 days ago
Salary
Not specified
Job Description
Job Requirements
- Azure Databricks:
- ○ Expert-level experience managing workspaces, clusters, and job scheduling.
- ○ Solid understanding of data lakehouse architectures and Delta Lake.
- ○ Proven experience in Performance Tuning, Spark Optimization and Cost Reduction.
- PySpark: Advanced proficiency in Spark DataFrame APIs and Spark SQL for large-scale data processing involving various data formats.
- SQL Mastery: Exceptional ability to write, tune, and troubleshoot complex queries.
- PostgreSQL: Hands-on experience with relational database design, indexing, and performance optimization.
- ETL/ELT Frameworks: Proven track record of building scalable data pipelines from scratch.
- Workflow Orchestration: Experience with Apache Airflow for managing complex task dependencies.
- Containerization: Familiarity with Azure Kubernetes Service (AKS) for deploying containerized data services.
- Infrastructure as Code (IaC): Knowledge of Terraform or Bicep for managing Azure resources.
- 10+ years of experience in Data Engineering or Software Engineering.
- 3+ years as a formal technical Lead managing an agile team and implementing E2E solutions.
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field.
- Strong communication skills with the ability to explain complex technical concepts to non-technical stakeholders.
- Strong problem-solving skills and attention to detail.
Related Guides
Related Categories
Related Job Pages
More Data Engineer Jobs
Data Platform Engineer (26-27)
IDEA Public SchoolsAt IDEA, the Staff Experience Team uses our Core Values to promote human connection and a culture of integrity, respect, and belonging for all Team and Family members.
The Data Platform Engineer will design, build, and maintain automated ELT pipelines ingesting data into Snowflake, configuring cloud-native tools and custom Python pipelines as necessary. This role also involves administering Snowflake environments, optimizing performance and cost, and managing infrastructure using infrastructure-as-code patterns.
Maleda Tech is seeking an experienced Data Engineer to support the development and expansion of a large-scale data ecosystem used to inform strategic decision-making. In this role, you will design and maintain data pipelines that integrate internal and third-party datasets, enabl...
Data Architect – MuleSoft Integration, Migration Specialist
ICFWe are not a typical consulting firm and our people are not typical consultants.
Data Architect specializing in MuleSoft integration for government contracts
Data Engineer III
eSimplicityAn engineering firm that delivers high-quality Healthcare IT, Cybersecurity, and Telecommunication solutions.
The role involves developing, expanding, and optimizing data and data pipeline architecture, including maintaining ETL processes and building Proofs of Concept using various cloud technologies. Responsibilities also include operating large-scale data processing pipelines, performing data quality analysis, and building infrastructure for optimal data extraction, transformation, and loading.
