• Understand Cogstate data sources and develop data pipelines using Databricks to bring all data into the data lake. • Design, develop, implement, and tune large-scale distributed systems and pipelines that process large volumes of data; focusing on scalability, low-latency, and fault-tolerance in every system. • Developing scalable and re-usable frameworks for ingesting data into Azure Databricks, incorporating standards and best practices into engineering solutions. • Databricks engineering - query tuning, performance tuning, troubleshooting, and debugging pipelines. • Deep understanding of ETL/ELT design methodologies, architecture, strategy, and tactics for complex ETL solutions, including CI/CD skills. • Develop high performance scripts in PySpark to achieve objectives of enterprise data, BI, data visualization and analytics needs. • Data processing/transformation using various technologies such as Apache Spark, SQL, Python/Scala and Azure cloud services. • Manage code versions in source control and coordinate changes across teams by leveraging Github. • Participate in architecture design and discussions, provide logical and physical data design, and database modelling. • Be part of the Agile team to ensure availability of data to internal and external users. • Organize and manage data shares. • Solve complex data issues around data integration, data quality, and other data processing incidents. • Work with business system owners to resolve source data issues and refine transformation rules.