Data Engineer - Scotiabank, Toronto

AI Summary

This Data Engineer role at Scotiabank in Toronto, Ontario, is crucial for the Enterprise Data & AI Technology organization. The position involves designing, building, and maintaining scalable data pipelines, leveraging SaaS platforms, and ensuring data integrity. Successful candidates will have strong experience with cloud-based data platforms like Azure, Lakehouse platforms such as Databricks, and robust programming skills in SQL and Python, with at least four years of relevant professional experience.

About the Role

Joining a purpose-driven and inclusive team, the Data Engineer plays a critical role within Scotiabank's Enterprise Data & AI Technology organization, which is a significant enterprise-wide strategic initiative. This team is instrumental in driving data-enabled decision making, fostering AI innovation, and modernizing technology across the Bank. In this dynamic position, you will be responsible for designing, building, and maintaining scalable and resilient data pipelines, ensuring they effectively support the ingestion, processing, and distribution of large datasets. A key aspect of the role involves leveraging various SaaS platforms and tools to develop, configure, and automate data workflows, significantly improving efficiency across the entire data engineering lifecycle. You will also collaborate closely with stakeholders and product managers to gather detailed requirements, design optimal ingestion patterns, and seamlessly onboard new data sources into Scotiabank's cloud data platform. Furthermore, the role requires implementing robust observability capabilities and continuously optimizing application services and pipeline performance to ensure reliability and speed. Performing thorough data integrity validation is also a core duty, ensuring the accuracy, reliability, and consistency of all processed data within the Scotiabank ecosystem. This role offers an opportunity to contribute to a high-performing culture committed to results.

Skills and Experience

To succeed in this vital Data Engineer role at Scotiabank, candidates must possess a comprehensive set of technical skills and practical experience. We are looking for individuals with proven experience in building sophisticated data pipelines and composable cloud-based data platforms, specifically utilizing Azure technologies such as Azure Data Factory and ADLS. Extensive experience working with Lakehouse platforms, including Databricks and Unity Catalog, along with Auto Loader, is essential. Candidates should demonstrate strong expertise in data engineering principles, programming, and both ETL and ELT processes for efficient data extraction and processing. This includes hands-on experience working with structured, semi-structured, and unstructured data across various sources. Strong programming and scripting skills in languages like SQL, Python, Java, and Scala are highly valued. A solid understanding of cloud infrastructure and microservices architecture is also required. The ability to create detailed technical design documentations, encompassing both logical and physical views, is important. Collaboration is key in this environment, so experience working effectively with DevOps, Scrum, and Product Teams is necessary. We seek a demonstrated team player with strong communication skills and a track record of successful delivery in product development. The capacity to translate complex technical problems into clear business language is a significant asset. Expertise in problem-solving and debugging is paramount. Specifically, candidates should have at least four years of experience working with Data Warehouse or Data Platforms, four years of experience creating ELT/ETL data pipelines and handling diverse data types, four years of experience working on continuous integration and continuous deployment (CI/CD) pipelines, and proficiency with source control systems such as Github, Bitbucket, and Terraform. Furthermore, a minimum of four years of experience in Cloud environments, particularly Azure, and four years of experience with Lakehouse technologies like Databricks, are critical for this position. This breadth of experience ensures the candidate can contribute effectively across Scotiabank's advanced data platforms.