Senior Data Engineer in Cybernetic Controls Ltd

Пряме повідомлення учасника, який розмістив вакансію, з Cybernetic Controls Ltd Bushra Shahzaib Bushra Shahzaib International Recruitment Specialist | Administrative Expert | Freelance Content Writer OverviewCCL builds high quality automation frameworks and bespoke software solutions for firms, with an aim to help organizations improve their operational processes and reduce overall risk and increase efficiency. Another key objective of CCL is to enable greater transparency for firms and regulators. CCL is also passionate about removing waste and making things simple.At CCL, we think it is just as important that you feel that we are the right company for you, as well as you being right for us. So we want to share our company values with you. These values were created as a collaborative effort between our employees and are integral to our company. These behaviors underpin everything we do and help create a positive culture within CCL. Read more on theCybernetic Controls website.Our Client:We are a multi-award winning RegTech company on a mission to transform the quality of regulatory reporting in the financial services industry. We’ve combined regulatory expertise with advanced technology to develop our market-leading quality assurance services. Unique in being able to fully assess data quality, our services are used by some of the world’s largest investment banks, asset managers, hedge funds and brokers, helping them to reduce costs, improve quality and increase confidence in their regulatory reporting.Job summaryOur client is seeking a Senior Data Engineer to join our fast-growing team. The successful candidate will join the testing team to work on ETL and development tasks. This is an exciting and challenging opportunity to build out new pipelines combining and processing large amounts of structured data from a variety of sources with the power ofPySparkat your fingertips.Key ResponsibilitiesArchitect and build pipelines using AWS cloud computing solutions that make data available with robustness, maintainability, efficiency, scalability, availability and security.Develop Python and/or Spark (Preferably PySpark, but Spark-Scala is also good to have) code that implements complex data transformations.Design and maintain databases and APIs for storage and transmission of data between applications.Monitor pipelines in production (and develop tools to facilitate this).Work collaboratively with other team members (brainstorming, troubleshooting, and code review).Liaise with other development teams to ensure the integrity of data pipelines.Skills, Knowledge & ExpertiseSkills:Excellent Python and PySpark programming (including Pandas/PySpark dataframes, web and database connections)Excellent understanding of ETL processes within Amazon Web Services (AWS)Apache Spark, AWS Glue, Athena, S3, Step Functions, Lake FormationSoftware development lifecycle best practicesTest-driven developmentServerless computing (AWS Lambda, API Gateway, SQS, SNS, EventBridge, S3, etc.)SQL and NoSQL database design and management (DynamoDB, MySQL)Strong SQL coding skills (Spark-SQL, Presto SQL, MySQL, etc.)Infrastructure as code (CloudFormation)Experience in Shell Scripting (preferably Linux)Version control with Git/GithubAgile principles, processes and toolsExcellent written and verbal communication skills.Experience:Designing, deploying and managing complex production data pipelines that interact with a range of data sources (file systems, web, database, users).Strong experience with Amazon Web Services (AWS)5 years’ work in a similar role.Knowledge:Data modeling, data pipeline architecture, Big Data implementation.Software development lifecycle best practices.Financial knowledge would be an asset. Qualifications/Training:Bachelor’s degree or equivalent in Computer Science or a related subject.What you’ll get in returnCompetitive salary packagePrivate healthcare contributionAnnual pay reviewRegular team socialsWorking within a culture of innovation and collaborationOpportunity to play a key role in a pioneering growth company Show more Show less Посадовий рівень Старший середній рівень Тип зайнятості Повний робочий день Посадові обов’язки Інформаційні технології Галузі Розробка програмного забезпечення