Data Engineer (Azure Databricks) in LimestoneDigital

Limestone Digital is looking for a Data Engineer to expand our team.

Requirements:

- Strong proficiency in Azure Data Factory (ADF) and Azure Synapse for designing and implementing data pipelines and workflows;

- Expertise in SQL for querying and manipulating data within relational databases and data warehouses;

- Experience with Databricks for big data processing, data exploration, and advanced analytics;

- Knowledge of Azure Events Hub for ingesting and processing real-time data streams;

- Familiarity with Azure Functions for developing serverless computing solutions and event-driven architectures;

- Understanding of Azure PureView for data quality management and data profiling;

- Proficiency in Azure Machine Learning for building and deploying machine learning models in a cloud environment;

- Ability to design and develop scalable, reliable, and efficient data processing solutions on the Azure cloud platform;

- Proficient in data modeling and database design principles to ensure data integrity and optimize query performance;

- Knowledge of data governance and data security best practices to ensure compliance and data privacy;

- Experience with data integration and data transformation techniques to extract, transform, and load (ETL) data from various sources into target data repositories;

- Familiarity with machine learning concepts and techniques to support data scientists in model development and deployment;

- Understanding of cloud computing platforms (Azure and/or AWS) and their services for data engineering and data processing tasks.

Responsibilities:

- Work closely with our engineers and client to develop a high-quality result for our customers;

- Communicating with external web services and collecting date;

- Develop and write efficient, readable code in (ADF/Synapse, SQL, Databricks, Events Hub, Azure Functions, PureView);

- Work with third-party libraries;

- Self-improvement development processes.

Nice to have:

- Proficient in programming Python for scripting and automation in data engineering workflows;

- Strong communication and collaboration skills to work effectively with cross-functional teams and stakeholders;

- Ability to stay up-to-date with the latest trends and advancements in data engineering and cloud technologies.