Lead Data Software Engineer in EPAM Systems

Our client, a world-renowned multinational technology company located in California and one of the Big 5 technology companies, is expanding its partnership with EPAM. Client specialization: consumer electronics software online services You will work with world-class experts and technologies from the heart of Silicon Valley, developing products and services used by billions of people every day. We are looking for an experienced Lead Software Data Engineer to join our team Skills 5+ years of hands-on experience with Big Data: Hadoop (HDFS, YARN or Mesos), SparkKnowledge and hands-on experience with SQL in a Big Data context : Spark SQL, Hive QLKnowledge of SparkOptimization of Spark execution plans using Spark UIGood knowledge of Scala, including Scala functional libraries (Cats)Knowledge of the basics of batch processing and ETL principles in data warehousesExperience with data orchestration and knowledge of data completeness conditionsExperience with data cleaning (handling corrupted, incorrect and duplicated data)Knowledge of migration of schemas and evolution of data setsProficiency in spoken and written English at least B2 levelAbility to quickly learn and learn a new set of tools and technologies used in the project: service platforms, telemetry providers, Spark-as-a-Service, system collection and much more Responsibilities Development of data pipelines, tasks for creating new output datasets as part of the development of new functionality Adaptation of existing data pipelines for integration with various platforms, tools and services Editing of errors in the code and data that were generated due to incorrect logic or implementation Rapid data analysis and research to help the product management team Monitoring and eliminating production problems with existing data pipelines Development and implementation of quality data checks (quality control) Planning the scope of new developments: estimating the required resources, planning the time frames for tasks manage, in all aspects: code quality, complexity, maintenance, documentation Communicate with other data engineers on the team, colleagues and project managers regarding status, issues, estimates, time frames, etc. will be an advantage Understanding of functional programming ideas and principles Experience writing and using web services Experience working with Teradata, Vertica, Oracle or TableauExperience with Spark Streaming and KafkaExperience or knowledge of the following technologies: Apache Iceberg, Trino (Presto), Druid, Cassandra, Blob storage, such as AWSExperience with SplunkKnowledge or experience with Azkaban or AirflowWe are you you can Work according to a flexible schedule remotely or in any of our comfortable offices or co-working spaces in Ukraine Get the necessary equipment to perform work tasks Change projects and the technology stack within EPAM Get experience in various business domains (Insurance, E-commerce, Healthcare, Finance, Traveling, Media , Artificial Intelligence and others) Consider options for relocation to more than 30 countries of the world Join volunteer, charity programs and communities (as pertechnical directions, as well as by interests) We will help you in your professional development Plan your individual career path together with the manager Get regular feedback from colleagues Improve your English for free with certified teachers (Speaking Clubs, courses on preparation for client interviews, etc.) Get the opportunity to undergo free training and certification with AWS, GCP or Azure Clouds Use the internal training program E-learn (18,200+ specialized training and mentoring programs) Get access to corporate accounts on LinkedIn Learning, Get Abstract and other partner resources Learn at Solution Architecture School with by practicing architects as teachers Develop as a leader, join Delivery Management, Resource Management, Leadership Essentials school and others Get taught in internal communities (500+ meetings, technical discussions, brainstorming sessions, online events and conferences every year) What we offer Vacation and sick days (including without a medical certificate) A wide range of insurance programs of the Voluntary Medical Insurance (VMI) with the provision of both medical treatment and various preventive options (including sports activities) Issuance of medical insurance for family members at corporate rates Support from the company during significant life events events (birth or adoption of a child, marriage, etc.) Psychological comfort support: discounts on the services of mental health specialists or coaches, thematic trainings E-kids - a free training program for children of EPAM-ers in programming languages ​​Pleasant corporate gifts About EPAMERAM strives to provide its global team with more than 60,000+ professionals in more than 45 countries, opportunities for professional growth from the first day of cooperation. Our colleagues are the source of ERAM's success, so we value cooperation, always try to understand our clients' business and strive for the highest quality standards. No matter where you are, you'll join a dedicated, diverse community that will help you reach your full potential Show more Show less Position level Senior secondary level Type of employment Full-time Job responsibilities Engineering and Information Technologies Industries IT services and IT Consulting