Middle Java Data Engineer with Kafka (DWH) in SoftConstruct Ukraine

13 серпня 2024

Middle Java Data Engineer with Kafka (DWH)

віддалено

SoftConstruct Ukraine is a large product company with Armenian roots.

We develop solutions in the fields of Big Data, Data Science and CryptoExchange. We combine our expertise in iGaming software development with advanced technologies to develop original software such as sports statistics, risk analysis, security. In most cases, we deal with atypical projects for the market, try to find and generate new technology-based solutions in our field.

We support our colleagues who actively participate in all spheres of our country’s defense, as well as help in the development of solutions, technical security support — the information sector of the state. We are ready to offer work for IT specialists who need it.

If you are looking for a new challenging position in  Java specialization, we would be more than happy to offer you our position of Middle Java Engineer .

Project:

Our Data Platform is a part of b2b platform and engineered to deliver good performance, simplicity, concurrency, and cost-effectiveness for data collection, rapid analytics, and extracting data-driven insights for business users.
Built on a Lambda architecture, it adeptly manages vast quantities of data by leveraging both batch and stream-processing methods. This dual approach ensures a balance of latency, throughput, and fault tolerance. Batch processing provides comprehensive and accurate views of large data sets, while real-time stream processing offers immediate insights from online data.

In numbers: - Up to 1 PB of data from 500+ partners in long-term storage - 30+ M new records daily - Latency for insight based on data delivery time <5 sec

The team consists of:
Java engineers, Python engineers, DevOps engineers, Data analysts, Data architect, Project manager, Product manager.

Competencies / Technical Stack:
Kafka Streams, Kafka Connect, GCP, Google Dataflow (Apache Beam), BigQuery, SQL, Kubernetes, Java, DataHub, RabbitMQ, Redis, Neo4j.

Main responsibilities:
— Develop, implement, and automate distributed system deployments for event collection andprocessing from multiple sources;
— Design and manage data schemas;
— Create and maintain Kafka Connectors, Producers, and Consumers for efficient data streaming and processing;
— Optimize the performance of distributed systems to handle large volumes of data efficiently;
— Identify and resolve issues in data pipelines and distributed systems to maintain smooth operation;
— Collaborate with DevOps teams to deploy, monitor, and scale Kafka clusters as required.

Must-haves:
— Excellent analytical and troubleshooting abilities;
— Experience with Kafka technology, with hands-on production experience and a deep understanding of the Kafka architecture and internals of how it works, along with interplay of architectural components: brokers, Zookeeper, producers/consumers, Kafka Connect, KafkaStreams;
— Proficiency in the Schema Registry;
— Understanding of monitoring Kafka infrastructure along with related components (Connectors, KStreams, and other producer/consumer apps);
— Experience with container orchestration tools (e.g., Kubernetes).

Nice to haves:
— Experience with other streaming data processing frameworks (e.g., Spark Streaming, Flink);
— Experience with schema management tools (e.g., Avro, Protobuf);
— Experience with DevSecOps and cloud-native technology with tools such as Docker,Kubernetes, Jenkins, and Terraform.

What can we offer you:
— Interesting and challenging work in a product and data driven company;
— Plenty of opportunities to learn, grow and progress in your career;
— Annual leave: 20 working days per year;
— All paid sick days with a doctor’s note (no limit on the number of times. But for 1 time it is paid up to 2 weeks);
— Registration within DiiaCity (Gig-contract);
— Individual English classes at EnglishDom online school (compensation within 50% of the amount of expenses);
— €450 / year budget for training and / or sports (compensation within 50% of the amount of expenses) after probationary period;
— Free health insurance after a probationary period.

Sounds interesting? Then apply now, don’t wait up!
SoftConstruct is a cool place to work, where you’ll have fun, make friends, and grow professionally.