Senior/Lead Data Engineer in Quantum

Quantum isaglobal technology partner delivering high-end software products that address real-world problems. Weadvance emerging technologies for outside-the-box solutions. Wefocus onMachine Learning, Computer Vision, Deep Learning, GIS, MLOps, Blockchain, and more.Here atQuantum, weare dedicated tocreating state-of-art solutions that effectively address the pressing issues faced bybusinesses and the world. Todate, our team ofexceptional people has already helped many organizations globally attain technological leadership.Weconstantly discover new ways tosolve never-ending business challenges byadopting new technologies, even when there isn’t yet abest practice. Ifyou share our passion for problem-solving and making animpact, joinus and enjoy getting toknow our wealth ofexperience!About the positionQuantum isexpanding the team and has brilliant opportunities for the Lead Data Engineer. The client isatechnological research company that utilizes proprietary AI-based analysis and language models toprovide comprehensive insights into global stocks inall languages. Our mission istobridge the knowledge gap inthe investment world and empower investors ofall types tobecome “super-investors.”Through our generativeAI technology implemented into brokerage platforms and other financial institutions’ infrastructures, weoffer instant fundamental analyses ofglobal stocks alongside bespoke investment strategies, enabling informed investment decisions for millions ofinvestors worldwide.Must have skills:Atleast 5+years inaData Engineer role orrelevant experience with leading skillsExpertise with Spark/PySparkSolid knowledge ofPython (Java can beanoption, but the codebase isonPython) and Object-oriented Design understandingPractical experience inSpark job performance optimization, knowledge ofSpark Deployment, Manage Resources for Spark jobsUnderstanding the difference between OLAP and OLTP: Operational system vsData processing systemKnowledge ofData model abstractions: data warehouse, lakehouse. And data models: Medallion architecture, start schema, wide tablesFamiliarity with SQL& NoSQL technologiesAtleast anUpper-Intermediate level ofEnglish (spoken and written)Would beaplus:Apache Storm, Apache Flink, GCP Data Flow, Apache beamApache Hadoop, Apache Hive, Snowflake, Dremio, RedshiftAirflow, Prefect, AirbyteGood working knowledge ofAWS Glue orAmazon EMR and related toolsInthe context ofdata analysis experience with Apache Pulsar, Kafka, KinesisPresto/Trino, AthenaUnderstanding ofthe difference between batches and streams. Aswell— the difference between aqueue and adata streamData quality control: data lineage, data quality, etcKnowledge Architecture patterns like lambda, kappa, data meshYour tasks will include:Make architecture for production solutions, suggest solutions for improvementBuilding and maintaining data pipelines for the extraction, transformation, and loading ofdataManaging databases, ensuring data consistency, integrity, and securityDetect and solve performance issues onarchitectural, design, and code levelsDevelop solid unit and integration tests according toorganizational standardsGenerate technical solutions for tasksMake project plans, estimate budgetsCollaborates with designers, developers, and product ownersTake part inpresale, analyze tasks, communicate with customersWeoffer:Delivering high-end software projects that address real-world problemsSurrounding experts who are ready tomove forward professionallyProfessional growth plan and team leader supportTaking ownership ofR&D and socially significant projectsParticipation inworldwide tech conferences and competitionsTaking part inregular educational activitiesBeing apart ofamulticultural company with afun and lighthearted atmosphereWorking from anywhere with flexible working hoursPaid vacation and sick leave daysJoin Quantum and take astep toward your data-driven future.