Middle/Senior Data Pipeline Engineer in StartupSoft

Middle/Senior Data Pipeline Engineer Львів, Тернопіль, віддалено About Project Delphi isaresearch group based out ofCarnegie Mellon University dedicated todeveloping the theory and practice ofepidemic tracking and forecasting. Pre-pandemic they worked mostly oninfluenza, dengue and norovirus; they’ve now picked upafocus onCOVID. They procure unique data streams that reflect epidemic (orpandemic) activity, extract relevant indicators, and make these publicly and continuously available. They and others then use these indicators for nowcasting (situational awareness) and short-term forecasting. The main goal ofDelphi Research Group improving the world’s ability todeal with the next pandemic. Tech stack MySQL, Python Project stage Live product Team composition 9full-time engineers (1DevOps, 1database, 2pipeline/statistics/masters, sys admin) + graduate students and professors. The team ismostly inEST Qualifications and skills Proficiency indata pipeline tools Experience with Python and SQL Strong understanding ofdata structures, algorithms, and system design Familiarity with data storage solutions (e.g., databases, data lakes) Excellent problem-solving skills and attention todetail Effective communication skills, both synchronous and asynchronous Responsibilities Monitor and maintain the health and efficiency ofdata pipelines Troubleshoot and perform root cause analysis for data discrepancies and pipeline issues Communicate with data providers tounderstand data discrepancies and manage changes indata delivery Implement fixes and enhancements toimprove data quality and pipeline performance Collaborate with data scientists and analysts tounderstand data needs and implement effective data solutions Develop strategies for data validation and quality assurance Optimize data flow and collection toimprove system efficiency Document and manage data pipeline architectures, including maintenance and update protocols Use tools such asSQL, version control and CI/CD, containerization, task schedulers, python frameworks, and cloud services for data pipeline management Ensure compliance with data governance and security standards Why StartupSoft? Work with top entrepreneurs and startups from Silicon Valley and EU Become anintegral part ofthe core team— bemore than someone who just writes code Potential toreceive equity inhigh-growth startups Contribute towell-funded products with the potential toimpact millions Beautiful offices inthe best locations with high-end amenities Tight-knit team with aninclusive and transparent culture Only the latest, modern technologies with nolegacy code and management bureaucracy Above average salary, benefits &medical insurance