Python Developer: Web Scraping. Scrapy in iPublisher

iPublisher continues to develop. We are relaunching the Clever mobile real estate project. This is a CRM for managing the work of real estate brokers. Part of this project is the collection of data from real estate websites. Currently, we want to expand this project to other cities. As part of the team, you will work on the collection, storage, processing and analysis of huge data sets. The main task will be to develop a parser. The technology stack: Python; Zennopostercapmonster.cloud, know scrapy (the actual tool for parsing) Django; Celery; MySQL; Docker. About the role: As a Python engineer, you will join a team that is passionate about your work, and together you will implement advanced technologies into an easy-to-use application. We have a white parser application. We need to fix some of the broken parsers (for example, Olh), then increase the number of sites from which we parse data. What you will do:—build and troubleshoot web collection tools;—expand data collection sites (parsing); circumvention of protection against parsing—improvement of parsing algorithms; fixing broken parsers—analyzing target websites, including reverse engineering, identifying defenses and using best practices for data acquisition;—creating and maintaining solutions for storing collected information;—analyzing the data you receive, extracting named objects, performing classifications, benchmarking etc. Our goal is to collect data, optimize its storage and use, and be able to analyze it. Working conditions:—part-time;—remote work;—really great team and partnership.