Senior Web Crawler

Pulse is the FinTech solution redefining financial management for SMEs and advisors.

Our unique visual approach breathes new life into the world of finance, making it more than just numbers and reports. Pulse transforms financial analysis into a vibrant, interactive experience.

Pulse promotes collaborative efforts, opening doors for financial experts to unite in making smarter decisions. Immerse yourself in the innovative realm of financial storytelling with Pulse, where SMEs and advisors find the perfect ally for pushing the boundaries of traditional financial management.

We are seeking a highly skilled and motivated Web Crawler Developer to join our dynamic team. As a Web Crawler Developer, you will play a crucial role in developing and maintaining cutting-edge web crawling solutions to gather data from various online sources. Your work will directly contribute to the success of our data-driven initiatives, providing valuable insights and facilitating informed decision-making processes.

▪ Design, develop, and maintain efficient and scalable web crawling systems to gather data from diverse online sources.

▪ Collaborate with cross-functional teams to understand data requirements and develop customized crawling strategies tailored to specific project needs.

▪ Identify and address potential challenges and limitations in web crawling processes, such as handling dynamic content, managing rate limits, and implementing anti-scraping measures.

▪ Optimize crawling algorithms and implement parallelization techniques to ensure high-speed and reliable data extraction.

▪ Conduct thorough data quality assurance and validation to ensure accuracy, completeness, and consistency of the gathered data.

▪ Monitor and analyse web crawling performance metrics, proactively identify bottlenecks, and implement necessary improvements for enhanced efficiency and scalability.

▪ Stay up-to-date with the latest trends and advancements in web crawling technologies and techniques, and incorporate them into the development process when appropriate.

▪ Collaborate with data engineers and data scientists to integrate crawled data into downstream systems and facilitate data analysis and modelling activities.

▪ Document the crawling architecture, processes, and methodologies, and provide training and support to internal stakeholders to ensure effective utilization of the data gathered.

▪ Bachelor’s or Master’s degree in Computer Science, Software Engineering, or a related field.

▪ Strong experience in developing web crawling systems using Python, Java, or other relevant programming languages.

▪ In-depth knowledge of web technologies, including HTML, CSS, JavaScript, and AJAX, and familiarity with popular web frameworks and libraries.

▪ Proficiency in working with APIs, web services, and databases to extract and store data.

▪ Familiarity with web scraping frameworks and tools such as Scrapy, Beautiful Soup, or Selenium.

▪ Solid understanding of data structures, algorithms, and data processing techniques for efficient and scalable web crawling.

▪ Experience in handling distributed crawling systems and working with cloud-based platforms like AWS or Azure is highly desirable.