Data Crawling Lead

Purpose of the Role

Your role will be to build out a data acquisition team as part of a larger product team. You will bring your expertise in data crawling, analytical mind, keen eye for detail and leadership abilities into building a team that will acquire significant amounts of data to feed groundbreaking products redefining enterprise software. This is a startup environment and we expect you to be able to work hands-on as needed and mentor and build a functional team. 

Duties and Responsibilities

• Work closely with the clients to identify and acquire data, crawl information from multiple online data sources
• Define the strategy and tools to be used for web crawlers, web scrapers and other automation tools, to help extract the content
• Lead a team of web crawling engineers and be responsible for their mentorship
• Drive change by staying up to date with the data science trends and technologies

Required Experience & Knowledge

• Previous experience leading a team and defining workload
• Previous hands-on experience working in a web crawling role is a must
• Senior expertise in Java and experience working with high volume web crawling/scraping
• Experience with Java related frameworks such as Spring, Hibernate
• Experience working with open source tools such as ApacheNutch, StormCrawler etc
• Experience working with relational databases such as MySQL, PostgreSQL, Oracle and noSQL databases
• Experience with queuing systems like RabbitMQ / AMQP, Kafka, JMS, Amazon Kinesis
• Experience with JSON, REST API, HTML, XML

Advantage

• Familiarity with supplying data to support machine learning efforts a plus
• Familiarity with Elasticsearch, AWS and the process of scaling and approaches 

Skills and Attributes

• Fluency in English (both written and spoken) is a must
• Client-oriented approach, high integrity and commitment
• Excellent communication and teamwork skills
• Detail-oriented and focused on quality
• Excellent organizational skills

Required Education & Qualifications

• BA or MSc Degree, or relevant experience