Databricks Engineer

Responsibilities:

• Design and deploy large-scale data processing frameworks and pipelines on Databricks to support enterprise-wide BI and data science initiatives.
• Create MLOps frameworks for model development, training, deployment, and monitoring.
• Optimize Databricks clusters for performance, scalability, and resource efficiency.
• Monitor cluster health, performance metrics, and resource usage, adjusting cluster sizes as needed.
• Design efficient data processing systems and pipelines using Databricks, APIs, and other cloud services.
• Act as the Databricks expert and uphold platform standards.
• Collaborate with Data and Platform architects on defining the product roadmap.
• Provide user training and data coaching for effective Databricks usage.

Qualifications

• Bachelor's degree in Computer Science or related field.
• 7+ years of IT experience, including 5+ years in Databricks development and administration, Spark, and cloud platforms (preferably Azure).
• Proficiency in Databricks admin and development, with expertise in Resource and Performance Management.
• Experience with Kafka and Spark, and strong in Python and Pyspark.
• Hands-on experience in machine learning pipelines and scripting languages.
• Familiarity with Java, Python, SQL/NoSQL databases, and Big Data tools.
• Passion for new tech, continuous improvement, and collaborative problem-solving.
• Expertise in test automation, continuous integration, Docker, GitHub, Jenkins, and Unix/Linux.
• Proactive, strong work ethic, and team player.
• Leadership in large project initiatives, willingness to mentor, and support for team members from different timezones.