Lead Data Engineer – AWS Python

Job Description

US citizens, Green Card Holders, and those authorized to work in the US are encouraged to apply. We are unable to sponsor H1b candidates at this time

Job Title: Lead Data Engineer – AWS Python

Location: Holmdel, NJ, twice a month

Duration: Full-time

Job Description:

As the Data Engineering Lead, you will play a critical role in leading and contributing to our data engineering competency to ensure the highest quality, integrity, efficiency, and scalability of our data services and infrastructure. This position requires a hands-on leader with a passion for data excellence, a deep technical understanding, and the ability to guide and mentor a team to provide data as a platform service to the broader organization and customers.

Responsibilities:

* Data Architecture: Lead the design, implementation, and operation of robust data architecture, ETL/ELT pipelines, and data warehousing strategies using Airflow and AWS cloud services.
* Team Management and Leadership: Manage a small team of Data and Business Intelligence engineers, fostering a collaborative and results-driven environment. Provide mentorship and encourage growth. Manage data expenses. Develop and report KPIs.
* Technical Expertise and Contribution: Be an individual contributor for our data projects. Provide hands-on guidance and expertise to the team in Python, SQL, and AWS cloud services, ensuring high-quality code and software engineering best practices. Help manage and pay off tech debt.
* Data and BI Services and Integration: Collaborate with product engineers and internal data stakeholders to make application data available for product features and work closely with business analysts to orchestrate Business Intelligence reporting using Looker.
* Quality Standards: Uphold and refine data quality standards, engineering best practices, and KPIs while optimizing data pipelines for performance, scalability, and data quality.
* Innovation: Challenge existing structures and thought processes, staying updated with the latest developments in technology, and recommending innovative solutions.
* Communication: Clearly and concisely communicate ideas to both technical and non-technical stakeholders, sharing knowledge and expertise both upstream and downstream.
* Cloud Infrastructure Optimization: Work closely with DevOps to optimize cloud infrastructure performance and scalability and help to save costs.
* Code Reviews: Participate in code reviews to ensure code quality and knowledge sharing among team members.
* Documentation: Maintain comprehensive documentation for data processes and systems. Establish and enforce documentation best practices.
* Testing: Implement robust testing frameworks for data pipelines and quality assurance. Increase test coverage across all data engineering projects. Develop automated testing scripts for data integrity and performance.
* Monitoring: Design and implement logging and monitoring systems for data workflows. Set up real-time alerting mechanisms for data pipeline issues.
* Data Governance, Compliance, and Security: Manage and maintain internal and external security best practices. Ensure compliance with Data Usage license agreements
Qualifications:

* Experience: 6+ years of experience as a Data Engineer with increasing responsibility and a strong track record of technical leadership.
* Technical Skills: Advanced skills in Python 3 (5+ years recent), SQL (5+ years recent), and data cloud services (AWS preferred, 3+ years). Extensive experience with data pipeline development (batch processing), management, data warehousing (Redshift, Postgres), and orchestration (Airflow) is a must. Experience working heavily with analytics and BI reports.
* Leadership: 2+ years of experience in a team leadership role
* Passion: Passionate about the latest developments in technology, with a commitment to data quality and high personal code/development standards.
* Team Player: A friendly, collaborative, and passionate team player who can adapt to a fast-paced environment and is experienced with Scrum/Agile methodologies. Can both give and receive respectful, constructive feedback. Committed to prioritizing team success over personal gain.
* Leveraging Partnerships: Experienced in evaluating and executing strategic outsourcing opportunities as a complement to in-house capabilities, ensuring optimal resource utilization without compromising team integrity or project quality.
* Continuous Learning: A commitment to staying updated with industry trends and emerging technologies.
* Communication: Excellent communication skills, with the ability to articulate ideas and solutions effectively.
Nice to Have:

* Working knowledge of Kubernetes and Docker
* Experience with AWS Athena & AWS GLUE
* Experience with AWS RDS Aurora
* Experience with Snowflake or GCP Big Query

A reasonable, good-faith estimate of the minimum and maximum for this position is $175K/year to $190K/year with benefits