OneMain Financial: Databricks Lead Engineer

TheLeadData Engineer Databricks Expert will report to the Enterprise Data Services team andwill building convert legacy data pipelines from SAS to Databricks to help the analytics and data science team in OneMain organization.

The candidate will be central critical role for establishing the databricks infrastructure for building data pipelines, provide best practices and recommendations for using databricks, work closely with data scientists and be the point of contact for all things related to databricks.

Core ResponsibilitiesDesign, develop, implement, and tune large-scale distributed systems and pipelines that process large volume of data; focusing on scalability, low-latency, and fault-tolerance in every system builtDeveloping scalable and re-usable frameworks for ingesting large data in Databricks.

Exposure to delta lake framework.

Databricks engineering query tuning, performance tuning, troubleshooting, and debugging Spark or other big data solutions.Provides technical leadership, fosters a team environment, and provides mentorship and feedback to technical resources.Deep understanding of ETL/ELT design methodologies, architecture, strategy, and tactics for complex ETL solution.Data processing/transformation using various technologies such as spark and cloud Services.Understand current data engineering pipelines using legacy SAS tools and convert to modern pipelines using Databricks.

Qualifications5+ years experience as data engineer.

Must have 2+ Years in implementing data engineering solutions with Databricks.This is hands on role building data pipelines using Databricks.

Hands-on technical experience with Apache Spark.Must have deep expertise in one of the programming languages for data processes (Python, Scala).

Experience with Python, PySpark, Hadoop, Hive and/or Spark to write data pipelines and data processing layersMust have worked with relational databases like Snowflake.

Good SQL experience for writing complex SQL transformation.Performance Tuning of Spark SQL running on S3/Data Lake/Delta Lake/ storage and Strong Knowledge on Databricks and Cluster Configurations.Nice to have Databricks administration including security and infrastructure features of databricks.Experience with Development Tools for CI/CD, Unit and Integration testing, Automation and Orchestration Who We Are:OneMain Financial is the countrys largest lending-exclusive financial company, a trusted name in lending for over 100 years.

Since 2005 alone, we have looked beyond customers credit scores to lend more than $152 billion to 16.2 million people looking for simple, affordable loans.

With branches across 44 states, we’re proud partners of the families and communities we serve.

They turn to us to help meet important financial needs, including debt consolidation, medical expenses, household bills and auto purchases.

Its all about doing the right thing a mission that hasnt changed for more than 100 years.

Benefits:Because we want our team members to bring us their very best every day, we believe they deserve the right opportunities and benefits.

Thats why we packed our comprehensive benefits package for full and some part-timers with:Health and wellbeing options for team members and their dependentsUp to 4% matching 401(k)Tuition reimbursementContinuing educationBonus eligibilityPaid time offPaid volunteer time #LI-DB1