Data Engineer

Job Description:

Responsible for performing more complex analysis and modeling for multiple products with the goal of maximizing profits and asset growth and minimizing risk and operating losses and/or other financial and marketing exposures. Develops complex program models to extract data and uses multiple databases to acquire statistical and financial data.

Utilizes portfolio trends to propose policy/procedural changes within segmentation structure to produce optimal results. Excels at risk/reward trade off. Build relationships with external agencies.

Participates in the rollout of company-wide pilot programs developed as a result of programmed models. Duties primarily include the regular use of discretion, independent judgment, the ability to communicate with multiple levels of management and the utilization of core leadership behaviors.

SAS and/or MicroStrategy experience preferred; 2-5 years experience preferred.

• Have strong technical foundation with in-depth knowledge in Big Data Hadoop, Data Reporting, Data Design, Data Analysis, Data governance, Data integration and Data quality.
• Experience in setting, configuring and monitoring of Hadoop cluster of Cloudera, Hortonworks distribution.
• Deep and extensive knowledge with HDFS, Spark, MapReduce, Hive, HBase, Sqoop, Yarn, Oozie.
• Thorough knowledge on Hadoop architecture and various components such as HDFS, Name Node, Data Node, Application Master, Resource Manager, Node Manager, Job Tracker, Task Tracker and MapReduce programming paradigm.
• Good understanding on Hadoop MR1 and MR2 (YARN) Architecture.
• Experience in developing scalable solutions using NoSQL databases including HBASE and COSMOS DB.
• Efficient in working with Hive data warehouse tool creating tables, data distributing by implementing Partitioning and Bucketing strategy, writing and optimizing the HiveQL queries.
• Experienced in performing analytics on structured data using Hive queries, operations, Joins, tuning queries, SerDe’s and UDF.
• Good experience working with different Hadoop file formats like Sequence File, RCFile, ORC, AVRO and Parquet.
• Experience in using modern Big-Data tools like SparkSQL to convert schema-less data into more structured files for further analysis. Experience in Spark Streaming to receive real time data and store the stream data into HDFS.
• Good Knowledge on Hadoop Cluster architecture and monitoring the cluster.
• In-depth understanding of Data Structure and Algorithms.
• Experience in managing and reviewing Hadoop log files.
• Excellent understanding and knowledge of NOSQL databases like HBase.
• Implemented in setting up standards and processes for Hadoop based application design and implementation.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
• Experience in working with Apache Sqoop to import and export data to and from HDFS and Hive.
• Good working experience in designing Oozie workflows for cleaning data and storing into Hive tables for quick analysis.
• Primarily responsible for designing, implementing, Testing, and maintaining database solution for Azure.
• Primarily involved in Data Migration process using Azure by integrating with Github repository and Jenkins.
• Hands on experience with Real time streaming using Kafka, Spark streaming into HDFS.
• Developed analytical components using SparkSql and Spark Stream.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Spark SQL using Scala.
• Good knowledge streaming data using Flume and Kafka from multiple sources into HDFS.
• Knowledge of processing and analyzing real-time data streams/flows using Kafka and HBase.
• Experience with Informatica Power Center Big Data Edition (BDE) for high-speed Data Ingestion and Extraction.
• Hands on experience with Amazon EMR, Cloudera (CDH4 & CDH5), and Horton Works Hadoop Distributions
• Proficient in all phases of software development including design, configuration, testing, debugging, implementation, release, and support of large-scale, Pega platform applications. Candidate must have 2-5 years of experience building and implementing model-driven, enterprise-level business solutions using Pega PRPC.  Qualified candidate will also need to be able to interface closely with our data science team on deeper analytical needs. The individual is also expected to be able to communicate with multiple levels of management and able to succinctly summarize complex and highly technical requests.

Required Job Skills:
• Proficiency with Hadoop, MapReduce, HDFS, Sqoop, Hive/Impala
• Hands on experience working with extracting and integrating large disconnected and unstructured datasets
• Experience in building processes that support data transformations and data modelling
• Intermediate knowledge in Python – having used Python for scripting, transformation and analysis of data
• Hands on experience with Spark/Pyspark

Job Band:

H5

Shift: 

1st shift (United States of America)

Hours Per Week:

40

Weekly Schedule:

Referral Bonus Amount:

0 –>

Job Description:

Responsible for performing more complex analysis and modeling for multiple products with the goal of maximizing profits and asset growth and minimizing risk and operating losses and/or other financial and marketing exposures. Develops complex program models to extract data and uses multiple databases to acquire statistical and financial data.

Utilizes portfolio trends to propose policy/procedural changes within segmentation structure to produce optimal results. Excels at risk/reward trade off. Build relationships with external agencies.

Participates in the rollout of company-wide pilot programs developed as a result of programmed models. Duties primarily include the regular use of discretion, independent judgment, the ability to communicate with multiple levels of management and the utilization of core leadership behaviors.

SAS and/or MicroStrategy experience preferred; 2-5 years experience preferred.

• Have strong technical foundation with in-depth knowledge in Big Data Hadoop, Data Reporting, Data Design, Data Analysis, Data governance, Data integration and Data quality.
• Experience in setting, configuring and monitoring of Hadoop cluster of Cloudera, Hortonworks distribution.
• Deep and extensive knowledge with HDFS, Spark, MapReduce, Hive, HBase, Sqoop, Yarn, Oozie.
• Thorough knowledge on Hadoop architecture and various components such as HDFS, Name Node, Data Node, Application Master, Resource Manager, Node Manager, Job Tracker, Task Tracker and MapReduce programming paradigm.
• Good understanding on Hadoop MR1 and MR2 (YARN) Architecture.
• Experience in developing scalable solutions using NoSQL databases including HBASE and COSMOS DB.
• Efficient in working with Hive data warehouse tool creating tables, data distributing by implementing Partitioning and Bucketing strategy, writing and optimizing the HiveQL queries.
• Experienced in performing analytics on structured data using Hive queries, operations, Joins, tuning queries, SerDe’s and UDF.
• Good experience working with different Hadoop file formats like Sequence File, RCFile, ORC, AVRO and Parquet.
• Experience in using modern Big-Data tools like SparkSQL to convert schema-less data into more structured files for further analysis. Experience in Spark Streaming to receive real time data and store the stream data into HDFS.
• Good Knowledge on Hadoop Cluster architecture and monitoring the cluster.
• In-depth understanding of Data Structure and Algorithms.
• Experience in managing and reviewing Hadoop log files.
• Excellent understanding and knowledge of NOSQL databases like HBase.
• Implemented in setting up standards and processes for Hadoop based application design and implementation.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
• Experience in working with Apache Sqoop to import and export data to and from HDFS and Hive.
• Good working experience in designing Oozie workflows for cleaning data and storing into Hive tables for quick analysis.
• Primarily responsible for designing, implementing, Testing, and maintaining database solution for Azure.
• Primarily involved in Data Migration process using Azure by integrating with Github repository and Jenkins.
• Hands on experience with Real time streaming using Kafka, Spark streaming into HDFS.
• Developed analytical components using SparkSql and Spark Stream.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Spark SQL using Scala.
• Good knowledge streaming data using Flume and Kafka from multiple sources into HDFS.
• Knowledge of processing and analyzing real-time data streams/flows using Kafka and HBase.
• Experience with Informatica Power Center Big Data Edition (BDE) for high-speed Data Ingestion and Extraction.
• Hands on experience with Amazon EMR, Cloudera (CDH4 & CDH5), and Horton Works Hadoop Distributions
• Proficient in all phases of software development including design, configuration, testing, debugging, implementation, release, and support of large-scale, Pega platform applications. Candidate must have 2-5 years of experience building and implementing model-driven, enterprise-level business solutions using Pega PRPC.  Qualified candidate will also need to be able to interface closely with our data science team on deeper analytical needs. The individual is also expected to be able to communicate with multiple levels of management and able to succinctly summarize complex and highly technical requests.

Required Job Skills:
• Proficiency with Hadoop, MapReduce, HDFS, Sqoop, Hive/Impala
• Hands on experience working with extracting and integrating large disconnected and unstructured datasets
• Experience in building processes that support data transformations and data modelling
• Intermediate knowledge in Python – having used Python for scripting, transformation and analysis of data
• Hands on experience with Spark/Pyspark

Job Band:

H5

Shift: 

1st shift (United States of America)

Hours Per Week:

40

Weekly Schedule:

Referral Bonus Amount:

0

Job Description: Responsible for performing more complex analysis and modeling for multiple products with the goal of maximizing profits and asset growth and minimizing risk and operating losses and/or other financial and marketing exposures. Develops complex program models to extract data and uses multiple databases to acquire statistical and financial data.

Utilizes portfolio trends to propose policy/procedural changes within segmentation structure to produce optimal results. Excels at risk/reward trade off. Build relationships with external agencies.

Participates in the rollout of company-wide pilot programs developed as a result of programmed models. Duties primarily include the regular use of discretion, independent judgment, the ability to communicate with multiple levels of management and the utilization of core leadership behaviors.

SAS and/or MicroStrategy experience preferred; 2-5 years experience preferred.

• Have strong technical foundation with in-depth knowledge in Big Data Hadoop, Data Reporting, Data Design, Data Analysis, Data governance, Data integration and Data quality.
• Experience in setting, configuring and monitoring of Hadoop cluster of Cloudera, Hortonworks distribution.
• Deep and extensive knowledge with HDFS, Spark, MapReduce, Hive, HBase, Sqoop, Yarn, Oozie.
• Thorough knowledge on Hadoop architecture and various components such as HDFS, Name Node, Data Node, Application Master, Resource Manager, Node Manager, Job Tracker, Task Tracker and MapReduce programming paradigm.
• Good understanding on Hadoop MR1 and MR2 (YARN) Architecture.
• Experience in developing scalable solutions using NoSQL databases including HBASE and COSMOS DB.
• Efficient in working with Hive data warehouse tool creating tables, data distributing by implementing Partitioning and Bucketing strategy, writing and optimizing the HiveQL queries.
• Experienced in performing analytics on structured data using Hive queries, operations, Joins, tuning queries, SerDe’s and UDF.
• Good experience working with different Hadoop file formats like Sequence File, RCFile, ORC, AVRO and Parquet.
• Experience in using modern Big-Data tools like SparkSQL to convert schema-less data into more structured files for further analysis. Experience in Spark Streaming to receive real time data and store the stream data into HDFS.
• Good Knowledge on Hadoop Cluster architecture and monitoring the cluster.
• In-depth understanding of Data Structure and Algorithms.
• Experience in managing and reviewing Hadoop log files.
• Excellent understanding and knowledge of NOSQL databases like HBase.
• Implemented in setting up standards and processes for Hadoop based application design and implementation.
• Experience in importing and exporting data using Sqoop from HDFS to Relational Database Systems and vice-versa.
• Experience in working with Apache Sqoop to import and export data to and from HDFS and Hive.
• Good working experience in designing Oozie workflows for cleaning data and storing into Hive tables for quick analysis.
• Primarily responsible for designing, implementing, Testing, and maintaining database solution for Azure.
• Primarily involved in Data Migration process using Azure by integrating with Github repository and Jenkins.
• Hands on experience with Real time streaming using Kafka, Spark streaming into HDFS.
• Developed analytical components using SparkSql and Spark Stream.
• Involved in converting Hive/SQL queries into Spark transformations using Spark RDDs, Spark SQL using Scala.
• Good knowledge streaming data using Flume and Kafka from multiple sources into HDFS.
• Knowledge of processing and analyzing real-time data streams/flows using Kafka and HBase.
• Experience with Informatica Power Center Big Data Edition (BDE) for high-speed Data Ingestion and Extraction.
• Hands on experience with Amazon EMR, Cloudera (CDH4 & CDH5), and Horton Works Hadoop Distributions
• Proficient in all phases of software development including design, configuration, testing, debugging, implementation, release, and support of large-scale, Pega platform applications. Candidate must have 2-5 years of experience building and implementing model-driven, enterprise-level business solutions using Pega PRPC.  Qualified candidate will also need to be able to interface closely with our data science team on deeper analytical needs. The individual is also expected to be able to communicate with multiple levels of management and able to succinctly summarize complex and highly technical requests.

Required Job Skills:
• Proficiency with Hadoop, MapReduce, HDFS, Sqoop, Hive/Impala
• Hands on experience working with extracting and integrating large disconnected and unstructured datasets
• Experience in building processes that support data transformations and data modelling
• Intermediate knowledge in Python – having used Python for scripting, transformation and analysis of data
• Hands on experience with Spark/Pyspark Shift:

1st shift (United States of America)

Hours Per Week: 

40