Python + PySpark - Junior Developer Job at Compunnel, Inc., New York, NY

NmNvMnlqM0cvQ2FVeE9HMUp5VU5IZGFIU1E9PQ==
  • Compunnel, Inc.
  • New York, NY

Job Description

The Python + PySpark + AWS Consultant will be responsible for designing, developing, and optimizing data engineering solutions using Python and PySpark. The role requires hands-on experience in building scalable ETL pipelines, working with cloud platforms such as AWS or Azure, and ensuring data integrity and performance optimization. The candidate should have strong SQL skills and a deep understanding of data engineering principles. Key Responsibilities

  • Develop and maintain data processing workflows using Python and PySpark
  • Design and implement ETL pipelines for structured and unstructured data
  • Optimize Spark-based data processing for efficiency and scalability
  • Deploy and manage data solutions on AWS or Azure
  • Write and optimize SQL queries for data transformation and analysis
  • Troubleshoot and resolve performance issues in data pipelines
  • Work closely with cross-functional teams to ensure data reliability and integrity
Required Qualifications
  • 5+ years of experience in data engineering
  • Strong proficiency in Python and object-oriented programming
  • Hands-on experience with PySpark for large-scale data processing
  • Proficiency in SQL for data manipulation and query performance tuning
  • Experience with AWS or Azure for cloud-based data solutions
  • Knowledge of ETL processes and data pipeline automation
  • Experience with Hadoop is acceptable
Preferred Qualifications
  • Experience in optimizing Spark jobs for performance and cost efficiency
  • Familiarity with DevOps practices for data engineering
  • Understanding of data governance, security, and compliance best practices
#J-18808-Ljbffr Compunnel, Inc.

Job Tags

Similar Jobs

Optum

Primary Care Physician | Optum CA Job at Optum

 ...Reliant Medical Group, part of the Optum family of businesses, is seeking a Primary Care Physician to join our team in Rialto, CA. Optum is a clinician-led care organization that is changing the way clinicians work and live. As a member of the Optum Care Delivery... 

EMAPPLY

Bookkeeper (Remote) - EMPCB01 Job at EMAPPLY

This Role is Open to all South AfricansRemote Bookkeeper (full time contractor)Monday to Friday, 9:00 AM - 5:00 PM Eastern Standard Time (New York City Time).30-40 hours per week, depending on agreement with the supervisor.Salary: R17 000 - R25 000 Basic salary per... 

Total Western

Electrician - Foreman (Sparks, NV) Job at Total Western

 ...972, Total-Western, Inc. is a premier provider of industrial construction and maintenance services, supporting a wide range of industries...  ...pursuit of excellence. Position Overview: The Electrician - Foreman provides on-site leadership and supervision for electrical...