Databricks Jobs

I run an online learning program that needs a seasoned Databricks professional—someone with at least five years of hands-on experience—to guide a cohort through an intensive data-engineering track. The engagement spans 25 consecutive days, and I will schedule 3-5 live hours each day. If the group makes rapid progress, we may extend slightly to finish capstone work, so a bit of flexibility is important. The focus is exclusively on Data Engineering inside the Databricks ecosystem: workspace setup, Delta Lake, Spark optimizations, job orchestration, and best practices for ingesting, transforming, and serving data at scale. The participants already grasp the fundamentals, so I’m aiming for an intermediate depth—enough to move them from “hello world” noteboo...

Azure Data Architect Consultant (Part-time) Engagement: Part-time contract, 10 to 20 hours per week | Location: Remote (India) | Experience: 9 to 14 years About the role We are looking for an experienced Azure data architect to consult on enterprise data platform engagements alongside our delivery team. You will review architectures, diagnose performance problems and guide our engineers, working directly with client stakeholders when needed. What you will do Review and shape Azure data architectures across ingestion, warehouse or lakehouse layers, modelling and reporting Lead performance and cost optimisation on Azure Synapse dedicated SQL pools: diagnosing slow queries from execution plans, fixing data skew and data movement, and choosing distribution, partitioning and indexing strategies...

I’m looking for hands-on help building and optimising data-engineering workflows inside Databricks. My immediate focus is on ingesting data that arrives in two ways—batch loads from relational databases and real-time streams—then transforming, testing and storing it so downstream analytics teams can trust the output. Here is what I need from you: • Design and implement robust ingestion pipelines that pull incremental data from the source databases and process the streaming feeds with minimal latency. • Develop scalable transformation logic inside notebooks (PySpark / SQL) and package reusable jobs so they can be scheduled and monitored through Databricks Workflows. • Advise on cluster configuration, performance tuning and cost-efficient autoscaling as...