PySpark Interview Questions and Answers
Last updated 2026-11-19
PySpark interviews test whether you understand how Spark executes your code, not just the DataFrame API. The same transformation can take two minutes or two hours depending on shuffles, partition counts and data skew, so interviewers ask what happens under the hood and how you would fix a slow or failing job.
The answers below pair explanations with code you can run on a local session. They sit naturally beside data engineer interview questions and the SQL practice in how to crack the SQL interview, since most PySpark tasks are SQL logic in another syntax.
| Typical rounds | PySpark coding, optimisation discussion, pipeline design |
|---|---|
| Core skills | DataFrames, joins, partitioning, skew, window functions |
| Questions here | 14 worked questions |
The mental model
- Transformations are lazy and build a plan; actions trigger execution.
- Narrow transformations (filter, select, map) work within a partition; wide ones (groupBy, join, distinct) need a shuffle across the network.
- A job is split into stages at shuffle boundaries, and each stage into one task per partition.
- Cost comes from shuffles, skew, spilling to disk and too many or too few partitions.
How to answer a performance question
Name the symptom (one long task, spills, out-of-memory), name the likely cause (skew, large shuffle, collect on a big result, huge broadcast), then name the fix and how you would verify it in the Spark UI: stage timeline, task duration distribution and shuffle read and write sizes.
PySpark Interview Questions and Answers: questions with worked answers
1 What is lazy evaluation, and what are transformations and actions?
easyTransformations such as select, filter, join and groupBy do not run immediately; Spark records them in a logical plan. Actions such as count, collect, show and write trigger the optimiser (Catalyst) to build a physical plan and execute it. Laziness lets Spark combine steps, push filters down to the data source and skip unused columns. A side effect is that an error in a transformation may only appear when an action runs, and that each action re-executes the plan unless the data is cached.
2 Explain narrow and wide transformations, and what a shuffle is.
mediumIn a narrow transformation each output partition depends on one input partition, so no data moves between executors: map, filter and select. A wide transformation needs data from many input partitions, such as groupBy, join and distinct, and Spark redistributes rows by key across the cluster, which is a shuffle. A shuffle writes to disk and moves data over the network, so it is the most expensive operation. Each shuffle ends a stage, and reducing the number of shuffles is the main optimisation.
3 DataFrame versus RDD: when would you use each?
easyDataFrames carry a schema and go through the Catalyst optimiser and Tungsten execution engine, so they are faster and more compact for structured data. RDDs are low-level collections of arbitrary objects with no optimiser. Use DataFrames or Spark SQL by default, and RDDs only for unstructured data or custom partitioning logic that the DataFrame API cannot express. Typed Datasets exist in Scala and Java, but not in Python.
4 How does a broadcast join work, and when is it a good idea?
mediumThe driver sends a small table to every executor, which joins it locally against each partition of the large table, so the large table is not shuffled. It is the best choice when one side fits comfortably in executor memory. Spark does it automatically when a table is below spark.sql.autoBroadcastJoinThreshold, which is 10 MB by default, and you can force it with a hint. Broadcasting a table that is too large causes out-of-memory errors on the driver or executors.
pythonfrom pyspark.sql import functions as F joined = orders.join(F.broadcast(countries), on='country_code', how='left') joined.explain() # look for BroadcastHashJoin5 What is data skew and how do you fix it?
hardSkew is when a few keys hold a large share of rows, so one task in a shuffle stage processes far more data than the rest, and the stage waits for that straggler. In the Spark UI you see one task running much longer, and shuffle read sizes that differ by a large factor. Fixes: enable adaptive query execution, which can split skewed partitions in joins; broadcast the smaller side; filter out junk keys such as nulls; or salt the key, which spreads a hot key over N sub-keys.
pythonN = 8 big_s = big.withColumn('salt', (F.rand() * N).cast('long')) small_s = small.crossJoin(spark.range(N).withColumnRenamed('id', 'salt')) joined = big_s.join(small_s, on=['key', 'salt']) # then drop salt6 repartition versus coalesce.
easyrepartition(n) performs a full shuffle and can increase or decrease the partition count, producing evenly sized partitions, and it can partition by columns. coalesce(n) only decreases the number of partitions by merging neighbours without a full shuffle, which is cheaper but can leave uneven sizes. Use coalesce to reduce the number of output files at the end of a job, and repartition to rebalance or to increase parallelism before a heavy stage.
7 How many partitions should a job have? Estimate for 500 GB of data.
mediumA common target is partitions of roughly 100 to 200 MB. For 500 GB, that is 500 x 1024 / 128 = 4,000 partitions at 128 MB. Each partition becomes a task, so with 100 cores you would run about 40 waves of tasks. Too few partitions underuse the cluster and cause memory pressure; too many create scheduling overhead and small files. The default spark.sql.shuffle.partitions is 200, which is too low for big shuffles, though adaptive execution can coalesce small partitions after the shuffle.
8 cache versus persist, and when caching helps.
mediumcache() stores a DataFrame at the default storage level (memory and disk for DataFrames) so that later actions reuse it instead of recomputing the lineage. persist lets you choose the level, such as MEMORY_ONLY or DISK_ONLY. Caching helps when the same expensive result feeds several actions, such as an iterative algorithm, and it is wasteful when data is used once. Caching is lazy, so the first action fills it, and unpersist frees the memory when you are done.
pythonclean = raw.filter('amount > 0').dropDuplicates(['order_id']).cache() clean.count() # materialises the cache daily = clean.groupBy('order_date').sum('amount') clean.unpersist()9 Return each customer's latest order with a window function.
mediumDefine a window partitioned by customer and ordered by timestamp descending, add row_number, and keep the first row. The window triggers a shuffle on the partition key, so make sure the key is not heavily skewed. As in SQL, add a tie-breaker to the ordering so the result is deterministic.
pythonfrom pyspark.sql import Window, functions as F w = Window.partitionBy('customer_id').orderBy(F.col('order_ts').desc(), F.col('order_id').desc()) latest = (orders.withColumn('rn', F.row_number().over(w)) .filter('rn = 1').drop('rn'))10 Why are Python UDFs slow, and what are the alternatives?
mediumA regular Python UDF forces Spark to serialise each row from the JVM to a Python worker and back, one row at a time, and the optimiser cannot see inside it. Prefer built-in functions in pyspark.sql.functions, which run in the JVM and are optimised. When custom Python is needed, a pandas UDF (vectorised, Arrow-based) processes batches of values and is usually far faster. If the logic is common, consider Spark SQL expressions such as when, regexp_extract or transform.
pythonimport pandas as pd from pyspark.sql.functions import pandas_udf @pandas_udf('double') def to_celsius(f: pd.Series) -> pd.Series: return (f - 32) * 5.0 / 9.011 Describe the Spark architecture: driver, executors and cluster manager.
easyThe driver runs your main program, builds the plan and schedules tasks. Executors are JVM processes on worker nodes that run tasks and store cached data. The cluster manager, such as YARN, Kubernetes or standalone, allocates resources. The driver breaks a job into stages and each stage into tasks, one per partition, and sends them to executors. Calling collect on a large DataFrame pulls everything into the driver and is a common cause of driver out-of-memory failures.
12 A Spark job fails with out-of-memory errors. What do you check?
hardIdentify whether the driver or an executor failed. On the driver, look for collect, toPandas or a large broadcast. On executors, look for skewed partitions, huge partitions from too few shuffle partitions, wide rows, or exploding joins that multiply rows. Fixes: increase partitions, handle skew, avoid collecting, select only needed columns early, increase executor memory or memoryOverhead (important for PySpark because Python workers use off-heap memory), and avoid caching more than fits. Confirm in the Spark UI which stage and which tasks spilled.
13 Write a groupBy aggregation with null handling and de-duplication.
easyDrop duplicates on the business key first, replace or exclude nulls deliberately, then aggregate with named outputs. count of a column ignores nulls while count of a literal counts rows. dropDuplicates without a subset compares all columns, and with a subset it keeps an arbitrary row among duplicates, so sort with a window if you need a specific one.
pythonfrom pyspark.sql import functions as F out = (orders.dropDuplicates(['order_id']) .fillna({'discount': 0}) .groupBy('city') .agg(F.count('*').alias('orders'), F.countDistinct('customer_id').alias('customers'), F.round(F.avg('amount'), 2).alias('avg_amount')))14 How do you write partitioned Parquet output and why?
mediumpartitionBy writes one directory per value of the column, such as order_date=2026-11-19, so queries that filter on the date read only the matching directories (partition pruning). Choose a column with moderate cardinality; a high-cardinality column creates a huge number of small files. Repartition by the same column before writing so each date is written by few tasks, which limits file count. Use mode overwrite with dynamic partition overwrite to replace only the partitions you write.
pythonspark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic') (df.repartition('order_date') .write.mode('overwrite') .partitionBy('order_date') .parquet('s3://bucket/orders_clean/'))
Frequently asked questions
Do I need Scala for Spark interviews?
Not for most roles. PySpark and Spark SQL are widely used, and the architecture questions are the same in both languages.
Is Spark SQL knowledge enough?
It covers much of the DataFrame work, but you also need to explain partitions, shuffles, joins, caching and tuning.
How do I practise without a cluster?
Run a local session with pyspark on your machine, generate a few million rows, and use the Spark UI at port 4040 to study stages and shuffles.
What happens after Spark in a data engineering loop?
Questions on Airflow, Kafka and warehouse design usually follow. The data engineer page covers those.
Related guides
More in Role-wise Interview Questions by Job and Tech Stack
- React Hooks Interview Questions and Answers
- React Interview Questions and Answers (React 19)
- React Native Interview Questions and Answers
- Redux Interview Questions and Answers (Redux Toolkit)
- SDET Interview Questions and Answers
- Selenium Interview Questions and Answers
- Site Reliability Engineer (SRE) Interview Questions and Answers
- SOC Analyst Interview Questions and Answers
- Software Engineer Fresher Interview Questions and Answers
- Spring Boot Interview Questions and Answers (Spring Boot 3)
- Statistics for Data Science Interview Questions and Answers
- Tableau Interview Questions and Answers