Autocoder

PySpark Interview Questions and Answers

Last updated 2026-11-19

PySpark interviews test whether you understand how Spark executes your code, not just the DataFrame API. The same transformation can take two minutes or two hours depending on shuffles, partition counts and data skew, so interviewers ask what happens under the hood and how you would fix a slow or failing job.

The answers below pair explanations with code you can run on a local session. They sit naturally beside data engineer interview questions and the SQL practice in how to crack the SQL interview, since most PySpark tasks are SQL logic in another syntax.

Typical roundsPySpark coding, optimisation discussion, pipeline design
Core skillsDataFrames, joins, partitioning, skew, window functions
Questions here14 worked questions

The mental model

How to answer a performance question

Name the symptom (one long task, spills, out-of-memory), name the likely cause (skew, large shuffle, collect on a big result, huge broadcast), then name the fix and how you would verify it in the Spark UI: stage timeline, task duration distribution and shuffle read and write sizes.

PySpark Interview Questions and Answers: questions with worked answers

  1. 1

    What is lazy evaluation, and what are transformations and actions?

    easy

    Transformations such as select, filter, join and groupBy do not run immediately; Spark records them in a logical plan. Actions such as count, collect, show and write trigger the optimiser (Catalyst) to build a physical plan and execute it. Laziness lets Spark combine steps, push filters down to the data source and skip unused columns. A side effect is that an error in a transformation may only appear when an action runs, and that each action re-executes the plan unless the data is cached.

  2. 2

    Explain narrow and wide transformations, and what a shuffle is.

    medium

    In a narrow transformation each output partition depends on one input partition, so no data moves between executors: map, filter and select. A wide transformation needs data from many input partitions, such as groupBy, join and distinct, and Spark redistributes rows by key across the cluster, which is a shuffle. A shuffle writes to disk and moves data over the network, so it is the most expensive operation. Each shuffle ends a stage, and reducing the number of shuffles is the main optimisation.

  3. 3

    DataFrame versus RDD: when would you use each?

    easy

    DataFrames carry a schema and go through the Catalyst optimiser and Tungsten execution engine, so they are faster and more compact for structured data. RDDs are low-level collections of arbitrary objects with no optimiser. Use DataFrames or Spark SQL by default, and RDDs only for unstructured data or custom partitioning logic that the DataFrame API cannot express. Typed Datasets exist in Scala and Java, but not in Python.

  4. 4

    How does a broadcast join work, and when is it a good idea?

    medium

    The driver sends a small table to every executor, which joins it locally against each partition of the large table, so the large table is not shuffled. It is the best choice when one side fits comfortably in executor memory. Spark does it automatically when a table is below spark.sql.autoBroadcastJoinThreshold, which is 10 MB by default, and you can force it with a hint. Broadcasting a table that is too large causes out-of-memory errors on the driver or executors.

    python
    from pyspark.sql import functions as F
    joined = orders.join(F.broadcast(countries), on='country_code', how='left')
    joined.explain()   # look for BroadcastHashJoin
  5. 5

    What is data skew and how do you fix it?

    hard

    Skew is when a few keys hold a large share of rows, so one task in a shuffle stage processes far more data than the rest, and the stage waits for that straggler. In the Spark UI you see one task running much longer, and shuffle read sizes that differ by a large factor. Fixes: enable adaptive query execution, which can split skewed partitions in joins; broadcast the smaller side; filter out junk keys such as nulls; or salt the key, which spreads a hot key over N sub-keys.

    python
    N = 8
    big_s = big.withColumn('salt', (F.rand() * N).cast('long'))
    small_s = small.crossJoin(spark.range(N).withColumnRenamed('id', 'salt'))
    joined = big_s.join(small_s, on=['key', 'salt'])   # then drop salt
  6. 6

    repartition versus coalesce.

    easy

    repartition(n) performs a full shuffle and can increase or decrease the partition count, producing evenly sized partitions, and it can partition by columns. coalesce(n) only decreases the number of partitions by merging neighbours without a full shuffle, which is cheaper but can leave uneven sizes. Use coalesce to reduce the number of output files at the end of a job, and repartition to rebalance or to increase parallelism before a heavy stage.

  7. 7

    How many partitions should a job have? Estimate for 500 GB of data.

    medium

    A common target is partitions of roughly 100 to 200 MB. For 500 GB, that is 500 x 1024 / 128 = 4,000 partitions at 128 MB. Each partition becomes a task, so with 100 cores you would run about 40 waves of tasks. Too few partitions underuse the cluster and cause memory pressure; too many create scheduling overhead and small files. The default spark.sql.shuffle.partitions is 200, which is too low for big shuffles, though adaptive execution can coalesce small partitions after the shuffle.

  8. 8

    cache versus persist, and when caching helps.

    medium

    cache() stores a DataFrame at the default storage level (memory and disk for DataFrames) so that later actions reuse it instead of recomputing the lineage. persist lets you choose the level, such as MEMORY_ONLY or DISK_ONLY. Caching helps when the same expensive result feeds several actions, such as an iterative algorithm, and it is wasteful when data is used once. Caching is lazy, so the first action fills it, and unpersist frees the memory when you are done.

    python
    clean = raw.filter('amount > 0').dropDuplicates(['order_id']).cache()
    clean.count()                 # materialises the cache
    daily = clean.groupBy('order_date').sum('amount')
    clean.unpersist()
  9. 9

    Return each customer's latest order with a window function.

    medium

    Define a window partitioned by customer and ordered by timestamp descending, add row_number, and keep the first row. The window triggers a shuffle on the partition key, so make sure the key is not heavily skewed. As in SQL, add a tie-breaker to the ordering so the result is deterministic.

    python
    from pyspark.sql import Window, functions as F
    w = Window.partitionBy('customer_id').orderBy(F.col('order_ts').desc(), F.col('order_id').desc())
    latest = (orders.withColumn('rn', F.row_number().over(w))
                    .filter('rn = 1').drop('rn'))
  10. 10

    Why are Python UDFs slow, and what are the alternatives?

    medium

    A regular Python UDF forces Spark to serialise each row from the JVM to a Python worker and back, one row at a time, and the optimiser cannot see inside it. Prefer built-in functions in pyspark.sql.functions, which run in the JVM and are optimised. When custom Python is needed, a pandas UDF (vectorised, Arrow-based) processes batches of values and is usually far faster. If the logic is common, consider Spark SQL expressions such as when, regexp_extract or transform.

    python
    import pandas as pd
    from pyspark.sql.functions import pandas_udf
    
    @pandas_udf('double')
    def to_celsius(f: pd.Series) -> pd.Series:
        return (f - 32) * 5.0 / 9.0
  11. 11

    Describe the Spark architecture: driver, executors and cluster manager.

    easy

    The driver runs your main program, builds the plan and schedules tasks. Executors are JVM processes on worker nodes that run tasks and store cached data. The cluster manager, such as YARN, Kubernetes or standalone, allocates resources. The driver breaks a job into stages and each stage into tasks, one per partition, and sends them to executors. Calling collect on a large DataFrame pulls everything into the driver and is a common cause of driver out-of-memory failures.

  12. 12

    A Spark job fails with out-of-memory errors. What do you check?

    hard

    Identify whether the driver or an executor failed. On the driver, look for collect, toPandas or a large broadcast. On executors, look for skewed partitions, huge partitions from too few shuffle partitions, wide rows, or exploding joins that multiply rows. Fixes: increase partitions, handle skew, avoid collecting, select only needed columns early, increase executor memory or memoryOverhead (important for PySpark because Python workers use off-heap memory), and avoid caching more than fits. Confirm in the Spark UI which stage and which tasks spilled.

  13. 13

    Write a groupBy aggregation with null handling and de-duplication.

    easy

    Drop duplicates on the business key first, replace or exclude nulls deliberately, then aggregate with named outputs. count of a column ignores nulls while count of a literal counts rows. dropDuplicates without a subset compares all columns, and with a subset it keeps an arbitrary row among duplicates, so sort with a window if you need a specific one.

    python
    from pyspark.sql import functions as F
    out = (orders.dropDuplicates(['order_id'])
                 .fillna({'discount': 0})
                 .groupBy('city')
                 .agg(F.count('*').alias('orders'),
                      F.countDistinct('customer_id').alias('customers'),
                      F.round(F.avg('amount'), 2).alias('avg_amount')))
  14. 14

    How do you write partitioned Parquet output and why?

    medium

    partitionBy writes one directory per value of the column, such as order_date=2026-11-19, so queries that filter on the date read only the matching directories (partition pruning). Choose a column with moderate cardinality; a high-cardinality column creates a huge number of small files. Repartition by the same column before writing so each date is written by few tasks, which limits file count. Use mode overwrite with dynamic partition overwrite to replace only the partitions you write.

    python
    spark.conf.set('spark.sql.sources.partitionOverwriteMode', 'dynamic')
    (df.repartition('order_date')
       .write.mode('overwrite')
       .partitionBy('order_date')
       .parquet('s3://bucket/orders_clean/'))

Frequently asked questions

Do I need Scala for Spark interviews?

Not for most roles. PySpark and Spark SQL are widely used, and the architecture questions are the same in both languages.

Is Spark SQL knowledge enough?

It covers much of the DataFrame work, but you also need to explain partitions, shuffles, joins, caching and tuning.

How do I practise without a cluster?

Run a local session with pyspark on your machine, generate a few million rows, and use the Spark UI at port 4040 to study stages and shuffles.

What happens after Spark in a data engineering loop?

Questions on Airflow, Kafka and warehouse design usually follow. The data engineer page covers those.

Related guides

More in Role-wise Interview Questions by Job and Tech Stack

See all in this section