Objective

An insurance agency recently acquired another firm and needs to merge their customer data, which is currently stored in two separate files.

Task

Combine all rows from both input DataFrames into a single output DataFrame. The schemas for both datasets are identical. Save your unified result as result_df.

File Path

• Dataset 1: /home/interview/customers_1.csv
• Dataset 2: /home/interview/customers_2.csv
• Starter script: /home/interview/merge_customers.py

Schema

Both input DataFrames and the Expected Output share this schema:

Example

Given this sample input:

customers_1

customers_2

The output would be:

from pyspark.sql import SparkSession

spark = SparkSession.builder.appName("PrepareshSpark").getOrCreate()
spark.sparkContext.setLogLevel("ERROR")

df1 = spark.read.csv("/home/interview/customers_1.csv", header=True, inferSchema=True)
df2 = spark.read.csv("/home/interview/customers_2.csv", header=True, inferSchema=True)

result_df = df1.union(df2)

# --- Do not edit below this line ---
result_df.coalesce(1).write.csv("/home/interview/output", header=True, mode="overwrite")
spark.stop()

Explanation

Step 1: Merging with union()

result_df = df1.union(df2)

The union() method combines two DataFrames vertically, appending the rows of df2 to the bottom of df1. For union() to work correctly, both DataFrames must have the exact same number of columns, and the columns must be in the exact same order.

Step 2: PySpark union vs SQL UNION

If you come from a SQL background, you might expect union() to automatically remove duplicate rows. It does not. In PySpark, df1.union(df2) behaves exactly like SQL's UNION ALL. It blindly appends the data, duplicates included. If you need to remove duplicates after merging, you must explicitly call .distinct() like this: df1.union(df2).distinct().

Step 3: A Safer Alternative: unionByName()

While union() resolves columns by their position, PySpark also offers unionByName(), which resolves columns by their names.

result_df = df1.unionByName(df2)

If df1 has (customer_id, age) and df2 has (age, customer_id), union() would mix up the data types, but unionByName() would correctly align the age data with age and customer_id with customer_id. Since our schemas are identical in this problem, both methods produce the exact same result.