Objective

A background check company has a database of the checks they have performed. They want to process this data for further analysis. You are given a DataFrame background_checks that stores these checks.

Task

The criminal_record, employment_history, education_history and address fields are comma-separated varchar fields that hold multiple pieces of information.

Write a PySpark function that transforms the background_checks DataFrame to count the number of items in each of those comma-separated fields. Create the following new columns:

• crime_count: The count of crimes in the criminal_record column.
• jobs_count: The count of jobs in the employment_history column.
• degrees_count: The count of degrees in the education_history column.
• places_lived_count: The count of addresses in the address column.

Save your resulting DataFrame as result_df. Ensure the output exactly matches the requested Output Schema, and order the final output by check_id in ascending order.

File Path

• Background Checks Dataset: /home/interview/background_checks.csv
• Starter script: /home/interview/parse_checks.py

Schema

background_checks.csv

Expected Output Schema

Example

Given this sample input:

background_checks

The expected output would be: