Parallel K-means

For this project, I investigated how to choose the number of K-means clusters based on their stability when the data changes. The analysis used 45,152 survey respondents and 468 features, with repeated subsampling for cluster counts from two to ten.

I implemented a Jaccard similarity calculation that counts shared pairs without constructing a full pairwise matrix, giving expected linear runtime. I used Numba to compile the calculation and joblib to parallelize 1,800 K-means fits on Berkeley's SCF cluster through Slurm. Three clusters gave the most stable solution under the chosen setup.

This project was created as Lab 3 in the course "Applied Statistics and Machine Learning" (STAT 215A) at UC Berkeley in fall 2026.

If you want to learn more about the analysis, you can find the report here. You can also download the code here.

Below are the similarity distributions for the different cluster counts, based on 100 subsample pairs per count.