CSC507 · Systems & performance · COURSEWORK
Large-File Processing & Concurrency
A study of summing corresponding values from pairs of billion-line text files. The aim was to compare execution strategies and understand the costs of processing large datasets.
How it works
The coursework compares a sequential script, a threaded implementation, and processing individual file chunks. Each approach reads values, calculates sums, and writes output for comparison.
A design decision
Splitting the workload changes both execution and file handling. A useful comparison must account for interpreter overhead, I/O, resource contention, coordination, splitting, and merging—not only the time spent inside an individual worker.
REPORTED FINDINGS
What the experiment showed
The report records 641.23 seconds for sequential processing and 1,417.49 seconds for the threaded implementation. Individual chunks took 63.50–68.06 seconds. The threaded implementation was slower in these reported runs.
Limits of the result
The roughly 68-second combined parallel result in the coursework was an extrapolation from individual chunks. The full concurrent workflow, including splitting and merging, was not established by an end-to-end measurement. No combined parallel speedup is claimed here.
The next iteration
Document the execution environment, verify output values as well as line counts, and benchmark the full concurrent workflow with reproducible inputs and repeated trials. A synthetic-record import pipeline is a possible future application.
These are planned improvements, not current capabilities.