Reading Performance

Reading time series data¶

Zarr stores can be read in various ways. Here, however, we focus in using the Xarray library to read and load in-memory large time series over a single geographic location, as per the needs of PVGIS.

How does PVGIS 6 read external time series ?

An error occurred: name 'png' is not defined

Following are various measurements of the time it takes to read and load large time series in-memory, thus replicating the essential step of retrieving data in PVGIS.

Performance measurements¶

A single location¶

⏱ The following table presents the speed of reading complete time series over a single geographic location using ...

Alternative tabular presentation

Poor performance¶

Sometimes, a coincidence of unfavoring factors can lead to poor performance. This can be anything from a saturated number of preocessing requests, an inefficient queuing of processes by the scheduler of the operating system and more.

The following figures is one such bad example with measurements of the time to read the same chunk sizes listed above (i.e. Zarr stores with the same chunk size configuration)

Multiple locations¶

The following experiments, each repeated twice, assess the reading performance over multiple geographic locations, though one after another and not in-parallel. They serve as evidence to support the claim that one of the most important factors that impacts reading performance, is the structural configuration of the time series data itself, independent of the location of a geographic location of interest in space (latitude, longitude).

Metadata of the experiment

• Performed on 1 February 2025
• Data retrieved from Zarr stores
• Stored on an SSD readin at @ 560 MB / s
• XFS File System with a block size of 4096 bytes

Scripted operations using timeit¶

with open('timing_read_operations_from_zarr_stores_again.txt', 'w') as f:
    for file in Path('.').glob('*.zarr'):
        dataset = xr.open_zarr(file)
        chunks = dataset.SIS.encoding['chunks']
        print(f"Dataset : {file},  Chunks : {chunks}", file=f)

        for longitude, latitude in zip(longitude_values, latitude_values):
            print(f' - {longitude=}, {latitude=}', file=f)

            # Measure the execution time for the .sel() command over 100 runs
            times = []
            for _ in range(100):  # Run the selection operation 100 times
                start_time = timeit.default_timer()  # Start the timer
                dataset.SIS.sel(lon=longitude, lat=latitude, method='nearest').load()  # Perform the selection
                elapsed_time = timeit.default_timer() - start_time  # Calculate elapsed time
                times.append(elapsed_time)  # Store the elapsed time

            median_time = np.median(times)  # Calculate the median of the recorded times
            print(f'Median time for {longitude=}, {latitude=}: {median_time:.4f} seconds for 100 runs', file=f)

        print(file=f)  # Print a blank line

Uncompressed data¶

Median reading time of location time series from uncompressed data

Experiment 1

Experiment 2

Compressed data¶

Experiment 1

Experiment 2

Using ipython's timeit magic¶

We repeat the same experiments and measure the average reading time in ms (unless stated otherwise) using ipython's timeit magic.

for file in Path('.').glob('*.zarr'):
    dataset = xr.open_zarr(file)
    chunks = dataset.SIS.encoding['chunks']
    print(f"Dataset : {file},  Chunks : {chunks}")
    for longitude, latitude in zip(longitude_values, latitude_values):
        print(f'- {longitude=}, {latitude=}')
        %timeit -r 100 dataset.SIS.sel(lon=longitude, lat=latitude, method='nearest').load()
    print()

Uncompressed data¶

Compressed data¶

The source "timings" have been edited to make for a cleaner table. Example, some figures have been converted from s to ms and some from μs to ms.