analysis-voidlinux-popcorn/popcorn.qmd

---
title: "Voidlinux popcorn"
subtitle: "Analysis of voidlinux package and kernel statistics"
---

This notebook analyses the daily package repository statistics files,
colloquially known as 'popcorn' files, that are generated by the Void Linux
package manager `xbps` and uploaded by users who have opted in to share.

```{python}
# | echo: false
import os
from typing import Any, Awaitable, Mapping

import lets_plot as lp
import polars as pl
from lets_plot import LetsPlot
from marimo import Cell


def run_cell(cell: Cell) -> tuple[Any, Mapping[str, Any]]:
    ret = cell.run()
    if isinstance(ret, Awaitable):
        raise NotImplementedError
    else:
        output, defs = ret
    return (output, defs)


fig_width, fig_height = (
    int(os.getenv("QUARTO_FIG_WIDTH") or 7),
    int(os.getenv("QUARTO_FIG_HEIGHT") or 5),
)


def pplot(cell: Cell) -> Any:
    outp, _ = run_cell(cell)
    return (
        outp
        + lp.flavor_darcula()
        + lp.ggsize(width=fig_width * 1000, height=fig_height * 1000)
    )


LetsPlot.setup_html()
```

## Daily statistics file size

The simplest operation we can do is look at the overall file size for each of the daily
statistics files over time. The files consist of a long list of packages which have been checked
from the repositories that day, along with the number of package instances. It also consists of
the same list separated by specifically installed versions of packages, so if somebody has
v0.9.1 and somebody else v0.9.3 instead this would count both packages separately.

Another count is the number of different Kernels that have been used on that day, with their
exact kernel name including major version, minor version and any suffix.

These are the major things that will lead to size increases in the file, but not just for an
increased amount of absolute users, packages or uploads --- we will get to those shortly.

No, an increase in file size here mainly suggests an increase in the 'breadth' of files on offer
in the repository, whether that be a wider variety of program versions or more different
packages that people are interested in, and those that the community chooses to use.

So while the overall amount of packages gives a general estimate of the interest in the
distribution, this can show a more 'distributor'-aligned view on how many different aisles of
the buffet people are eating from.

```{python}
# | echo: true
from notebooks.popcorn import plt_filesize
pplot(plt_filesize)
```

As we can see, the difference over time is massive. Especially early on, between 2019 and the
start of 2021, the amount of different packages and package versions used grew rapidly, with the
pace picking up once again starting 2023.

There are a few outlier days with a size of 0 kB, which we will remove from the data. In all
likelihood, those days were not reported correctly or there was some kind of issue on the
backend so the stats for those days are lost.

There are also a few days where the modification date of the file does not correspond to the
represented statistical date but those are kept. This rather points to certain times when the
files have been moved on the backend, or recreated externally but does not mean the data are
bad.

```{python}
from notebooks.popcorn import tab_pkg
outp, defs = tab_pkg.run()
outp
```

## Package statistics

Now that we have an idea of how the overall interest in the distribution has changed over time,
let's look at the actual package statistics.

The popcorn files contain two main pieces of information: the number of installs per package
(e.g. how many people have rsync installed) and the number of unique installs (i.e. unique
machines providing statistics). We will look at both of these in turn.

```{python}
from notebooks.popcorn import plt_weekly_packages
pplot(plt_weekly_packages)
```

```{python}
from notebooks.popcorn import plt_pkg_relative
pplot(plt_pkg_relative)
```

The amount of packages installed on all machines increases strongly over time.

```{python}
from notebooks.popcorn import plt_weekday_packages
pplot(plt_weekday_packages)
```

```{python}
from notebooks.popcorn import plt_month_packages
pplot(plt_month_packages)
```

```{python}
from notebooks.popcorn import plt_top_packages
pplot(plt_top_packages)
```

```{python}
from notebooks.popcorn import plt_package_distribution
pplot(plt_package_distribution)
```

```{python}
from notebooks.popcorn import plt_top_packages
_, defs = plt_top_packages.run()
df_pkg_dl = defs["df_pkg_dl"]
def get_num(df: pl.LazyFrame) -> int:
    return df.count().collect(engine="streaming").item(0, 0)

one_ten_installs = df_pkg_dl.sort("count", descending=False).filter(
    (pl.col("count") >= 1) & (pl.col("count") < 10)
)
ten_twenty_installs = df_pkg_dl.sort("count", descending=False).filter(
    (pl.col("count") >= 10) & (pl.col("count") < 20)
)
twenty_thirty = df_pkg_dl.sort("count", descending=False).filter(
    (pl.col("count") >= 20) & (pl.col("count") < 30)
)
thirty_plus = df_pkg_dl.sort("count", descending=False).filter((pl.col("count") >= 30))
```

There are `{python} f"{get_num(one_ten_installs):,}"` packages which have between one
and ten installations in the data, `{python} f"{get_num(ten_twenty_installs):,}"`
packages between eleven and 20 installations, and
`{python} f"{get_num(twenty_thirty):,}"` packages between 21 and 30 installations.
`{python} f"{get_num(thirty_plus):,}"` packages have over 30 installations.

## Kernel Analysis

```{python}
from notebooks.popcorn import plt_kernel_versions
pplot(plt_kernel_versions)
```

When looking at the kernel versions used, we see a very strong jump between major kernel version
4 and major kernel version 5.

For this analysis we had to exclude {kernel_df_v99.select(pl.len()).item()} rows which were
apparently from the future, as they were running variations of major kernel version 99. In all
likelihood there is a custom kernel version out there which reports its own major version as 99.
The strange version starts appearing on {kernel_df_v99.select("date").row(0)[0]} and shows up
all the way until {kernel_df_v99.select("date").row(-1)[0]}.

```{python}
from notebooks.popcorn import plt_kernel_timeline
pplot(plt_kernel_timeline)
```

```{python}
from datetime import date
from notebooks.popcorn import plt_kernel_timeline
_, defs = plt_kernel_timeline.run()
weekly_kernel_df = defs["weekly_kernel_df"]

last_kernel4: date = weekly_kernel_df.filter(pl.col("major_ver") == "4")[-1][
    "date"
].item()
first_kernel5: date = weekly_kernel_df.filter(pl.col("major_ver") == "5")[0][
    "date"
].item()
last_kernel5: date = weekly_kernel_df.filter(pl.col("major_ver") == "5")[-1][
    "date"
].item()
```

A timeline analysis of the kernels used to report daily downloads shows that people generally
adopt new major kernel versons at roughly the same time. This change is especially stark between
major kernel versions 5 and 6, which seem to have traded place in usage almost over night.

The first time that major version 5 of the kernel shows up is on {first_kernel5}. From here, it
took a long time for the last of the version 4 kernels to disappear, coinciding with the big
switch between major version 5 and 6. The last time a major version 4 is seen is on
{last_kernel4}, while the last major version 5 kernels still pop up.
It would seem, then, that the people still running kernel version 4 used the opportunity of
everybody switching to the stable version of 6 to also upgrade their machines.

## Odds and Ends

There are some missing days in the statistics.

```{python}
from notebooks.popcorn import tab_missing_days

outp, defs = tab_missing_days.run()
outp
```

## Outline

- intro
- filesize
- unique installations reported from
- packages -> perhaps find new subcategories
  - global
  - relative (pkg/unique)
  - top packages
  - rare packages?
  - install distribution
- packages per time unit (find clever title, e.g. 'accumulated packages')
  - per year?
  - weekday
  - month of year (combine with weekday?)
- kernels
  - overall kernel version installations
  - kernels over time

- misc
  - missing days
  - moved days

- things we can't see (limitations)
  - packages on offer in the repositories
    - this could shed light on the bumps of users and relative package ownership

Modified date != descriptive (named) date

```{python}
from notebooks.popcorn import plt_modified_times
pplot(plt_modified_times)
```