---
title: "Voidlinux popcorn"
subtitle: "Analysis of voidlinux package and kernel statistics"
toc: true
---

This notebook analyses the daily package repository statistics files,
colloquially known as 'popcorn' files, that are generated by the Void Linux
tool [`PopCorn`](https://voidlinux.org/packages/?arch=x86_64&q=popcorn) and
uploaded by users who have opted in to share with it.

The tool collects a variety of statistics from the user which it anonymizes
(you only get a uuid and to ensure your stats are only uploaded once, and
that's it) and mixes with all other daily uploaded statistics on the server.
The result is an overview of the package and kernel ecosystem of the Void Linux
distribution collected from a multitude of unique installations.

Before we get into the pretty graphs I would like to point out that you too can
participate in this collection, and thereby help the distribution maintainers
know which packages are more relevant or, perhaps more important, less relevant
for end users.

To upload your own statistics anonymously, simply `xbps-install PopCorn` and
enable its service to run in the background with `ln -s /etc/sv/popcorn
/var/service`.

```{python}
# | echo: false
import os
from typing import Any, Awaitable, Mapping

import lets_plot as lp
import polars as pl
from lets_plot import LetsPlot
from marimo import Cell


def run_cell(cell: Cell) -> tuple[Any, Mapping[str, Any]]:
    ret = cell.run()
    if isinstance(ret, Awaitable):
        raise NotImplementedError
    else:
        output, defs = ret
    return (output, defs)


fig_width, fig_height = (
    int(os.getenv("QUARTO_FIG_WIDTH") or 7),
    int(os.getenv("QUARTO_FIG_HEIGHT") or 5),
)


def pplot(cell: Cell) -> Any:
    outp, _ = run_cell(cell)
    return (
        outp
        + lp.flavor_darcula()
        + lp.ggsize(width=fig_width * 1000, height=fig_height * 1000)
    )


LetsPlot.setup_html()
```

## Daily statistics file size

Before we take an in-depth look at the packages, let's quickly get an overview
of the stats files themselves. These are the raw `JSON` files that we are using
throughout the rest of the article, reporting installed packages and kernels,
and their respective versions.

A look at the overall file size for each of the daily statistics files over
time reveals not necessarily changes in the absolute use of packages (e.g.
`rsync` being installed more or less often). Whether it has been downloaded
once or 100 times, the file size does not change drastically. Instead it
increases much more drastically when both `rsync` and `rclone` are installed,
or a variety of different versions for one of the packages are installed.
Similarly for different versions of the kernel.

An increase in file size here mainly suggests an increase in the 'breadth' of
files on offer in the repository, whether that be a _wider variety_ of packages
or more different package _versions_ that people are interested in, and that
the community chooses to use.

So while the overall amount of packages and installations --- we'll get to those
--- gives a general estimate of the interest in the distribution, we see at a
glance a more 'user-choice'-aligned view of how many aisles the buffet offers,
and how many of those the users are eating from.

```{python}
# | echo: true
from notebooks.popcorn import plt_filesize
pplot(plt_filesize)
```

As we can see, the difference over time is massive. Especially early on,
between 2019 and the start of 2021, the amount of different packages and
package versions used grew rapidly, with the pace also picking up once again
starting 2023.

From a reported filesize of around 50KB in the first days before the end of
2019 we have easily tripled the filesize to over 150KB needed for the report
per day. Nowadays we have reached just about 400KB daily report size, over 8
times the size beginning 2018.

There are a few outlier days with a size of 0 KB on the server, which we had to
remove from the data. In all likelihood, those days were not reported correctly
or there was some kind of issue on the backend so the stats for those days are
lost.

<!-- TODO: is this still true? -->
We take a look at the missing days
among other things at the end of this article.

There are also a few days where the modification date of the file does not
correspond to the represented statistical date but those are kept. This rather
points to certain times when the files have been moved on the backend,
recreated externally, or otherwise externally modified but does not mean the
data contained are bad.

Let's not over-emphasize the results here: it is a very coarse view on the
packages and kernels underneath. Still, the shape of the curve above is a good
indication for what awaits us in other stats: 2019 to 2021 were massive years
of growth with some slow-down in 2021. From 2022 onwards the growth picks back
up again, if at a more mellow pace.

## Package statistics

Now that we have an idea of how the overall reported sizes in the distribution
have changed over time, let's focus on the actual package statistics.

The popcorn files contain two main pieces of information: the number of
installs per package (e.g. how many people have `rsync` installed) and the
number of unique installs (i.e. how many people provide their statistics). We
will look at both of these in turn.

```{python}
from notebooks.popcorn import plt_weekly_packages
pplot(plt_weekly_packages)
```

The number of packages overall strongly rises until early 2021,
when it stagnates a little before rising more slowly again afterwards.
The pattern strongly mirrors the curve we saw before for the daily filesize.

Turning to the daily unique uploads, we can see a similar pattern, though even
more strongly pronounced.

```{python}
from notebooks.popcorn import plt_unique_installs
pplot(plt_unique_installs)
```

Unique installations rise sharply until early 2020. Then they not just stagnate
but shrink for the next three years. It is only early 2023 when the numbers
recover and begin rising again slowly.

We also have one day on 05 July 2024 which has significantly fewer unique
uploads (36 only) than all the other days around it. I have no clue if
something happened to data collection or everybody collectively decided to
leave their PC offline just for that day, but the numbers are back to normal
the day after.[^independence-day]

[^independence-day]: I suppose one interpretation would be people taking their
  4th of July celebrations very seriously, and thus not being present in the
  statistics for the day after. However, I am not sure if this would reflect so
  strongly in data collection, and it additionally pre-supposes the data
  collected predominantly stemming from the United States. Lastly, one would
  suppose this having a similar effect every year if that was the case.

This curve also goes some way to explaining the dip in overall package
installations previously. When there are fewer people uploading their daily
statistics the absolute number of package installations will be somewhat
reduced as a result, unless for some reason the remaining people all of a
sudden start having many more packages installed.

Let's check that out next, by actually looking at the installed packages _per
user_ for each day.

```{python}
from notebooks.popcorn import plt_pkg_relative
pplot(plt_pkg_relative)
```

Combining both stats to look at the installed packages at a more individual
level per user, we see this confirmed. There is no similarly strong dip for the
relative package ownership as there was for the absolute package numbers.

Indeed, with the exception of a small more rapid increase in individual package
ownership in 2019, we see a much more stable increase in per-user packages than
the absolute numbers and no similarly big slump over three years.

Instead we see different patterns of rises and dips. Both in the beginning of
2020, and the beginning of 2024, we can see first a strong rise and then an
equally strong fall in the average number of user-owned packages at once.
This could point to one of multiple options:

Perhaps users have been collectively trying out more new packages over the end
of year holidays or with the start of the new year. New year, new workflow
could presumably be something a few users decide to do, and this may be a
reflection of it. Or, equally, using the new year to learn new software. The
subsequent dip would then mean an end to the period of trying out new stuff, or
adopting the new packages and dropping the old ones.

At the same time, with the relatively limited absolute number of installations,
it is also quite likely that the representation is skewed by a single user or a
couple users having a much larger package ownership than everybody else. This
may signify new users checking out Void Linux and downloading a large variety
of packages in the process.

<!-- TODO: still accurate? -->
For a breakdown of the absolute numbers of packages on systems by weekday and
month of the year instead of over time, see the Appendix below.

Beyond pure installation numbers, let's take a look at the actual top-installed
packages on users' systems.

<!-- TODO: perhaps the pre-made ISOs play a role, especially Feb2024? no hang on feb 2025 -->

```{python}
from notebooks.popcorn import plt_top_packages
pplot(plt_top_packages)
```

The top packages are unsurprisingly
the `base-system` and `xtools` packages, followed by `wget`, `htop` and
`rsync`.[^popcorn-removal]

[^popcorn-removal]: I have removed the PopCorn package itself from the data.
  Funnily enough, since _everybody_ who is represented in the data has to have
  PopCorn installed or the data wouldn't be collected in the first place, if we
  extrapolate from the collected data naively this means more people have PopCorn
  installed than the base-system. Of course, viewed over the majority of Void
  Linux installations this is hogwash. We have the absolute numbers and only
  around 150 people ever have PopCorn installed. But it nicely represents some of
  the danger of over-interpreting the results before us without also reflecting
  on sample bias.

In my opinion the list of top packages reflect the technical audience of Void
Linux and does not hold too many surprises. Almost everyone uses `socklog` and
most people have the `nonfree` repo enabled. `firefox` is the most installed
browser, and everyone at least has `alsa-utils` installed, even if they're not
using `alsa` as their primary sound provider.

I am somewhat surprised by the prevalence of `git`, though this package is in
turn required by many others. Among them some of the other top packages such as
`xtools`, so it does make sense.

It, along with the prevalence of `tmux` (even above `zip`!) does once again
speak to the technical nature of Void Linux users, however, at least for those
opting into data collection.

Almost everyone keeps the `base-system` package installed but, importantly,
not _everyone_. The package is not represented for each installation, with a
sizable chunk of people having removed it.

Lastly, I am also pleasantly surprised by the appearance of `gimp` in the top
20 packages.

The 'rarest' 20 packages shows a snapshot of packages which have been installed
by _someone_ but only that single someone. In other words, there are quite a
few packages which nobody in the sample has installed but those are not
represented here. Instead, the rare packages tend to show those that somebody
built themselves, or only tried briefly. They provide more of a snapshot of the
kind of custom shenanigans users get up to within the `xbps` package system, or
could be viewed as a potential 'wishlist' of packages not yet officially
available.

Let's turn to the 'distribution' of package installations.

```{python}
from notebooks.popcorn import plt_package_distribution
pplot(plt_package_distribution)
```

Visualized above is the package installation frequency (or density) distribution.
On the Y-axis we see the amount of packages while on the X-axis we see the amount of installations.
What this means is that we see _how often_ packages tend to be installed,
and where the majority of packages is grouped.[^density-approximation]

[^density-approximation]: In the package density count above, since we are
  accumulating over the absolute numbers of all installations of all users, the
  overall high numbers are really _high_, i.e. above 150,000. Since we are
  sorting the package counts into a finite number of bins to make visualizing it
  possible, the lowest bin overshoots the 0-mark and we get an estimation of
  minus-installation counts. Of course, this is not possible, no package in the
  data has been installed negative amount of times --- to my knowledge!

_Many_ packages are installed 0 to 10 times.
Some packages are installed above 10 times,
fewer yet above 100 times,
and so on,
and this distribution is what we see here.

```{python}
from notebooks.popcorn import plt_top_packages
_, defs = plt_top_packages.run()
df_pkg_dl = defs["df_pkg_dl"]
def get_num(df: pl.LazyFrame) -> int:
    return df.count().collect(engine="streaming").item(0, 0)

one_ten_installs = df_pkg_dl.sort("count", descending=False).filter(
    (pl.col("count") >= 1) & (pl.col("count") < 10)
)
ten_twenty_installs = df_pkg_dl.sort("count", descending=False).filter(
    (pl.col("count") >= 10) & (pl.col("count") < 20)
)
twenty_thirty = df_pkg_dl.sort("count", descending=False).filter(
    (pl.col("count") >= 20) & (pl.col("count") < 30)
)
thirty_plus = df_pkg_dl.sort("count", descending=False).filter((pl.col("count") >= 30))
```

To be more precise with the numbers:
There are `python f"{get_num(one_ten_installs):,}"` packages which have between one
and ten installations in the data, `python f"{get_num(ten_twenty_installs):,}"`
packages between eleven and 20 installations, and
`python f"{get_num(twenty_thirty):,}"` packages between 21 and 30 installations.
`python f"{get_num(thirty_plus):,}"` packages have over 30 installations.

## Kernel Analysis

```{python}
from notebooks.popcorn import plt_kernel_versions
pplot(plt_kernel_versions)
```

When looking at the kernel versions used, we see a very strong jump between major kernel version
4 and major kernel version 5.

For this analysis we had to exclude {kernel_df_v99.select(pl.len()).item()} rows which were
apparently from the future, as they were running variations of major kernel version 99. In all
likelihood there is a custom kernel version out there which reports its own major version as 99.
The strange version starts appearing on {kernel_df_v99.select("date").row(0)0} and shows up
all the way until {kernel_df_v99.select("date").row(-1)[0]}.

```{python}
from notebooks.popcorn import plt_kernel_timeline
pplot(plt_kernel_timeline)
```

```{python}
from datetime import date
from notebooks.popcorn import plt_kernel_timeline
_, defs = plt_kernel_timeline.run()
weekly_kernel_df = defs["weekly_kernel_df"]

last_kernel4: date = weekly_kernel_df.filter(pl.col("major_ver") == "4")[-1][
    "date"
].item()
first_kernel5: date = weekly_kernel_df.filter(pl.col("major_ver") == "5")[0][
    "date"
].item()
last_kernel5: date = weekly_kernel_df.filter(pl.col("major_ver") == "5")[-1][
    "date"
].item()
```

A timeline analysis of the kernels used to report daily downloads shows that people generally
adopt new major kernel versons at roughly the same time. This change is especially stark between
major kernel versions 5 and 6, which seem to have traded place in usage almost over night.

The first time that major version 5 of the kernel shows up is on {first_kernel5}. From here, it
took a long time for the last of the version 4 kernels to disappear, coinciding with the big
switch between major version 5 and 6. The last time a major version 4 is seen is on
{last_kernel4}, while the last major version 5 kernels still pop up.
It would seem, then, that the people still running kernel version 4 used the opportunity of
everybody switching to the stable version of 6 to also upgrade their machines.

## Appendix: Odds and Ends

The above graphics are the main ones that I think could be useful, entertaining, or somewhere in between.
However, when exploring data, many more visualizations come to light.
Most of them are a little more 'boring' than the ones selected above,
but may still be of interest for technical deep-dives or more specific investigations.
They are collected here, in my pseudo-appendix to the main article.

### The PopCorn files

Let's have a closer look at the provided PopCorn statistics files themselves.

The files consist of a long list of packages which have been reported to the
central server that day, along with the number of package instances. The amount
of unique installations from which these statistics are derived are given as a
sum. It also consists of the same list once again, but separated by
specifically installed versions of packages.

So if somebody has v0.9.1 and somebody else v0.9.3 instead this list counts the
number of both packages with their versions separately. Another count is the
number of different Kernels that have been used on that day, with their exact
kernel name including major version, minor version and any suffix.

```json
{
  "UniqueInstalls": 2,
  "Packages": {
    "ImageMagick": 1,
    "PopCorn": 2,
    "acpi": 1,
    "alsa-utils": 2,
  ...
    "xurls": 1,
    "youtube-dl": 1
  },
  "Versions": {
    "ImageMagick": {
      "6.9.9.40_1": 1
    },
    "PopCorn": {
      "0.2_1": 2
    },
    "acpi": {
      "1.7_3": 1
    },
    "alsa-utils": {
      "1.1.5_1": 1,
      "1.1.6_1": 1
    },
  ...
  },
  "XuKernel": {
    "4.16.6_1": 1
  }
```

When grouped by the packages and aggregated over all days, this results in a
table, for example the following is the table for the package count list:

```{python}
from notebooks.popcorn import tab_pkg
outp, defs = tab_pkg.run()
outp
```

When taking a look at the file sizes of the PopCorn report files we did so for
each day individually above. But we can also look at the accumulative growth
instead: here we just add up all the files reported so far for each day, and
show the resulting growth line.

```{python}
from notebooks.popcorn import plt_filesize_cumulative
outp, defs = plt_filesize_cumulative.run()
outp
```

A cumulative view gives a less granular look at the individual daily changes but
provides a more macro-level view on how big the statistics have grown to be overall.
We can see that, as each individually reported day adds up to 400KB nowadays, the
cumulative size is up to almost 700MB currently.

There are some missing days in the statistics.

```{python}
from notebooks.popcorn import tab_missing_days

outp, defs = tab_missing_days.run()
outp
```

### Packages monthwise and per weekday

```{python}
from notebooks.popcorn import plt_weekday_packages
pplot(plt_weekday_packages)
```

```{python}
from notebooks.popcorn import plt_month_packages
pplot(plt_month_packages)
```

## Outline

- intro
- filesize
- unique installations reported from
- packages -> perhaps find new subcategories
  - global
  - relative (pkg/unique)
  - top packages
  - rare packages?
  - install distribution
- packages per time unit (find clever title, e.g. 'accumulated packages')
  - per year?
  - weekday
  - month of year (combine with weekday?)
- kernels
  - overall kernel version installations
  - kernels over time

- misc
  - missing days
  - moved days

- things we can't see (limitations)
  - packages on offer in the repositories
    - this could shed light on the bumps of users and relative package ownership

Modified date != descriptive (named) date

```{python}
from notebooks.popcorn import plt_modified_times
pplot(plt_modified_times)
```
