SàT progress note 2019-W25

goffi 5 years ago jabber-xmpp-en SàT project libre SàT progress

Hello everybody, time for a new progress note.

This week I've been working with Flatpak. Flatpak is a new way to package applications using containerization, which has pros and cons: for a low overhead the applications can be run on most GNU/Linux distributions, they run in the same reproducible environment, and they have limited access to outside. I prefer using software packaged in my distribution when they're available (better integration, no overhead at all), but Flatpak is a nice way to try quickly something or to have access to software not packaged in our distribution.

About SàT, the main goal is to have super simple installation so people can try Cagou (desktop/mobile interface) easily. I'm also working on packages for other frontends mainly jp (CLI) and Primitivus (TUI), so they can be tested quickly. I hope this can encourage people to try them, and help gain some visibility.

Let's talk technical now. I have already worked on Flatpak nearly 1 year ago, and could launch several frontends using hand made manifests (the files instructing Flatpak how to build your application). I didn't found the time to polish it and it has never been made public.

This time, I wanted to have something as automated as possible: there are several frontends, and dependencies may change, I don't want to have to do everything by hand each time I update the packages.

The main difficulty when building flatpak, is that you have to deal with what is available in the "runtime" (the base of the distribution you'll use), and without network (you can't make internet connection during building process). In the case of Python, that means that you have to download every dependencies (including their own dependencies, etc.) and then install them in the right order (if you have wrong order, a module installation will fail because of missing dependency). This task is more complicated that it seems.

Flatpak ecosystem propose a script to make this task easier: Flatpak PIP Generator. It retrieves dependencies of a Python package, download them, generate the hash and retrieve download URL to prepare a list of "modules" usable in the manifest. The problem is that the list is not ordered according to dependencies, and the JSON produced is not directly usable. I quickly moved to make my own script specially tailored for SàT needs.

The first issue with dependencies is that there is no good way to find them. They are normally declared in the setup.py script which is the standard way to install a Python package, but you can't import this script to analyze it (it would execute its code and the usual if __name__ == '__main__' way to avoid that is never used with setup.py).

Well you have PyPI doing the analyze for you, and if you use its JSON API you can retrieve the mandatory requirements + optional ones using the ['info']['requires_dist'] field of the metadata object. Except that this field is far from always being complete, for instance it's not set for Twisted (I have not investigated for the reasons, maybe because Twisted is using an other file dynamically read). Twisted is far from being the only package without this data, so it's unusable in my case.

I've searched a bit on Internet and made different tries, and at the end the best I've found to reliably get the dependencies used by a Python project is to install it in a virtual environment and to parse the output of pip show <package_name>.
This command output a text with a Requires: line showing the actual requirements used in this project. Thanks to that I could solve the dependencies hell problem and my script is able to order correctly and automatically the modules needed in my manifest. The only modules not managed are setup requirements (requirements needed only for building the modules), but there are only a few of them and it's easy to specify them by hand.

I'm actually doing it in two times: first I download (using pip download) the requirements of the Python package to get the list and version needed, then I install the package in a virtual environment to analyze the output of pip show (pip handle a cache, so packages are not downloaded twice).

So my manifest generation script which I was expecting to do quickly took actually most of my free time this week and it's still not finished. I'm adding other options and I have to faces troubles likes Mercurial not being managed by Flatpak (for this case I have to retrieve myself the repository and upload it manually). I'm also sparing you other troubles (like how to organize the packages, SàT being a specific architecture I've first made my own runtime before being discouraged of doing this on the Flatpak mailing list).

At the end it's a lot of time spent just to make installation easier, and I have to do everything myself. I hope this kind of stories I'm explaining in my notes helps people to understand why it takes so long to develop a software, specially when you're on your free time. I hope also this work will help to popularize the project, and, maybe, bring some helping hands :)

See you soon.