GNOME Disks: Integrate Resize and Repair

Juli 25th, 2017 by Kai

The basic prototype patches let GNOME Disks resize partitions and their filesystems, check filesystems and repair them – at least when built against UDisks master where the actual work is performed. It needs some fixes on how jobs and UI states correspond but here a glimpse on how the resize looks like.

The current pending mockups for the whole UI will be discussed in person at GUADEC. Where to show the result of the filesystem check or repair and where to show the jobs are some of the questions.

In the implementation some problems pop up as usual, this time with some impact on the robustness but hopefully nothing again at the level of dealing with libparted geometry configuration.
The prototype needs some polishing as well, and in the underlaying layers I would like to add NTFS support and improve the FAT resize support. Finally it would be nice if the UDisks jobs indicate the progress which is not supported in libblockdev yet.

Resizing other block devices than partitions is also left to do (e.g. LUKS, backing loop file). Offering a filesystem resize where the block device can’t be fitted is a strange experience because it’s in general impossible to tell up to which size a filesystem occupies space in this block, meaning the result would look the same except for a difference in free space. Thus for the beginning only filesystems in partitions are resizable.

Development News

Recently 3.25.4 was released and the last changes improved the build experience a lot – Iñigo Martínez gave a meson port as surprise gift. Let’s see whether it’s enough time for the ongoing work to land on master for 3.26.

UDisks Repair and Resize API

Juni 27th, 2017 by Kai

Almost all actions in GNOME Disks rely on the UDisks system service. That way the authorization policy is handled by PolKit and Disks does not have to spawn any root processes. Because format and mount jobs are registered there to the device objects it is possible to initiate them from command line via gio mount, udisksctl or another D-Bus client such that Disks can still show these ongoing operations in the UI.

There are cases where D-Bus services are not possible (e.g. in an OS installer) and therefore libblockdev is a new library with similar functionality. The last UDisks release offloads many tasks to it instead of spawning utility binaries itself. In libblockdev the preference is to reuse libraries like libparted if possible and otherwise spawn utility binaries.

To have resize and repair support in Disks I had to introduce querying for installed support of an action and the action itself in UDisks and thus in libblockdev. The advantages are only small changes in Disks and Cockpit will also benefit. Now there are PRs pending for a generic repair and check function, a partition resize function and query functions for repair, resize and consistency check support of filesystem types in libblockdev. Then the PRs pending for UDisks can make use of them to expose the following API methos:

org.freedesktop.UDisks2.Manager:
CanFormat(fstype) -> (available, required_utility)
CanResize(fstype) -> (available, mode, required_utility)
CanCheck(fstype) -> (available, required_utility)
CanRepair(fstype) -> (available, required_utility)

org.freedesktop.UDisks2.Partition:
Resize(size, options)

org.freedesktop.UDisks2.Filesystem:
Resize(size, options)
Check(options) -> consistent
Repair(options) -> repaired

The resize mode flags indicate if online or offline shrinking/growing is supported. The size argument for partitions should guarantee that content of this size will fit in and alignment might only make it bigger. The progress percentage property of the running job is not yet set because this needs parsing stdout in libblockdev, so just a spinner in Disks for now. It’s possible to test them from d-feet, e.g. to see if the F2FS format is available and what utility is missing if not:

On the command line it would be gdbusmdbus2 or dbus-send or the D-Bus library of your choice (e.g. in Python3 from gi.repository import Gio). Here assuming an almost empty unmounted ext4 partition 1 on a loop device:

$ # repair first (the option dict {} is unused)
$ gdbus call --system --dest org.freedesktop.UDisks2 \
 --object-path /org/freedesktop/UDisks2/block_devices/loop0p1 \
 --method org.freedesktop.UDisks2.Filesystem.Repair {}
(true,)

$ # shrink filesystem to 50 MB
$ gdbus call --system --dest org.freedesktop.UDisks2 \
 --object-path /org/freedesktop/UDisks2/block_devices/loop0p1 \
 --method org.freedesktop.UDisks2.Filesystem.Resize 50000000 {}
()

$ # shrink partition to 50 MB
$ gdbus call --system --dest org.freedesktop.UDisks2 \
 --object-path /org/freedesktop/UDisks2/block_devices/loop0p1 \
 --method org.freedesktop.UDisks2.Partition.Resize 50000000 {}
()

GNOME Disks Development News

After Disks 3.25.2 was released there were only small changes in master. The standby menu entry was fixed to be usable again after the action took effect. Mitchell took care of large file support on 32-bit systems, an interesting topic. The biggest current change is the new format dialog. The new UDisks API must be integrated now in the UI. It would be nice to get NTFS (and progress) support in libblockdev and finally the Disks UI mockups done as well.

An Observation in UI Design

Juni 12th, 2017 by Kai

Reading the GNOME Human Interface Guidelines gives a good idea on how to arrange elements and reduce complexity. The HIGs also emphasize on having a clear goal which helps in deciding which elements need to be arranged at all. But I did not grasp the wideness of being purpose-driven for this goal of the application which might then mean to abstract from technical details on the way. So now I try to explain this observation here.

I think there are two opposite paradigms one can follow in UI design, and thus what users can expect from UIs. This is not a thorough research on the topic, so you can leave a comment if you have insights to share on whether this view is mistaken.
Maybe you will find your own examples during reading if you try to think about image, audio and video formats and their parameters, or network connection and encryption stuff in UIs.

The first paradigm is to make features accessible in the UI. This could just mean showing technical choices and configuration values coming from an API or command line tools. The UI serves as big toolbox with few limits, allowing to be creative and meet various use cases. But from the viewpoint of a newbie it is a big frustration to use these kind of applications because the options don’t have a meaning. Reaching the point where all implications of the parameters are understood is not possible in a few minutes. The whole research task for appropriate parameters has been offloaded to the user.

Therefore, the other paradigm comes into play and thinks from the main purpose of the UI and what elements it needs to reach the goal. That’s what I expect in GNOME. The technical details go to the background and may not be found as UI elements anymore. This needs clever ways to find optimal parameters, or studies of the technical and social/psychological conditions to reduce the workload for the user. First this looks like hiding ‚features‘ and configuration options (i.e. technical details) but it might be good to take a step back and look at the state of IT. Is it really necessary that everyone fiddles around with the available technical options? Can the decision for a configuration value be automated or evaluated? And how did we get into the situation of having so many competing ’standards‘ and a rag rug of solutions for the different layers involved and all this strange legacy stuff needed for compatibility.

Maybe we first need to work on a algorithms, libraries and single standards in order to be able to reduce UI complexity. But it’s not always possible to find unique technical solutions for the whole planet and we have to expose some of this strange acronyms and abbreviations in the UI and hope that users have an idea of their meaning. The result is a mixed approach of both paradigms where necessary.

Example: The format dialog in GNOME Disks

In the specific case of the format dialog for pen drives or hard disks you could simply decide to make features of the command line tools available in the UI. This would involve to expose the choice of the filesystem type and if it should be contained in an ecryption layer. Not speaking of overwhelming non-default parameters this toolbox is already hard to understand. For the new design of the format dialog I first proposed this toolbox paradigm approach (even if preference of the tools is indicated and not all are directly visible):

Assessing every option for the given use case and conditions is a lot of work and sometimes a final judgement can’t be made concerning issues with inter-OS compatibility, snapshots, Linux file permissions, encryption and performance (storage type, in-kernel drivers).
With the Opus audio codec joint efforts came finally to a good general solution, maybe too late.
But for filesystems it’s much more complex and the operating systems also have significant differences.

So if we want to use the result-oriented paradigm, we already see the problem that the UI has to guide the user through these OS compatibility issues. But this still means abstracting from selecting a concrete filesystem type and focusing more on use cases which are tied to filesystem choices (here still visible in brackets). The UI does not have to differ too much but important is the base idea. Here we see Allan’s design:

Ext4, NTFS and FAT are the current default filesystems in Disks but maybe exFAT is already more relevant and could e.g. replace both NTFS and FAT or even cover a distinct use case – that needs further research.

But also for Linux-only use cases there are other filesystems than Ext4 which could replace it as default choice or at least cover special demands like snapshots or being suited for flash storage.
Listing all these use cases would almost have the same effect like listing the filesystems with explanation, and that is what should be avoided for a common formating action of an external hard drive to back up some files.

A pragmatic middle path would e.g. as above show swap partitions and XFS for large amounts of data.
Yet I would argue that once people set up swap partitions or e.g. Btrfs volumes they can just handle the toolbox paradigm during that step. So as remembered from install wizards, the format dialog could do a paradigm shift and expose the toolbox if a custom filesystem setup is chosen instead of the defaults:

I think that this flexibility is needed because I don’t see a single filesystem to emerge soon which fits all the demands. But I would highly appreciate if people do not have to think about filesystems anymore because operating systems switched to let’s say Btrfs in collaboration and even your digital camera can handle it.

With this fallback available there is less pressure in finding optimal choices for the major use cases.
It can be observed if they need to be adjusted instead of campaigning for agreement on e.g. a single Linux-only and also a single inter-OS filesystem.
Still it would be nice to find out how user interaction can be reduced and less options presented by e.g. user surveys or applying automatic constraints based on the context.

GNOME Disks spins up for GSoC

Mai 17th, 2017 by Kai

This year’s Google Summer of Code features many valuable open source organizations and GNOME is one of them.
In GNOME’s project list you can find Disks, the GNOME utility for dealing with storage devices.
I’m happy that it worked out, because at first it was not in the list of ideas. The fact that Disks was considered unmaintained bothered me, but also was a big hint to finally get involved instead of mere bug reporting. So the plan emerged to propose working on two key features: Resize and repair filesystems.

Getting in contact with people through open bugs, IRC and the mailing lists went well and I’m grateful for the support (extra thanks to Michael!). Ondrej stepped in as mentor and made this possible, I already learned a lot and just took over maintainership. In the rest of this post I write a bit about the GSoC project outline and recent development in GNOME Disks.

Goal:
GNOME Disks should support repairing and resizing filesystems through UDisks, thus
enabling non-root users to safely set up their devices from a graphical interface.

Motivation:
GParted does not use UDisks, runs only as root and thus not under Wayland.

Outline

  • Work out new D-Bus interface functions in cooperation with the UDisks team so that
    UDisks offers resize/repair through libblockdev (FS plugin) and is also tooling aware by
    probing relevant binaries. The exact solution is to be negotiated, since Cockpit might
    make use of it too.
  • Use that new UDisks functionality in GNOME Disks, design and implement UI (follow
    HIG, good progress indication)
  • Focus on supporting just a few filesystems in the beginning

The first task is to rework the format dialog, which should stick with the principle to offer just a few sane defaults but also not forbid to choose others (list available filesystems and find a good solution for missing tooling).



(Needs Radio Buttons for the selection.)

Development News

Translation work is constantly going on – great to see that!

I’ve set up a GNOME Wiki page as website for Disks.

A bug preventing unmounting and locking is fixed as well as a bug in the format dialog.

Taking care of loop devices:

  • Explain the auto-clear option, i.e. detaching loop devices if they are not mounted anymore
  • Disable auto-clear during operations if it would lead to unwanted detachment
  • Create a new empty disk image from the AppMenu

UI improvements:

  • Show UUID
  • Hints for passphrase missmatch
  • Indicate maximal FS label length
  • Explain the meaning of the mounting and encryption options, i.e. overwriting the session defaults with fstab/crypttab entries
  • Use mimetypes for disk image selection (welcome Mitchell as new contributor!)

There are many ways to contribute to GNOME, please support the Outreachy by spreading the word!

Design of a Python-subset Compiler in Rust targeting ZPAQL

Oktober 19th, 2016 by Kai

B.Sc. Thesis, Computer Science Freie Universität Berlin, Kai Lüke

Abstract:

The compressed data container format ZPAQ embeds decompression algorithms as ZPAQL bytecode in the archive. This work contributes a Python-subset compiler written in Rust for the assembly language ZPAQL, discusses design decisions and improvements. On the way it explains ZPAQ and some theoretical and practical properties of context mixing compression by the example of compressing digits of π. As use cases for the compiler it shows a lossless compression algorithm for PNM image data, a LZ77 variant ported to Python from ZPAQL to measure compiler overhead and as most complex case an implementation of the Brotli algorithm. It aims to make the development of algorithms for ZPAQ more accessible and leverage the discussion whether the current specification limits the suitability of ZPAQ as an universal standard for compressed archives.

Download paper: Design of a Python-subset Compiler in Rust targeting ZPAQL, Kai Lüke

Converted to HTML with pdf2htmlEX: Design of a Python-subset Compiler in Rust targeting ZPAQL, Kai Lüke

Web Connectivity in Ghana – A survey

Juli 11th, 2015 by Kai

Abstract:

Starting from the research “Dissecting web latency in Ghana” of Zaki et al. the field of Internet connectivity in Ghana is explored and the different aspects like latency, DNS resolution times, complexity of web sites, caching and prefetching, TLS/SSL, TCP and cellular network performance, role of ISPs, Internet Exchange Points, are brought in relation. Measurement approaches and tailored solutions for areas with bad connectivity are listed. Possible ways of improvement are discussed.

Download paper: Web Connectivity in Ghana – A survey, Kai Lüke

Converted to HTML with pdf2htmlEX: Web Connectivity in Ghana – A survey, Kai Lüke

Categories and Subject Descriptors: C.2.6 [Internetworking]: Standards; C.4 [Performance of Systems]: Reliability, availability, and serviceability

Keywords: Developing Countries, Internet, Latency, Performance, TCP, Mobile, Cellular

modifying kernel space – uname hack revamped

August 27th, 2013 by Kai

This change of the original uname hack kernel module was laying unreleased and forgotten on the disk, but after some tweaking to support all options, here it comes — ugly and not meant for serious usage:

uname_kernel_hack_module.tar.gz (1,7 kB)

Some months ago I modified it to not change the machine from e.g. amd64 to i386 but the kernel version. Curiously to solve an issue it itself is having in the second line of the Makefile (uname -r); some actions rely on the kernel version to access the actual initramdisk or kernel headers. These might be different versions within guest system containers and led to the ugly hack of overwriting the kernel struct (yes, we are on Linux, not GNU Hurd) where the uname tool gets it’s release version from:

struct new_utsname {
char sysname[__NEW_UTS_LEN + 1];
char nodename[__NEW_UTS_LEN + 1];
char release[__NEW_UTS_LEN + 1];
char version[__NEW_UTS_LEN + 1];
char machine[__NEW_UTS_LEN + 1];
char domainname[__NEW_UTS_LEN + 1];
};

At initialisation the module saves the original value of e.g. sysname and overwrites the kernel space struct with the value you want. After usage, when the module is unloaded, the original value will be restored.

So when using the provided Makefile you could test how your system looks like when pretending to be something else:

make KRELEASE=2.5.2-0 KSYSNAME=Minix KVERSION="Debian 2.5 (2012-03-04)" KMACHINE=i386 test

The output while running make could look like:

uname -a
Linux hstnm 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07) x86_64 GNU/Linux
sudo /sbin/insmod unamehack.ko
[sudo] password for kai: 
uname -a
Minix hstnm 2.5.2-0 Debian 2.5 (2012-03-04) i386 GNU/Linux
sudo /sbin/rmmod unamehack
uname -a
Linux hstnm 3.10-2-amd64 #1 SMP Debian 3.10.5-1 (2013-08-07) x86_64 GNU/Linux

So a longer usage would require you to enter

make KMACHINE=i386 insertunamemodule

and revert the changes as soon as possible by

make removeunamemodule

So, good luck! You’ll better not use it and try to fix cases where it’s necessary.

„Faire“ Computer?

Juni 10th, 2012 by Kai

Foxconn bekommt durch Apple in letzter Zeit immer mehr Aufmerksamkeit bezüglich der Gewerkschaftsrechte und Arbeitsbedingungen der Fabriken in China. Gerade weil iLifestyle und Suizid so weit voneinander weg zu liegen scheinen. Kampagnen wie makeITfair (Germanwatch e.V.) u.a. sowie der Greenpeace »Guide to Greener Electronics« werfen den Blick auf die Schattenseiten der Technik in Herkunft der Materialien, Produktionsbedingungen, Giftstoffen, Energieverbrauch, Entsorgung und die mitgekaufte Verantwortung. Obwohl auch in Deutschland Reuse vor Recycle gilt, wird meist schon das Recycling in den Anfängen der abwechselnd von den Herstellern organisierten Entsorgungskette erstickt.

Gestern sah ich den Film »Blood in the Mobile«, der die für Elektronikprodukte essentiellen »Konfliktmineralien« aus dem Kongo zum Thema hat. Aufgebracht durch einen UN-Expertenbericht, der aussagt, dass ein direkter Zusammenhang zwischen Minenabbau für Mobiltelefone und Angriffen auf die Bevölkerung durch Milizen besteht (UN Group of Experts, Dez. 2008, S/2008/43), tritt Regisseur Frank Piasecki Poulsen mit dem Mobiltelefonhersteller seiner Wahl in Kontakt. Aber woher die verwendeten Mineralien stammen, kann und will niemand sagen, obwohl seit 10 Jahren das Problem intern diskutiert wird – zudem sei auch die ganze Elektroniksparte betroffen und auch bei den Konsumierenden läge die Verantwortung. Poulsen will verstehen und mit eigenen Augen sehen, wie die Realität vor Ort aussieht und bereist mit Kamerateam eine der Minen und zeigt somit die verschiedenen Problematiken auf. Lösungsansätze werden auch diskutiert und als Anfang ist durch einen US-Gesetzesantrag erreicht worden, dass die Zulieferungskette transparent veröffentlicht werden muss. Eine Projektseite ist hier.

Relativ neu ist die Doku »Behind the Screen: Das Leben meines Computers«, welche einige Aspekte der Herstellung und Entsorgung beispielhaft abdeckt. Da EU-Exporte für Elektroschrott verboten sind, landen als Gebrauchtware deklarierte Müllladungen auf Halden in Ghana und werden in Accra, Koforidua usw. zerlegt. Dieser Prozess setzt höchstgiftige Stoffe frei und betrifft somit die dort arbeitenden Kinder mehrfach. Auch die Bleikonzentration und weitere Kunststoffe wie Weichmacher belasten das Trinkwasser.

Doch bevor ein Device dort landet, wird es z.B. in Tschechien von vietnamesischen Leiharbeitern in Knebelverträgen hergestellt. Das verwendete Gold könnte aber wieder aus Ghana kommen. Hauptsache, wir sind auf der „richtigen“ Seite des digital divide? Mehr zum Thema Informatik & Gesellschaft auch hier, in der Zeitschrift des »Forums InformatikerInnen für Frieden und gesellschaftliche Verantwortung e.V.«.

Codecvergleich mit GStreamer durch SSIM

Mai 18th, 2012 by Kai

Während der Wirrungen bezüglich des HTML5-Videostandards hatte ich (noch lange bevor WebM existierte) einen Blick auf Dirac geworfen, einen alternativen Codec der BBC Research Labs. Nach subjektiven Vergleichen und einiger Benutzung, fehlte mir dennoch ein Verfahren, um mehr verlässliche Aussagen über die Qualität (sowie zu erwartende Größe) eines komprimierten Videos zu machen. Anstelle des „alten“ PSNR wurde SSIM zur Differenzanalyse zwischen Original und dem komprimierten Bild verwendet. GStreamer bietet sich durch starke Flexibilität an – ein Testframework wurde in Python mit Django umgesetzt. Leider steht der Out-of-the-box-Verwendung noch einiges im Wege; gerade die SSIM-Berechnung ist noch nicht als eigenständiges GStreamer-Element implementiert, was Geschwindigkeit und Stabilität verbessern würden. Auch die generelle Anbindung der GStreamer-Mainloop in die Webapplikation wäre besser durch einen daemon zu lösen. Daher kommt hier jetzt noch kein Quellcode…

Ausarbeitung (Kurs Digitales Video): Der Dirac-Wavelet-Codec und automatisierter Qualitätsvergleich mit Hilfe des GStreamer-Frameworks

Die erhobenen Daten sind nur beispielhaft und erheben keinen Anspruch auf vollständige Charakterisierung der Codecs. Denn der Zeitaufwand für SSIM ist groß – und somit ist die Eingabesequenz sehr kurz gewählt.

Arduino N64 Controller Library und Tetris-Port

März 26th, 2012 by Kai

Moved to GitHub: https://github.com/pothos/arduino-n64-controller-library

Die Arduino-Plattform (arduino.cc) bietet einen einfachen Einstieg in die Microcontroller-Programmierung und ist gerade im Nicht-Geek-Bereich eine Erfolgsgeschichte. Die vielen Bibliotheken ermöglichen es, hauptsächlich auf der „Organisationsebene“ zu bleiben – in kurzer Zeit war aus der Kombination von TVout, dem verfügbaren Tetris-Clone und einigen Modifikationen zum Speichern der Highscore im EEPROM eine archaische Spielkonsole aufgebaut. Dank dieser Anleitung konnte ein N64-Controller benutzt werden. Während für NES relativ viel Quellcode verfügbar ist, gibt es keine komfortable N64-Controller-Library, da wegen exaktem Timing Inline-Assembler verwendet wird und der PIN fest eingeschrieben ist. Meine Bibliothek ist vielleicht nicht in schönster Manier geschrieben und alles andere als optimal in der Codegröße, aber für eine Erstveröffentlichung sollte es ausreichen – komfortabel ist sie zumindest (auch alle möglichen PINs von 0 bis 13 können unabhängig voneinander für Controller benutzt werden).

Die Lizenz ist unklar, es handelt sich hauptsächlich um Modifikationen und Zusammenstellungen. Der Tetris-Port „Simple Tetris Clone“ ist z.B. unter der MIT-License veröffentlicht, die meisten Bibliotheken unter der (L)GPL.

Version 1 der Library (26.03.2012) 2 der Library (26.07.2012),

getestet auf Arduino Uno: N64Controller

Hier der Quelltext für das Spiel: N64Tetris

Beispielquellcode:

#include <N64Controller.h>
N64Controller player1 (12); // PIN 12
void setup() {
    player1.begin(); // Initialisierung
}
void loop() {
    delay(30);
    player1.update(); // Tastenzustand auslesen
    if (player1.button_A() && player1.button_D_down()
        || player1.button_Start()) { // bitte keinen Sinn suchen
        int xachse = player1.axis_x(); // kann negativ oder positiv sein,
                                   // je nach Ausrichtung des Analog-Sticks
    }
    // …
}