Louis-Philippe Véronneau - fosshttps://veronneau.org/2022-05-20T00:00:00-04:00Introducing metalfinder2022-05-20T00:00:00-04:002022-05-20T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2022-05-20:/introducing-metalfinder.html<p>After going to an incredible <a href="https://www.bandsintown.com/e/103112574">Arch Enemy / Behemoth / Napalm Death / Unto
Others</a> concert a few weeks ago, I decided I wanted to go to more
concerts.</p>
<p>I like music, and I <em>really</em> enjoy concerts. Sadly, I often miss great
performances because no one told me about it, or my local …</p><p>After going to an incredible <a href="https://www.bandsintown.com/e/103112574">Arch Enemy / Behemoth / Napalm Death / Unto
Others</a> concert a few weeks ago, I decided I wanted to go to more
concerts.</p>
<p>I like music, and I <em>really</em> enjoy concerts. Sadly, I often miss great
performances because no one told me about it, or my local newspaper didn't cover
the event enough in advance for me to get tickets.</p>
<p>Some online services lets you sync your Spotify account to notify you when a
new concert is announced, but I don't use Spotify. As a music geek, I have a
local music collection and if I need to stream it, I have a <a href="https://packages.debian.org/bullseye/supysonic">supysonic</a>
server.</p>
<p>Introducing <a href="https://gitlab.com/baldurmen/metalfinder">metalfinder</a>, a cli tool to find concerts using your local music
collection! At the moment, it scans your music collection, creates a list of
artists and queries <a href="https://www.bandsintown.com">Bandsintown</a> for concerts in your town. Multiple output
formats are supported, but I mainly use the ATOM one, as I'm a heavy feed
reader user.</p>
<p><img src="/media/blog/2022-05-20/atom.png" width="70%" style="margin-left:15%" title="Screenshot of the ATOM output in my feed reader" alt="Screenshot of the ATOM output in my feed reader"></p>
<p>The current metalfinder version (<code>1.1.1</code>) is a MVP: it works well enough, but I
still have <a href="https://gitlab.com/baldurmen/metalfinder/-/issues/">a lot of work to do</a>... If you want to give it a try, the
easiest way is to download it from PyPi. metalfinder is also <a href="https://ftp-master.debian.org/new/metalfinder_1.0.2-1.html">currently in
NEW</a> and I'm planning to have something feature complete in time for the
Bookworm freeze.</p>Migrating from ledger to hledger2022-02-05T00:00:00-05:002022-02-05T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2022-02-05:/migrating-from-ledger-to-hledger.html<p>I first started using ledger — the original <a href="https://plaintextaccounting.org/">plain-text accounting
software</a> — in 2017. Since then, I had been pretty happy with my
accounting routine, but grew a little annoyed by the repetitive manual work I
had to do to assign recurring transactions to the right account.</p>
<p>To make things easier, I …</p><p>I first started using ledger — the original <a href="https://plaintextaccounting.org/">plain-text accounting
software</a> — in 2017. Since then, I had been pretty happy with my
accounting routine, but grew a little annoyed by the repetitive manual work I
had to do to assign recurring transactions to the right account.</p>
<p>To make things easier, I had a collection of bash scripts to parse and convert
the CSV files from my bank's website<sup id="fnref:csv"><a class="footnote-ref" href="#fn:csv">1</a></sup> into ledger entries. They were of
course ugly, unreadable piles of sed-grep-regex and did not let met achieve the
automation complexity I longed for.</p>
<p>Dissatisfied with ledger's features, I decided to migrate to hledger. Contrary
to ledger, hledger comes with a <a href="https://hledger.org/hledger.html#csv-format">CSV parser</a> one can use to import
and classify recurring transactions automagically.</p>
<p>Having a proper DSL for this makes all the difference: I can easily add new
rules and understand the old ones. In the end, I get a very consistent result,
something my old bash scripts weren't great at.</p>
<p>Here is what my <code>debit.csv.rules</code> file looks like. It is used to tell hledger
how CSV data from my debit card should be parsed:</p>
<pre>
# skip the headings line:
skip 1
# assign fields
fields , , , date, , description, , amount2-in, amount2-out
# assign account where the money comes from
# 99 to get it at the bottom of the transaction
account99 assets:checking
include matchers.rules
</pre>
<p>As you can see, the rules matching transactions to specific accounts are
imported from another file, <code>matchers.rules</code>. I'm doing this since I want to be
able to use the same set of rules for debit and credit and use the cards
interchangeably.</p>
<p>Here's a snippet of my <code>matchers.rules</code> file:</p>
<pre>
# house
if Loyer/bail
account2 expenses:rent
# leisure
if PAYPAL .*STEAM GAMES
account2 expenses:leisure:videogame
if PAYPAL .*BANDCAMP
account2 expenses:leisure:music
</pre>
<p>Using this ruleset, a transaction looking like this:</p>
<pre>
"SOME ACCOUNT DESC","111111","EOP","2022/01/03",00002,"Loyer/bail","",521.00,"","","","",""
</pre>
<p>Would be transformed into:</p>
<pre>
2022-01-03 Loyer/bail
expenses:rent 521.00
assets:checking
</pre>
<p>Sadly, hledger's CSV rules won't let you do arithmetics. This can be useful
when you know a certain transaction needs to be split between accounts.</p>
<p>This is where <a href="https://hledger.org/hledger.html#auto-postings">auto postings</a> come in. They are a way to specify arbitrary
rules when an account is encountered.</p>
<p>Going back to my previous rent example, I split it 50/50 with my SO using this
rule:</p>
<pre>
= expenses:rent
assets:receivable:rent *0.5
assets:checking
</pre>
<p>After it is applied, the final transaction looks like this:</p>
<pre>
2022-01-03 Loyer/bail ; modified:
expenses:rent 521.00
assets:receivable:rent 260.50 ; generated-posting: = expenses:rent
assets:checking
</pre>
<p>Neat eh? Here is the little bash script I've written to automate all these
steps:</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/bin/bash</span>
<span class="c1">#</span>
<span class="c1"># Convert CSV to ledger using hledger</span>
<span class="nb">declare</span><span class="w"> </span>-a<span class="w"> </span><span class="nv">assets</span><span class="o">=(</span><span class="s2">"credit"</span><span class="w"> </span><span class="s2">"debit"</span><span class="o">)</span>
<span class="c1"># Read the array values with space</span>
<span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">assets</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span>
<span class="k">do</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="nb">test</span><span class="w"> </span>-f<span class="w"> </span><span class="s2">"</span><span class="nv">$i</span><span class="s2">.csv"</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="c1"># convert to ledger</span>
<span class="w"> </span><span class="nv">LEDGER</span><span class="o">=</span><span class="k">$(</span>hledger<span class="w"> </span>-f<span class="w"> </span><span class="s2">"</span><span class="nv">$i</span><span class="s2">.csv"</span><span class="w"> </span>--rules-file<span class="w"> </span>rules/<span class="s2">"</span><span class="nv">$i</span><span class="s2">.csv"</span>.rules<span class="w"> </span>print<span class="k">)</span>
<span class="w"> </span><span class="c1"># add auto_postings</span>
<span class="w"> </span><span class="nv">LEDGER</span><span class="o">=</span><span class="k">$(</span><span class="nb">printf</span><span class="w"> </span><span class="s2">"include rules/auto_postings.ledger\n\n</span><span class="nv">$LEDGER</span><span class="s2">\n"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>hledger<span class="w"> </span>print<span class="w"> </span>-f-<span class="w"> </span>--auto<span class="k">)</span>
<span class="w"> </span><span class="c1"># remove superfluous assets:checking lines</span>
<span class="w"> </span><span class="nv">LEDGER</span><span class="o">=</span><span class="k">$(</span><span class="nb">printf</span><span class="w"> </span><span class="s2">"</span><span class="nv">$LEDGER</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">'/assets:checking \+; generated.\+/d'</span><span class="k">)</span>
<span class="w"> </span><span class="nb">printf</span><span class="w"> </span><span class="s2">"</span><span class="nv">$LEDGER</span><span class="s2">"</span><span class="w"> </span>><span class="w"> </span><span class="s2">"</span><span class="nv">$i</span><span class="s2">.ledger"</span>
<span class="w"> </span><span class="k">else</span>
<span class="w"> </span><span class="nb">printf</span><span class="w"> </span><span class="s2">"File </span><span class="nv">$i</span><span class="s2">.csv does not exist\n"</span>
<span class="w"> </span><span class="k">fi</span>
<span class="k">done</span>
</code></pre></div>
<p>Migrating to hledger, I've cut down the time I spend on accounting from 1 hour
per month to about 10 minutes, all while making my workflow much cleaner.</p>
<p>Many thanks to the kind folks on <code>#hledger @irc.libera.chat</code> for the help!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:csv">
<p>One for my checking+savings account and another one for my credit card. <a class="footnote-backref" href="#fnref:csv" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Goodbye Nexus 52022-01-23T00:00:00-05:002022-01-23T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2022-01-23:/goodbye-nexus-5.html<p>I've blogged <a href="/running-android-9-pie-on-a-nexus-5-unlegacy-android.html">a few</a> <a href="/musings-on-long-term-software-support-and-economic-incentives.html">times</a> already about my Nexus 5, the Android device I
have/had been using for 8 years. Sadly, it died a few weeks ago, when the WiFi
chip stopped working. I could probably have attempted a mainboard swap, but at
this point, getting a new device …</p><p>I've blogged <a href="/running-android-9-pie-on-a-nexus-5-unlegacy-android.html">a few</a> <a href="/musings-on-long-term-software-support-and-economic-incentives.html">times</a> already about my Nexus 5, the Android device I
have/had been using for 8 years. Sadly, it died a few weeks ago, when the WiFi
chip stopped working. I could probably have attempted a mainboard swap, but at
this point, getting a new device seemed like the best choice.</p>
<p>In a world where most Android devices are EOL after less than 3 years, it is
amazing I was able to keep this device for so long, always running the latest
Android version with the latest security patch. The Nexus 5 originally shipped
with Android 4.4 and when it broke, I was running Android 11, with the November
security patch! I'm very grateful to the FOSS Android community that made this
possible, especially the LineageOS community.</p>
<p>I've replaced my Nexus 5 by a used Pixel 3a, mostly because of the similar form
factor, relatively affordable price and the presence of a headphone jack.
Google also makes flashing a custom ROM easy, although I had more trouble with
this than I first expected.</p>
<p>The first Pixel 3a I bought on eBay was a scam: I ordered an "Open Box" phone
and it arrived all scratched<sup id="fnref:scratch"><a class="footnote-ref" href="#fn:scratch">1</a></sup> and with a broken rear camera. The
second one I got (from the Amazon Renewed program) arrived in perfect condition,
but happened to be a Verizon model. As I found out, Verizon locks the bootloader
on their phones, making it impossible to install LineageOS<sup id="fnref:bootloader"><a class="footnote-ref" href="#fn:bootloader">2</a></sup>. The
vendor was kind enough to let me return it.</p>
<p>As they say, third time's the charm. This time around, I explicitly bought a
phone on eBay listed with a unlocked bootloader. I'm very satisfied with my
purchase, but all in all, dealing with all the returns and the shipping was
exhausting.</p>
<p>Hopefully this phone will last as long as my Nexus 5!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:scratch">
<p>There was literally a whole layer missing at the back, as if
someone had sanded the phone... <a class="footnote-backref" href="#fnref:scratch" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:bootloader">
<p>Apparently, an "Unlocked phone" means it is "SIM unlocked",
i.e. you can use it with any carrier. What I should have been looking for is
a "Factory Unlocked phone", one where the bootloader isn't locked :L <a class="footnote-backref" href="#fnref:bootloader" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Preventing an OpenPGP Smartcard from caching the PIN eternally2021-03-13T00:00:00-05:002021-03-13T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2021-03-13:/preventing-an-openpgp-smartcard-from-caching-the-pin-eternally.html<p>While I'm overall very happy about <a href="https://veronneau.org/new-year-new-openpgp-key.html">my migration to an OpenPGP hardware
token</a>, the process wasn't entirely seamless and I had to hack around
some issues, for example the PIN caching behavior in GnuPG.</p>
<p>As described <a href="https://dev.gnupg.org/T3362">in this bug</a> the <code>cache-ttl</code> parameter in GnuPG is not
implemented and thus does …</p><p>While I'm overall very happy about <a href="https://veronneau.org/new-year-new-openpgp-key.html">my migration to an OpenPGP hardware
token</a>, the process wasn't entirely seamless and I had to hack around
some issues, for example the PIN caching behavior in GnuPG.</p>
<p>As described <a href="https://dev.gnupg.org/T3362">in this bug</a> the <code>cache-ttl</code> parameter in GnuPG is not
implemented and thus does nothing. This means once you type in your PIN, it is
cached for as long as the token is plugged.</p>
<p>Security-wise, this is not great. Instead of manually disconnecting the token
frequently, I've come up with a script that restarts <code>scdameon</code> if the token
hasn't been used during the last X minutes.</p>
<p>It seems to work well and I call it using this cron entry:</p>
<p><code>*/5 * * * * my_user /usr/local/bin/restart-scdaemon</code></p>
<p>To get a log from <code>scdaemon</code>, you'll need a <code>~/.gnupg/scdaemon.conf</code> file that
looks like this:</p>
<pre>
debug-level basic
log-file /var/log/scdaemon.log
</pre>
<p>Hopefully it can be useful to others!</p>
<div class="highlight"><pre><span></span><code><span class="ch">#!/usr/bin/python3</span>
<span class="c1"># Copyright 2021, Louis-Philippe Véronneau <pollo@debian.org></span>
<span class="c1">#</span>
<span class="c1"># This script is free software: you can redistribute it and/or modify it under</span>
<span class="c1"># the terms of the GNU General Public License as published by the Free Software</span>
<span class="c1"># Foundation, either version 3 of the License, or (at your option) any later</span>
<span class="c1"># version.</span>
<span class="c1"># </span>
<span class="c1"># This script is distributed in the hope that it will be useful, but WITHOUT</span>
<span class="c1"># ANY WARRANTY; without even the implied warranty of MERCHANTABILITY or FITNESS</span>
<span class="c1"># FOR A PARTICULAR PURPOSE. See the GNU General Public License for more</span>
<span class="c1"># details.</span>
<span class="c1"># </span>
<span class="c1"># You should have received a copy of the GNU General Public License along with</span>
<span class="c1"># this script. If not, see <http://www.gnu.org/licenses/>.</span>
<span class="sd">"""</span>
<span class="sd">This script restarts scdaemon after X minutes of inactivity to reset the PIN</span>
<span class="sd">cache. It is meant to be ran by cron each X/2 minutes.</span>
<span class="sd">This is needed because there is currently no way to set a cache time for</span>
<span class="sd">smartcards. See https://dev.gnupg.org/T3362#137811 for more details.</span>
<span class="sd">"""</span>
<span class="kn">import</span> <span class="nn">os</span>
<span class="kn">import</span> <span class="nn">sys</span>
<span class="kn">import</span> <span class="nn">subprocess</span>
<span class="kn">from</span> <span class="nn">datetime</span> <span class="kn">import</span> <span class="n">datetime</span><span class="p">,</span> <span class="n">timedelta</span>
<span class="kn">from</span> <span class="nn">argparse</span> <span class="kn">import</span> <span class="n">ArgumentParser</span>
<span class="n">p</span> <span class="o">=</span> <span class="n">ArgumentParser</span><span class="p">(</span><span class="n">description</span><span class="o">=</span><span class="vm">__doc__</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">'-l'</span><span class="p">,</span> <span class="s1">'--log'</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">"/var/log/scdaemon.log"</span><span class="p">,</span>
<span class="n">help</span><span class="o">=</span><span class="s1">'Path to the scdaemon log file.'</span><span class="p">)</span>
<span class="n">p</span><span class="o">.</span><span class="n">add_argument</span><span class="p">(</span><span class="s1">'-t'</span><span class="p">,</span> <span class="s1">'--timeout'</span><span class="p">,</span> <span class="nb">type</span><span class="o">=</span><span class="nb">int</span><span class="p">,</span> <span class="n">default</span><span class="o">=</span><span class="s2">"10"</span><span class="p">,</span>
<span class="n">help</span><span class="o">=</span><span class="p">(</span><span class="s2">"Desired cache time in minutes."</span><span class="p">))</span>
<span class="n">args</span> <span class="o">=</span> <span class="n">p</span><span class="o">.</span><span class="n">parse_args</span><span class="p">()</span>
<span class="k">def</span> <span class="nf">get_last_line</span><span class="p">(</span><span class="n">scdaemon_log</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Returns the last line of the scdameon log file."""</span>
<span class="k">with</span> <span class="nb">open</span><span class="p">(</span><span class="n">scdaemon_log</span><span class="p">,</span> <span class="s1">'rb'</span><span class="p">)</span> <span class="k">as</span> <span class="n">f</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">SEEK_END</span><span class="p">)</span>
<span class="k">while</span> <span class="n">f</span><span class="o">.</span><span class="n">read</span><span class="p">(</span><span class="mi">1</span><span class="p">)</span> <span class="o">!=</span> <span class="sa">b</span><span class="s1">'</span><span class="se">\n</span><span class="s1">'</span><span class="p">:</span>
<span class="n">f</span><span class="o">.</span><span class="n">seek</span><span class="p">(</span><span class="o">-</span><span class="mi">2</span><span class="p">,</span> <span class="n">os</span><span class="o">.</span><span class="n">SEEK_CUR</span><span class="p">)</span>
<span class="n">last_line</span> <span class="o">=</span> <span class="n">f</span><span class="o">.</span><span class="n">readline</span><span class="p">()</span><span class="o">.</span><span class="n">decode</span><span class="p">()</span>
<span class="k">return</span> <span class="n">last_line</span>
<span class="k">def</span> <span class="nf">check_time</span><span class="p">(</span><span class="n">last_line</span><span class="p">,</span> <span class="n">timeout</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Returns True if scdaemon hasn't been called since the defined timeout."""</span>
<span class="c1"># We don't need to restart scdaemon if no gpg command has been run since</span>
<span class="c1"># the last time it was restarted.</span>
<span class="n">should_restart</span> <span class="o">=</span> <span class="kc">True</span>
<span class="k">if</span> <span class="s2">"OK closing connection"</span> <span class="ow">in</span> <span class="n">last_line</span><span class="p">:</span>
<span class="n">should_restart</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">else</span><span class="p">:</span>
<span class="n">last_time</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">strptime</span><span class="p">(</span><span class="n">last_line</span><span class="p">[:</span><span class="mi">19</span><span class="p">],</span> <span class="s1">'%Y-%m-</span><span class="si">%d</span><span class="s1"> %H:%M:%S'</span><span class="p">)</span>
<span class="n">now</span> <span class="o">=</span> <span class="n">datetime</span><span class="o">.</span><span class="n">now</span><span class="p">()</span>
<span class="n">delta</span> <span class="o">=</span> <span class="n">now</span> <span class="o">-</span> <span class="n">last_time</span>
<span class="k">if</span> <span class="n">delta</span> <span class="o"><=</span> <span class="n">timedelta</span><span class="p">(</span><span class="n">minutes</span> <span class="o">=</span> <span class="n">timeout</span><span class="p">):</span>
<span class="n">should_restart</span> <span class="o">=</span> <span class="kc">False</span>
<span class="k">return</span> <span class="n">should_restart</span>
<span class="k">def</span> <span class="nf">restart_scdaemon</span><span class="p">(</span><span class="n">scdaemon_log</span><span class="p">):</span>
<span class="w"> </span><span class="sd">"""Restart scdaemon and verify the restart process was successful."""</span>
<span class="n">subprocess</span><span class="o">.</span><span class="n">run</span><span class="p">([</span><span class="s1">'gpgconf'</span><span class="p">,</span> <span class="s1">'--reload'</span><span class="p">,</span> <span class="s1">'scdaemon'</span><span class="p">],</span> <span class="n">check</span><span class="o">=</span><span class="kc">True</span><span class="p">)</span>
<span class="n">last_line</span> <span class="o">=</span> <span class="n">get_last_line</span><span class="p">(</span><span class="n">scdaemon_log</span><span class="p">)</span>
<span class="k">if</span> <span class="s2">"OK closing connection"</span> <span class="ow">not</span> <span class="ow">in</span> <span class="n">last_line</span><span class="p">:</span>
<span class="n">sys</span><span class="o">.</span><span class="n">exit</span><span class="p">(</span><span class="s2">"Restarting scdameon has failed."</span><span class="p">)</span>
<span class="k">def</span> <span class="nf">main</span><span class="p">():</span>
<span class="w"> </span><span class="sd">"""Main function."""</span>
<span class="n">last_line</span> <span class="o">=</span> <span class="n">get_last_line</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">log</span><span class="p">)</span>
<span class="n">should_restart</span> <span class="o">=</span> <span class="n">check_time</span><span class="p">(</span><span class="n">last_line</span><span class="p">,</span> <span class="n">args</span><span class="o">.</span><span class="n">timeout</span><span class="p">)</span>
<span class="k">if</span> <span class="n">should_restart</span><span class="p">:</span>
<span class="n">restart_scdaemon</span><span class="p">(</span><span class="n">args</span><span class="o">.</span><span class="n">log</span><span class="p">)</span>
<span class="k">if</span> <span class="vm">__name__</span> <span class="o">==</span> <span class="s2">"__main__"</span><span class="p">:</span>
<span class="n">main</span><span class="p">()</span>
</code></pre></div>New Year, New OpenPGP Key2021-03-07T00:00:00-05:002021-03-07T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2021-03-07:/new-year-new-openpgp-key.html<pre>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Sun, 07 Mar 2021 13:00:17 -0500
I've recently set up a new OpenPGP key and will be transitioning away from my
old one.
It is a chance for me to start using a OpenPGP hardware token and to transition
to a new …</pre><pre>
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA512
Sun, 07 Mar 2021 13:00:17 -0500
I've recently set up a new OpenPGP key and will be transitioning away from my
old one.
It is a chance for me to start using a OpenPGP hardware token and to transition
to a new personal email address (my main public contact is still my
`@debian.org` address).
Please note that I've partially redacted some email addresses from this
statement to minimise the amount of spam I receive. It shouldn't be hard for
actual humans to follow the instructions below to find the complete addresses.
The old key will continue to be valid for a few months, but will eventually be
revoked.
You might know my old OpenPGP certificate as:
pub rsa4096/0x7AEAC4EC6AAA0A97 2014-12-22 [expires: 2021-06-02]
Key fingerprint = 677F 54F1 FA86 81AD 8EC0 BCE6 7AEA C4EC 6AAA 0A97
uid Louis-Philippe Véronneau <REDACTED@riseup.net>
uid Louis-Philippe Véronneau (alias) <REDACTED@riseup.net>
uid Louis-Philippe Véronneau (debian) <REDACTED@debian.org>
My new OpenPGP certificate is:
pub ed25519/0xE1E5457C8BAD4113 2021-03-06 [expires: 2022-03-06]
Key fingerprint = F64D 61D3 21F3 CB48 9156 753D E1E5 457C 8BAD 4113
uid Louis-Philippe Véronneau <REDACTED@veronneau.org>
uid Louis-Philippe Véronneau <REDACTED@debian.org>
These days, I mostly use my key for Debian and to sign git commit. I don't
really expect you to sign my new key if you had signed my old one.
I've published the new certificate on keys.openpgp.org as well as on my
personal website. You can fetch it like this:
$ wget -O- https://veronneau.org/media/openpgp.key | gpg --import
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEZ39U8fqGga2OwLzmeurE7GqqCpcFAmBFFM8ACgkQeurE7Gqq
CpcuchAAscAeszdtA+TlCI4YvK5nlk+nJnCnNBSnl7Et+jiNjq8kB/Fud+dWMTXC
Zag8oJkalbbxub0BT0bEAn+BiBunu58E0gd0Xq4syTbqZ5o5IN17S/tfxCD0k1hf
ewrnYZ2l0i5g4YvHGKC+Xv4D+Z84BylnIRaXHqlUdluOVfVYDfLybOAqoktO/KUH
I+vQBwXj0Fr/QAtgiz5Nwh/YHFiU9xMSvr5ozRwAFs6+xfIqFHuVPRRkEN5iVo4D
kkMIz+kFfkoh4aWIP4dgAu39XnEgxwTR9J+4yE8TzCCMzO7xCK0X6vqgPAxYMPvb
RuP4FnGWOnGnlcudCUAUkOaryrwRi+dPQTnNICHTYsvVc7dg+W0EhVUkwEuuEwpI
qtcB/Y5AGhqK0Cc11uXiFjIQwLTgwcUez4F0xrGeqsTtAM5gyRup2w0jbocTuYSh
ZRv/2zwrq/S3xVrUYGqdT+L5odmkBzz9zOwY5WlU2H9CMFOdh71XOv9wWQXan9ou
hLRodeOQ8MinIBP+sX36ol1zg/aP7mCHvRRSBzWt7l3WhVxgZFpNwIfp/RZqU0R4
IEq48mntFhPvHJjFmAKLKK/ckzNMtSn+HWQPJV3HTInKCTu5PTNMU3SAvPHOHEps
V6WWSOPB+1Lm/tlIULDc+0SopWoiWO4NObCSs8zMZHlYPBk5x/KIdQQBFgoAHRYh
BMqnQAcHqBawIC/DzfQlelCyHPqFBQJgRRTPAAoJEPQlelCyHPqFFVEA/1qScaAk
O+eBEE4q0BaJDsqweCS1XCcuQGkQCKi5Zv6kAQChQ96Ve7cKbN/wRkT9pdIhmx01
+CmIsnp3k6N0ZYLLCg==
=onl0
-----END PGP SIGNATURE-----
</pre>dput-ng or: How I Learned to Stop Worrying and Love the Hooks2021-02-21T00:00:00-05:002021-02-21T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2021-02-21:/dput-ng-or-how-i-learned-to-stop-worrying-and-love-the-hooks.html<p>As my contributions to Debian continue to grow in number, I find myself
uploading to the archive more and more often.</p>
<p>Although I'm pretty happy with my current sbuild-based workflow, twice in the
past few weeks I inadvertently made a binary upload instead of a source-only
one.<sup id="fnref:thanks"><a class="footnote-ref" href="#fn:thanks">1</a></sup></p>
<p>As it …</p><p>As my contributions to Debian continue to grow in number, I find myself
uploading to the archive more and more often.</p>
<p>Although I'm pretty happy with my current sbuild-based workflow, twice in the
past few weeks I inadvertently made a binary upload instead of a source-only
one.<sup id="fnref:thanks"><a class="footnote-ref" href="#fn:thanks">1</a></sup></p>
<p>As it turns out, I am not the only DD who has had this problem before. As
Nicolas Dandrimont kindly pointed to me, <code>dput-ng</code> supports pre and post
upload hooks that can be used to lint your uploads. Even better, it also ships
with a <code>check-debs</code> hook that lets you block binary uploads.</p>
<p>Pretty neat, right? In a perfect world, enabling the hook would only be a matter
of adding it in the hook list of <code>/etc/dput.d/metas/debian.json</code> and using the
following defaults:</p>
<pre>
"check-debs": {
"enforce": "source",
"skip": false
},
</pre>
<p>Sadly, <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983160">bug #983160</a> currently makes this whole setup more complex than
it should be and forces me to use two different <code>dput-ng</code> profiles pointing to
two different files in <code>/etc/dput.d/metas</code>: a default source-only one
(<code>ftp-master</code>) and a binary upload one (<code>ftp-master-binary</code>).</p>
<p>Otherwise, one could use a single profile that disallows binary uploads and when
needed, override the hook using something like this:</p>
<pre>
$ dput --override "check-debs.enforce=debs" foo_1.0.0-1_amd64.changes
</pre>
<p>I did start debugging the <code>--override</code> issue in <code>dput-ng</code>, but I'm not sure I'll
have time to submit a patch anytime soon. In the meantime, I'm happy to report I
shouldn't be uploading the wrong <code>.changes</code> file by mistake again!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:thanks">
<p>Thanks to Holger Levsen and Adrian Bunk for catching those and
notifying me. <a class="footnote-backref" href="#fnref:thanks" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>What are the incentive structures of Free Software?2021-02-17T00:00:00-05:002021-02-17T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2021-02-17:/what-are-the-incentive-structures-of-free-software.html<p>When I started my Master's degree in January 2018, I was confident I would be
done in a year and half. After all, I only had one year of classes and I
figured 6 months to write a thesis would be plenty.</p>
<p>Three years later, I'm finally done: the final …</p><p>When I started my Master's degree in January 2018, I was confident I would be
done in a year and half. After all, I only had one year of classes and I
figured 6 months to write a thesis would be plenty.</p>
<p>Three years later, I'm finally done: the final version of my thesis was
accepted on January 22<sup>nd</sup> 2021.</p>
<p>My thesis, entitled <em>What are the incentive structures of Free Software? An
economic analysis of Free Software's specific development model</em>, can be <a href="https://veronneau.org/media/blog/2021-02-17/mémoire.pdf">found
here</a> <sup id="fnref:french"><a class="footnote-ref" href="#fn:french">1</a></sup>. If you care about such things, both the data and the final
document can be built from source with the code in this <a href="https://gitlab.com/baldurmen/memoire-maitrise">git repository</a>.</p>
<h2>Results and analysis</h2>
<p>My thesis is divided in four main sections:</p>
<ol>
<li>an introduction to FOSS</li>
<li>a chapter discussing the incentive structures of Free Software (and arguing
the so called “<em>Tragedy of the Commons</em>” isn't inevitable)</li>
<li>a chapter trying to use empirical data to validate the theories presented in
the previous chapter</li>
<li>an annex on the various FOSS business models</li>
</ol>
<p>If you're reading this blog post, chances are you'll find both section 1 and 4
a tad boring, as you might already be familiar with these concepts.</p>
<h3>Incentives</h3>
<p>So, why <em>do</em> people contribute to Free Software? Unsurprisingly, it's
complicated. Many economists have studied this topic, but for some reason, most
research happened in the early 2000s.</p>
<p>Although papers don't all agree with each other and most importantly, about the
variables' importance, the main incentives<sup id="fnref:complex"><a class="footnote-ref" href="#fn:complex">2</a></sup> can be summarized by:</p>
<ul>
<li>expectation of monetary gain</li>
<li>writing FOSS as a hobby (that includes “<em>scratching your own itch</em>”)</li>
<li>liking the FOSS community and feeling a sense of belonging</li>
<li>altruism (writing FOSS for Good™)</li>
</ul>
<p>Giving weights to these variables is not an easy thing: the FOSS ecosystem is
highly heterogeneous and thus, people tend to write FOSS for different reasons.
Moreover, incentives tend to shift with time as the ecosystem does. People
writing Free Software in the 1990s probably did it for different reasons than
people in 2021.</p>
<p>These four variables can also be divided in two general categories: extrinsic
and intrinsic incentives. Monetary gain expectancy is an extrinsic incentive
(its value is delayed and mediated), whereas the three other ones are intrinsic
(they have an immediate value by themselves).</p>
<h3>Empirical analysis</h3>
<p>Theory is nice, but it's even better when you can back it up with data. Sadly,
most of the papers on the economic incentives of FOSS are either purely
theoretical, or use sample sizes so small they could as well be.</p>
<p>Using the data from the <a href="https://insights.stackoverflow.com/survey/2018/">StackOverflow 2018 survey</a>, I thus
tried to see if I could somehow confirm my previous assumptions.</p>
<p>With 129 questions and more than 100 000 respondents (which after statistical
processing yields between 28 000 and 39 000 observations per variable of
interest), the StackOverflow 2018 survey is a <em>very</em> large dataset compared to
what economists are used to work with.</p>
<p>Sadly, it wasn't entirely enough to come up with hard answers. There <em>is</em> a
strong and significant correlation between writing Free Software and having a
higher salary, but endogeneity problems<sup id="fnref:endogeneity"><a class="footnote-ref" href="#fn:endogeneity">3</a></sup> made it hard to give a
reliable estimate of how much money this would represent. Same goes for writing
code has a hobby: it seems there is a strong and significant correlation, but
the exact numbers I came up with cannot really be trusted.</p>
<p>The results on community as an incentive to writing FOSS were the ones that
surprised me the most. Although I expected the relation to be quite strong, the
coefficients predicted were in fact quite small. I theorise this is partly due
to only 8% of the respondents declaring they didn't feel like they belonged in
the IT community. With such a high level of adherence, the margin for
improvement has to be smaller.</p>
<p>As for altruism, I wasn't able get any meaningful results. In my opinion this
is mostly due to the fact there was no explicit survey question on this topic
and I tried to make up for it by cobbling data together.</p>
<p>Kinda anti-climatic, isn't it? I would've loved to come up with decisive
conclusions on this topic, but if there's one thing I learned while writing
this thesis, it is I don't know much after all.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:french">
<p>Note that the thesis is written in French. <a class="footnote-backref" href="#fnref:french" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:complex">
<p>Of course, life is complex and so are people's motivations. One
could come up with dozen more reasons why people contribute to Free Software.
The "fun" of theoretical modelisation is trying to make complex things
somewhat simpler. <a class="footnote-backref" href="#fnref:complex" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:endogeneity">
<p>I'll spare you the details, but this means there is no way to
know if this correlation is the result of a causal link between the two
variables. There are ways to deal with this problem (using an <a href="https://en.wikipedia.org/wiki/Instrumental_variables_estimation">instrumental
variables model</a> is a very popular one), but again, the survey didn't
provide the proper instruments to do so. For example, it could very well be
the correlation is due to omitted variables. If you are interested in this
topic (and can read French), I talk about this issue in section 3.2.8. <a class="footnote-backref" href="#fnref:endogeneity" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Book Review: Working in Public by Nadia Eghbal2020-11-06T00:00:00-05:002020-11-06T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2020-11-06:/book-review-working-in-public-by-nadia-eghbal.html<p>I have a lot of respect for Nadia Eghbal, partly because I can't help to be
jealous of her work on the economics of Free Software<sup id="fnref:os"><a class="footnote-ref" href="#fn:os">1</a></sup>. If you are not
already familiar with Eghbal, she is the author of <a href="https://www.fordfoundation.org/about/library/reports-and-studies/roads-and-bridges-the-unseen-labor-behind-our-digital-infrastructure/"><em>Roads and Bridges: The
Unseen Labor Behind Our Digital Infrastructure …</em></a></p><p>I have a lot of respect for Nadia Eghbal, partly because I can't help to be
jealous of her work on the economics of Free Software<sup id="fnref:os"><a class="footnote-ref" href="#fn:os">1</a></sup>. If you are not
already familiar with Eghbal, she is the author of <a href="https://www.fordfoundation.org/about/library/reports-and-studies/roads-and-bridges-the-unseen-labor-behind-our-digital-infrastructure/"><em>Roads and Bridges: The
Unseen Labor Behind Our Digital Infrastructure</em></a>, a great technical
report published for the Ford Foundation in 2016. You may also have caught her
<a href="https://www.youtube.com/watch?v=W2AR1owg0ao">excellent keynote</a> at LCA 2017, entitled <em>Consider the Maintainer</em>.</p>
<p>Her latest book, <em>Working in Public: The Making and Maintenance of Open Source
Software</em>, published by Stripe Press a few months ago, is a great read and if
this topic interests you, I highly recommend it.</p>
<p>The book itself is simply gorgeous; bright orange, textured hardcover binding,
thick paper, wonderful typesetting — it has everything to please. Well, nearly
everything. Sadly, it is only available on Amazon, exclusively in the United
States. A real let down for a book on Free and Open Source Software.</p>
<p>The book is divided in five chapters, namely:</p>
<ol>
<li>Github as a Platform</li>
<li>The Structure of an Open Source Project</li>
<li>Roles, Incentives and Relationships</li>
<li>The Work Required by Software</li>
<li>Managing the Costs of Production</li>
</ol>
<p><img src="/media/blog/2020-11-06/cover.jpg" width="70%" style="margin-left:15%" title="A picture of the book cover" alt="A picture of the book cover"></p>
<p>Contrary to what I was expecting, the book feels more like an extension of the
LCA keynote I previously mentioned than <em>Roads and Bridges</em>. Indeed, as made
apparent by the following quote, Eghbal doesn't believe funding to be the
primary problem of FOSS anymore:</p>
<blockquote>
<p><em>We still don't have a common understanding about </em>who's<em> doing the work,
</em>why<em> they do it, and </em>what<em> work needs to be done. Only when we understand
the underlying behavioral dynamics of open source today, and how it differs
from its early origins, can we figure out where money fits in. Otherwise,
we're just flinging wet paper towels at a brick wall, hoping that something
sticks.</em> — p.184</p>
</blockquote>
<p>That is to say, the behavior of maintainers and the challenges they face — not
the eternal money problem — is the real topic of this book. And it feels
refreshing. When was the last time you read something on the economics of Free
Software without it being mostly about what licences projects should pick and
how business models can be tacked on them? I certainly can't.</p>
<p>To be clear, I'm not sure I agree with Eghbal on this. Her having worked at
Github for a few years and having interviewed mostly people in the Ruby on
Rails and Javascript communities certainly shows in the form of a strong
selection bias. As she herself admits, this is a book on how software <em>on
Github</em> is produced. As much as this choice irks me (the Free Software
community certainly cannot be reduced to Github), this exercise had the merit
of forcing me to look at my own selection biases.</p>
<p>As such, reading <em>Working in Public</em> did to me something I wasn't expecting it
to do: it broke my Free Software echo chamber. Although I consider myself very
familiar with the world of Free and Open Source Software, I now understand my
somewhat ill-advised contempt for certain programming languages — mostly JS —
skewed my understanding of what FOSS in 2020 really is.</p>
<p>My Free Software world very much revolves around Debian, a project with a
strong and opinionated view of Free Software, rooted in a historical and
political understanding of the term. This, Eghbal argues, is not the case for a
large swat of developers anymore. They are <em>The Github Generation</em>, people
attached to Github as a platform first and foremost, and who feel "Open Source"
is just a convenient way to make things.</p>
<p>Although I could intellectualise this, before reading the book, I didn't really
<a href="https://en.wiktionary.org/wiki/grok">grok</a> how communities akin to npm have been reshaping the modern FOSS
ecosystem and how different they are from Debian itself. To be honest, I am not
sure I like this tangent and it is certainly part of the reason why I had a
tendency to dismiss it as a fringe movement I could safely ignore.</p>
<p>Thanks to Nadia Eghbal, I come out of this reading more humble and certainly
reminded that FOSS' heterogeneity is real and should not be idly dismissed.
This book is rich in content and although I could go on (my personal notes
clock-in at around 2000 words and I certainly disagree with a number of
things), I'll stop here for now. Go and grab a copy already!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:os">
<p>She insists on using the term <em>open source</em>, but I won't :) <a class="footnote-backref" href="#fnref:os" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Musings on long-term software support and economic incentives2020-10-19T00:00:00-04:002020-10-19T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2020-10-19:/musings-on-long-term-software-support-and-economic-incentives.html<p>Although I still read a lot, during my college sophomore years my reading
habits shifted from novels to more academic works. Indeed, reading dry
textbooks and economic papers for classes often kept me from reading anything
else substantial. Nowadays, I tend to binge read novels: I won't touch a book …</p><p>Although I still read a lot, during my college sophomore years my reading
habits shifted from novels to more academic works. Indeed, reading dry
textbooks and economic papers for classes often kept me from reading anything
else substantial. Nowadays, I tend to binge read novels: I won't touch a book
for months on end, and suddenly, I'll read 10 novels back to back<sup id="fnref:dune"><a class="footnote-ref" href="#fn:dune">1</a></sup>.</p>
<p>At the start of a novel binge, I always follow the same ritual: I take out my
e-reader from its storage box, marvel at the fact the battery is still pretty
full, turn on the WiFi and check if there are OS updates. And I have to admit,
Kobo Inc. (now Rakuten Kobo) has done a stellar job of keeping my e-reader up
to date. I've owned this model (a Kobo Aura 1<sup>st</sup> generation) for 7
years now and I'm still running the latest version of Kobo's Linux-based OS.</p>
<p>Having recently had trouble updating my Nexus 5 (also manufactured 7 years ago)
to Android 10<sup id="fnref:nexus"><a class="footnote-ref" href="#fn:nexus">2</a></sup>, I asked myself:</p>
<blockquote>
<p><em>Why is my e-reader still getting regular OS updates, while Google stopped
issuing security patches for my smartphone four years ago?</em></p>
</blockquote>
<p>To try to answer this, let us turn to economic incentives
theory.</p>
<p>Although not the be-all and end-all some think it is<sup id="fnref:incentives"><a class="footnote-ref" href="#fn:incentives">3</a></sup>, incentives
theory is not a bad tool to analyse this particular problem. Executives at
Google most likely followed a very business-centric logic when they decided to
drop support for the Nexus 5. Likewise, Rakuten Kobo's decision to continue
updating older devices certainly had very little to do with ethics or loyalty
to their user base.</p>
<p>So, what are the incentives that keep Kobo updating devices and why are they
different than smartphone manufacturers'?</p>
<h2>A portrait of the current long-term software support offerings for smartphones and e-readers</h2>
<p>Before delving deeper in economic theory, let's talk data. I'll be focusing on
2 brands of e-readers, Amazon's Kindle and Rakuten's Kobo. Although the
e-reader market is highly segmented and differs a lot based on geography,
Amazon was in 2015 the clear worldwide leader with 53% of the worldwide
e-reader sales, followed by Rakuten Kobo at 13%<sup id="fnref:ereader_sales"><a class="footnote-ref" href="#fn:ereader_sales">4</a></sup>.</p>
<p>On the smartphone side, I'll be differentiating between Apple's iPhones and
Android devices, taking Google as the barometer for that ecosystem. As mentioned
below, Google is sadly the leader in long-term Android software support.</p>
<h4>Rakuten Kobo</h4>
<p>According to <a href="https://help.kobo.com/hc/en-us/articles/360019690433">their website</a> and to this <a href="https://en.wikipedia.org/wiki/Kobo_eReader#Chronological_Overview">Wikipedia
table</a>, the only e-readers Kobo has deprecated are the original
Kobo eReader and the Kobo WiFi N289, both released in 2010. This makes their
oldest still supported device the Kobo Touch, released in 2011. In my book,
that's a pretty good track record. Long-term software support does not seem to
be advertised or to be a clear selling point in their marketing.</p>
<h4>Amazon</h4>
<p>According to <a href="https://www.amazon.com/gp/help/customer/display.html?nodeId=GKMQC26VQQMM8XSW">their website</a>, Amazon has dropped support for
all 8 devices produced before the Kindle Paperwhite 2<sup>nd</sup> generation,
first sold in 2013. To put things in perspective, the first Kindle came out in
2007, 3 years before Kobo started selling devices. Like Rakuten Kobo, Amazon
does not make promises of long-term software support as part of their
marketing. </p>
<h4>Apple</h4>
<p>Apple has a very clear <a href="https://support.apple.com/en-gb/HT201624">software support policy</a> for all their
devices:</p>
<blockquote>
<p><em>Owners of iPhone, iPad, iPod or Mac products may obtain a service and parts
from Apple or Apple service providers for five years after the product is no
longer sold – or longer, where required by law.</em></p>
</blockquote>
<p>This means in the worst-case scenario of buying an iPhone model just as it is
discontinued, one would get a minimum of 5 years of software support.</p>
<h4>Android</h4>
<p>Google's policy for their Android devices is to provide software support for <a href="https://support.google.com/pixelphone/answer/4457705?hl=en">3
years after the launch date</a>. If you buy a Pixel device just
before the new one launches, you could theoretically only get 2 years of
support. In 2018, Google decided OEMs would have to provide security updates
for <a href="https://www.theverge.com/2018/10/24/18019356/android-security-update-mandate-google-contract">at least 2 years</a> after launch, threatening not to license
Google Apps and the Play Store if they didn't comply.</p>
<h2>A question of cost structure</h2>
<p>From the previous section, we can conclude that in general, e-readers seem to
be supported longer than smartphones, and that Apple does a better job than
Android OEMs, providing support for about twice as long.</p>
<p>Even Fairphone, <strong>who's entire business is to build phones designed to last and
to be repaired</strong> was not able to keep the Fairphone 1 (2013) updated for more
than <a href="https://en.wikipedia.org/wiki/Fairphone_1#Software_updates">a couple years</a> and <a href="https://en.wikipedia.org/wiki/Fairphone_2#Software">seems to be struggling</a> to keep the
Fairphone 2 (2015) running an up to date version of Android.</p>
<p>Anyone who has ever worked in IT will tell you: maintaining software over time
is hard work and hard work by specialised workers is expensive. Most commercial
electronic devices are sold and developed by for-profit enterprises and
software support all comes down to a question of cost structure. If companies
like Google or Fairphone are to be expected to provide long-term support for
the devices they manufacture, they have to be able to fund their work somehow.</p>
<p>In a perfect world, people would be paying for the cost of said long-term
support, as it would likely be cheaper then buying new devices every few years
and would certainly be better for the planet. Problem is, manufacturers aren't
making them pay for it.</p>
<p>Economists call this type of problem externalities: things that should be
part of the cost of a good, but aren't for one a reason or another. A classic
example of an externality is pollution. Clearly pollution is bad and leads to
horrendous consequences, like climate change. Sane people agree we should
drastically cut our greenhouse gas emissions, and yet, we aren't.</p>
<p>Neo-classical economic theory argues the way to fix externalities like
pollution is to <em>internalise</em> these costs, in other words, to make people pay
for the "real price" of the goods they buy. In the case of climate change and
pollution, neo-classical economic theory is plain wrong (spoiler alert: it
often is), but this is where band-aids like the carbon tax comes from.</p>
<p>Still, coming back to long-term software support, let's see what would happen
if we were to try to internalise software maintenance costs. We can do this
multiple ways.</p>
<h3>1 - Include the price of software maintenance in the cost of the device</h3>
<p>This is the choice Fairphone makes. This might somewhat work out for them since
they are a very small company, but it cannot scale for the following reasons:</p>
<ol>
<li>
<p>This strategy relies on you giving your money to an enterprise <u>now</u>,
and trusting them to "Do the right thing" <u>years later</u>. As the years
go by, they will eventually look at their books, see how much ongoing
maintenance is costing them, drop support for the device, apologise and move
on. That is to say, enterprises have a clear economic incentive to promise
long-term support and not deliver. One could argue a company's reputation
would suffer from this kind of behaviour. Maybe sometime it does, but most
often people forget. Political promises are a great example of this.</p>
</li>
<li>
<p>Enterprises go bankrupt all the time. Even if company X promises 15 years of
software support for their devices, if they cease to exist, your device will
stop getting updates. The internet is full of stories of IoT devices getting
bricked when the parent company goes bankrupt and their servers disappear.
This is related to point number 1: to some degree, you have a disincentive
to pay for long-term support in advance, as the future is uncertain and
there are chances you won't get the support you paid for.</p>
</li>
<li>
<p>Selling your devices at a higher price to cover maintenance costs does not
necessarily mean you will make more money overall — raising more money to
fund maintenance costs being the goal here. To a certain point, smartphone
models are <a href="https://en.wikipedia.org/wiki/Substitute_good">substitute goods</a> and prices higher than market prices will
tend to drive consumers to buy cheaper ones. There is thus a disincentive to
include the price of software maintenance in the cost of the device.</p>
</li>
<li>
<p>People tend to be bad at rationalising the total cost of ownership over a
long period of time. Economists call this phenomenon
<a href="https://en.wikipedia.org/wiki/Hyperbolic_discounting">hyperbolic discounting</a>. In our case, it means people are far more likely
to buy a 500$ phone each 3 years than a 1000$ phone each 10 years. Again,
this means OEMs have a clear disincentive to include the price of long-term
software maintenance in their devices.</p>
</li>
</ol>
<p>Clearly, life is more complex than how I portrayed it: enterprises are not
perfect rational agents, altruism exists, not all enterprises aim solely for
profit maximisation, etc. Still, in a capitalist economy, enterprises wanting
to charge for software maintenance upfront have to overcome these hurdles one
way or another if they want to avoid failing.</p>
<h3>2 - The subscription model</h3>
<p>Another way companies can try to internalise support costs is to rely on a
subscription-based revenue model. This has multiple advantages over the previous
option, mainly:</p>
<ol>
<li>
<p>It does not affect the initial purchase price of the device, making it easier
to sell them at a competitive price.</p>
</li>
<li>
<p>It provides a stable source of income, something that is <em>very</em> valuable to
enterprises, as it reduces overall risks. This in return creates an incentive
to continue providing software support as long as people are paying.</p>
</li>
</ol>
<p>If this model is so interesting from an economic incentives point of view, why
isn't any smartphone manufacturer offering that kind of program? The answer is,
they are, but not explicitly<sup id="fnref:subscribe"><a class="footnote-ref" href="#fn:subscribe">5</a></sup>.</p>
<p>Apple and Google can fund part of their smartphone software support via the 30%
cut they take out of their respective app stores. A <a href="https://sensortower.com/blog/app-revenue-and-downloads-2019">report from Sensor
Tower</a> shows that in 2019, Apple made an estimated US$ 16 billion
from the App Store, while Google raked in US$ 9 billion from the Google Play
Store. Although the <a href="https://fortune.com/fortune500/2019">Fortune 500 ranking</a> tells us this respectively
is "only" 5.6% and 6.5% of their gross annual revenue for 2019, the profit
margins in this category are certainly higher than any of their other products.</p>
<p>This means Google and Apple have an important incentive to keep your device
updated for some time: if your device works well and is updated, you are more
likely to keep buying apps from their store. When software support for a device
stops, there is a risk paying customers will buy a competitor device and leave
their ecosystem.</p>
<p>This also explains why OEMs who don't own app stores tend not to provide
software support for very long periods of time. Most of them only make money
when you buy a new phone. Providing long-term software support thus becomes a
disincentive, as it directly reduces their sale revenues.</p>
<p>Same goes for Kindles and Kobos: the longer your device works, the more money
they make with their electronic book stores. In my opinion, it's likely Amazon
and Rakuten Kobo produce quarterly cost-benefit reports to decide when to drop
support for older devices, based on ongoing support costs and the recurring
revenues these devices bring in.</p>
<p>Rakuten Kobo is also in a more precarious situation than Amazon is: considering
Amazon's very important market share, if your device stops getting new updates,
there is a greater chance people will replace their old Kobo with a Kindle.
Again, they have an important economic incentive to keep devices running as long
as they are profitable.</p>
<h2>Can Free Software fix this?</h2>
<p>Yes and no. Free Software certainly isn't a magic wand one can wave to make
everything better, but does provide major advantages in terms of security, user
freedom and sometimes costs. The last piece of the puzzle explaining why Rakuten
Kobo's software support is better than Google's is technological choices.</p>
<p>Smartphones are incredibly complex devices and have become the main computing
platform of many. Similar to the web, there is a race for features and
complexity that tends to create bloat and make older devices slow and painful
to use. On the other hand, e-readers are simpler devices built for a single
task: display electronic books.</p>
<p>Control over the platform is also a key aspect of the cost structure of
providing software updates. Whereas Apple controls both the software and
hardware side of iPhones, Android is a sad mess of drivers and SoCs, all
providing different levels of support over time<sup id="fnref:snapdragon"><a class="footnote-ref" href="#fn:snapdragon">6</a></sup>.</p>
<p>If you take a look at the platforms the Kindle and Kobo are built on, you'll
quickly see they both use <a href="https://en.wikipedia.org/wiki/I.MX">Freescale I.MX SoCs</a>. These processors
are well known for their excellent upstream support in the Linux kernel and
their relative longevity, chips being produced for either 10 or 15 years. This
in turn makes updates much easier and less expensive to provide.</p>
<p>So clearly, open architectures, free drivers and open hardware helps
tremendously, but aren't enough on their own. One of the lessons we must learn
from the (amazing) LineageOS project is how lack of funding hurts everyone.</p>
<p>If there is no one to do the volunteer work required to maintain a version of
LOS for your device, it won't be supported. Worse, when purchasing a new
device, users cannot know in advance how many years of LOS support they will
get. This makes buying new devices a frustrating hit-and-miss experience. <a href="https://wiki.lineageos.org/devices/jflteatt">If
you are lucky</a>, you will get many years of support. Otherwise, you risk
your device becoming an expensive insecure paperweight.</p>
<p>So how do we fix this? Anyone with a brain understands throwing away perfectly
good devices each 2 years is not sustainable. Government regulations enforcing
a minimum support life would be a step in the right direction, but at the end
of the day, Capitalism is to blame. Like the aforementioned carbon tax, band-aid
solutions can make things somewhat better, but won't fix our current economic
system's underlying problems.</p>
<p>For now though, I'll leave fixing the problem of Capitalism to someone else.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:dune">
<p>My most recent novel binge has been focused on re-reading the <em>Dune</em>
franchise. I first read the 6 novels written by Frank Herbert when I was 13
years old and only had vague and pleasant memories of his work. Great stuff. <a class="footnote-backref" href="#fnref:dune" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:nexus">
<p>I'm back on LineageOS! Nice folks released an <a href="https://forum.xda-developers.com/google-nexus-5/orig-development/rom-lineageos-17-1-nexus-5-hammerhead-t4039273">unofficial LOS
17.1</a> port for the Nexus 5 last January and have kept it updated since
then. If you are to use it, I would also recommend updating TWRP <a href="https://forum.xda-developers.com/google-nexus-5/orig-development/recovery-twrp-hh-nexus-5-hammerhead-t4047653">to this
version</a> specifically patched for the Nexus 5. <a class="footnote-backref" href="#fnref:nexus" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:incentives">
<p>Very few serious economists actually believe neo-classical
rational agent theory is a satisfactory explanation of human behavior. In my
opinion, it's merely a (mostly flawed) lens to try to interpret certain
behaviors, a tool amongst others that needs to be used carefully, preferably
as part of a <a href="https://en.wikipedia.org/wiki/Pluralism_in_economics">pluralism of approaches</a>. <a class="footnote-backref" href="#fnref:incentives" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
<li id="fn:ereader_sales">
<p>Good data on the e-reader market is hard to come by and is
mainly produced by specialised market research companies selling their
findings at very high prices. Those particular statistics come from a
<a href="https://www.marketwatch.com/press-release/ereader-market-size-share-2020-by-development-history-business-prospect-trend-key-manufacturers-price-supply-demand-growth-factor-and-end-user-analysis-outlook-till-2024-2020-07-20">MarketWatch</a> analysis. <a class="footnote-backref" href="#fnref:ereader_sales" title="Jump back to footnote 4 in the text">↩</a></p>
</li>
<li id="fn:subscribe">
<p>If they were to tell people: <em>You need to pay us 5$/month if you
want to receive software updates</em>, I'm sure most people would not pay. Would
you? <a class="footnote-backref" href="#fnref:subscribe" title="Jump back to footnote 5 in the text">↩</a></p>
</li>
<li id="fn:snapdragon">
<p>Coming back to Fairphones, if they had so much problems
providing an Android 9 build for the Fairphone 2, it's because <a href="https://www.xda-developers.com/in-depth-capitulation-of-why-msm8974-devices-are-excluded-from-nougat/">Qualcomm
never provided</a> Android 7+ support for the Snapdragon 801 SoC it
uses. <a class="footnote-backref" href="#fnref:snapdragon" title="Jump back to footnote 6 in the text">↩</a></p>
</li>
</ol>
</div>Trying out Sourcehut2019-10-10T00:00:00-04:002019-10-10T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2019-10-10:/trying-out-sourcehut.html<p><a href="https://github.com/isbg/isbg/issues/131">Last month</a>, I decided it was finally time to move a project I maintain
from Github<sup id="fnref:migrating"><a class="footnote-ref" href="#fn:migrating">1</a></sup> to another git hosting platform.</p>
<p>While polling other contributors (I proposed moving to gitlab.com), someone
suggested moving to <a href="https://sourcehut.org/">Sourcehut</a>, a newish git hosting platform written
and maintained by Drew DeVault. I've been …</p><p><a href="https://github.com/isbg/isbg/issues/131">Last month</a>, I decided it was finally time to move a project I maintain
from Github<sup id="fnref:migrating"><a class="footnote-ref" href="#fn:migrating">1</a></sup> to another git hosting platform.</p>
<p>While polling other contributors (I proposed moving to gitlab.com), someone
suggested moving to <a href="https://sourcehut.org/">Sourcehut</a>, a newish git hosting platform written
and maintained by Drew DeVault. I've been following Drew's work for a while now
and although I had read a few blog posts on Sourcehut's development, I had
never really considered giving it a try. So I did!</p>
<p>Sourcehut is still in alpha and I'm expecting a lot of things to change in the
future, but here's my quick review.</p>
<h2>Things I like</h2>
<h3>Sustainable FOSS</h3>
<p>Sourcehut is 100% Free Software. Github is proprietary and I dislike Gitlab's
Open Core business model.</p>
<p>Sourcehut's business model also seems sustainable to me, as it relies on people
paying a monthly fee for the service. You'll need to pay if you want your code
hosted on <a href="https://sr.ht">https://sr.ht</a> once Sourcehut moves into beta. As I've written
previously, <a href="/paying-for-the-internet.html">I like that a lot</a>.</p>
<p>In comparison, Gitlab is mainly funded by venture capital and I'm afraid of the
long term repercussions this choice will have.</p>
<h3>Continuous Integration</h3>
<p>Continuous Integration is very important to me and I'm happy to say Sourcehut's
CI is pretty good! Like Travis and Gitlab CI, you declare what needs to happen
in a YAML file. The CI uses real virtual machines backed by QEMU, so you can
run many different distros and CPU archs!</p>
<p>Even nicer, you can actually SSH into a failed CI job to debug things. In
comparison, Gitlab CI's <a href="https://docs.gitlab.com/ce/ci/interactive_web_terminal/">Interactive Web Terminal</a> is ... web
based and thus not as nice. Worse, it seems it's still somewhat buggy as Gitlab
still hasn't enabled it on their gitlab.com instance.</p>
<p>Here's what the instructions to SSH into the CI look like when a job fails:</p>
<div class="highlight"><pre><span></span><code><span class="n">This</span><span class="w"> </span><span class="n">build</span><span class="w"> </span><span class="n">job</span><span class="w"> </span><span class="n">failed</span><span class="p">.</span><span class="w"> </span><span class="n">You</span><span class="w"> </span><span class="n">may</span><span class="w"> </span><span class="nf">log</span><span class="w"> </span><span class="k">into</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">failed</span><span class="w"> </span><span class="n">build</span><span class="w"> </span><span class="n">environment</span><span class="w"> </span><span class="k">within</span><span class="w"> </span><span class="mi">10</span>
<span class="n">minutes</span><span class="w"> </span><span class="k">to</span><span class="w"> </span><span class="n">examine</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">results</span><span class="w"> </span><span class="k">with</span><span class="w"> </span><span class="n">the</span><span class="w"> </span><span class="n">following</span><span class="w"> </span><span class="nl">command</span><span class="p">:</span>
<span class="n">ssh</span><span class="w"> </span><span class="o">-</span><span class="n">t</span><span class="w"> </span><span class="n">builds</span><span class="nv">@foo</span><span class="p">.</span><span class="n">bar</span><span class="w"> </span><span class="k">connect</span><span class="w"> </span><span class="n">NUMBER</span>
</code></pre></div>
<p>Sourcehut's CI is not as feature-rich or as flexible as Gitlab CI, but I feel
it is more powerful then Gitlab CI's default docker executor. Folks that run
integration tests or more complicated setups where Docker fails should
definitely give it a try.</p>
<p>From the few tests I did, Sourcehut's CI is also pretty quick (it's definitely
faster than Travis or Gitlab CI).</p>
<h3>No JS</h3>
<p>Although Sourcehut's web interface does bundle some Javascript, all features
work without it. Three cheers for that!</p>
<h2>Things I dislike</h2>
<h3>Features division</h3>
<p>I'm not sure I like the way features (the issue tracker, the CI builds, the git
repository, the wikis, etc.) are subdivided in different subdomains.</p>
<p>For example, when you create a git repository on <code>git.sr.ht</code>, you only get a git
repository. If you want an issue tracker for that git repository, you have to
create one at <code>todo.sr.ht</code> with the same name. That issue tracker isn't visible
from the git repository web interface.</p>
<p>That's the same for all the features. For example, you don't see the build
status of a merged commit when you look at it. This design choice makes you feel
like the different features aren't integrated to one another.</p>
<p>In comparison, Gitlab and Github use a more "centralised" approach: everything
is centered around a central interface (your git repository) and it feels more
natural to me.</p>
<h3>Discoverability</h3>
<p>I haven't seen a way to search <code>sr.ht</code> for things hosted there. That makes it
hard to find repositories, issues or even the Sourcehut source code!</p>
<h3>Merge Request workflow</h3>
<p>I'm a sucker for the Merge Request workflow. I really like to have a big green
button I can click on to merge things. I know some people prefer a more manual
workflow that uses <code>git merge</code> and stuff, but I find that tiresome.</p>
<p>Sourcehut chose a workflow based on sending patches by email. It's neat since
you can submit code without having an account. Sourcehut also provides mailing
lists for projects, so people can send patches to a central place.</p>
<p>I find that workflow harder to work with, since to me it makes it more
difficult to see what patches have been submitted. It also makes the review
process more tedious, since the CI isn't ran automatically on email patches.</p>
<h2>Summary</h2>
<p>All in all, I don't think I'll be moving ISBG to Sourcehut (yet?). At the
moment it doesn't quite feel as ready as I'd want it to be, and that's OK. Most
of the things I disliked about the service can be fixed by some UI work and I'm
sure people are already working on it.</p>
<p>Github was bought by MS for 7.5 billion USD and Gitlab is currently valued at
2.7 billion USD. It's not really fair to ask Sourcehut to fully compete just
yet :)</p>
<p>With Sourcehut, Drew DeVault is fighting the good fight and I wish him the most
resounding success. Who knows, maybe I'll really migrate to it in a few years!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:migrating">
<p>Github is a proprietary service, <a href="/lets-migrate-away-from-github.html">has been bought by Microsoft</a>
and gosh darn do I hate Travis CI. <a class="footnote-backref" href="#fnref:migrating" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Archiving 20 years of online content2019-09-19T00:00:00-04:002019-09-19T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2019-09-19:/archiving-20-years-of-online-content.html<p>Last Spring at work I was tasked with archiving all of the digital content made
by <em>Association pour une Solidarité Syndicale Étudiante (ASSÉ)</em>, a student union
federation that was planned to shut down six months later.</p>
<p>Now that I've done it, I have to say it was quite a demanding …</p><p>Last Spring at work I was tasked with archiving all of the digital content made
by <em>Association pour une Solidarité Syndicale Étudiante (ASSÉ)</em>, a student union
federation that was planned to shut down six months later.</p>
<p>Now that I've done it, I have to say it was quite a demanding task: ASSÉ was
founded in 2001 and neither had proper digital archiving policies nor good web
practices in general.</p>
<p>The goal was not only archiving those web resources, but also making sure they
were easily accessible online too. I thus decided to create <a href="https://asse-solidarite.qc.ca">a meta site</a>
regrouping and presenting all of them.</p>
<p>All in all, I archived:</p>
<ul>
<li>a Facebook page</li>
<li>a Twitter account</li>
<li>a YouTube account</li>
<li>multiple ephemeral websites</li>
<li>old handcrafted PHP4 websites that I had to partially re-write</li>
<li>a few crummy Wordpress sites</li>
<li>2 old phpBB Forum using PHP3</li>
<li>a large Mailman2 mailing list</li>
<li>a large Yahoo! Group mailing list</li>
</ul>
<p>Here are the three biggest challenges I faced during this project:</p>
<h2>The Twitter API has stupid limitations</h2>
<p>The Twitter API won't give you more than an account's last 3000 posts. When you
need to automate the retrieval of more than 5500 tweets, you know you're
entering a world of pain.</p>
<p>Long story short, I ended up writing this <em>crummy</em> shell script to parse the
HTML, statify all the Twitter links and push the resulting code to a Wordpress
site using <a href="https://github.com/ozh/ozh-tweet-archive-theme">Ozh' Tweet Archive Theme</a>. The URL list was generated
using the ArchiveTeam's web crawler.</p>
<p>Of course, once done I made the Wordpress into a static website. I personally
think the result <a href="https://oiseau.asse-solidarite.qc.ca/">looks purty</a>.</p>
<p>Here's the shell script I wrote - displayed here for archival purposes only.
Let's pray I don't ever have to do this again. Please don't run this, as it
might delete your grandma.</p>
<div class="highlight"><pre><span></span><code>cat<span class="w"> </span><span class="nv">$1</span><span class="w"> </span><span class="p">|</span><span class="w"> </span><span class="k">while</span><span class="w"> </span><span class="nb">read</span><span class="w"> </span>line
<span class="k">do</span>
<span class="w"> </span><span class="c1"># get the ID</span>
<span class="w"> </span><span class="nv">id</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$line</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">'s@https://mobile.twitter.com/.\+/status/@@'</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># download the whole HTML page</span>
<span class="w"> </span><span class="nv">html</span><span class="o">=</span><span class="k">$(</span>curl<span class="w"> </span>-s<span class="w"> </span><span class="nv">$line</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># get the date</span>
<span class="w"> </span><span class="nv">date</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$html</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-A<span class="w"> </span><span class="m">1</span><span class="w"> </span><span class="s1">'<div class="metadata">'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-o<span class="w"> </span><span class="s2">"[0-9].\+20[0-9][0-9]"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">'s/ - //'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>date<span class="w"> </span>-f<span class="w"> </span>-<span class="w"> </span>+<span class="s2">"%F %k:%M:%S"</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># extract the tweet</span>
<span class="w"> </span><span class="nv">tweet</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$html</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s1">'<div class="dir-ltr" dir="ltr">'</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># we strip the HTML tags for the title</span>
<span class="w"> </span><span class="nv">title</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$tweet</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/<[^>]*>//g'</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># get CSV list of tags</span>
<span class="w"> </span><span class="nv">tags</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$tweet</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-io<span class="w"> </span><span class="s2">"\#[a-z]\+"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">':a;N;$!ba;s/\n/,/g'</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># get a CSV list of links</span>
<span class="w"> </span><span class="nv">links</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$tweet</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-Po<span class="w"> </span><span class="s2">"title=\"http.*?>"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">'s/title=\"//; s/">//'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">':a;N;$!ba;s/\n/,/g'</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># get a CSV list of usernames</span>
<span class="w"> </span><span class="nv">usernames</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$tweet</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span>-Po<span class="w"> </span><span class="s2">">\@.*?<"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">'s/>//; s/<//'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">':a;N;$!ba;s/\n/,/g'</span><span class="k">)</span>
<span class="w"> </span><span class="nv">image_link</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$html</span><span class="s2">"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>grep<span class="w"> </span><span class="s2">"<img src=\"https://pbs.twimg.com/media/"</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">'s/:small//'</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># remove twitter cruft</span>
<span class="w"> </span><span class="nv">tweet</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$tweet</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span><span class="s1">'s/<div class="dir-ltr" dir="ltr"> /<p>/'</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>perl<span class="w"> </span>-pe<span class="w"> </span><span class="s1">'s@<a href="/hashtag.*?dir="ltr">@<span class="hashtag hashtag_local">@g'</span><span class="k">)</span>
<span class="w"> </span><span class="c1"># expand links</span>
<span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-z<span class="w"> </span><span class="nv">$links</span><span class="w"> </span><span class="o">]</span>
<span class="k">then</span>
<span class="w"> </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">','</span><span class="w"> </span><span class="nb">read</span><span class="w"> </span>-ra<span class="w"> </span>link<span class="w"> </span><span class="o"><<<</span><span class="w"> </span><span class="s2">"</span><span class="nv">$links</span><span class="s2">"</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">link</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">tweet</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$tweet</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>perl<span class="w"> </span>-pe<span class="w"> </span><span class="s2">"s@<a href=\"*.+?rel=\"nofollow noopener\"dir=\"ltr\"data-*.+?</a>@<a href='</span><span class="nv">$i</span><span class="s2">'></span><span class="nv">$i</span><span class="s2"></a>@"</span><span class="k">)</span>
<span class="w"> </span><span class="k">done</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="c1"># replace hashtags by links</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-z<span class="w"> </span><span class="nv">$tags</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">','</span><span class="w"> </span><span class="nb">read</span><span class="w"> </span>-ra<span class="w"> </span>tag<span class="w"> </span><span class="o"><<<</span><span class="w"> </span><span class="s2">"</span><span class="nv">$tags</span><span class="s2">"</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">tag</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">plain</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/#//'</span><span class="k">)</span>
<span class="w"> </span><span class="nv">tweet</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$tweet</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s2">"s@</span><span class="nv">$i</span><span class="s2">@#<a href=\"https://oiseau.asse-solidarite.qc.ca/index.php/tag/</span><span class="nv">$plain</span><span class="s2">\"></span><span class="nv">$plain</span><span class="s2">@"</span><span class="k">)</span>
<span class="w"> </span><span class="k">done</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="c1"># replace usernames by links</span>
<span class="w"> </span><span class="k">if</span><span class="w"> </span><span class="o">[</span><span class="w"> </span>!<span class="w"> </span>-z<span class="w"> </span><span class="nv">$usernames</span><span class="w"> </span><span class="o">]</span>
<span class="w"> </span><span class="k">then</span>
<span class="w"> </span><span class="nv">IFS</span><span class="o">=</span><span class="s1">','</span><span class="w"> </span><span class="nb">read</span><span class="w"> </span>-ra<span class="w"> </span>username<span class="w"> </span><span class="o"><<<</span><span class="w"> </span><span class="s2">"</span><span class="nv">$usernames</span><span class="s2">"</span>
<span class="w"> </span><span class="k">for</span><span class="w"> </span>i<span class="w"> </span><span class="k">in</span><span class="w"> </span><span class="s2">"</span><span class="si">${</span><span class="nv">username</span><span class="p">[@]</span><span class="si">}</span><span class="s2">"</span>
<span class="w"> </span><span class="k">do</span>
<span class="w"> </span><span class="nv">plain</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$i</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sed<span class="w"> </span>-e<span class="w"> </span><span class="s1">'s/\@//'</span><span class="k">)</span>
<span class="w"> </span><span class="nv">tweet</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$tweet</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>perl<span class="w"> </span>-pe<span class="w"> </span><span class="s2">"s@<a href=\"/</span><span class="nv">$plain</span><span class="s2">.*?</a>@<span class=\"username username_linked\">\@<a href=\"https://twitter.com/</span><span class="nv">$plain</span><span class="s2">\"></span><span class="nv">$plain</span><span class="s2"></a></span>@i"</span><span class="k">)</span>
<span class="w"> </span><span class="k">done</span>
<span class="w"> </span><span class="k">fi</span>
<span class="w"> </span><span class="c1"># replace images</span>
<span class="w"> </span><span class="nv">tweet</span><span class="o">=</span><span class="k">$(</span><span class="nb">echo</span><span class="w"> </span><span class="nv">$tweet</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>perl<span class="w"> </span>-pe<span class="w"> </span><span class="s2">"s@<a href=\"http://t.co*.+?data-pre-embedded*.+?</a>@<span class=\"embed_image embed_image_yes\"></span><span class="nv">$image_link</span><span class="s2"></span>@"</span><span class="k">)</span>
<span class="nb">echo</span><span class="w"> </span><span class="nv">$tweet</span><span class="w"> </span><span class="p">|</span><span class="w"> </span>sudo<span class="w"> </span>-u<span class="w"> </span>twitter<span class="w"> </span>wp-cli<span class="w"> </span>post<span class="w"> </span>create<span class="w"> </span>-<span class="w"> </span>--post_title<span class="o">=</span><span class="s2">"</span><span class="nv">$title</span><span class="s2">"</span><span class="w"> </span>--post_status<span class="o">=</span><span class="s2">"publish"</span><span class="w"> </span>--tags_input<span class="o">=</span><span class="s2">"</span><span class="nv">$tag</span><span class="s2">"</span><span class="w"> </span>--post_date<span class="o">=</span><span class="s2">"</span><span class="nv">$date</span><span class="s2">"</span><span class="w"> </span>><span class="w"> </span>tmp
<span class="nv">post_id</span><span class="o">=</span><span class="k">$(</span>grep<span class="w"> </span>-Po<span class="w"> </span><span class="s2">"[0-9]{4}"</span><span class="w"> </span>tmp<span class="k">)</span>
sudo<span class="w"> </span>-u<span class="w"> </span>twitter<span class="w"> </span>wp-cli<span class="w"> </span>post<span class="w"> </span>meta<span class="w"> </span>add<span class="w"> </span><span class="nv">$post_id</span><span class="w"> </span>ozh_ta_id<span class="w"> </span><span class="nv">$id</span>
<span class="nb">echo</span><span class="w"> </span><span class="s2">"</span><span class="nv">$post_id</span><span class="s2"> created"</span>
rm<span class="w"> </span>tmp
<span class="k">done</span>
</code></pre></div>
<h2>Does anyone ever update phpBBs?</h2>
<p>What's worse than a phpBB forum? Two phpBB 2.0.x forums using PHP3 and last
updated in 2006.</p>
<p>I had to resort to unholy methods just to be able to get those things running
again to be able to <code>wget</code> the crap out of them.</p>
<p>By the way, the magic <code>wget</code> command to grab a whole website looks like this:</p>
<pre>
wget --mirror -e robots=off --page-requisites --adjust-extension -nv --base=./ --convert-links --directory-prefix=./ -H -D www.foo.org,foo.org http://www.foo.org/
</pre>
<p>Depending on the website you are trying to archive, you might have to play with
other obscure parameters. I sure had to. All the credits for that command goes
to <a href="https://wiki.koumbit.net/Fossilisation#Prendre_une_copie_statique_du_site">Koumbit's wiki page on the dark arts of website statification</a>.</p>
<h2>Archiving mailing lists</h2>
<p>mailman2 is pretty great. You can get a dump of an email list pretty easily and
mailman3's web frontend, the lovely <a href="https://gitlab.com/mailman/hyperkitty">hyperkitty</a>, is well, lovely.
Importing a legacy mailman2 <code>mbox</code> went without a hitch thanks to the awesome
<code>hyperkitty_import</code> importer. Kudos to the Debian Mailman Team for packaging
this in Debian for us.</p>
<p>But what about cramming a Yahoo! Group mailing list in hyperkitty? I wouldn't
recommend it. After way too many hours spent battling character encoding errors
I just decided people that wanted to read obscure emails from 2003 would have
to deal with broken accents and shit. <a href="https://support.asse-solidarite.qc.ca/list/asse-support@groupesyahoo.ca/">But hey, it kinda works!</a></p>
<p>Oh, and yes, archiving a Yahoo! Group with <a href="https://sourceforge.net/projects/grabyahoogroup/">an old borken Perl script</a>
wasn't an easy task. Hell, I kept getting blacklisted by Yahoo! for scraping
too much data to their liking. I ended up patching together the results of
multiple runs over a few weeks to get the full mbox and attachments.</p>
<p>By the way, if anyone knows how to tell hyperkitty to stop at a certain year
(i.e. not display links for 2019 when the list stopped in 2006), please ping
me.</p>Paying for The Internet2019-08-06T00:00:00-04:002019-08-06T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2019-08-06:/paying-for-the-internet.html<p>For a while now, I've been paying for The Internet. Not the internet connection
provided by my ISP, mind you, but for the stuff I enjoy online and the services
I find useful.</p>
<p>Most of the Internet as we currently know it is funded by ads. I hate ads and …</p><p>For a while now, I've been paying for The Internet. Not the internet connection
provided by my ISP, mind you, but for the stuff I enjoy online and the services
I find useful.</p>
<p>Most of the Internet as we currently know it is funded by ads. I hate ads and I
take a vicious pride in blocking them with the help of great projects like
<a href="https://github.com/gorhill/uBlock/">uBlock Orign</a> and <a href="https://noscript.net/">NoScript</a>. More fundamentally, I believe
the web shouldn't be funded via ads:</p>
<ul>
<li>they control your brain (that alone should be enough to ban ads)</li>
<li>they create morally wrong economic incentives towards consumerism</li>
<li>they create important security risks and make websites gather data on you</li>
</ul>
<p><img src="/media/blog/2019-08-06/sale_pub_sexiste.jpg" title="A sticker with a feminist anti-ads message" alt="A sticker with a feminist anti-ads message" height="30%" width="30%" style="float:right"></p>
<p>I could go on like this, but I feel those are pretty strong arguments. Feel free
to disagree.</p>
<p>So I've started paying. Paying for my emails. Paying for the comics I enjoy
online <sup id="fnref:comics"><a class="footnote-ref" href="#fn:comics">1</a></sup>. Paying for the few YouTube channels I like. Paying for the
newspapers I read.</p>
<p>At the moment, The Internet costs me around 260 USD per year. Luckily for me,
I'm privileged enough that it doesn't have a significant impact on my finances.
I also pay for a lot of the software I use and enjoy by making patches and
spending time working on them. I feel that's a valid way to make The Internet a
more sustainable place.</p>
<p>I don't think individual actions like this one have a very profound impact on
how things work, but like riding your bike to work or eating locally produced
organic food, it opens a window into a possible future. A better future.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:comics">
<p>I currently like these comics enough to pay for them:</p>
<ul>
<li><a href="https://existentialcomics.com/">Existential Comics</a></li>
<li><a href="http://thedevilspanties.com/">The Devil's Panties</a></li>
<li><a href="http://www.lunarbaboon.com/">Lunarbaboon</a></li>
<li><a href="https://questionablecontent.net/">Questionable Content</a></li>
<li><a href="https://www.smbc-comics.com/">SMBC Comics</a></li>
<li><a href="https://www.mrlovenstein.com/">Mr. Lovenstein</a></li>
</ul>
<p><a class="footnote-backref" href="#fnref:comics" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
</ol>
</div>Running Android 9 (Pie) on a Nexus 5 -- Unlegacy Android2019-04-24T00:00:00-04:002019-04-24T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2019-04-24:/running-android-9-pie-on-a-nexus-5-unlegacy-android.html<p>Like most good stories, this one starts in a bar with friends. <a href="https://veronneau.org/debconf-videoteam-sprint-fosdem-report.html">I had the chance
to go to FOSDEM this year</a> and I managed to get a bunch of people from
the DebConf videoteam to meet me at Brew Dog Bruxelles.</p>
<p>Like the respectable geeks we are, the first …</p><p>Like most good stories, this one starts in a bar with friends. <a href="https://veronneau.org/debconf-videoteam-sprint-fosdem-report.html">I had the chance
to go to FOSDEM this year</a> and I managed to get a bunch of people from
the DebConf videoteam to meet me at Brew Dog Bruxelles.</p>
<p>Like the respectable geeks we are, the first thing we did after exchanging
greeting was to connect to the local Wi-Fi. That's when Stefano Rivera looked
at my phone and casually said: “Wow, that's running Android 7, it's <em>ancient!</em>”.</p>
<p>I've had my Nexus 5 for a while now. If I recall correctly, I bought it sometime
in Q1 2014. Although already dated at the time, it was a great phone: fast,
cheap and easy to repair. The Nexus 5 hasn't been supported by Google for years,
but until recently LineageOS did a great job at keeping the OS patched and
updated. Sadly, it seems my device won't get ported to LineageOS 16. <a href="https://lineageos.org/Changelog-22/">As
announced here</a>, the 14.1 branch won't get security support anymore.</p>
<p>Lucky for me, I found the amazing <a href="https://forum.xda-developers.com/google-nexus-5/orig-development/rom-unlegacy-android-project-t3593425">Unlegacy Android</a> project. Their
goal is to build and patch the Android Open Source Project (AOSP) for older
devices. The difference with LineageOS is that they do not customize the ROMs,
thus making patch porting easier and less troublesome. A few models are
supported, but their main focus is the Nexus 5.</p>
<h2>Installing Unlegacy Android</h2>
<p>If you have a Nexus 5 and want to upgrade to a bleeding edge version of Android,
you have two options:</p>
<ol>
<li>You can install one of the builds provided by the Unlegacy Android project.
Since they still consider Android 9 experimental (even though it works
pretty much flawlessly), those builds are hosted on a <a href="https://notredame.app.box.com/s/26a4bygh9vbaw7jjq08xr5evomvaw5ww">shady box.com cloud
storage</a>.</li>
<li>You can build the ROM yourself, following the <a href="https://wiki.lineageos.org/devices/sailfish/build">LineageOS wiki</a>
instruction while replacing the code repository with the <a href="https://github.com/Unlegacy-Android">Unlegacy</a>
one.</li>
</ol>
<p>Although it's a bit more work, I prefer to build the project myself. Something
about downloading an update from a random box.com cloud storage doesn't feel
right to me. Once a month, when the monthly <a href="https://source.android.com/security/bulletin/">Android Security Patch</a>
is merged in the Unlegacy repository, I boot a VM on my server and build a new
OTA update.</p>
<p>Unlegacy Android also provides official <a href="https://builds.unlegacy-android.org/">Android 7 and Android 8
builds</a> if you are into that.</p>
<h2>What about bugs?</h2>
<p>I've been using Unlegacy Android with the latest Android 9 builds for 3 months
now and the only bugs I've experienced are:</p>
<ul>
<li>On an encrypted device, TWRP isn't able to decrypt the partitions, rendering
it somewhat useless. You can still update the device via ADB sideload though.</li>
<li>Bluetooth voice calls (HFP) are garbled. Playing music via A2DP works fine
though.</li>
</ul>
<p>I also seen more random app crashes than when I was using LineageOS 14.1, but
nothing extreme. The overall experience is great: my device is snappy, runs the
latest AOSP build and I neither have bloatware nor Google Apps.</p>
<p>All in all, considering new phones running Android 9 start at 500 USD and that
you can buy a new Nexus 5 for less than 100 USD on ebay, it's a pretty
compelling choice.</p>Roundcube fr_FEM locale 1.4-rc12019-03-01T00:00:00-05:002019-03-01T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2019-03-01:/roundcube-fr_fem-locale-14-rc1.html<p>Roundcube <code>1.4-rc1</code> was released today and with it, I've released version
<code>1.4-rc1</code> of my <code>fr_FEM</code> (French gender-neutral) locale.</p>
<p>I'm really excited by the new Roundcube version, as it adds a bunch of neat
encryption stuff and more importantly, now uses a new responsive skin. I'm sure
mobile users …</p><p>Roundcube <code>1.4-rc1</code> was released today and with it, I've released version
<code>1.4-rc1</code> of my <code>fr_FEM</code> (French gender-neutral) locale.</p>
<p>I'm really excited by the new Roundcube version, as it adds a bunch of neat
encryption stuff and more importantly, now uses a new responsive skin. I'm sure
mobile users will enjoy that new feature a lot.</p>
<p>If you happen to run Roundcube, please test this new locale release! Feedback is
always welcome.</p>
<p>You can find the locale <a href="https://github.com/baldurmen/roundcube_fr_FEM">here</a>.</p>From Cooperation to Competition or How Code Became Proprietary2018-12-29T00:00:00-05:002018-12-29T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2018-12-29:/from-cooperation-to-competition-or-how-code-became-proprietary.html<p>After two semesters of hard work, I'm happy to say I've finished all the classes
I had to take for my Master's degree. If I work hard enough, I should be able to
finish writing my thesis by the end of August.</p>
<p>Last night, I handed out the final paper …</p><p>After two semesters of hard work, I'm happy to say I've finished all the classes
I had to take for my Master's degree. If I work hard enough, I should be able to
finish writing my thesis by the end of August.</p>
<p>Last night, I handed out the final paper for my <em>History of the Economic
Thought</em> class. Titled "From Cooperation to Competition or How Code Became
Proprietary", it is a research paper on the evolution of the Copyright Law in
the United States with regards to computer code.</p>
<p>Here's the abstract:</p>
<hr>
<p>With the emergence of computer science as a academic research domain in the 60s,
a new generation of students gain access to computers. This generation will
become the first wave of hackers, a community based on a strong ethos promoting
curiosity and knowledge sharing. The hazy legal framework of computer code
intellectual property at the time enables this community to grow and to put
forward cooperation as a mean of economic production. Meanwhile in the 60s, the
American politicians take advantage of the revision of the <em>Copyright Act</em> to
think about the implications of applying copyright to computer code. After
failing to tackle computer issues in the 1976 revision of the <em>Copyright Act</em>,
the Senate creates the Commission on New Technological Uses of Copyrighted Works
(CONTU). The final report of the CONTU will eventually lead computer code to be
recognised as a literary work and thus copyrightable. A few years later,
American courts finally create a strong case law on these issues with the 1983
Apple v. Franklin court case.</p>
<hr>
<p>Sadly, I haven't had time to translate the whole paper to English yet (it's in
French). Maybe if enough people are interested by theses issues I'll take the
time to do so.</p>
<p>If you want to read my paper in French, you can download the PDF <a href="https://veronneau.org/media/blog/2018-12-29/paper.pdf">here</a>.</p>Let's migrate away from GitHub2018-06-03T00:00:00-04:002018-06-03T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2018-06-03:/lets-migrate-away-from-github.html<p>As many of you heard today, <a href="https://www.bloomberg.com/news/articles/2018-06-03/microsoft-is-said-to-have-agreed-to-acquire-coding-site-github">Microsoft is acquiring GitHub</a>. What
this means for the future of GitHub is not yet clear, but <a href="https://about.gitlab.com/2018/06/03/microsoft-acquires-github/">the folks at
Gitlab</a> think Microsoft's end goal is to integrate GitHub in their
Azure empire. To me, this makes a lot of sense.</p>
<p>Even though I …</p><p>As many of you heard today, <a href="https://www.bloomberg.com/news/articles/2018-06-03/microsoft-is-said-to-have-agreed-to-acquire-coding-site-github">Microsoft is acquiring GitHub</a>. What
this means for the future of GitHub is not yet clear, but <a href="https://about.gitlab.com/2018/06/03/microsoft-acquires-github/">the folks at
Gitlab</a> think Microsoft's end goal is to integrate GitHub in their
Azure empire. To me, this makes a lot of sense.</p>
<p>Even though I still reluctantly use GitHub for some projects, I migrated all
my personal repositories to Gitlab instances a while ago<sup id="fnref:1"><a class="footnote-ref" href="#fn:1">1</a></sup>. Now is time for
you to do the same and ditch GitHub.</p>
<p><img src="/media/blog/2018-06-03/ms-lovent-linux.png" title="Microsoft loven't Linux" alt="Microsft loven't Linux" height="25%" width="25%" style="float:right"></p>
<p>Some people might be fine with Microsoft's takeover, but to me it's the straw
that breaks the camel's back. For a few years now, MS has been running a large
marketing campaign on how they love Linux and suddenly decided to embrace Free
Software in all of its forms. More like MS BS to me.</p>
<p>Let us take a moment to remind ourselves that:</p>
<ul>
<li>Windows is still a huge proprietary monster that rips billions of people from
their privacy and rights every day.</li>
<li>Microsoft is known for spreading FUD about "the dangers" of Free Software in
order to keep governments and schools from dropping Windows in favor of FOSS.</li>
<li>To secure their monopoly, Microsoft hooks up kids on Windows by giving out
"free" licences to primary schools around the world. Drug dealers use the same
tactics and give out free samples to secure new clients.</li>
<li>Microsoft's Azure platform - even though it can run Linux VMs - is still a
giant proprietary hypervisor.</li>
</ul>
<p>I know moving git repositories around can seem like a pain in the ass, but the
folks at Gitlab are riding the wave of people leaving GitHub and made the the
migration easy <a href="https://docs.gitlab.com/ee/user/project/import/github.html">by providing a GitHub importer</a>.</p>
<p>If you don't want to use Gitlab's main instance (<a href="https://gitlab.org">gitlab.org</a>), here
are two other alternative instances you can use for Free Software projects:</p>
<ul>
<li>The <a href="https://salsa.debian.org">Debian Gitlab instance</a> is available for every FOSS project and
not only for Debian-related ones<sup id="fnref:2"><a class="footnote-ref" href="#fn:2">2</a></sup>. As long as the project respects the
<a href="https://en.wikipedia.org/wiki/Debian_Free_Software_Guidelines">Debian Free Software Guidelines</a>, you can use the instance and its CI
runners.</li>
<li>Riseup maintains a Gitlab instance for radical projects named <a href="https://0xacab.org">0xacab</a>. If
your <a href="https://riseup.net/en/about-us/politics">ethos aligns with Riseup's</a>, chances are they'll be happy to host
your projects there.</li>
</ul>
<p>Friends don't let friends use GitHub anymore.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:1">
<p>Gitlab is pretty good, but it should not be viewed as a panacea: it's
still an open-core product made by a for-profit enterprise that could one
day be sold to a large corp like Oracle or Microsoft. <a class="footnote-backref" href="#fnref:1" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:2">
<p>See the <a href="https://wiki.debian.org/Salsa/FAQ#What_can_be_hosted_on_salsa">Salsa FAQ</a> for more details. <a class="footnote-backref" href="#fnref:2" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
</ol>
</div>Minimal SQL privileges2018-03-17T00:00:00-04:002018-03-17T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2018-03-17:/minimal-sql-privileges.html<p>Lately, I have been working pretty hard on a paper I have to hand out at the end
of my university semester for the machine learning class I'm taking. I will
probably do a long blog post about this paper in May if it turns out to be good,
but …</p><p>Lately, I have been working pretty hard on a paper I have to hand out at the end
of my university semester for the machine learning class I'm taking. I will
probably do a long blog post about this paper in May if it turns out to be good,
but for the time being I have some time to kill while my latest boosting model
runs.</p>
<p>So let's talk about something I've started doing lately: creating issues on FOSS
webapp project trackers when their documentation tells people to grant all
privileges to the database user.</p>
<p>You know, something like:</p>
<div class="highlight"><pre><span></span><code><span class="k">GRANT</span><span class="w"> </span><span class="k">ALL</span><span class="w"> </span><span class="k">PRIVILEGES</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="k">database</span><span class="p">.</span><span class="o">*</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="s1">'username'</span><span class="nv">@'localhost'</span><span class="w"> </span><span class="k">IDENTIFIED</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="s1">'password'</span><span class="p">;</span>
</code></pre></div>
<p>I'd like to say I've never done this and always took time to specify a
restricted subset of privileges on my servers, but I'd be lying. To be honest, I
woke up last Christmas when someone told me it was an insecure practice.</p>
<p>When you take a few seconds to think about it, there are quite a few database
level <a href="https://mariadb.com/kb/en/library/grant/#database-privileges">SQL privileges</a> and I don't see why I should grant them all to a
webapp if it only needs a few of them.</p>
<p>So I started asking projects to do something about this and update their
documentation with a minimal set of SQL privileges needed to run correctly. The
Drupal project <a href="https://api.drupal.org/api/drupal/INSTALL.mysql.txt/7.x">does this quite well</a> and tells you to:</p>
<div class="highlight"><pre><span></span><code><span class="k">GRANT</span><span class="w"> </span><span class="k">SELECT</span><span class="p">,</span><span class="w"> </span><span class="k">INSERT</span><span class="p">,</span><span class="w"> </span><span class="k">UPDATE</span><span class="p">,</span><span class="w"> </span><span class="k">DELETE</span><span class="p">,</span><span class="w"> </span><span class="k">CREATE</span><span class="p">,</span><span class="w"> </span><span class="k">DROP</span><span class="p">,</span><span class="w"> </span><span class="k">INDEX</span><span class="p">,</span><span class="w"> </span><span class="k">ALTER</span><span class="p">,</span><span class="w"> </span><span class="k">CREATE</span><span class="w"> </span><span class="k">TEMPORARY</span><span class="w"> </span><span class="k">TABLES</span><span class="w"> </span><span class="k">ON</span><span class="w"> </span><span class="n">databasename</span><span class="p">.</span><span class="o">*</span><span class="w"> </span><span class="k">TO</span><span class="w"> </span><span class="s1">'username'</span><span class="nv">@'localhost'</span><span class="w"> </span><span class="k">IDENTIFIED</span><span class="w"> </span><span class="k">BY</span><span class="w"> </span><span class="s1">'password'</span><span class="p">;</span>
</code></pre></div>
<p>When I first reached out to the upstream devs of these projects, I was sure I'd
be seen as some zealous nuisance. To my surprise, everyone thought it was a good
idea and fixed it.</p>
<p>Shout out to <a href="https://github.com/nextcloud/documentation/issues/648">Nextcloud</a>, <a href="https://github.com/mattermost/mattermost-server/issues/8432">Mattermost</a> and <a href="https://github.com/kanboard/kanboard/issues/3699">KanBoard</a> for taking this
seriously!</p>
<p>If you are using a webapp and the documentation states you should grant all
privileges to the database user, here is a template you can use to create an
issue and ask them to change it:</p>
<pre>
Hi!
The installation documentation says that you should grant all SQL privileges to
the database user:
GRANT ALL PRIVILEGES ON database.* TO 'username'@'localhost' IDENTIFIED BY 'password';
I was wondering what are the true minimal SQL privileges WEBAPP needs to run
properly.
I don't normally like to grant all privileges for security reasons and would
really appreciate it if you could publish a minimal SQL database privileges
list.
I guess I'm expecting something like [Drupal][drupal] does.
GRANT SELECT, INSERT, UPDATE, DELETE, CREATE, DROP, INDEX, ALTER, CREATE TEMPORARY TABLES ON databasename.* TO 'username'@'localhost' IDENTIFIED BY 'password';
At the database level, [MySQL/MariaDB][mariadb] supports:
* `ALTER`
* `CREATE`
* `CREATE ROUTINE`
* `CREATE TEMPORARY TABLES`
* `CREATE VIEW`
* `DELETE`
* `DELETE HISTORY`
* `DROP`
* `EVENT`
* `INDEX`
* `INSERT`
* `LOCK TABLES`
* `REFERENCES`
* `SELECT`
* `SHOW VIEW`
* `TRIGGER`
* `UPDATE`
Does WEBAPP really need database level privileges like EVENT or CREATE ROUTINE?
If not, why should I grant them?
Thanks for your work on WEBAPP!
[drupal]: https://api.drupal.org/api/drupal/INSTALL.mysql.txt/7.x
[mariadb]: https://mariadb.com/kb/en/library/grant/#database-privileges
</pre>Roundcube fr_FEM locale 1.3.52018-03-16T00:00:00-04:002018-03-16T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2018-03-16:/roundcube-fr_fem-locale-135.html<p>Roundcube <code>1.3.5</code> was released today and with it, I've released version <code>1.3.5</code>
of my <code>fr_FEM</code> (French gender-neutral) locale.</p>
<p>This latest version is actually the first one that can be used with a
production version of Roundcube: the first versions I released were based on
the latest …</p><p>Roundcube <code>1.3.5</code> was released today and with it, I've released version <code>1.3.5</code>
of my <code>fr_FEM</code> (French gender-neutral) locale.</p>
<p>This latest version is actually the first one that can be used with a
production version of Roundcube: the first versions I released were based on
the latest commit in the master branch at the time instead of an actual
release. Not sure why I did that.</p>
<p>I've also changed the versioning scheme to follow Roundcube's. Version <code>1.3.5</code>
of my localisation is thus compatible with Roundcube <code>1.3.5</code>. Again, I should
have done that from the start.</p>
<p>The fine folks at <a href="https://riseup.net">Riseup</a> actually started using <code>fr_FEM</code> as the default
French locale on their instance and I'm happy to say the UI integration seems
to be working pretty well.</p>
<p>Sandro Knauß (hefee), who is working on the Debian Roundcube package, also told
me he'd like to replace the default Roundcube French locale by <code>fr_FEM</code> in
Debian. Nice to see people think a gender-neutral locale is a good idea!</p>
<p>Finally, since this was the first time I had to compare two different releases
of Roundcube to see if the 20 files I care about had changed, I decided to
write a <a href="https://github.com/baldurmen/roundcube_fr_FEM/blob/master/roundcube_diff.sh">simple script</a> that leverages git to do this automatically.
Running <code>./roundcube_diff.sh -p git_repo -i 1.3.4 -f 1.3.5 -l fr_FR -o
roundcube_diff.txt</code> outputs a nice file that tells you if new localisation
files have been added and displays what changed in the old ones.</p>
<p>You can find the locale <a href="https://github.com/baldurmen/roundcube_fr_FEM">here</a>.</p>Playing with water2018-03-14T00:00:00-04:002018-03-14T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2018-03-14:/playing-with-water.html<p><img src="/media/blog/2018-03-14/h2o_job.png" title="H2o Flow gradient boosting job" alt="H2o Flow gradient boosting job" height="30%" width="30%" style="float:right"></p>
<p>I'm currently taking a machine learning class and although it is an insane
amount of work, I like it a lot. I initially had planned to use <a href="https://en.wikipedia.org/wiki/R_(programming_language)">R</a> to play
around with the database I have, but the teacher recommended I use <a href="https://www.h2o.ai">H2o</a>, a
FOSS machine learning framework.</p>
<p>I was …</p><p><img src="/media/blog/2018-03-14/h2o_job.png" title="H2o Flow gradient boosting job" alt="H2o Flow gradient boosting job" height="30%" width="30%" style="float:right"></p>
<p>I'm currently taking a machine learning class and although it is an insane
amount of work, I like it a lot. I initially had planned to use <a href="https://en.wikipedia.org/wiki/R_(programming_language)">R</a> to play
around with the database I have, but the teacher recommended I use <a href="https://www.h2o.ai">H2o</a>, a
FOSS machine learning framework.</p>
<p>I was a bit sceptical at first since I'm already pretty good with R, but then I
found out you could simply import H2o as an R library. H2o replaces most R
functions by its own parallelized ones to cut down on processing time (no more
<code>doParallel</code> calls) and uses an "external" server you have to run on the side
instead of running R calls directly.</p>
<p><img src="/media/blog/2018-03-14/h2o_model.png" title="H2o Flow gradient boosting model" alt="H2o Flow gradient boosting model" height="30%" width="30%" style="float:left"></p>
<p>I was pretty happy with this situation, that is until I actually started using
H2o in R. With the huge database I'm playing with, the library felt clunky and I
had a hard time doing anything useful. Most of the time, I just ended up with
long Java traceback calls. Much love.</p>
<p>I'm sure in the right hands using H2o as a library could have been incredibly
powerful, but sadly it seems I haven't earned my black belt in R-fu yet.</p>
<p><img src="/media/blog/2018-03-14/h2o_var_importance.png" title="H2o Flow variable importance weights" alt="H2o Flow variable importance weights" height="30%" width="30%" style="float:right"></p>
<p>I was pissed for at least a whole day - not being able to achieve what I wanted
to do - until I realised H2o comes with a WebUI called Flow. I'm normally not
very fond of using web thingies to do important work like writing code, but Flow
is simply incredible.</p>
<p>Automated graphing functions, integrated ETA when running resource intensive
models, descriptions for each and every model parameters (the parameters are
even divided in sections based on your familiarly with the statistical models in
question), Flow seemingly has it all. In no time I was able to run 3 basic
machine learning models and get actual interpretable results.</p>
<p>So yeah, if you've been itching to analyse very large databases using state of
the art machine learning models, I would recommend using H2o. Try Flow at first
instead of the Python or R hooks to see what it's capable of doing.</p>
<p>The only downside to all of this is that H2o is written in Java and depends on
Java 1.7 to run... That, and be warned: it requires a metric fuckton of
processing power and RAM. My poor server struggled quite a bit even with 10
available cores and 10Gb of RAM...</p>French Gender-Neutral Translation for Roundcube2018-01-22T00:00:00-05:002018-01-22T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2018-01-22:/french-gender-neutral-translation-for-roundcube.html<p>Here's a quick blog post to tell the world I'm now doing a French
gender-neutral translation for Roundcube.</p>
<p>A while ago, someone wrote on the Riseup translation list to complain against
the current <code>fr_FR</code> translation. French is indeed a very gendered language and
it is common place in radical spaces …</p><p>Here's a quick blog post to tell the world I'm now doing a French
gender-neutral translation for Roundcube.</p>
<p>A while ago, someone wrote on the Riseup translation list to complain against
the current <code>fr_FR</code> translation. French is indeed a very gendered language and
it is common place in radical spaces to use gender-neutral terminologies.</p>
<p>So yeah, here it is: <a href="https://github.com/baldurmen/roundcube_fr_FEM">https://github.com/baldurmen/roundcube_fr_FEM</a></p>
<p>I haven't tested the UI integration yet, but I'll do that once the Riseup folks
integrate it to their Roundcube instance.</p>