Louis-Philippe Véronneau - pythonhttps://veronneau.org/2023-05-29T00:00:00-04:00Python 3.11, pip and (breaking) system packages2023-05-29T00:00:00-04:002023-05-29T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2023-05-29:/python-311-pip-and-breaking-system-packages.html<p>As we get closer to <a href="https://micronews.debian.org/2023/1682655083.html">Debian Bookworm's release</a>, I thought I'd share
one change in Python 3.11 that will surely affect many people.</p>
<p>Python 3.11 implements the new <a href="https://peps.python.org/pep-0668/">PEP 668</a>, <em>Marking Python base environments
as “externally managed”</em><sup id="fnref:kudos"><a class="footnote-ref" href="#fn:kudos">1</a></sup>. If you use <code>pip</code> regularly on Debian, it's
likely you'll …</p><p>As we get closer to <a href="https://micronews.debian.org/2023/1682655083.html">Debian Bookworm's release</a>, I thought I'd share
one change in Python 3.11 that will surely affect many people.</p>
<p>Python 3.11 implements the new <a href="https://peps.python.org/pep-0668/">PEP 668</a>, <em>Marking Python base environments
as “externally managed”</em><sup id="fnref:kudos"><a class="footnote-ref" href="#fn:kudos">1</a></sup>. If you use <code>pip</code> regularly on Debian, it's
likely you'll eventually hit the <code>externally-managed-environment</code> error:</p>
<pre>
error: externally-managed-environment
× This environment is externally managed
╰─> To install Python packages system-wide, try apt install
python3-xyz, where xyz is the package you are trying to
install.
If you wish to install a non-Debian-packaged Python package,
create a virtual environment using python3 -m venv path/to/venv.
Then use path/to/venv/bin/python and path/to/venv/bin/pip. Make
sure you have python3-full installed.
If you wish to install a non-Debian packaged Python application,
it may be easiest to use pipx install xyz, which will manage a
virtual environment for you. Make sure you have pipx installed.
See /usr/share/doc/python3.11/README.venv for more information.
note: If you believe this is a mistake, please contact your Python installation or OS distribution provider. You can override this, at the risk of breaking your Python installation or OS, by passing --break-system-packages.
hint: See PEP 668 for the detailed specification.
</pre>
<p>With this PEP, Python tools can now distinguish between packages that have been
installed by the user with a tool like <code>pip</code> and ones installed using a
distribution's package manager, like <code>apt</code>.</p>
<p>This is generally great news: it was previously too easy to break a system by
mixing the two types of packages. This PEP will simplify our role as a
distribution, as well as improve the overall Python user experience in Debian.</p>
<p>Sadly, it's also likely this change will break some of your scripts, especially
CI that (legitimately) install packages via <code>pip</code> alongside system packages.
For example, I use the following <code>gitlab-ci</code> snippet to make sure my PRs don't
break my build process<sup id="fnref:build"><a class="footnote-ref" href="#fn:build">2</a></sup>:</p>
<pre>
build:flit:
stage: build
script:
- apt-get update && apt-get install -y flit python3-pip
- FLIT_ROOT_INSTALL=1 flit install
- metalfinder --help
</pre>
<p>With Python 3.11, this snippet will error out, as <code>pip</code> will refuse to install
packages alongside the system's. The fix is to tell <code>pip</code> it's OK to "break"
your system packages, either using the <code>--break-system-packages</code> parameter, or
the <code>PIP_BREAK_SYSTEM_PACKAGES=1</code> environment variable<sup id="fnref:search"><a class="footnote-ref" href="#fn:search">3</a></sup>.</p>
<p>This, of course, is not something you should be using in production to restore
the old behavior! The "proper" way to fix this issue, as the
<code>externally-managed-environment</code> error message aptly (har har) informs you, is
to use virtual environments.</p>
<p>Happy hacking!</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:kudos">
<p>Kudos to our own Matthias Klose, Stefano Rivera and Elana Hashman,
who worked on designing and implementing this PEP! <a class="footnote-backref" href="#fnref:kudos" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:build">
<p>Which is something that bit me before... You push some changes to
your git repository, everything seems fine and all the tests pass, so you
merge it and make a new git tag.
When the time comes to build and upload this tag to PyPi, you find out some
minor thing broke your build system (which you weren't testing) and you
have to scramble to make a point-release to fix the issue. Sad! <a class="footnote-backref" href="#fnref:build" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:search">
<p>Don't go searching for this environment variable in <code>pip</code>'s code
though, as you won't find it! All of <code>pip</code>'s command line options <a href="https://pip.pypa.io/en/stable/topics/configuration/#environment-variables">can be
passed as env vars</a> using the <code>PIP_<UPPER_LONG_NAME></code> format.
Useful for tools that use <code>pip</code> indirectly, like <code>flit</code>. <a class="footnote-backref" href="#fnref:search" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Debian Python Team 2022 Sprint Report2022-12-08T00:00:00-05:002022-12-08T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2022-12-08:/debian-python-team-2022-sprint-report.html<p>This is the report for the <a href="https://wiki.debian.org/Sprints/2022/PythonTeam">Debian Python Team remote sprint</a>
that took place on December 2-3-4 2022.</p>
<p>Many thanks to those who participated, namely:</p>
<ul>
<li>Étienne Mollier (emollier)</li>
<li>Taihsiang Ho (tai271828)</li>
<li>Athos Ribeiro (athos)</li>
<li>Stuart Prescott (stuart)</li>
<li>Louis-Philippe Véronneau (pollo)</li>
<li>Ileana Dumitrescu (ildumi)</li>
<li>James Valleroy (jvalleroy)</li>
<li>Emmanuel Arias (eamanu)</li>
<li>Kurt …</li></ul><p>This is the report for the <a href="https://wiki.debian.org/Sprints/2022/PythonTeam">Debian Python Team remote sprint</a>
that took place on December 2-3-4 2022.</p>
<p>Many thanks to those who participated, namely:</p>
<ul>
<li>Étienne Mollier (emollier)</li>
<li>Taihsiang Ho (tai271828)</li>
<li>Athos Ribeiro (athos)</li>
<li>Stuart Prescott (stuart)</li>
<li>Louis-Philippe Véronneau (pollo)</li>
<li>Ileana Dumitrescu (ildumi)</li>
<li>James Valleroy (jvalleroy)</li>
<li>Emmanuel Arias (eamanu)</li>
<li>Kurt Kremitzki (kkremitzki)</li>
<li>Mohammed Bilal (rmb)</li>
<li>Stefano Rivera (tumbleweed)</li>
<li>Jeroen Ploemen (jcfp)</li>
</ul>
<p>Here is a list of issues we worked on:</p>
<h2>pybuild autodep8 feature</h2>
<p>About a year ago, <a href="https://salsa.debian.org/python-team/tools/dh-python/-/merge_requests/27">Antonio Terceiro contributed code</a> to <code>pybuild</code> to
make it possible to automatically run the upstream test suite as autopkgtests.</p>
<p>This feature has now been merged and uploaded to unstable. Although you can
find out more about it in the <a href="https://manpages.debian.org/unstable/dh-python/pybuild-autopkgtest.1.en.html">pybuild-autopkgtest manpage</a>, an email
providing more details should be sent to the debian-python mailing list
relatively soon.</p>
<h2>Fixing packages that run tests via <code>python3 setup.py test</code></h2>
<p>Last August, Stefano Rivera <a href="https://lists.debian.org/debian-python/2022/08/msg00046.html">poked the team </a> about the deprecation
of the <code>python3 setup.py test</code> command to run tests in <code>pybuild</code>. Although this
feature has been <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=982298">deprecated upstream for 6 years now</a>, many packages
in the archive still use it to run the upstream test suite during build.</p>
<p>Around 29 of the 67 packages that are team-maintained by the Debian Python Team
were fixed during the sprint. Ideally, all of them would be before the feature
is removed from <code>pybuild</code>.</p>
<p>if a package you maintain still runs this command, please consider fixing it!</p>
<h2>Fixing packages that use <code>nose</code></h2>
<p><code>nose</code>, provided by the <code>python3-nose</code> package, is an obsolete testing
framework for Python and has been <a href="https://github.com/nose-devs/nose/commit/0f40fa995384afad">unmaintained since 2015</a>.</p>
<p>During the sprint, people worked on fixing some of the many bugs filled against
packages still running tests via <code>nose</code>, but there are still <a href="https://bugs.debian.org/cgi-bin/pkgreport.cgi?users=python-modules-team@lists.alioth.debian.org;tag=nose-rm">around 240
packages</a> affected by this issue in the archive.</p>
<p>Again, if a package you maintain still runs this command, please consider
fixing it!</p>
<h2>Removal of the remaining Python2 packages</h2>
<p>With the upload of <code>dh-python 5.20221202</code>, Stefano Rivera officially removed
support for <code>dh_python2</code> and <code>dh_pypy</code>, thus closing the <a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=942959">"Python2 removal in
sid/bullseye"</a> bug.</p>
<p>It seems some work still needs to be done for <a href="https://release.debian.org/transitions/html/python2-rm.html">complete Python2
removal from Sid</a>, but I expect this will be done in time for the
Bookworm release.</p>
<h2>Working on Lintian tags for the Team</h2>
<p>During the sprint, I managed to work on some Lintian issues that we had
targeted, namely:</p>
<ul>
<li>the addition of a <a href="https://salsa.debian.org/lintian/lintian/-/merge_requests/429">"missing-cpython-extension"</a> tag to flag packages
that do not build CPython extensions for all the supported Python versions.</li>
<li>fixing a false-positive in the
<a href="https://salsa.debian.org/lintian/lintian/-/merge_requests/426">"missing-prerequisite-for-pyproject-backend"</a> tag.</li>
</ul>
<p>I also worked on a few other Lintian tags, but they were unrelated to the
Debian Python Team itself.</p>
<p>I'm also happy to report many of the tags I wrote for the team in the past few
months were merged by the awesome Russ Allbery and should land in unstable as
soon as a new release is made.</p>
<p>I'm particularly looking forward the new <a href="https://salsa.debian.org/lintian/lintian/-/merge_requests/404">"uses-python-distutils"</a> tag
that should help us flag packages that still use the deprecated <code>distutils</code>
library.</p>
<h2>Patching distro-tracker (tracker.debian.org) to show pending team MRs</h2>
<p>It's often hard to have a good overview of pending merge requests when working
with team-maintained packages, as by default, Salsa doesn't notify anyone when
a MR is opened.</p>
<p>Although our workflow typically does not involve creating merge requests, <a href="https://salsa.debian.org/groups/python-team/packages/-/merge_requests">some
people still do</a> and they end up sitting there, unnoticed.</p>
<p>During the sprint, Kurt Kremitzki worked on solving this issue by having
<code>distro-tracker</code> show the pending MRs on our <a href="https://tracker.debian.org/teams/python/">team's tracker page</a>.</p>
<p>Sadly, it seems little progress was made, as the removal of
<code>python3-django-jsonfield</code> from the archive and breaking changes in
<code>python3-selenium</code> has broken the test suite.</p>
<h2>Migrate packages building with the flit plugin to the generic pyproject one</h2>
<p><code>pybuild</code> has been supporting building with PEP-517 style <code>pyproject.toml</code> files
via a generic plugin (<code>pybuild-plugin-pyproject</code>) for a while now.</p>
<p>As this plugin supersedes the old <code>flit</code> plugin, we've been thinking of
deprecating it in time for the Bookworm release.</p>
<p>To make this possible, most of the packages in the archive that still used this
plugin were migrated to the generic one and I opened bugs on the last handful
of packages that were not team-maintained.</p>
<h2>Other work</h2>
<p>Many other things were done during the sprint, such as:</p>
<ul>
<li>improving existing packages (for example, adding or fixing autopkgtests)</li>
<li>triaging and fixing <a href="https://bugs.debian.org/cgi-bin/pkgreport.cgi?tag=python3.11&users=debian-python@lists.debian.org">Python 3.11 bugs</a></li>
<li>updating packages to the latest upstream release</li>
<li>sponsoring uploads</li>
</ul>
<h2>Thanks</h2>
<p>Thanks again to everyone who joined the sprint, and three big cheers for all
the folks who <a href="https://www.debian.org/donations">donate to Debian</a> and made it possible for us to have a
food budget for the event.</p>Introducing metalfinder2022-05-20T00:00:00-04:002022-05-20T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2022-05-20:/introducing-metalfinder.html<p>After going to an incredible <a href="https://www.bandsintown.com/e/103112574">Arch Enemy / Behemoth / Napalm Death / Unto
Others</a> concert a few weeks ago, I decided I wanted to go to more
concerts.</p>
<p>I like music, and I <em>really</em> enjoy concerts. Sadly, I often miss great
performances because no one told me about it, or my local …</p><p>After going to an incredible <a href="https://www.bandsintown.com/e/103112574">Arch Enemy / Behemoth / Napalm Death / Unto
Others</a> concert a few weeks ago, I decided I wanted to go to more
concerts.</p>
<p>I like music, and I <em>really</em> enjoy concerts. Sadly, I often miss great
performances because no one told me about it, or my local newspaper didn't cover
the event enough in advance for me to get tickets.</p>
<p>Some online services lets you sync your Spotify account to notify you when a
new concert is announced, but I don't use Spotify. As a music geek, I have a
local music collection and if I need to stream it, I have a <a href="https://packages.debian.org/bullseye/supysonic">supysonic</a>
server.</p>
<p>Introducing <a href="https://gitlab.com/baldurmen/metalfinder">metalfinder</a>, a cli tool to find concerts using your local music
collection! At the moment, it scans your music collection, creates a list of
artists and queries <a href="https://www.bandsintown.com">Bandsintown</a> for concerts in your town. Multiple output
formats are supported, but I mainly use the ATOM one, as I'm a heavy feed
reader user.</p>
<p><img src="/media/blog/2022-05-20/atom.png" width="70%" style="margin-left:15%" title="Screenshot of the ATOM output in my feed reader" alt="Screenshot of the ATOM output in my feed reader"></p>
<p>The current metalfinder version (<code>1.1.1</code>) is a MVP: it works well enough, but I
still have <a href="https://gitlab.com/baldurmen/metalfinder/-/issues/">a lot of work to do</a>... If you want to give it a try, the
easiest way is to download it from PyPi. metalfinder is also <a href="https://ftp-master.debian.org/new/metalfinder_1.0.2-1.html">currently in
NEW</a> and I'm planning to have something feature complete in time for the
Bookworm freeze.</p>