Louis-Philippe Véronneau - puppethttps://veronneau.org/2021-01-09T00:00:00-05:00puppetserver 6: a Debian packaging post-mortem2021-01-09T00:00:00-05:002021-01-09T00:00:00-05:00Louis-Philippe Véronneautag:veronneau.org,2021-01-09:/puppetserver-6-a-debian-packaging-post-mortem.html<p>I have been a Puppet user for a couple of years now, first at work, and
eventually for my personal servers and computers. Although it can have a steep
learning curve, I find Puppet both nimble and very powerful. I also prefer it
to Ansible for its speed and the …</p><p>I have been a Puppet user for a couple of years now, first at work, and
eventually for my personal servers and computers. Although it can have a steep
learning curve, I find Puppet both nimble and very powerful. I also prefer it
to Ansible for its speed and the agent-server model it uses.</p>
<p>Sadly, Puppet Labs hasn't been the most supportive upstream and tends to move
pretty fast. Major versions rarely last for a whole Debian Stable release and
the upstream <code>.deb</code> packages are full of vendored libraries.<sup id="fnref:openssl"><a class="footnote-ref" href="#fn:openssl">1</a></sup></p>
<p>Since 2017, Apollon Oikonomopoulos has been the one doing most of the work on
Puppet in Debian. Sadly, he's had less time for that lately and with Puppet 5
being deprecated in January 2021, Thomas Goirand, Utkarsh Gupta and I have been
trying to package Puppet 6 in Debian for the last 6 months.</p>
<p>With Puppet 6, the old ruby Puppet server using Passenger is not supported
anymore and has been replaced by <code>puppetserver</code>, written in Clojure and running
on the JVM. That's quite a large change and although <code>puppetserver</code> does reuse
some of the Clojure libraries <code>puppetdb</code> (already in Debian) uses, packaging it
meant quite a lot of work.</p>
<h2>Work in the Clojure team</h2>
<p>As part of my efforts to package <code>puppetserver</code>, I had the pleasure to join the
Clojure team and learn a lot about the Clojure ecosystem.</p>
<p>As I mentioned earlier, a lot of the Clojure dependencies needed for
<code>puppetserver</code> were already in the archive. Unfortunately, when Apollon
Oikonomopoulos packaged them, the <code>leiningen</code> build tool hadn't been packaged
yet. This meant I had to rebuild a lot of packages, on top of packaging some
new ones.</p>
<p>Since then, thanks to the efforts of Elana Hashman, <code>leiningen</code> has been
packaged and lets us run the upstream testsuites and create <code>.jar</code> artifacts
closer to those upstream releases.</p>
<p>During my work on <code>puppetserver</code>, I worked on the following packages:</p>
<details>
<summary>
List of packages
</summary>
<ul>
<li><code>backport9</code></li>
<li><code>bidi-clojure</code></li>
<li><code>clj-digest-clojure</code></li>
<li><code>clj-helper</code></li>
<li><code>clj-time-clojure</code></li>
<li><code>clj-yaml-clojure</code></li>
<li><code>cljx-clojure</code></li>
<li><code>core-async-clojure</code></li>
<li><code>core-cache-clojure</code></li>
<li><code>core-match-clojure</code></li>
<li><code>cpath-clojure</code></li>
<li><code>crypto-equality-clojure</code></li>
<li><code>crypto-random-clojure</code></li>
<li><code>data-csv-clojure</code></li>
<li><code>data-json-clojure</code></li>
<li><code>data-priority-map-clojure</code></li>
<li><code>java-classpath-clojure</code></li>
<li><code>jnr-constants</code></li>
<li><code>jnr-enxio</code></li>
<li><code>jruby</code></li>
<li><code>jruby-utils-clojure</code></li>
<li><code>kitchensink-clojure</code></li>
<li><code>lazymap-clojure</code></li>
<li><code>liberator-clojure</code></li>
<li><code>ordered-clojure</code></li>
<li><code>pathetic-clojure</code></li>
<li><code>potemkin-clojure</code></li>
<li><code>prismatic-plumbing-clojure</code></li>
<li><code>prismatic-schema-clojure</code></li>
<li><code>puppetlabs-http-client-clojure</code></li>
<li><code>puppetlabs-i18n-clojure</code></li>
<li><code>puppetlabs-ring-middleware-clojure</code></li>
<li><code>puppetserver</code></li>
<li><code>raynes-fs-clojure</code></li>
<li><code>riddley-clojure</code></li>
<li><code>ring-basic-authentication-clojure</code></li>
<li><code>ring-clojure</code></li>
<li><code>ring-codec-clojure</code></li>
<li><code>shell-utils-clojure</code></li>
<li><code>ssl-utils-clojure</code></li>
<li><code>test-check-clojure</code></li>
<li><code>tools-analyzer-clojure</code></li>
<li><code>tools-analyzer-jvm-clojure</code></li>
<li><code>tools-cli-clojure</code></li>
<li><code>tools-reader-clojure</code></li>
<li><code>trapperkeeper-authorization-clojure</code></li>
<li><code>trapperkeeper-clojure</code></li>
<li><code>trapperkeeper-filesystem-watcher-clojure</code></li>
<li><code>trapperkeeper-metrics-clojure</code></li>
<li><code>trapperkeeper-scheduler-clojure</code></li>
<li><code>trapperkeeper-webserver-jetty9-clojure</code></li>
<li><code>url-clojure</code></li>
<li><code>useful-clojure</code></li>
<li><code>watchtower-clojure</code></li>
</ul>
</details>
<p>If you want to learn more about packaging Clojure libraries and applications,
I rewrote the Debian Clojure <a href="https://wiki.debian.org/Clojure/PackagingTutorial">packaging tutorial</a> and added a section
about the quirks of using <code>leiningen</code> without a dedicated <code>dh_lein</code> tool.</p>
<h2>Work left to get puppetserver 6 in the archive</h2>
<p>Unfortunately, I was not able to finish the <code>puppetserver</code> 6 packaging work.
It is thus unlikely it will make it in Debian Bullseye. If the issues described
below are fixed, it would be possible to to package <code>puppetserver</code> in
<code>bullseye-backports</code> though.</p>
<p>So what's left?</p>
<h3>jruby</h3>
<p>Although I tried my best (kudos to Utkarsh Gupta and Thomas Goirand for the
help), <code>jruby</code> in Debian is still broken. It does build properly, but the
testsuite fails with multiple errors:</p>
<ul>
<li><code>ruby-psych</code> is broken (<a href="https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=959571">#959571</a>)</li>
<li>there are some random java failures on a few tests (no clue why)</li>
<li>tests ran by <code>raklelib/rspec.rake</code> fail to run, maybe because the <code>--pattern</code>
command line option isn't compatible with our version of <code>rake</code>? Utkarsh seemed
to know why this happens.</li>
</ul>
<p><code>jruby</code> testsuite failures aside, I have not been able to use the <code>jruby.deb</code> the
package currently builds in <code>jruby-utils-clojure</code> (testsuite failure). I had the
same exact failure with the (more broken) <code>jruby</code> version that is currently in
the archive, which leads me to think this is a <code>LOAD_PATH</code> issue in
<code>jruby-utils-clojure</code>. More on that below.</p>
<p>To try to bypass these issues, I <a href="https://salsa.debian.org/clojure-team/jruby-utils-clojure/-/commit/1499fcf367887442bf5848470458f2be02e21907">tried to <em>vendor</em></a> <code>jruby</code> into
<code>jruby-utils-clojure</code>. At first I understood <em>vendoring</em> meant including
upstream pre-built artifacts (<code>jruby-complete.jar</code>) and shipping them directly.</p>
<p>After talking with people on the <code>#debian-mentors</code> and <code>#debian-ftp</code> IRC
channels, I now understand why this isn't a good idea (and why it's not
permitted in Debian). Many thanks to the people who were patient and kind enough
to discuss this with me and give me alternatives.</p>
<p>As far as I now understand it, <em>vendoring</em> in Debian means "to have an embedded
copy of the source code in another package". Code shipped that way still needs
to be built from source. This means we need to build <code>jruby</code> ourselves, one way
or another. <em>Vendoring</em> <code>jruby</code> in another package thus isn't terribly helpful.</p>
<p>If fixing <code>jruby</code> the proper way isn't possible, I would suggest trying to
build the package using embedded code copies of the external libraries <code>jruby</code>
needs to build, instead of trying to use the Debian libraries.<sup id="fnref:ruby2.5"><a class="footnote-ref" href="#fn:ruby2.5">2</a></sup> This
should make it easier to replicate what upstream does and to have a final
<code>.jar</code> that can be used.</p>
<h3>jruby-utils-clojure</h3>
<p>This package is a first-level dependency for <code>puppetserver</code> and is the glue
between <code>jruby</code> and <code>puppetserver</code>.</p>
<p>It builds fine, but the testsuite fails when using the Debian <code>jruby</code> package. I
think the problem is caused by a <code>jruby</code> <code>LOAD_PATH</code> issue.</p>
<p>The Debian <code>jruby</code> package plays with the <code>LOAD_PATH</code> a little to try use
Debian packages instead of downloading gems from the web, as upstream <code>jruby</code>
does. This seems to clash with the <code>gem-home</code>, <code>gem-path</code>, and
<code>jruby-load-path</code> variables in the <code>jruby-utils-clojure</code> package. The testsuite
plays around with these variables and some Ruby libraries can't be found.</p>
<p>I tried to fix this, but failed. Using the upstream <code>jruby-complete.jar</code> instead
of the Debian <code>jruby</code> package, the testsuite passes fine.</p>
<p>This package could clearly be uploaded to NEW right now by ignoring the
testsuite failures (we're just packaging static <code>.clj</code> source files in the
proper location in a <code>.jar</code>).</p>
<h3>puppetserver</h3>
<p><code>jruby</code> issues aside, packaging puppetserver itself is 80% done. Using the
upstream <code>jruby-complete.jar</code> artifact, the testsuite fails with a weird
Clojure error I'm not sure I understand, but I haven't debugged it for very
long.</p>
<p>Upstream uses git submodules to vendor puppet (agent), hiera (3), facter and
puppet-resource-api for the testsuite to run properly. I haven't touched that,
but I believe we can either:</p>
<ul>
<li>link to the Debian packages</li>
<li>fix the Debian packages if they don't include the right files (maybe in a new
binary package that just ships part of the source code?)</li>
</ul>
<p>Without the testsuite actually running, it's hard to know what files are needed
in those packages.</p>
<h2>What now</h2>
<p><strong>Puppet 5 is now deprecated.</strong></p>
<p>If you or your organisation cares about Puppet in Debian,<sup id="fnref:care"><a class="footnote-ref" href="#fn:care">3</a></sup> <code>puppetserver</code>
really isn't far away from making it in the archive.</p>
<p>Very talented Debian Developers are always eager to work on these issues and
can be contracted for very reasonable rates. If you're interested in
contracting someone to help iron out the last issues, don't hesitate to reach
out via one of the following:</p>
<ul>
<li>the <a href="https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-puppet-devel">debian-puppet</a> mailing list</li>
<li>the <a href="https://lists.debian.org/debian-jobs/">debian-jobs</a> mailing list</li>
<li>the <a href="https://www.debian.org/consultants/index.html">Debian Consultants index</a></li>
<li><a href="https://www.fossjobs.net/">fossjobs.net</a></li>
</ul>
<p>As for I, I'm happy to say I got a new contract and will go back to teaching
Economics for the Winter 2021 session. I might help out with some general Debian
packaging work from time to time, but it'll be as a hobby instead of a job.</p>
<h2>Thanks</h2>
<p>The work I did during the last 6 weeks would be not have been possible without
the support of the Wikimedia Foundation, who were gracious enough to contract
me. My particular thanks to Faidon Liambotis, Moritz Mühlenhoff and John Bond.</p>
<p>Many, many thanks to Rob Browning, Thomas Goirand, Elana Hashman, Utkarsh Gupta
and Apollon Oikonomopoulos for their direct and indirect help, without which
all of this wouldn't have been possible.</p>
<div class="footnote">
<hr>
<ol>
<li id="fn:openssl">
<p>For example, the upstream package for the Puppet Agent vendors
OpenSSL. <a class="footnote-backref" href="#fnref:openssl" title="Jump back to footnote 1 in the text">↩</a></p>
</li>
<li id="fn:ruby2.5">
<p>One of the problems of using Ruby libraries already packaged in
Debian is that <code>jruby</code> currently only supports Ruby 2.5. Ruby libraries in
Debian are currently expected to work with Ruby 2.7, with the transition to
Ruby 3.0 planned after the Bullseye release. <a class="footnote-backref" href="#fnref:ruby2.5" title="Jump back to footnote 2 in the text">↩</a></p>
</li>
<li id="fn:care">
<p>If you run Puppet, you clearly should care: the <code>.deb</code> packages
upstream publishes really aren't great and I would not recommend using them. <a class="footnote-backref" href="#fnref:care" title="Jump back to footnote 3 in the text">↩</a></p>
</li>
</ol>
</div>Shared Puppet client modules group2016-05-22T00:00:00-04:002016-05-22T00:00:00-04:00Louis-Philippe Véronneautag:veronneau.org,2016-05-22:/shared-puppet-client-modules-group.html<p>SOGÉÉCOM has about 15 client computers running Linux, i.e just a bit too much to maintain them manually. We could use a <a href="http://fai-project.org/">FAI</a>, but this means we would have to reinstall all machines each time a small modification is made to our configs.</p>
<p>The obvious solution was using a …</p><p>SOGÉÉCOM has about 15 client computers running Linux, i.e just a bit too much to maintain them manually. We could use a <a href="http://fai-project.org/">FAI</a>, but this means we would have to reinstall all machines each time a small modification is made to our configs.</p>
<p>The obvious solution was using a configuration management system. Since we were already using <a href="https://puppet.com">Puppet</a> successfully to maintain our servers, we decide it would do the job for clients too.</p>
<p>And guess what, it works beautifully.</p>
<p>Inspired by the <em>awsome</em> <a href="https://gitlab.com/groups/shared-puppet-modules-group">Puppet shared modules groups</a>, from which we use some nice modules like <code>apt</code>, we decided it would be nice to pool resources and thus created a <a href="https://gitlab.com/groups/shared-puppet-client-modules-group">Puppet shared <strong>client</strong> modules groups</a>. In the months to come, we will work hard polishing the different modules we wrote to maintain the different aspects of our clients in order to make them public.</p>
<p>If you use Puppet to maintain Linux clients and wish to share your modules, please join us! We hang out on irc.indymedia.org channel #puppet and will shortly have a mailing list.</p>