blog/content/posts/installing-gerrit-and-keycloak.md

---
date: 2019-10-15T00:00:00-05:00
title: "Installing Gerrit and Keycloak for GDB"
tags: [en_us, english, debian, free-software, planet-fedora, gerrit, keycloak, howto]
---

Back in September, we had
the [GNU Tools Cauldron](https://gcc.gnu.org/wiki/cauldron2019) in the
gorgeous city of Montréal (perhaps I should write a post specifically
about it...).  One of the sessions we had was the **GDB BoF**, where
we discussed, among other things, how to improve our patch review
system.

I have my own personal opinions about the current review system we use
(mailing list-based, in a nutshell), and I haven't felt very confident
to express it during the discussion.  Anyway, the outcome was that at
least 3 global maintainers have used or are currently using
the [Gerrit Code Review](https://www.gerritcodereview.com) system for
other projects, are happy with it, and that we should give it a try.
Then, when it was time to decide who wanted to configure and set
things up for the community, I volunteered.  Hey, I'm already running
the Buildbot master for GDB, what is the problem to manage yet another
service?  Oh, well.

Before we dive into the details involved in configuring and running
gerrit in a machine, let me first say that I don't totally support the
idea of migrating from mailing list to gerrit.  I volunteered to set
things up because I felt the community (or at least the its most
active members) wanted to try it out.  I don't necessarily agree with
the choice.

Ah, and I'm writing this post mostly because I want to be able to
close the 300+ tabs I had to open on my Firefox during these last
weeks, when I was searching how to solve the myriad of problems I
faced during the set up!

The initial plan
----------------

My very initial plan after I left the session room was to talk to
the [sourceware.org](https://sourceware.org) folks and ask them if it
would be possible to host our gerrit there.  Surprisingly, they
already have a gerrit instance up and running.  It's been set up back
in 2016, it's running an old version of gerrit, and is pretty much
abandoned.  Actually, saying that it has been configured is an
overstatement: it doesn't support authentication, user registration,
barely supports projects, etc.  It's basically what you get from a
pristine installation of the gerrit RPM package in RHEL 6.

I won't go into details here, but after some discussion it was clear
to me that the instance on sourceware would not be able to meet our
needs (or at least what I had in mind for us), and that it would be
really hard to bring it to the quality level I wanted.  I decided to
go look for other options.

The OSCI folks
--------------

Have I mentioned the [OSCI project](https://osci.io/) before?  They
are absolutely **awesome**.  I really love working with them, because
so far they've been able to meet every request I made!  So, kudos to
them!  They're the folks that host
our [GDB Buildbot master](https://gdb-buildbot.osci.io).  Their
infrastructure is quite reliable (I never had a single problem), and
Marc Dequénes (Duck) is very helpful, friendly and quick when replying
to my questions :-).

So, it shouldn't come as a surprise the fact that when I decided to
look for other another place to host gerrit, they were my first
choice.  And again, they delivered :-).

Now, it was time to start thinking about the gerrit set up.

User registration?
------------------

Over the course of these past 4 weeks, I had the opportunity to learn
a bit more about how gerrit does things.  One of the first things that
negatively impressed me was the fact that gerrit doesn't handle user
registration by itself.  It is possible to have a very rudimentary
user registration "system", but it relies on the site administration
manually registering the users (via `htpasswd`) and managing
everything by him/herself.

It was quite obvious to me that we would need some kind of access
control (we're talking about a GNU project, with a copyright
assignment requirement in place, after all), and the best way to
implement it is by having registered users.  And so my quest for the
best user registration system began...

Gerrit supports some user **authentication** schemes, such
as [OpenID](https://en.wikipedia.org/wiki/OpenID) (**not** OpenID
Connect!), [OAuth2](https://en.wikipedia.org/wiki/OAuth#OAuth_2.0)
(via plugin)
and
[LDAP](https://en.wikipedia.org/wiki/Lightweight_Directory_Access_Protocol).
I remembered hearing
about [FreeIPA](https://en.wikipedia.org/wiki/FreeIPA) a long time
ago, and thought it made sense using it.  Unfortunately, the project's
community told me that installing FreeIPA on a Debian system is really
hard, and since our VM is running Debian, it quickly became obvious
that I should look somewhere else.  I felt a bit sad at the beginning,
because I thought FreeIPA would really be our silver bullet here, but
then I noticed that it doesn't really offer a self-service user
registration.

After exchanging a few emails with Marc, he told me
about [Keycloak](https://en.wikipedia.org/wiki/Keycloak).  It's a
full-fledged Identity Management and Access Management software,
supports OAuth2, LDAP, **and** provides a self-service user
registration system, which is exactly what we needed!  However, upon
reading the description of the project, I noticed that it is written
in Java (JBOSS, to be more specific), and I was afraid that it was
going to be very demanding on our system (after all, gerrit is also a
Java program).  So I decided to put it on hold and take a look at
using LDAP...

Oh, man.  Where do I start?  Actually, I think it's enough to say that
I just **tried**
installing [OpenLDAP](https://en.wikipedia.org/wiki/OpenLDAP), but
gave up because it was too cumbersome to configure.  Have you ever
heard that LDAP is really complicated?  I'm afraid this is true.  I
just didn't feel like wasting a lot of time trying to understand how
it works, only to have to solve the "user registration" problem later
(because of course, OpenLDAP is just an LDAP server).

OK, so what now?  Back to Keycloak it is.  I decided that instead of
thinking that it was too big, I should actually install it and check
it for real.  Best decision, by the way!

Setting up Keycloak
-------------------

It's pretty easy to set Keycloak up.  The official website provides a
`.tar.gz` file which contains the whole directory tree for the
project, along with helper scripts, `.jar` files, configuration, etc.
From there, you just need to follow the documentation, edit the
configuration, and voilà.

For our specific setup I chose to use PostgreSQL instead of the
built-in database.  This is a bit more complicated to configure,
because you need to download the JDBC driver, and install it in a
strange way (at least for me, who is used to just editing a
configuration file).  I won't go into details on how to do this here,
because it's easy to find on the internet.  Bear in mind, though, that
the official documentation is really incomplete when covering this
topic!  [This](https://devopstales.github.io/sso/keycloak2/) is one of
the guides I used, along
with
[this other one](http://www.janua.fr/how-to-install-keycloak-with-mariadb/) (which
covers MariaDB, but can be adapted to PostgreSQL as well).

Another interesting thing to notice is that Keycloak expects to be
running on its own virtual domain, and not under a subdirectory (e.g,
`https://example.org` instead of `https://example.org/keycloak`).  For
that reason, I chose to run our instance on another port.  It is
supposedly possible to configure Keycloak to run under a subdirectory,
but it involves editing a lot of files, and I confess I couldn't make
it fully work.

A last thing worth mentioning: the official documentation says that
Keycloak needs Java 8 to run, but I've been using OpenJDK 11 without
problems so far.

Setting up Gerrit
-----------------

The fun begins now!

The gerrit project also offers a `.war` file ready to be deployed.
After you download it, you can execute it and initialize a gerrit
project (or application, as it's called).  Gerrit will create a
directory full of interesting stuff; the most important for us is the
`etc/` subdirectory, which contains all of the configuration files for
the application.

After initializing everything, you can try starting gerrit to see if
it works.  This is where I had my first trouble.  Gerrit also requires
Java 8, but unlike Keycloak, it doesn't work out of the box with
OpenJDK 11.  I had to make a small but important addition in the file
`etc/gerrit.config`:

	[container]
        ...
        javaOptions = "--add-opens=jdk.management/com.sun.management.internal=ALL-UNNAMED"
        ...

After that, I was able to start gerrit.  And then I started trying to
set it up for OAuth2 authentication using Keycloak.  This took a
**very long** time, unfortunately.  I was having several problems with
Gerrit, and I wasn't sure how to solve them.  I
tried
[asking for help](https://groups.google.com/forum/#!topic/repo-discuss/_n5Y2vDppX8) on
the official mailing list, and was able to make some progress, but in
the end I figured out what was missing: I had forgotten to add the
`AddEncodedSlashes On` in the Apache configuration file!  This was
causing a very strange error on Gerrit (as you can see, a
`java.lang.StringIndexOutOfBoundsException`!), which didn't make
sense.  In the end, my Apache config file looks like this:

	<VirtualHost *:80>
		ServerName gnutoolchain-gerrit.osci.io

		RedirectPermanent / https://gnutoolchain-gerrit.osci.io/r/
	</VirtualHost>

	<VirtualHost *:443>
		ServerName gnutoolchain-gerrit.osci.io

		RedirectPermanent / /r/

		SSLEngine On
		SSLCertificateFile /path/to/cert.pem
		SSLCertificateKeyFile /path/to/privkey.pem
		SSLCertificateChainFile /path/to/chain.pem

		# Good practices for SSL
		# taken from: <https://mozilla.github.io/server-side-tls/ssl-config-generator/>

		# intermediate configuration, tweak to your needs
		SSLProtocol             all -SSLv3
		SSLCipherSuite          ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA:ECDHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-RSA-AES256-SHA256:DHE-RSA-AES256-SHA:ECDHE-ECDSA-DES-CBC3-SHA:ECDHE-RSA-DES-CBC3-SHA:EDH-RSA-DES-CBC3-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:DES-CBC3-SHA:!DSS
		SSLHonorCipherOrder     on
		SSLCompression          off
		SSLSessionTickets       off

		# OCSP Stapling, only in httpd 2.3.3 and later
		#SSLUseStapling          on
		#SSLStaplingResponderTimeout 5
		#SSLStaplingReturnResponderErrors off
		#SSLStaplingCache        shmcb:/var/run/ocsp(128000)

		# HSTS (mod_headers is required) (15768000 seconds = 6 months)
		Header always set Strict-Transport-Security "max-age=15768000"

		ProxyRequests Off
		ProxyVia Off
		ProxyPreserveHost On
		<Proxy *>
			Require all granted
		</Proxy>

		AllowEncodedSlashes On
			ProxyPass /r/ http://127.0.0.1:8081/ nocanon
			#ProxyPassReverse /r/ http://127.0.0.1:8081/r/
	</VirtualHost>

I confess I was almost giving up Keycloak when I finally found
the problem...

Anyway, after that things went more smoothly.  I was finally able to
make the user authentication work, then I made sure Keycloak's user
registration feature also worked OK...

Ah, one interesting thing: the user logout wasn't really working as
expected.  The user was able to logout from gerrit, but not from
Keycloak, so when the user clicked on "Sign in", Keycloak would tell
gerrit that the user was already logged in, and gerrit would
automatically log the user in again!  I was able to solve this by
redirecting the user to Keycloak's logout page, like this:

    [auth]
        ...
        logoutUrl = https://keycloak-url:port/auth/realms/REALM/protocol/openid-connect/logout?redirect_uri=https://gerrit-url/
        ...

After that, it was already possible to start worrying about configure
gerrit itself.  I don't know if I'll write a post about that, but let
me know if you want me to.

Conclusion
----------

If you ask me if I'm totally comfortable with the way things are set
up now, I can't say that I am 100%.  I mean, the set up seems robust
enough that it won't cause problems in the long run, but what bothers
me is the fact that I'm using technologies that are alien to me.  I'm
used to setting up things written in Python, C, C++, with very simple
yet powerful configuration mechanisms, and an easy to discover what's
wrong when something bad happens.

I am reasonably satisfied with the Keycloak logs things, but Gerrit
leaves a lot to be desired in that area.  And both projects are
written in languages/frameworks that I am absolutely not comfortable
with.  Like, it's really tough to debug something when you don't even
know where the code is or how to modify it!

All in all, I'm happy that this whole adventure has come to an end,
and now all that's left is to maintain it.  I hope that the GDB
community can make good use of this new service, and I hope that we
can see a positive impact in the quality of the whole patch review
process.

My final take is that this is all worth as long as the Free Software
and the User Freedom are the ones who benefit.

P.S.: Before I forget, our gerrit instance is running
at
[https://gnutoolchain-gerrit.osci.io](https://gnutoolchain-gerrit.osci.io).