How to setup a Keycloak server with external MySQL database on AWS ECS Fargate in clustered mode with the Quarkus distribution
Edit: I have updated this so it applies to the latest version of Keycloak, 24.0.1 as of writing April 16th 2024. Just some minor adjustments from 23.
As the title says the only difference from my previous post is that this one applies for Keycloak version 17 and above. Keycloak went from running on Wildfly to Quarkus. You can read more about the changes in their release notes: https://www.keycloak.org/docs/latest/release_notes/index.html
This change is very noticeable, especially running on Fargate. The startup time of Keycloak has been reduced drastically, it is now way faster. But the configuration is a little bit different, so I will go through the steps required to upgrade from 16 (would also apply for fresh installs).
So let us get down to business, in this post I will not repeat the steps from my previous post regarding creation of services, load balancing etc on AWS. For that just head over to my previous post. The main difference is that instead of defining a docker image directly from Docker Hub in the task definition (jboss/keycloak) like we did in the previous post we have to build our own this time and upload it to ECR. Everything else should be the same (take a note on health checks and the removal of “/auth”).
Step 1, you probably want to have your own repository for Keycloak. Once inside create a Dockerfile like this:
The Dockerfile
If you are curious about the documentation on how to install packages, see here: https://www.keycloak.org/server/containers
FROM quay.io/keycloak/keycloak:24.0.1@sha256:8e38bc8a4e0606e38a37401422dfbf414e2b73797952dfe94c9b56e2f9207897 AS builder
ENV KC_DB=mysql
ENV KC_HEALTH_ENABLED=true
# JDBC-PING cluster setup
COPY ./cache-ispn-jdbc-ping.xml /opt/keycloak/conf/cache-ispn-jdbc-ping.xml
RUN /opt/keycloak/bin/kc.sh build --cache-config-file=cache-ispn-jdbc-ping.xml
FROM registry.access.redhat.com/ubi9/ubi:9.3-1361.1699548029@sha256:5dc85ec81a0d2cc5d19164f80b8d287b176483fd09a88426ca2f698bb2bd09de AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y && \
dnf --installroot /mnt/rootfs clean all && \
rpm --root /mnt/rootfs -e --nodeps setup
FROM quay.io/keycloak/keycloak:24.0.1@sha256:8e38bc8a4e0606e38a37401422dfbf414e2b73797952dfe94c9b56e2f9207897
ENV KC_CACHE_CONFIG_FILE=cache-ispn-jdbc-ping.xml
COPY --from=ubi-micro-build /mnt/rootfs /
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
COPY --from=builder /opt/keycloak/conf/cache-ispn-jdbc-ping.xml /opt/keycloak/conf
WORKDIR /opt/keycloak
COPY ./themes/ /opt/keycloak/themes/ # only applies if you have custom themes
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]We are running a multi-staged build to build our version of Keycloak 24. As you maybe already have noticed the new versions of Keycloak are not available on Docker Hub, but has moved to Quay (see here for the latest versions, current is 24.0.1).
For an overview over environment variables I recommend reading this page. I will not go through all as you can easily read up on them there.
KC_FEATURES
Here you can enable features like token-exchange. Check them out, there is a lot of exiting stuff there. But if you just are using standard Keycloak you can drop this environment variable completely.
KC_CACHE_CONFIG_FILE
You can read more about caching here. You can specify your own cache config file. Something we need to do to support clustering on Fargate with JDBC_PING. To my disappointment JDBC_PING is not supported without custom config. S3_PING is supported, but using a bucket just for that feels messy and I do not like it.
Github user xgp created a gist I used as inspiration (thanks a lot!). The gist was for Postgres, so I had to tweak it a little bit for MySQL.
Step 2, create a file called cache-ispn-jdbc-ping.xml and paste this content:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
xmlns="urn:infinispan:config:11.0">
<!-- custom stack goes into the jgroups element -->
<jgroups>
<stack name="jdbc-ping-tcp" extends="tcp">
<JDBC_PING connection_driver="com.mysql.cj.jdbc.Driver"
connection_username="${env.KC_DB_USERNAME}"
connection_password="${env.KC_DB_PASSWORD}"
connection_url="${env.KC_DB_URL}"
info_writer_sleep_time="500"
initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data VARBINARY(255), constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
remove_all_data_on_view_change="true"
stack.combine="REPLACE"
stack.position="MPING" />
</stack>
</jgroups>
<cache-container name="keycloak">
<!-- custom stack must be referenced by name in the stack attribute of the transport element -->
<transport lock-timeout="60000" stack="jdbc-ping-tcp"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<distributed-cache name="sessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="authenticationSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="clientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="loginFailures" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<local-cache name="authorization">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<replicated-cache name="work">
<expiration lifespan="-1"/>
</replicated-cache>
<local-cache name="keys">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="3600000"/>
<memory max-count="1000"/>
</local-cache>
<distributed-cache name="actionTokens" owners="2">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="-1" lifespan="-1" interval="300000"/>
<memory max-count="-1"/>
</distributed-cache>
</cache-container>
</infinispan>Now we have everything we need to build our Dockerfile. To use it on AWS we need to head over to ECR and create a repository for our Keycloak docker images (I have called mine keycloak). Once that is done you just build it.
A note on health checks
Keycloak previously did not provide any health check endpoint (in my previous post I just used “/auth”), that has changed. So I would recommend to use that endpoint in the container health check and the load balancer health check “/health”. Just set the KC_HEALTH_ENABLED environment variable to true.
Step 3, run the build command:
docker build -t "{AWS Account ID}.dkr.ecr.{AWS Region}.amazonaws.com/{ECR Repository name}:{Tag}" .Example:
docker build -t "772104948942.dkr.ecr.eu-central-1.amazonaws.com/keycloak:24.0.1" .Next step would be to upload it to AWS ECR. A prerequisite would be to install and configure AWS CLI 2 (see here for instructions). Once AWS CLI is up and running we are ready for step 4.
Step 4, login to ECR and push:
You will find the login command if you click on the “View push commands” button. Just copy and paste the login command to your terminal and run it. Then you are able to upload your image to ECR by doing docker push. Like this:
docker push 772104948942.dkr.ecr.eu-central-1.amazonaws.com/keycloak:24.0.1Task definition
Now that our image is ready to be used, lets configure our new task definition.
Step 5, create a new task definition. My ecs-task.json for our staging environment:
Edit: Updated for Keycloak 24, KC_PROXY with value edge is now replaced by KC_PROXY_HEADERS and KC_HTTP_ENABLED, as per the migration guide.
{
"family": "Keycloak-dev",
"networkMode": "awsvpc",
"cpu": "1024",
"memory": "2048",
"taskRoleArn": "arn:aws:iam::772104948942:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::772104948942:role/ecsTaskExecutionRole",
"containerDefinitions": [
{
"name": "keycloak",
"image": "772104948942.dkr.ecr.eu-central-1.amazonaws.com/keycloak:21.1.0",
"command": [
"start-dev"
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/Keycloak-dev",
"awslogs-region": "eu-central-1",
"awslogs-stream-prefix": "ecs",
"awslogs-multiline-pattern": "^*(TRACE|DEBUG|INFO|WARN|ERROR)"
}
},
"environment": [
{
"name": "KC_HOSTNAME",
"value": "keycloak-staging.mydomain.com"
},
{
"name": "KC_HTTP_RELATIVE_PATH",
"value": "/auth"
},
{
"name": "KC_HEALTH_ENABLED",
"value": "true"
},
{
"name": "KC_DB",
"value": "mysql"
},
{
"name": "KC_PROXY_HEADERS",
"value": "xforwarded"
},
{
"name": "KC_HTTP_ENABLED",
"value": "true"
},
{
"name": "KC_HOSTNAME_STRICT_BACKCHANNEL",
"value": "true"
},
{
"name": "KC_LOG_LEVEL",
"value": "INFO"
}
],
"secrets": [
{
"valueFrom": "KEYCLOAK_DEV_DB_ADDR_STAGING",
"name": "KC_DB_URL"
},
{
"valueFrom": "KEYCLOAK_DEV_DB_PASSWORD",
"name": "KC_DB_PASSWORD"
},
{
"valueFrom": "KEYCLOAK_DEV_DB_USERNAME",
"name": "KC_DB_USERNAME"
}
],
"portMappings": [
{
"hostPort": 8080,
"protocol": "tcp",
"containerPort": 8080
}
],
"healthCheck": {
"retries": 3,
"command": [
"CMD-SHELL",
"curl -f http://localhost:8080/auth/health || exit 1"
],
"timeout": 5,
"interval": 60,
"startPeriod": 300
}
}
],
"requiresCompatibilities": [
"FARGATE"
]
}As you can see there is a lot of configuration going on here with environment variables. I have stored my secrets in AWS Parameter Store (“valueFrom” is the name of the parameter in Parameter Store, “name” is the environment variable used inside the container). For instance connection to the database, proxy configuration for running behind load balancers on AWS and so on.
One thing to notice is the KC_HTTP_RELATIVE_PATH environment variable. Previous versions of keycloak had “/auth” as part of their url, that is now removed. So by setting this to “/auth” it will not break your API’s or applications that still expect the url to contain “/auth”. This environment variable should be removed if you are not doing an upgrade. For fresh installs this is not a problem, for us on the other hand with a lot of API’s and applications its another story. We would need to remove “/auth” from the keycloak url in a lot of services and redeploy, the cost/benefit analysis was pretty easy, just add this variable if you are upgrading and do not want to redeploy all your services.
Another important aspect is the command part of the task definition, it is really important. I am talking about this part:
start-devThis starts Keycloak in development mode. The only difference in a production environment would be to remove -dev, like this:
startA quick note on fresh install
If you are doing a fresh install remember to add the environment variables KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD with a username and password for your admin user to your task definition. If you use Keycloak with a database you only have to add them the first time you run the container, after that the user is stored in the database and the environment variables could be removed from the task definition.
Sources I recommend reading
Keycloaks documentation takes you a long way, and the community the rest.
Finishing thoughts
This was my second article on Medium. I did not have much time writing this, so it is not as newbie friendly as my last one where I used a lot of screenshots and took time explaining. I apologize for that. But just ask in the comment section if anything is unclear. If you are familiar with AWS I think most of it should be easy to follow. Hopefully it will help someone having trouble configuring the Quarkus distribution of Keycloak on AWS.
Edit: A tip if you are upgrading (not relevant for fresh installs). I had some trouble upgrading to 23 from 21 because of deployment pattern on AWS. So my two tasks for 21 ran at the same time as my two new ones on 23. That cluttered the JGROUPSPING table in the keycloak database. So a tip if the container crashes due to it tries to connect to the old instances, just wipe the JGROUPSPING table so its empty and stop the old tasks. Make sure the new ones has a clean JGROUPSPING table then you should be good.
