How to setup a Keycloak server with external MySQL database on AWS ECS Fargate in clustered mode with the Quarkus distribution
Edit: I have updated this so it applies to the latest version of Keycloak, 25.0.4 as of writing August 30th 2024. Just some minor adjustments from 24.
As the title says the only difference from my previous post is that this one applies for Keycloak version 17 and above. Keycloak went from running on Wildfly to Quarkus. You can read more about the changes in their release notes: https://www.keycloak.org/docs/latest/release_notes/index.html
This change is very noticeable, especially running on Fargate. The startup time of Keycloak has been reduced drastically, it is now way faster. But the configuration is a little bit different, so I will go through the steps required to upgrade from 16 (would also apply for fresh installs).
So let us get down to business, in this post I will not repeat the steps from my previous post regarding creation of services, load balancing etc on AWS. For that just head over to my previous post. The main difference is that instead of defining a docker image directly from Docker Hub in the task definition (jboss/keycloak) like we did in the previous post we have to build our own this time and upload it to ECR. Everything else should be the same (take a note on health checks and the removal of “/auth”).
Step 1, you probably want to have your own repository for Keycloak. Once inside create a Dockerfile like this:
The Dockerfile
If you are curious about the documentation on how to install packages, see here: https://www.keycloak.org/server/containers
FROM quay.io/keycloak/keycloak:25.0.4@sha256:bf788a3b7fd737143f98d4cb514cb9599c896acee01a26b2117a10bd99e23e11 AS builder
ENV KC_DB=mysql
ENV KC_HEALTH_ENABLED=true
RUN /opt/keycloak/bin/kc.sh build
FROM registry.access.redhat.com/ubi9/ubi:9.4-1181@sha256:763f30167f92ec2af02bf7f09e75529de66e98f05373b88bef3c631cdcc39ad8 AS ubi-micro-build
RUN mkdir -p /mnt/rootfs
RUN dnf install --installroot /mnt/rootfs curl --releasever 9 --setopt install_weak_deps=false --nodocs -y && \
dnf --installroot /mnt/rootfs clean all && \
rpm --root /mnt/rootfs -e --nodeps setup
FROM quay.io/keycloak/keycloak:25.0.4@sha256:bf788a3b7fd737143f98d4cb514cb9599c896acee01a26b2117a10bd99e23e11
# JDBC-PING cluster setup
COPY ./cache-ispn-jdbc-ping.xml /opt/keycloak/conf/cache-ispn-jdbc-ping.xml
ENV KC_CACHE_CONFIG_FILE=cache-ispn-jdbc-ping.xml
COPY --from=ubi-micro-build /mnt/rootfs /
COPY --from=builder /opt/keycloak/lib/quarkus/ /opt/keycloak/lib/quarkus/
WORKDIR /opt/keycloak
COPY ./themes/ /opt/keycloak/themes/
ENTRYPOINT ["/opt/keycloak/bin/kc.sh"]We are running a multi-staged build to build our version of Keycloak 25. As you maybe already have noticed the new versions of Keycloak are not available on Docker Hub, but has moved to Quay (see here for the latest versions, current is 25.0.4).
If you are coming over from Keycloak 24 there is a difference here in Keycloak 25, you no longer include the cache config file in the build step.
For an overview over environment variables I recommend reading this page. I will not go through all as you can easily read up on them there.
KC_FEATURES
Here you can enable features like token-exchange. Check them out, there is a lot of exiting stuff there. But if you just are using standard Keycloak you can drop this environment variable completely.
KC_CACHE_CONFIG_FILE
You can read more about caching here. You can specify your own cache config file. Something we need to do to support clustering on Fargate with JDBC_PING. To my disappointment JDBC_PING is not supported without custom config. S3_PING is supported, but using a bucket just for that feels messy and I do not like it.
Github user xgp created a gist I used as inspiration (thanks a lot!). The gist was for Postgres, so I had to tweak it a little bit for MySQL.
Step 2, create a file called cache-ispn-jdbc-ping.xml and paste this content:
<?xml version="1.0" encoding="UTF-8"?>
<infinispan
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="urn:infinispan:config:11.0 http://www.infinispan.org/schemas/infinispan-config-11.0.xsd"
xmlns="urn:infinispan:config:11.0">
<!-- custom stack goes into the jgroups element -->
<jgroups>
<stack name="jdbc-ping-tcp" extends="tcp">
<JDBC_PING connection_driver="com.mysql.cj.jdbc.Driver"
connection_username="${env.KC_DB_USERNAME}"
connection_password="${env.KC_DB_PASSWORD}"
connection_url="${env.KC_DB_URL}"
info_writer_sleep_time="500"
initialize_sql="CREATE TABLE IF NOT EXISTS JGROUPSPING (own_addr varchar(200) NOT NULL, cluster_name varchar(200) NOT NULL, ping_data VARBINARY(255), constraint PK_JGROUPSPING PRIMARY KEY (own_addr, cluster_name));"
remove_all_data_on_view_change="true"
stack.combine="REPLACE"
stack.position="MPING" />
</stack>
</jgroups>
<cache-container name="keycloak">
<!-- custom stack must be referenced by name in the stack attribute of the transport element -->
<transport lock-timeout="60000" stack="jdbc-ping-tcp"/>
<local-cache name="realms">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<local-cache name="users">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<distributed-cache name="sessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="authenticationSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="clientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="offlineClientSessions" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<distributed-cache name="loginFailures" owners="2">
<expiration lifespan="-1"/>
</distributed-cache>
<local-cache name="authorization">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<memory max-count="10000"/>
</local-cache>
<replicated-cache name="work">
<expiration lifespan="-1"/>
</replicated-cache>
<local-cache name="keys">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="3600000"/>
<memory max-count="1000"/>
</local-cache>
<distributed-cache name="actionTokens" owners="2">
<encoding>
<key media-type="application/x-java-object"/>
<value media-type="application/x-java-object"/>
</encoding>
<expiration max-idle="-1" lifespan="-1" interval="300000"/>
<memory max-count="-1"/>
</distributed-cache>
</cache-container>
</infinispan>Now we have everything we need to build our Dockerfile. To use it on AWS we need to head over to ECR and create a repository for our Keycloak docker images (I have called mine keycloak). Once that is done you just build it.
A note on health checks
Keycloak previously did not provide any health check endpoint (in my previous post I just used “/auth”), that has changed. So I would recommend to use that endpoint in the container health check and the load balancer health check “/health”. Just set the KC_HEALTH_ENABLED environment variable to true.
If you come from Keycloak 24 to Keycloak 25 you also need to update the healthCheck from the ECS task definition which I showed in the previous article since the health check now runs on a new port (9000). See here:
"command": [
"CMD-SHELL",
"curl -f http://localhost:9000/auth/health || exit 1"
],Remember to strip the /auth part of the url if you do not have the KC_HTTP_RELATIVE_PATH environment variable set.
Also remember to update the target group health check, you need to override the port (not use traffic port) and set it to 9000. Or else your tasks would eventually fail ELB health checks.
Step 3, run the build command:
docker build -t "{AWS Account ID}.dkr.ecr.{AWS Region}.amazonaws.com/{ECR Repository name}:{Tag}" .Example:
docker build -t "772104948942.dkr.ecr.eu-central-1.amazonaws.com/keycloak:24.0.1" .Next step would be to upload it to AWS ECR. A prerequisite would be to install and configure AWS CLI 2 (see here for instructions). Once AWS CLI is up and running we are ready for step 4.
Step 4, login to ECR and push:
You will find the login command if you click on the “View push commands” button. Just copy and paste the login command to your terminal and run it. Then you are able to upload your image to ECR by doing docker push. Like this:
docker push 772104948942.dkr.ecr.eu-central-1.amazonaws.com/keycloak:24.0.1Task definition
Now that our image is ready to be used, lets configure our new task definition.
Step 5, create a new task definition. My ecs-task.json for our staging environment:
Edit: Updated for Keycloak 24, KC_PROXY with value edge is now replaced by KC_PROXY_HEADERS and KC_HTTP_ENABLED, as per the migration guide.
Update for Keycloak 25, need to add https:// to KC_HOSTNAME and if you are doing an upgrade (not fresh install) and have the KC_HTTP_RELATIVE_PATH set, you must append that as well. Previous value: “keycloak-staging.mydomain.com” new value: “https://keycloak-staging.mydomain.com/auth”. Also need to rename KC_HOSTNAME_STRICT_BACKCHANNEL to KC_HOSTNAME_BACKCHANNEL_DYNAMIC. More information is found in the upgrade guide.
{
"family": "Keycloak-dev",
"networkMode": "awsvpc",
"cpu": "1024",
"memory": "2048",
"taskRoleArn": "arn:aws:iam::772104948942:role/ecsTaskExecutionRole",
"executionRoleArn": "arn:aws:iam::772104948942:role/ecsTaskExecutionRole",
"containerDefinitions": [
{
"name": "keycloak",
"image": "772104948942.dkr.ecr.eu-central-1.amazonaws.com/keycloak:21.1.0",
"command": [
"start-dev"
],
"logConfiguration": {
"logDriver": "awslogs",
"options": {
"awslogs-group": "/ecs/Keycloak-dev",
"awslogs-region": "eu-central-1",
"awslogs-stream-prefix": "ecs",
"awslogs-multiline-pattern": "^*(TRACE|DEBUG|INFO|WARN|ERROR)"
}
},
"environment": [
{
"name": "KC_HOSTNAME",
"value": "https://keycloak-staging.mydomain.com/auth"
},
{
"name": "KC_HTTP_RELATIVE_PATH",
"value": "/auth"
},
{
"name": "KC_HEALTH_ENABLED",
"value": "true"
},
{
"name": "KC_DB",
"value": "mysql"
},
{
"name": "KC_PROXY_HEADERS",
"value": "xforwarded"
},
{
"name": "KC_HTTP_ENABLED",
"value": "true"
},
{
"name": "KC_HOSTNAME_BACKCHANNEL_DYNAMIC",
"value": "true"
},
{
"name": "KC_LOG_LEVEL",
"value": "INFO"
}
],
"secrets": [
{
"valueFrom": "KEYCLOAK_DEV_DB_ADDR_STAGING",
"name": "KC_DB_URL"
},
{
"valueFrom": "KEYCLOAK_DEV_DB_PASSWORD",
"name": "KC_DB_PASSWORD"
},
{
"valueFrom": "KEYCLOAK_DEV_DB_USERNAME",
"name": "KC_DB_USERNAME"
}
],
"portMappings": [
{
"hostPort": 8080,
"protocol": "tcp",
"containerPort": 8080
}
],
"healthCheck": {
"retries": 3,
"command": [
"CMD-SHELL",
"curl -f http://localhost:8080/auth/health || exit 1"
],
"timeout": 5,
"interval": 60,
"startPeriod": 300
}
}
],
"requiresCompatibilities": [
"FARGATE"
]
}As you can see there is a lot of configuration going on here with environment variables. I have stored my secrets in AWS Parameter Store (“valueFrom” is the name of the parameter in Parameter Store, “name” is the environment variable used inside the container). For instance connection to the database, proxy configuration for running behind load balancers on AWS and so on.
One thing to notice is the KC_HTTP_RELATIVE_PATH environment variable. Previous versions of keycloak had “/auth” as part of their url, that is now removed. So by setting this to “/auth” it will not break your API’s or applications that still expect the url to contain “/auth”. This environment variable should be removed if you are not doing an upgrade. For fresh installs this is not a problem, for us on the other hand with a lot of API’s and applications its another story. We would need to remove “/auth” from the keycloak url in a lot of services and redeploy, the cost/benefit analysis was pretty easy, just add this variable if you are upgrading and do not want to redeploy all your services.
Another important aspect is the command part of the task definition, it is really important. I am talking about this part:
start-devThis starts Keycloak in development mode. The only difference in a production environment would be to remove -dev, like this:
startA quick note on fresh install
If you are doing a fresh install remember to add the environment variables KEYCLOAK_ADMIN and KEYCLOAK_ADMIN_PASSWORD with a username and password for your admin user to your task definition. If you use Keycloak with a database you only have to add them the first time you run the container, after that the user is stored in the database and the environment variables could be removed from the task definition.
Sources I recommend reading
Keycloaks documentation takes you a long way, and the community the rest.
Finishing thoughts
This was my second article on Medium. I did not have much time writing this, so it is not as newbie friendly as my last one where I used a lot of screenshots and took time explaining. I apologize for that. But just ask in the comment section if anything is unclear. If you are familiar with AWS I think most of it should be easy to follow. Hopefully it will help someone having trouble configuring the Quarkus distribution of Keycloak on AWS.
Edit: A tip if you are upgrading (not relevant for fresh installs). I had some trouble upgrading to 23 from 21 because of deployment pattern on AWS. So my two tasks for 21 ran at the same time as my two new ones on 23. That cluttered the JGROUPSPING table in the keycloak database. So a tip if the container crashes due to it tries to connect to the old instances, just wipe the JGROUPSPING table so its empty and stop the old tasks. Make sure the new ones has a clean JGROUPSPING table then you should be good.
