Engage 2022 – Domino TOTP/2FA – Best Practices and Pitfalls

It’s hard to describe how well the Engage conference is organized and how fabulous the event is. Engage is the place where many good people gather, who gave me the opportunity to learn from them, and over the years many of them became my friends. Therefore, it is something special for me to present at Engage, and for that, I am thankful and honored.

My colleague and dear friend Martin Leyrer and I talked about Domino and TOTP, below you can take a look at the slides we have used:

We have also recorded our session, sorry for the bad audio. You can take a look at the recording below:

I’m Speaking at Engage 2022 in Bruges! Yay!

For the second time, I will be speaking at the EngageUG conference. I’m honored and thrilled at the same time. Come join Martin Leyrer and me for the “Domino TOTP/2FA – Best Practices and Pitfalls” session, at which we will give our best to share all our knowledge and experience with TOTP on Domino!

If this year’s conference is going to be half as good as any other before it, we are going to have a great time! Learn what’s NEXT for HCL Digital Solutions and meet your friends and colleagues.

You haven’t registered? Don’t worry, there is still time, head over to the EngageUG website and book yourself a ticket! πŸ™‚

Kubernetes – Host Entries

As Kubernetes pods do not make use of the Kubernetes nodes/hosts “host” file (/etc/hosts), which can be a challenge or a blessing, do not despair if you find yourself in an environment in which some DNS entries are missing. There is an easy workaround to “get you going”.

Although I have to stress out, that this is not a long-term solution, and you should use DNS as opposed to the following solution.

If you are using “coreDNS” on Kubernetes, edit the “configmap” of the service, by issuing the following command:

kubectl edit cm -n kube-system coredns

After that, just add the section which is marked red in the following screenshot:

And edit it to suit your needs.

Note: “fallthrough” parameter is important as it will make sure that the requests which can’t be handled by hosts configuration in the “coreDNS” pods, will be sent to the configured DNS servers.

After you are happy with the configuration, just recreate the “coreDNS” pods, by running the following command:

kubectl -n kube-system delete pods -l k8s-app=kube-dns

Note: this will cause a brief interruption of the name resolution service as the pods are being recreated.

You can test the DNS inside the Kubernetes network by creating a pod for this purpose that has all the “networking” tools you need, I decided to run with “infoblox/dnstools”.

kubectl run -it –rm –restart=Never –image=infoblox/dnstools:latest dnstools

HCL Connections – Orient Me “Loop”

In the course of the recent deployment of HCL Connections Component Pack, I ran into an issue with the Orient Me application.

After deploying Orient Me, every try to open the new Orient Me homepage would result in a “loop”, the user was being redirected from the Orient Me (/social) page to the Homepage application (/homepage) and back again to the Orient Me page. The root cause for the problem was not a bug or an error in the HCL Connections code, but rather the configuration of the HCL Connections Blue Stack and other components in the IT landscape of this environment.

After further investigating the issue, we have seen the following messages in one of the “orient-web-client” pods:

Mon, 14 Mar 2022 13:15:37 GMT hsts deprecated The “includeSubdomains” parameter is deprecated. Use “includeSubDomains” (with a capital D) instead. at node_modules/loopback/lib/server-app.js:74:25
{“pid”:7,”hostname”:”orient-web-client-57594bc9df-ck6qq”,”name”:”@connections/cachy-service/src/client/redis-client.js”,”level”:30,”time”:1647263737775,”msg”:”om-readiness-probe Initializing [%s]”,”v”:1}
Web server listening at: http://localhost:8000
2022-03-14T13:15:37.883Z – ESC[32minfoESC[39m: [orient-web-client] Cachy-gatekeeper register response: true
2022-03-14T13:17:41.020Z – ESC[34mdebugESC[39m: [auth-service] decodeJWT: no JWT found, returning no_jwt_token
2022-03-14T13:17:41.022Z – ESC[34mdebugESC[39m: [auth-service] >auth.service: about to call setJWT to generate or re-generate token…
2022-03-14T13:17:41.022Z – ESC[34mdebugESC[39m: [auth-service] Setting status to be 401
2022-03-14T13:17:41.024Z – ESC[34mdebugESC[39m: [auth-service] setJWT failed with err: [no_auth_token]: no_auth_token
2022-03-14T13:17:41.025Z – ESC[31merrorESC[39m: [orient-web-client] >server: ensureLogin callback with error: UnauthorizedError: [get_token_failed]: Unable to get token undefined
Unhandled error for request HEAD /social/auth/token: UnauthorizedError: [get_token_failed]: Unable to get token
2022-03-14T13:17:41.355Z – ESC[34mdebugESC[39m: [orient-web-client] ENTRY: AuthenticationMiddleware.handleAuthentication undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [orient-web-client] ENTRY: AuthenticationMiddleware.extractLDAPCookies undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [orient-web-client] EXIT: AuthenticationMiddleware.extractLDAPCookies for user id: undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [auth-service] decodeJWT: no JWT found, returning no_jwt_token
2022-03-14T13:17:41.357Z – ESC[33mwarnESC[39m: [auth-service] JWT token is bad, can not proceed for potential CSRF!
2022-03-14T13:17:41.358Z – ESC[34mdebugESC[39m: [orient-web-client] handleAuthentication ensureLogin error { [UnauthorizedError: [not_authorized]: Not authorized]
name: ‘UnauthorizedError’,
code: ‘not_authorized’,

What was obvious, was that the user didn’t have a JWT Cookie going to the Orient Me web client and the Orient Me web client could not generate the same as it could not recognize the username of the user “hitting” the service.

Thankfully, with the guidance and help of my colleague, Martin Leyrer, we managed to resolve this issue. In order to resolve the problem, we have done the following.

We checked and modified all DNS entries of all hosts and services (NGINX, IBM HTTP Server, etc…) involved and made sure these were resolving correctly.

The “dynamicHosts” property in the “LotusConnections-Config.xml” was enabled, as described in the official documentation, and we corrected the “interService” URLs (in our case the URL resolving to IBM HTTP Server), in the same configuration file.

On the WebSphere Application Servers, the LTPA Cookie names were not set to the default. As this could cause similar issues, we have changed those to default, “LtpaToken” and “LtpaToken2”.

Note: As version 2 of the LTPA Cookie is much more secure, we recommend only using this version as opposed to version 1.

The last change we have done was in the IBM HTTP Server configuration (“httpd.conf” file), here we have added the “ServerName” parameter to the <VirtualHost *:443> section, after that, the configuration looked like the following:

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 443
<VirtualHost *:443>
ServerName ibm_ihs_hostname.domain.com
RewriteEngine on

After doing all the changes mentioned above, we were able to resolve the issue and could successfully render and use the Orient Me application. πŸ˜‰

I sincerely hope this will help and save you some pain! πŸ™‚

HCL Connections CP – Enabling Elasticsearch Metrics

Recently, when enabling Elasticsearch metrics, part of the HCL Connections Component Pack, I ran into an issue with the “config_blue_metrics.py” script. Which was failing every time I tried to run it (sadly, I’ve “lost” the output and the direct error message which was shown directly after running the python script). The details on how to run this script are described in the official documentation.

Nevertheless, after looking into the WebSphere Metrics Application logs, “AppsCluster” if your HCL Connections Blue Stack environment is installed as a “Medium deployment”, I’ve noticed the following error:

[10/3/22 17:26:54:477 IST] 0000016c LotusConnecti E Unable to access the required data
javax.servlet.ServletException: java.io.FileNotFoundException: SRVE0190E: File not found: /configsetter

at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:248)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.lconn.core.web.customization.CustomizationFilter.doFilter(CustomizationFilter.java:142)

Caused by: java.io.FileNotFoundException: SRVE0190E: File not found: /configsetter
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)

Thanks to my colleague, Herwig W. Schauer, we have found a quite simple solution for this issue. We have changed the default “Context Root For Web Modules” for “Metrics” and “MetricsUI” application. To do this, just open the WAS ISC and navigate to the following entry “Enterprise Applications –> MetricsUI –> Context Root For Web Modules”. We have changed the mentioned configuration in the following way.

We have set the “Metrics” application context root from “/metrics” and “/metrics/service to “/metricsOLD” and “/metricsOLD/service” respectively:

After making the changes mentioned above, we proceeded to make similar changes for the “MetricsUI” application, here we changed the Context Root from “/metricssc/service/rest” and “/metricssc” to “/metrics/service/rest” and “/metrics”:

After the needed configuration is in place, make sure that the WAS nodes are synchronized and restart the “Metrics” and “MetricsUI” applications. By doing so, the issue should be resolved and the “config_blue_metrics.py” script should be able to run without any errors.

Note: You still may get a warning message after running the script: “Warning elasticsearch not found in Kubnetes.”, but this is fine and does not lead to any further problems.

HCL Sametime – RunFaster=1

Everybody likes when software performs well and feels “snappy”, guided by that mantra I’ve found that with the help of one “sametime.ini” parameter for LDAP tuning, you can improve the “login” performance of the clients and the time it takes to load Sametime Business Cards considerably.

By setting the “ST_DB_LDAP_CONNECTIONS_NUMBER” parameter, to a value greater than one, you can increase the maximum number of the LDAP connections HCL Sametime components can use to access the LDAP repository. Per default, every HCL Sametime module, which needs LDAP access, will use one connection to access the repository (“StAuthentication.dll” is the exception, as it uses two connections). Details about that can be read from the official documentation.

In most Sametime environments I’ve deployed, I have set the parameter “ST_DB_LDAP_CONNECTIONS_NUMBER” to two or three, depending on the Sametime Community server and LDAP server hardware. This led, at least subjectively, to 40-60% faster login times and about the same faster response when loading the Sametime Business Cards.

Make sure to monitor the hardware utilization and the performance of both the HCL Sametime Community server(s) and of the LDAP server(s), as changing this parameter will have a direct impact on it. Also, consider that setting the value too high can have negative effects.

HCL Sametime – Setting the Community ID

If you are planning to deploy HCL Sametime Community service in a cluster or HA architecture, setting a Community ID is a must.

Ideally, this should be an FQDN used for accessing the Community servers, something which is easy to remember, and your users can relate to. So, think ahead and use a name that can be used to access the service externally and internally, I would never use the hostname of the Sametime Community server, in most cases, this is hard to remember and can’t be used to access the service over the internet. A “nice” and simple DNS alias like “chat.company.com” is generally the best way to go.

This is especially important when setting up an HA environment, in this case, you also must make sure that the FQDN used, points to the Load Balancer. The consequences for not doing this, in a HA environment, can be catastrophic. In an event of the failover, the Sametime Embedded or Connect client may recognize the server failing over to, as a member of a different community. This would result, among other issues, in not displaying the user’s Sametime contact list.

The Community ID can be set via “sametime.ini” and the process is quite simple, as described in the official product documentation.

HCL Traveler – Cleaning up the “lotustraveler.nsf” database

If you are seeing some old and invalid objects in the “lotustraveler.nsf”, on the HCL Traveler server, try running “tell traveler cleanup show” command. It will show you which entries are obsolete and could be deleted.

Provided that you are OK with the result, run the “cleanup” command without the “show” option, this command will delete the entries previously shown.

To find out more about the HCL Traveler “cleanup” command, take a look at the official documentation.