HCL Connections 8 – PDF Export Issues After Installing CNX in a Clustered WAS Environment

Recently I encountered an issue with PDF Export, right after the installation of HCL Connections applications in a multi-node, clustered, IBM WebSphere Application Server environment. This problem only occurs in a multi-node WAS environment.

In the HCL Connections GUI, in the “PDF Export Access” settings of the”Edit Community” menu (Community –> Community Actions –> Edit Community –> PDF Export Access), the following error was displayed:

Error 500: org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: com/ibm/ess/ic/ic360/security/tai/Ic360ImpersonateUserTAI

At the same time, the following error is logged in the WAS application server logs, where the IC360 application is running:

ServletWrappe E com.ibm.ws.webcontainer.servlet.ServletWrapper service SRVE0014E: Uncaught service() exception root cause ic360: org.springframework.web.util.NestedServletException: Handler dispatch failed; nested exception is java.lang.NoClassDefFoundError: com/ibm/ess/ic/ic360/security/tai/Ic360ImpersonateUserTAI

The cause for the problem were the missing java virtual machine properties, which were only set on one of the IC360 WAS applications servers. In my case, as I was installing HCL Connections using “large deployment” topology, the java virtual machine properties were only set for “IC350Cluster_server2”. Those settings can be found under “Application servers –> IC360Cluster_server2 –> Process definition –> Java Virtual Machine –> Custom properties”, as shown in the screenshot below:

On the other hand, these crucial settings were missing on the other IBM WebSphere Application Cluster member, as can be seen in the screenshot below:

To resolve the problem, make sure that the same JVM “Custom properties” are set across all WAS application servers hosting the IC360 Application. In my case I had to set the same settings for “IC360Cluster_server1” application server as they were set by the installer for “IC360Cluster_server2”, as displayed in the screenshot below:

After that, make sure that all IBM WebSphere nodes are synchronized and restart the IC360 WebSphere Application Server cluster.

The steps to resolve this issue are also documented in the KB0085725 article.

I hope this helps!

Advertisement

HCL Domino TOTP/2FA – Implementation, Best Practices and Pitfalls – Webinar

My colleague Martin Leyrer and I will be hosting a webinar “HCL Domino TOTP/2FA – Implementation, Best Practices and Pitfalls”. The session will start on September the 15th, at 4 PM CEST. So, if you are interested in TOTP/2FA Implementation using HCL Domino natively, make sure to register and join our Webinar:

Registration Form

We will be delighted to have your presence!

HCL Domino – Contact Sync Issues

Recently, we came across some issues with contact synchronization between mobile devices using HCL Traveler, mail databases of HCL Notes users, and address books of the HCL Notes Roaming users.

To be exact, these are two separate problems which are described in the following Knowledge Base articles:

KB0099431

KB0097255

You might have the issues mentioned in the KB articles above, but haven’t noticed them yet, as the HCL Notes and HCL Traveler users will only have problems with synchronizing certain contacts “across the board”, namely those which are created on HCL Traveler devices. The issue will become more apparent with the users having more than one mobile device activated on HCL Traveler, as the contacts created on one of the devices will not sync to the other and vice versa.

There is a workaround for both issues, as stated in the KB articles mentioned above, which is to add the “AccessContacts” role to the owner of the mail database as well as to the roaming address book database, assuming the same user is also a roaming user. You can either do this manually or via LotusScript code provided by Domino Development, which you can find in the following Knowledge Base article:

KB0099761

Many thanks to the HCL Traveler team for confirming the issue and developing the workaround so quickly, as well as to the HCL Domino Development team for writing the code to implement the workaround.

New Fixes for HCL Notes 12.0.1 German Template

As of yesterday, a new version of HCL Notes 12.0.1 German mail template is available, which incorporates the fixes for the following SPRs:

SPR  # PDARCBQ86U >>  DOMI: MSTeams meeting is not getting updated with new URL when user opens the accepted reschedule invite

SPR  # PDARCC68MC >>  DOMI: Reschedule meeting notice displays the old url for MSTeams meeting when chair accepts the counter

You can find the new version of the HCL Notes 12.0.1 German mail template in the KB0097354 article.

Hope this helps! 🙂

HCL SafeLinx – Encrypted Communication Between the SafeLinx Client and the SafeLinx Server

After deploying HCL SafeLinx, one of the first things you should do, is to configure the communication between the HCL SafeLinx Administrator client and the HCL SafeLinx Access Manager, so that it takes place in an encrypted and secure manner.

For this, only a few simple steps are needed.

• Generate a new p12 keystore together with a new private key and SSL certificate, which we will use for accessing the HCL SafeLinx Access Manager.
  • For this, we need OpenSSL or similar software.
  • Example using OpenSSL:

openssl req -newkey rsa:4096 -nodes -sha256 -keyout sf.key -x509 -days 3650 -subj “/C=DE/ST=Florida/O=NOW/CN=<insert_your_sf_fqdn_here>” -out sf.crt

This will create a new private key “sf.key” and a certificate “sf.crt”. The Subject name of the certificate, in this case, is not important, use it for your reference.

With the next command, we will create a new p12 keystore using the private key and the certificate we created earlier.

openssl pkcs12 -export -out sfNew.p12 -inkey sf.key -in sf.crt

• Copy the p12 keystore to the HCL SafeLinx server.
  • You can place it into the installation folder of HCL SafeLinx server or outside of it.
• Configure the HCL SafeLinx Access Manager to use the new p12 keystore.
  • To do this, we will use the HCL SafeLinx Administrator.
  • Connect to the HCL SafeLinx Server, switch to the “Resources” tab and open “Access Manager” properties:

Open the Access Manager properties by double-clicking on the “Access Manager” entry.

• Change the p12 keystore.
  • Switch to the “TLS” tab and enter the path to the newly created p12 keystore, as well as the password you have set for the database.

Here you can also modify the TLS settings for the connection, for example, I am only using TLS 1.3.

Note: If you have copied the p12 keystore into the installation folder of the HCL SafeLinx server, then you can use the relative path to the file, as in the screenshot above.

• Create a new “secure” connection profile in the HCL SafeLinx Administrator client.
  • Open the “Login Profile Details” menu:

Go to “File”, then select “Login Profiles…”.

• In the next menu, select “Add Secure Profile…”:

This will open a new window.

• Now we must enter the path to the p12 keystore on the client, as well as the password for opening the database:

After filling out all the information needed, click the “OK” button.

As of now, you should be able to use HCL SafeLinx Administrator to connect to the HCL SafeLinx server over an encrypted connection.

Engage 2022 – Domino TOTP/2FA – Best Practices and Pitfalls

It’s hard to describe how well the Engage conference is organized and how fabulous the event is. Engage is the place where many good people gather, who gave me the opportunity to learn from them, and over the years many of them became my friends. Therefore, it is something special for me to present at Engage, and for that, I am thankful and honored.

My colleague and dear friend Martin Leyrer and I talked about Domino and TOTP, below you can take a look at the slides we have used:

Engage.UG 2022 – Domino TOTP/2FA – Best Practices and Pitfalls from Milan Matejic

We have also recorded our session, sorry for the bad audio. You can take a look at the recording below:

I’m Speaking at Engage 2022 in Bruges! Yay!

For the second time, I will be speaking at the EngageUG conference. I’m honored and thrilled at the same time. Come join Martin Leyrer and me for the “Domino TOTP/2FA – Best Practices and Pitfalls” session, at which we will give our best to share all our knowledge and experience with TOTP on Domino!

If this year’s conference is going to be half as good as any other before it, we are going to have a great time! Learn what’s NEXT for HCL Digital Solutions and meet your friends and colleagues.

You haven’t registered? Don’t worry, there is still time, head over to the EngageUG website and book yourself a ticket! 🙂

Kubernetes – Host Entries

As Kubernetes pods do not make use of the Kubernetes nodes/hosts “host” file (/etc/hosts), which can be a challenge or a blessing, do not despair if you find yourself in an environment in which some DNS entries are missing. There is an easy workaround to “get you going”.

Although I have to stress out, that this is not a long-term solution, and you should use DNS as opposed to the following solution.

If you are using “coreDNS” on Kubernetes, edit the “configmap” of the service, by issuing the following command:

kubectl edit cm -n kube-system coredns

After that, just add the section which is marked red in the following screenshot:

And edit it to suit your needs.

Note: “fallthrough” parameter is important as it will make sure that the requests which can’t be handled by hosts configuration in the “coreDNS” pods, will be sent to the configured DNS servers.

After you are happy with the configuration, just recreate the “coreDNS” pods, by running the following command:

kubectl -n kube-system delete pods -l k8s-app=kube-dns

Note: this will cause a brief interruption of the name resolution service as the pods are being recreated.

You can test the DNS inside the Kubernetes network by creating a pod for this purpose that has all the “networking” tools you need, I decided to run with “infoblox/dnstools”.

kubectl run -it –rm –restart=Never –image=infoblox/dnstools:latest dnstools

HCL Connections – Orient Me “Loop”

In the course of the recent deployment of HCL Connections Component Pack 7.0.0.2, I ran into an issue with the Orient Me application.

After deploying Orient Me, every try to open the new Orient Me homepage would result in a “loop”, the user was being redirected from the Orient Me (/social) page to the Homepage application (/homepage) and back again to the Orient Me page. The root cause for the problem was not a bug or an error in the HCL Connections code, but rather the configuration of the HCL Connections Blue Stack and other components in the IT landscape of this environment.

After further investigating the issue, we have seen the following messages in one of the “orient-web-client” pods:

Mon, 14 Mar 2022 13:15:37 GMT hsts deprecated The “includeSubdomains” parameter is deprecated. Use “includeSubDomains” (with a capital D) instead. at node_modules/loopback/lib/server-app.js:74:25
{“pid”:7,”hostname”:”orient-web-client-57594bc9df-ck6qq”,”name”:”@connections/cachy-service/src/client/redis-client.js”,”level”:30,”time”:1647263737775,”msg”:”om-readiness-probe Initializing [%s]”,”v”:1}
Web server listening at: http://localhost:8000
2022-03-14T13:15:37.883Z – ESC[32minfoESC[39m: [orient-web-client] Cachy-gatekeeper register response: true
2022-03-14T13:17:41.020Z – ESC[34mdebugESC[39m: [auth-service] decodeJWT: no JWT found, returning no_jwt_token
2022-03-14T13:17:41.022Z – ESC[34mdebugESC[39m: [auth-service] >auth.service: about to call setJWT to generate or re-generate token…
2022-03-14T13:17:41.022Z – ESC[34mdebugESC[39m: [auth-service] Setting status to be 401
2022-03-14T13:17:41.024Z – ESC[34mdebugESC[39m: [auth-service] setJWT failed with err: [no_auth_token]: no_auth_token
2022-03-14T13:17:41.025Z – ESC[31merrorESC[39m: [orient-web-client] >server: ensureLogin callback with error: UnauthorizedError: [get_token_failed]: Unable to get token undefined
Unhandled error for request HEAD /social/auth/token: UnauthorizedError: [get_token_failed]: Unable to get token
2022-03-14T13:17:41.355Z – ESC[34mdebugESC[39m: [orient-web-client] ENTRY: AuthenticationMiddleware.handleAuthentication undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [orient-web-client] ENTRY: AuthenticationMiddleware.extractLDAPCookies undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [orient-web-client] EXIT: AuthenticationMiddleware.extractLDAPCookies for user id: undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [auth-service] decodeJWT: no JWT found, returning no_jwt_token
2022-03-14T13:17:41.357Z – ESC[33mwarnESC[39m: [auth-service] JWT token is bad, can not proceed for potential CSRF!
2022-03-14T13:17:41.358Z – ESC[34mdebugESC[39m: [orient-web-client] handleAuthentication ensureLogin error { [UnauthorizedError: [not_authorized]: Not authorized]
name: ‘UnauthorizedError’,
code: ‘not_authorized’,

What was obvious, was that the user didn’t have a JWT Cookie going to the Orient Me web client and the Orient Me web client could not generate the same as it could not recognize the username of the user “hitting” the service.

Thankfully, with the guidance and help of my colleague, Martin Leyrer, we managed to resolve this issue. In order to resolve the problem, we have done the following.

We checked and modified all DNS entries of all hosts and services (NGINX, IBM HTTP Server, etc…) involved and made sure these were resolving correctly.

The “dynamicHosts” property in the “LotusConnections-Config.xml” was enabled, as described in the official documentation, and we corrected the “interService” URLs (in our case the URL resolving to IBM HTTP Server), in the same configuration file.

On the WebSphere Application Servers, the LTPA Cookie names were not set to the default. As this could cause similar issues, we have changed those to default, “LtpaToken” and “LtpaToken2”.

Note: As version 2 of the LTPA Cookie is much more secure, we recommend only using this version as opposed to version 1.

The last change we have done was in the IBM HTTP Server configuration (“httpd.conf” file), here we have added the “ServerName” parameter to the <VirtualHost *:443> section, after that, the configuration looked like the following:
…
LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 443
<VirtualHost *:443>
ServerName ibm_ihs_hostname.domain.com
SSLEnable
RewriteEngine on
…

After doing all the changes mentioned above, we were able to resolve the issue and could successfully render and use the Orient Me application. 😉

I sincerely hope this will help and save you some pain! 🙂

HCL Connections CP – Enabling Elasticsearch Metrics

Recently, when enabling Elasticsearch metrics, part of the HCL Connections Component Pack 7.0.0.2, I ran into an issue with the “config_blue_metrics.py” script. Which was failing every time I tried to run it (sadly, I’ve “lost” the output and the direct error message which was shown directly after running the python script). The details on how to run this script are described in the official documentation.

Nevertheless, after looking into the WebSphere Metrics Application logs, “AppsCluster” if your HCL Connections Blue Stack environment is installed as a “Medium deployment”, I’ve noticed the following error:

[10/3/22 17:26:54:477 IST] 0000016c LotusConnecti E Unable to access the required data
javax.servlet.ServletException: java.io.FileNotFoundException: SRVE0190E: File not found: /configsetter
at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:248)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.lconn.core.web.customization.CustomizationFilter.doFilter(CustomizationFilter.java:142)
…
Caused by: java.io.FileNotFoundException: SRVE0190E: File not found: /configsetter
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
at
…

Thanks to my colleague, Herwig W. Schauer, we have found a quite simple solution for this issue. We have changed the default “Context Root For Web Modules” for “Metrics” and “MetricsUI” application. To do this, just open the WAS ISC and navigate to the following entry “Enterprise Applications –> MetricsUI –> Context Root For Web Modules”. We have changed the mentioned configuration in the following way.

We have set the “Metrics” application context root from “/metrics” and “/metrics/service to “/metricsOLD” and “/metricsOLD/service” respectively:

After making the changes mentioned above, we proceeded to make similar changes for the “MetricsUI” application, here we changed the Context Root from “/metricssc/service/rest” and “/metricssc” to “/metrics/service/rest” and “/metrics”:

After the needed configuration is in place, make sure that the WAS nodes are synchronized and restart the “Metrics” and “MetricsUI” applications. By doing so, the issue should be resolved and the “config_blue_metrics.py” script should be able to run without any errors.

Note: You still may get a warning message after running the script: “Warning elasticsearch not found in Kubnetes.”, but this is fine and does not lead to any further problems.