HCL Connections – Orient Me “Loop”

In the course of the recent deployment of HCL Connections Component Pack 7.0.0.2, I ran into an issue with the Orient Me application.

After deploying Orient Me, every try to open the new Orient Me homepage would result in a “loop”, the user was being redirected from the Orient Me (/social) page to the Homepage application (/homepage) and back again to the Orient Me page. The root cause for the problem was not a bug or an error in the HCL Connections code, but rather the configuration of the HCL Connections Blue Stack and other components in the IT landscape of this environment.

After further investigating the issue, we have seen the following messages in one of the “orient-web-client” pods:

Mon, 14 Mar 2022 13:15:37 GMT hsts deprecated The “includeSubdomains” parameter is deprecated. Use “includeSubDomains” (with a capital D) instead. at node_modules/loopback/lib/server-app.js:74:25
{“pid”:7,”hostname”:”orient-web-client-57594bc9df-ck6qq”,”name”:”@connections/cachy-service/src/client/redis-client.js”,”level”:30,”time”:1647263737775,”msg”:”om-readiness-probe Initializing [%s]”,”v”:1}
Web server listening at: http://localhost:8000
2022-03-14T13:15:37.883Z – ESC[32minfoESC[39m: [orient-web-client] Cachy-gatekeeper register response: true
2022-03-14T13:17:41.020Z – ESC[34mdebugESC[39m: [auth-service] decodeJWT: no JWT found, returning no_jwt_token
2022-03-14T13:17:41.022Z – ESC[34mdebugESC[39m: [auth-service] >auth.service: about to call setJWT to generate or re-generate token…
2022-03-14T13:17:41.022Z – ESC[34mdebugESC[39m: [auth-service] Setting status to be 401
2022-03-14T13:17:41.024Z – ESC[34mdebugESC[39m: [auth-service] setJWT failed with err: [no_auth_token]: no_auth_token
2022-03-14T13:17:41.025Z – ESC[31merrorESC[39m: [orient-web-client] >server: ensureLogin callback with error: UnauthorizedError: [get_token_failed]: Unable to get token undefined
Unhandled error for request HEAD /social/auth/token: UnauthorizedError: [get_token_failed]: Unable to get token
2022-03-14T13:17:41.355Z – ESC[34mdebugESC[39m: [orient-web-client] ENTRY: AuthenticationMiddleware.handleAuthentication undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [orient-web-client] ENTRY: AuthenticationMiddleware.extractLDAPCookies undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [orient-web-client] EXIT: AuthenticationMiddleware.extractLDAPCookies for user id: undefined
2022-03-14T13:17:41.356Z – ESC[34mdebugESC[39m: [auth-service] decodeJWT: no JWT found, returning no_jwt_token
2022-03-14T13:17:41.357Z – ESC[33mwarnESC[39m: [auth-service] JWT token is bad, can not proceed for potential CSRF!
2022-03-14T13:17:41.358Z – ESC[34mdebugESC[39m: [orient-web-client] handleAuthentication ensureLogin error { [UnauthorizedError: [not_authorized]: Not authorized]
name: ‘UnauthorizedError’,
code: ‘not_authorized’,

What was obvious, was that the user didn’t have a JWT Cookie going to the Orient Me web client and the Orient Me web client could not generate the same as it could not recognize the username of the user “hitting” the service.

Thankfully, with the guidance and help of my colleague, Martin Leyrer, we managed to resolve this issue. In order to resolve the problem, we have done the following.

We checked and modified all DNS entries of all hosts and services (NGINX, IBM HTTP Server, etc…) involved and made sure these were resolving correctly.

The “dynamicHosts” property in the “LotusConnections-Config.xml” was enabled, as described in the official documentation, and we corrected the “interService” URLs (in our case the URL resolving to IBM HTTP Server), in the same configuration file.

On the WebSphere Application Servers, the LTPA Cookie names were not set to the default. As this could cause similar issues, we have changed those to default, “LtpaToken” and “LtpaToken2”.

Note: As version 2 of the LTPA Cookie is much more secure, we recommend only using this version as opposed to version 1.

The last change we have done was in the IBM HTTP Server configuration (“httpd.conf” file), here we have added the “ServerName” parameter to the <VirtualHost *:443> section, after that, the configuration looked like the following:

LoadModule ibm_ssl_module modules/mod_ibm_ssl.so
Listen 443
<VirtualHost *:443>
ServerName ibm_ihs_hostname.domain.com
SSLEnable
RewriteEngine on

After doing all the changes mentioned above, we were able to resolve the issue and could successfully render and use the Orient Me application. 😉

I sincerely hope this will help and save you some pain! 🙂

HCL Connections CP – Enabling Elasticsearch Metrics

Recently, when enabling Elasticsearch metrics, part of the HCL Connections Component Pack 7.0.0.2, I ran into an issue with the “config_blue_metrics.py” script. Which was failing every time I tried to run it (sadly, I’ve “lost” the output and the direct error message which was shown directly after running the python script). The details on how to run this script are described in the official documentation.

Nevertheless, after looking into the WebSphere Metrics Application logs, “AppsCluster” if your HCL Connections Blue Stack environment is installed as a “Medium deployment”, I’ve noticed the following error:

[10/3/22 17:26:54:477 IST] 0000016c LotusConnecti E Unable to access the required data
javax.servlet.ServletException: java.io.FileNotFoundException: SRVE0190E: File not found: /configsetter

at com.ibm.ws.webcontainer.filter.FilterInstanceWrapper.doFilter(FilterInstanceWrapper.java:248)
at com.ibm.ws.webcontainer.filter.WebAppFilterChain.doFilter(WebAppFilterChain.java:91)
at com.ibm.lconn.core.web.customization.CustomizationFilter.doFilter(CustomizationFilter.java:142)


Caused by: java.io.FileNotFoundException: SRVE0190E: File not found: /configsetter
at com.ibm.ws.webcontainer.extension.DefaultExtensionProcessor._processEDR(DefaultExtensionProcessor.java:977)
at


Thanks to my colleague, Herwig W. Schauer, we have found a quite simple solution for this issue. We have changed the default “Context Root For Web Modules” for “Metrics” and “MetricsUI” application. To do this, just open the WAS ISC and navigate to the following entry “Enterprise Applications –> MetricsUI –> Context Root For Web Modules”. We have changed the mentioned configuration in the following way.

We have set the “Metrics” application context root from “/metrics” and “/metrics/service to “/metricsOLD” and “/metricsOLD/service” respectively:

After making the changes mentioned above, we proceeded to make similar changes for the “MetricsUI” application, here we changed the Context Root from “/metricssc/service/rest” and “/metricssc” to “/metrics/service/rest” and “/metrics”:

After the needed configuration is in place, make sure that the WAS nodes are synchronized and restart the “Metrics” and “MetricsUI” applications. By doing so, the issue should be resolved and the “config_blue_metrics.py” script should be able to run without any errors.

Note: You still may get a warning message after running the script: “Warning elasticsearch not found in Kubnetes.”, but this is fine and does not lead to any further problems.

HCL Connections Docs – 2.0.1 Update Issue

Lately I had an issue upgrading from HCL Docs 2.0 CR3 iFix009 to 2.0.1. While upgrading “Docs Editor” application, using upgrade/install scripts, I was blessed with the following error code:

2021-03-05 17:23:16,464 INFO –>IM:RegisterDocsInNews
2021-03-05 17:23:16,465 INFO Register Docs application in Connections News Started
2021-03-05 17:23:40,280 INFO Register Docs application in Connections News completed
2021-03-05 17:23:47,996 INFO –>IM:ReviseLotusConnectionsConfig
2021-03-05 17:23:47,997 INFO Register Docs application in LCC.xml
2021-03-05 17:24:03,913 INFO Exception: ‘ascii’ codec can’t encode character u’\xa0′ in position 32627: ordinal not in range(128)
2021-03-05 17:24:03,913 ERROR Error while executing command, now rollback previous changes…
2021-03-05 17:24:03,913 INFO –>IM:Rollback RegisterDocsInNews
2021-03-05 17:24:03,914 INFO Rollback successfully.
2021-03-05 17:24:03,914 INFO Remove temp dir…
2021-03-05 17:24:03,915 INFO Exception: Upgrade failed. Check log file for more detail.
2021-03-05 17:24:03,915 INFO –>IM:FAILED

Although it took me a lot of time to solve this issue, it is quite simple.  I don’t know how, but I had some “Non-breaking space” characters in the LotusConnections-config.xml file for which, as it turns out, Python 2.7 does not show a lot of love.

To solve this problem I have done the following:

  1. Login to the WAS Deployment Manager and check out the LotusConnections-config.xml.
  2. Create a backup of the file.
  3. Delete all “Non-breaking space characters.
    • I managed to do this only in vim editor, by using the following setting “set list listchars=nbsp:!”.
    • Delete all newly shown “Non-breaking space” characters.
    • For more information, take a look at the Stack Exchange post.
  4. Check the file back in.
  5. Synchronize all WAS nodes in the cell.
  6. Restart the installation.

After that, the installation should complete. I hope this helps. 🙂

Note that the Error code shown above lists the exact position in the “LotusConnections-config.xml” file where the problematic character was found in the first place, but this can be misleading as you could have more than one “Non-breaking space” character in the file, so make sure to delete all of those characters.

HCL Connections 7 – PDF Export Issues

After upgrading to HCL Connections 7, the new PDF Export feature didn’t work. By clicking on the new “PDF” button inside the Wiki Page and trying to export it as a PDF, I would get an error in the GUI. In my two environments where I have encountered this, I had to do the following steps.

  1. Install “wkhtmltopdf” software and additional fonts.

This step is very well documented in the official documentation. Albeit a bit hidden.

2. Modify the IC360 configuration parameters.

In order to do this, login to the WAS Administration Console or ISC, and navigate to “Resources –>Resource Environment –> Resource environment entries –> ic360 –> Custom Properties”. Make sure that the following configuration parameters are correct:

“http.hostname” should correspond with the FQDN you are using to access HCL Connections, for example “social.domain.com”.

“http.auth.admin.user” set to WAS Administrator username.

“http.auth.admin.password” needs to have the value of WAS Administrator password. When changing the password, just delete the whole value, including “{xor}” in the field, after saving, the password will automatically be encrypted with XOR.

For additional information about IC360 configuration parameters, refer to the official documentation.

3. Sync and restart IC360 Applications.

After doing the steps mentioned above, you should be able to export the HCL Connections content to PDF files.

HCL Connections – Do Not Forget To Install A New APNs Certificate For 2021

By default, the APNs certificates are valid for one year. The current APNs certificate for HCL Connections is going to expire on the 21st of March. You can update the APNs certificate by installing an Interim Fix provided by the HCL Connections development team, which can be obtained from the HCL Software License & Delivery Portal. There is an Interim Fix for basically all relevant versions of HCL Connections, 7, 6.5 CR1 and 6.0 CR6 and 6.0 CR3. For all details, take a look at the official article.

As soon as the APNs certificate expires, the users of the HCL Connections mobile app on iOS will not be able to get push notifications, which means that they will not get alerts or notifications from HCL Connections while the mobile application is running in the background.

Installing Interim Fixes for HCL Connections is fairly straightforward, for a step-by-step guide, refer to the official documentation. This particular Interim Fix will take about one to five minutes to install, depending on the environment.

HCL Connections Invite – Issues when using TDS/SDS as User Repository

Take care when deploying the HCL Connections Invite application using the TDS (Tivoli Directory Server)/SDS (IBM Security Directory Server) as user repository. The following information from the official documentation is wrong:

The following value includes the LDAP property used to map the user in LDAP to their Profiles GUID. In an Active Directory environment, for example, the default property is objectGUID. Other known defaults are entryuuid (IBM Security Director Integrator, formerly TDI),…

<profiles-connector>
             <entry>
                  <attributes>
                       <attribute>
                            <value>${ldap:objectGUID}</value>

First, the value of the Attribute in the “selfregistration-config.xml” is missing, this would be: “<key>com.ibm.snx_profiles.base.guid</key>”.

By reading the documentation, you would set the <value> parameter to ${ldap:entryuuid}, if you do that, you would get the following error:

00000660 SelfRegistrat W   Failed to get all the required attributes! required: ‘{​${​ldap:entryuuid}​=entryuuid, ${​ldap:sn}​=sn, ${​ldap:cn}​=cn}​’, received attributes: ‘{​sn=sn: Mail, cn=cn: <e-mail>}​’

The correct value for the parameter, when using TDS/SDS, would be “${ldap:ibm-entryuuid}”, so you should endup with the following:

<attribute>
                    <key>com.ibm.snx_profiles.base.guid</key>
                    <value>${ldap:ibm-entryuuid}</value>
                    <type>text</type>
                </attribute>

Although there are some comments in the “selfregistration-config.xml” showing the right information, a few colleagues/friends including me found them to be a bit “misleading”. This will surely be corrected in the future.

HCL Connections & Kerberos Authentication Protocol Issue

After implementing Kerberos Authentication protocol for HCL Connections, as described in the official documentation (HCL Connections and IBM WebSphere documentation) and restarting the whole environment, the “synchronization status” of the Nodes in the IBM WebSphere ISC Console appeared to be “unknown”. All the HCL Connections Applications were running, there were no errors in GUI and the SSO was working without any issues. From the logs of the node agents we could see that the synchronization with the deployment manager was taking place and completing successfully, only in the “SystemOut.log” of the WepSphere deployment manager I found the following error message:

000015c Krb5WSSecurit E SECJ9314E: An unexpected exception occurred when trying to run initSecContext() method : GSSException: org.ietf.jgss.GSSException, major code: 11, minor code: 41


major string: General failure, unspecified at GSSAPI level
minor string: Kerberos error formatting credential for delegation: 41
at com.ibm.security.jgss.i18n.I18NException.throwGSSException(Unknown Source)
at com.ibm.security.jgss.mech.krb5.g.a(Unknown Source)
at com.ibm.security.jgss.mech.krb5.g.a(Unknown Source)
at com.ibm.security.jgss.mech.krb5.g.initSecContext(Unknown Source)
at com.ibm.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
at com.ibm.security.jgss.GSSContextImpl.initSecContext(Unknown Source)
at com.ibm.ISecurityLocalObjectTokenBaseImpl.Krb5WSSecurityContextImpl$1.run(Krb5WSSecurityContextImpl.java:508)
at java.security.AccessController.doPrivileged(AccessController.java:770)
at javax.security.auth.Subject.doAs(Subject.java:570)
at com.ibm.ISecurityLocalObjectTokenBaseImpl.Krb5WSSecurityContextImpl.initSecContext(Krb5WSSecurityContextImpl.java:273)
at com.ibm.ISecurityLocalObjectTokenBaseImpl.Krb5WSSecurityContextImpl.initSecContext(Krb5WSSecurityContextImpl.java:157)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.processInternal(SOAPConnectorClient.java:1285)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.getSecurityHeader(SOAPConnectorClient.java:1095)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplateOnce(SOAPConnectorClient.java:783)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:697)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invokeTemplate(SOAPConnectorClient.java:687)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.queryNames(SOAPConnectorClient.java:599)
at com.ibm.ws.management.connector.soap.SOAPConnectorClient.invoke(SOAPConnectorClient.java:524)
at com.sun.proxy.$Proxy40.queryNames(Unknown Source)
at com.ibm.ws.management.AdminClientImpl.queryNames(AdminClientImpl.java:108)
at com.ibm.ws.management.AdminServiceImpl.queryNames(AdminServiceImpl.java:679)
at com.ibm.ws.management.connector.AdminServiceDelegator.queryNames(AdminServiceDelegator.java:113)
at sun.reflect.GeneratedMethodAccessor88.invoke(Unknown Source)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:55)
at java.lang.reflect.Method.invoke(Method.java:508)
at com.ibm.ws.management.connector.soap.SOAPConnector.invoke(SOAPConnector.java:503)
at com.ibm.ws.management.connector.soap.SOAPConnector.service(SOAPConnector.java:335)
at com.ibm.ws.management.connector.soap.SOAPConnection.handleRequest(SOAPConnection.java:65)
at com.ibm.ws.http.HttpConnection.readAndHandleRequest(HttpConnection.java:733)
at com.ibm.ws.http.HttpConnection.run(HttpConnection.java:522)
at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1892)

After a quick Google search I found that this boils down to a problem with a hostname defined or spelled in upper case letters. As described in the following article:

https://www.ibm.com/support/pages/apar/PM66457

This article also mentions a custom variable which could be set to remedy this issue.

But before trying the custom variable out, I wanted to make sure that this was really the issue. I checked the DNS Server and all SPN records, I could not find any issues there. The output of the “hostname” and “hostnamectl” commands (the whole environment is running on RHEL), were all returning the hostname written in (all) lowercase characters. Only when looking into the “hosts” file of the deployment manager I could find the issue, there was a “127.0.1.1” entry stylized in “CamelCase”. I replaced the upper case letters in the “hosts” file with lower case letters and as soon as I have done that, all the nodes in the ISC console had the status “synchronized” and the error mentioned above did not appear anymore. The custom variable was not needed.

I hope this saves you some time! 🙂

HCL Notes Connections Plug-In – Login via SPNEGO not possible

If you are trying to use SPNEGO as the means for authenticating HCL Notes Connections Plug-In and you run into an issue, check if “Use the alias host name for the application server” option is used on Websphere Application Server and if that is the case try turning it off.

For some reason the HCL Notes Connections Plug-In cannot use SPNEGO authentication mechanism when this option is used on WAS. You can find this setting under “Global Security – Web and SIP security – SPNEGO web authentication”.

Depending on your DNS configuration and SPN entries created, this option may be needed for HCL Connections or other applications, so make sure you test everything appropriately after changing the SPNEGO settings.

HCL Connections – URL Preview Tip

If you have implemented or planing to implement the “URL Preview” feature for HCL Connections, make sure you import the Root Certificate for “Let’s Encrypt” certificate authority inside the java trust store.

By default, the root certificate of “Let’s Encrypt” certificate authority is not in the java trust store which the HCL Connections documentation suggests to use for URL Preview. Which means that if you do not import it manually, the URL preview won’t work for websites which are using “Let’s Encrypt” certificates.

HCL Connections Invite – Trace/Debug Parameters

If you are having issues with HCL Connections Invite, than the following Websphere debug or trace code might help:

com.ibm.lconn.registration.*=finest

HCL Connections Invite is assigned, per Default, to the WebSphere Homepage Cluster/Server. So this is where you have to set the parameter mentioned above.

This is how you can set the debug parameter during runtime. No restart needed.