Q1. My Domino server crashed/hanged. What should I do now?
User tries to access the application and gets error “ Domino server not responding “ or “ Remote Server not responding “.
Causes:
Server Hang / Server Crash.
So, what is a Server Hang?
Hang is a situation where the Domino server is still running and can see domino console, but one or more tasks on the server are not responding to requests. Most of the time, a hang occurs due to a low-level loop or a permanent unavailability of a resource, causing serious performance issues.
A few causes that can result in the server hang are given below:
- Too many applications running at same time
- Multiple agents processing
- Heavy traffic
- Too many users accessing the server at the same time
- Slow network
- Server tasks not responding
- Semaphore errors
Fix:
Fixing Domino server hang varies upon different situations and depends on the error message that throws on the server console. In most cases quitting the respective task and starting them again resolves the issue.
Incase of heavy traffic we would have to review the number of concurrent users using the server. This is normally controlled by setting up the ini parameter on the domino server: Server_Maxusers which is normally a good practice to come across such situations.
An ideal Domino Admin environment to get over such situations would be to setup clustered environments that can withstand the load based on the number of users accessing the server.
A few links to understand and setup the Domino Admin is given below :
https://www.ibm.com/developerworks/lotus/documentation/d-ls-dominoclustering/index.html
Server Crash:
A server hang not properly fixed results in server crash. Server crash normally results in the complete shutdown of Domino server without any warning or any specific presumptions.
In most cases domino servers are configured to automatically recover using the Fault recovery mode and the server also generates a clear report called NSD which helps in trouble shooting the actual cause of the crash and fixing them from future occurrences.
NSD: Notes System Diagnostic
NSD (Notes System Diagnostic) is one of the primary diagnostics used for the Lotus Domino Product Suite. It is used to troubleshoot crashes, hangs and severe performance problems for Domino Server & Notes Client.
Refer the link to get an understanding on NSD and how the same can be used to analyse and fix the Domino servers from future crashes.
Predominant reasons for Domino server crash:
- Unhealthy applications
- Database size
- Server disk space issue
- Frequent hang on any server task.
NSDs can be generated both manually and also configured in server document which can be done with the links given below.
https://www-10.lotus.com/ldd/dominowiki.nsf/dx/How_to_run_a_manual_NSD_on_different_platforms
There are also various tools that can be used for analyzing NSD which are available online and easily points to different solutions from IBM to fix the issue. Below links would help to download and work on NSDs.
Best Practices to avoid Domino server hang and server crashes.
- Run frequent Diagnostics on Domino server to determine the disk space.
- Schedule regular maintenance tasks on applications can be done using program documents on daily or weekly basis.
- Perform a proper restart of Domino server on Weekly basis.
- Schedule a task like backup and agents without each of them hindering each other.
- Upgrade server with proper patches and latest versions and fixes if any required.
The above scenarios are frequent in Versions of Domino from Version 6 to Version 9.
Q2. I got a Semaphore error in Domino Admin. What’s wrong and how do I fix it?
Semaphore :
The most common is where a task locks the semaphore and then crashes. This can also happen if a task locks the semaphore and then goes into an endless loop or it gets an error and forgets to unlock it. Semaphore deadlock can occur when two tasks try to lock two different semaphores in a different order.
Most predominant error that occurs in the Domino application server is
Occurrences:
To explain in simpler terms when multiple agents or tasks try to process at the same time without releasing the flag that is engaged in Domino server resulting in an endless loop. Most of this results in Domino Hang or Crash.
Link to a very detailed article to understand semaphore is given below.
https://www-01.ibm.com/support/docview.wss?uid=swg21094630
Best practices to avoid Semaphore issues:
- Take care that no two agents are scheduled at the same time
- Maintain proper health of DBs by performing regular maintenance tasks.
- Avoid having DBs of large sizes.
- Perform proper archiving activity to have DB sizes under control.
The above scenarios are frequent in versions of Domino from version 6 and above to version 9.
Q3. Key pointers during Domino Server Consolidation
Domino Server Administration Consolidation:
A predominant task for the Domino Admins is to save the cost of storage and ensure optimal utilization of the server while maximising uptime. By thwarting any major issues on server pro-actively, the administrator can avoid any disruption to business continuity.
Why Server Consolidation?
A few reasons why we perform server consolidation:
- Reduce the number of dormant and non-productive servers.
- Reduce the cost of the storage.
- Enhance the throughput of the server.
- Decommission outdated servers to be used for any other purpose.
- Prepare the existing environment post migration to other platforms.
Steps to be taken care for Server Consolidation:
- Number of databases, depends on the size
- Size of databases, max 52 GB recommended.
- Technical specification of the server.
- Number of tasks on the server
- Number of users accessing the server.
- Number of URLs hosted in the server
- Whether in WAN, LAN or facing the internet.
- Number of clusters.
- Backup schedule and the time for backup.
- Current understanding of the Architecture.
- Analysing the new architecture if there are any changes.
A detailed document on server consolidation can be referred with the below link:
https://www-01.ibm.com/support/docview.wss?uid=swg27008851
How to consolidate multiple domains within domain?
Domain Migration:
Domino Domain Migration is performed to merge all the users in one primary domain, If the organization has multiple domains.
There are a few pointers that needs to be followed before we consider an environment for Domain migration:
- Decide on a proper domain to which all the servers and contents needs to be migrated.
- Number of users and groups
- Connection documents
- Administration tasks on the server.
- URLs configured for any of the domains under scope of migration
A brief on the steps to migrate the domain can be perused below :
https://www.ibm.com/developerworks/lotus/library/ls-Merging_Domino_domains/index.html
https://www-10.lotus.com/ldd/dominowiki.nsf/dx/10282009054817AMWEBDHQ.htm
Q4. My agent managers in Domino Admin are not getting triggered on the right time. How do I fix it?
Agent Manager:
Agents are scripted codes that are designed by the developer to perform specific server tasks on the database.
Most of the Agents are automated and run on a schedule as designed and configured by the developer. They are designed to run on Server or on Demand depending on the need and the frequency of executing the agent.
The task of an agent manager is to make sure that any designed agent is performing the said task as programmed by the developer and ensure the they are completed on time.
Predominantly Admins face a lot of issues concerned to Agents which not only deteriorates the performance of server but results in overall crash of the server.
Few pointers to remember when working with Agents:
- Admins should be aware of the number of agents scheduled on the server and also have a track of the schedules, there are console commands to give these information, refer link below:
https://goo.gl/VKbWWS
- Backup to be scheduled making sure that they are not hindering any agents, normally developers would have good idea on the agents that has been deployed in a particular server.
- Avoiding too many agents scheduled on the same server, proper coordination with the Admins and developer to make sure that they are not clashing at the same interval of time.
- If clustered environment, it would be ideal to schedule agents to run on cluster and applications on another server.
- Monitoring agent logs on regular basis to avoid malfunctioning of any agents, probably getting the help of developers to understand and fix the agent thus avoiding server issues due to same.
There are good tools that can help in scheduling the agent one among them is given below:
https://rprwyatt.com/product/essential-agent-master
This is a simple tool that can give all the details of the agents running in a Domino server along with schedules and timings so that Admins can manage agents effectively.
Q5. While trying to access a link in the Domino Admin platform, I got an error saying “SSL certificate expired”. Why is it coming?
SSL Expiry on Domino server:
SSL is a tool that Domino provides by itself as s self-contained pack in Domino. Although there are external partners who provide SSL signers but Domino can handle it by themselves even without an external Signer.
The way to setup an SSL is pretty simple which can be understood with the given link :
https://www-01.ibm.com/support/docview.wss?uid=swg21114148
Application servers hosting URLs predominantly come across issues wherein the SSL certificate expires after a given time. Generally, SSLs are valid for a minimum of one year and can be setup for 2 or 5 years, but normally most of the organizations go in for yearly renewal since the external signers keep changing based on budgeting and costing.
Few preferred SSL signers to name are Verisign, Symantec who are most preferred partners any organizations associate with.
SSL certificate expiry can be fixed by Admins who have Managed PKI authority certificate from the partner website.
The usual process would be to get the kyr file and sth file from the domino server and upload the keypair in the Symantec website for the Managed PKI to approve the renewal and the tenure of the renewal. Once this is done replace the kyr in the Domino data directory and restart the http task.
Detailed steps for performing is given in the link below :
Few pointers:
- Make sure to fill in proper country code and relevant info in the CA database while generating the keypair since any variation in data will not be accepted by the SSL signer.
- Make sure to remove any old kyr and sth files before placing the new one.
- Make sure to restart the http task once the renewed files are placed in the data directory
Q6. I encountered an HTTP hang. What should I do?
HTTP Hang:
If you are an Domino Apps Admin, this is something very predominant issue we will face with Apps server.
During this the http task is in hung state and results in serious server performance. They are classified as two types:
- Performance – Slow http response and process though the task is running.
- Semaphore Deadlock – Two processes waiting for each other to complete and http will not respond to any of the console commands to requests.
It would be very important for the Admins to first enable the debugging if we need to troubleshoot this issue.
Access to domlog.nsf , Task Manager info , OS level diagnostics , ini parameters and server document info would be needed to effectively troubleshoot http hang.
Normally trouble shooting http hang goes in phases as given below:
Phase 1: Determining if the hang is due to an agent
Phase 2: Finding the originating thread/URL hang
Phase 3: Determining the cause of the agent hang
The above information can be derived from the semdebug that is generated when running the NSD and can be taken from the IBM technical folder.
Once these are done, we might have to raise a PMR with IBM who would give us appropriate fix and scripts if required for further analysis or fixture. The link below will help us in effective troubleshooting of http hang.
https://www-01.ibm.com/support/docview.wss?uid=swg27010969
Basic care to avoid http hang:
- Regular maintenance of Domino databases
- Keeping the database size under limit
- Perform regular server restarts as a part of maintenance
- Scheduling proper program documents for maintenance of critical databases on which the task is running.
Q7. I don’t want multiple passwords. How do I integrate it with Windows?
ADSSO :
Domino Server can be integrated with other platforms to achieve certain technical ease in accessing information. This is a concept where the Domino server is integrated with Active Directory to achieve authentication using Windows credentials.
Domino Admin has an inbuilt ADsynch tool that can be used to achieve a single sign on facility to access applications on Domino server.
All we require is to register the following dll file in the server which needs the Ad synch and we are done with the job.
Regsvr32 nadsync.dll
Once this is registered, the AD synch gets enabled in the AD server and we just need to enable the feature by doing a right click.
Care must be taken that the proper schema is selected in AD that is analogous to Domino else the user name and password field won’t match as desired.
A very detailed technical document is given by IBM that will help in setting this AD SSO and can be used to achieve the requirements.
https://www.ibm.com/developerworks/lotus/library/domino-adsync/index.html
Apart from these there are other Microsoft tools that can be used and the below link should help for the same:
Read the articled titled “5 ways to get better ROI from Domino Infrastructure” Click Here
Nice Post
Pretty nice post. I just stumbled upon your weblog and wanted to say that I have really enjoyed browsing your blog posts. After all I will be subscribing to your feed and I hope you write again very soon!