KBA00021224

This article details fine troubleshooting steps for Hiplink and CAD paging not sending texts or pages out. Agency admins, SAAs, and user should use this article to troubleshoot Hiplink and CAD Paging before contacting Technical Services. The majority of Hiplink and CAD paging problems can be resolved with these suggestions. 

NOTE: Some of this article will refer to Hiplink specific processes. If your agency is not using Hiplink, you will not need to check these.

Troubleshooting the setup

See Hiplink Paging - How it works and how to set it up for detailed info:

• Verify that pgplan has the correct type, nature, agency and groups (from pggroups) set up.
• Verify that pggroups has the group ID, groups, units/individuals set up.
• Verify that pgpager has correct pager ID, pager number, description, carrier ID, and unit OR officer.
  NOTE: The value must be unit OR officer, not both, If you have both it will cause conflicts in the software and potential failures.
• Finally cdunit should be checked to make sure that the unit entry has the correct officer set up.

Troubleshooting the CAD paging interface

Detailed information can be found in the Paging Interface manual.

If the problem was with a manual page. Can you replicate the issue? Is it because of bad formatting of the command? See the table below for the correct command formats:



If the problem was with an automatic page

Pages are automatically sent depending on the type of page:

• Paging Plan pages. A paging plan page is sent whenever a CAD call meets the conditions of the paging plan set up by the SAA.
• Dispatched Unit pages. A dispatched unit page is sent to a unit whenever the unit is assigned to a call.
• Completed Call pages. A completed call page is sent to a unit when it reaches any status the SAA has defined in the pgcmplt apparam.

Troubleshooting Automatic Paging Rules

• Off-duty units are not paged by automatic paging
  • If the unit shows as off duty in cdunit and syunit this explains why it did not send.
  • This can also be verified in the paging logs
    • Unix/Linux
      • /sds/app/force/custom/cadpaging/ubin/pageAuto.debug
    • Windows
      • \SpillmanServer\app\force\custom\cadpaging\ubin\pageAuto.debug
    • Example: 11/29/2017 21:13:53       Cannot page ABC, ABC is not on duty.
• A unit or person receives only one automatic page for each call status. NOTE: The officer will receive a second pgcmplt page because that is a different type of page.
  • If a unit/person is receiving more than one it's possible that there are multiple CAD paging processes running
  • Verify this by opening sydaemon and finding the Auto cad paging daemon.
    
    
     
• Here you can see the process ID as well as the process name
• Next you will want to do a process name search:
  
  Unix/Linux
  
  ps -ef | grep <Exec Name>
  
  Windows
  
  tasklist | findstr <Exec Name>
  
  If you see multiple processes listing do the following:
  • Stop the paging daemons.
  • Kill any remaining processes found by the previous command.
  • Start the paging daemon.
     
• If the dispatcher changes the nature of a call, the interface pages all group members who are to receive pages for calls of the new nature, even persons who received a page for the old nature.
• If this isn't working you'll need to verify the call nature and group members are set up properly. See the section above about troubleshooting the setup.

The CLI_log.txt can be in a few locations depending on the setup of hiplink. Being as it's not really our software it seems to vary from site to site, but currently the running locations it may exist are as follows:

Linux:
/sds/app/force/custom/hiplink/bin
/usr/local/hiplink/bin
/tmp

Windows:
\SpillmanServer\app\force\custom\HipLink\bin\CLI_log.txt

if all else fails in finding it on Linux you can search for it using:

find -name /sds/app/ "CLI_log.txt"
or
find -name /usr/local "CLI_log.txt"
or
find -name /tmp "CLI_log.txt"

on Windows you'd use the Windows Explorer search bar.

Daemons Randomly Die

Beginning in 2024.1, a new function is intended to keep Perl processes from timing out. This has somehow had the unintended effect of causing HipLink daemons to randomly die. For the time being, to prevent this, open /sds/app/tomcat/properties/Spillman.properties and delete the line PerlProcessTimeout=600, then restart the Spillman standalone.

Final Troubleshooting Note

There will be instances where a page was not sent out, but everything looks correct. The best way to see where the problem might lie is by double-checking the logs again for the auto paging.

The key log you will be looking for looks something like this:

04/02/2018 10:27:46 Auto : ***** SENT    7 PAGES OUT OF 7 *****

You'll need to search the logs for the date/time stamp of when the page was sent out and verify what that line says. Generally you will see either something like the above or something like this:

04/02/2018 10:27:46 Auto : ***** SENT    0 PAGES OUT OF 7 *****

If you see the former then the CAD Paging part of the interface is working and sending. From here it's likely the failure is somewhere outside of the interface that needs to be checked. If you see the latter, it's proof that there's something up with the CAD Paging interface sending and thus confirms the need to verify the above troubleshooting steps. If everything above is correct and the latter is happening Technical Support can assist in further troubleshooting.

Another area to check if you see the latter message would be the pageComm.debug log file. Find the time of the sent message and see what info it may have there. This is where you may need to have Technical Support help to investigate further.

Troubleshooting Hiplink specific issues

Hiplink has it's own additional part of the interface that is generally supported by Hiplink and thus Spillman Technical Services might not be able to fully provide support for issues on the Hiplink end of the interface. However, there are a few things in this article to help in troubleshooting Hiplink and what to find.

While you are seeing the error 04/02/2018 10:27:46 Auto : ***** SENT    0 PAGES OUT OF 7 ***** it will be valuable to check the pageComm.debug log file and see what details might be within this log file. In many instances Hiplink has failed to accept the messages for some reason. We can see this with a simple log entry in this log:

02/11/2018 13:17:43 cliSend( , )->Message failed.

If we see this message it is indicating that CAD Paging had attempted to pass the page to the Hiplink CLI using the cliSend method in the code, but something in the CLI failed that caused the message to fail. We can investigate a bit further to find what may be the issue within the CLI by checking the Hiplink CLI logs found on the server in /sds/app/force/custom/hiplink/bin/CLI_log.txt or \SpillmanServer\app\force\custom\HipLink\bin\CLI_log.txt. NOTE: The logs might be in a different location on your server, but these are generally the location to look for them.

Once we are looking in this log we can search the date/time stamp to see what info is in the log at that time. In most cases we should find a four digit error code that will be a Hiplink indicator of a failure. 

The error codes that Hiplink has are found in this document from Hiplink: http://www.hiplink.net/guides/Programmers_Guide.pdf 

If you see them they should help in determining what needs to be done or you can reach out to Hiplink for further help resolving these issues