Credit Card Uptime and System Improvements
For those of you who have been experiencing credit card issues over the last two months, we wanted to write a brief explanation of what has transpired and update you on the solutions we’re implementing at Club Prophet to ensure maximum uptime going forward.
First, we know the pain these issues can cause. Credit card and software uptime is the lifeblood of your business. We take these issues very seriously and are working on multiple solutions around the clock to make our environment “best in class” and reliable for the thousands of facilities we serve.
Some of these recent outages are out of our control and caused by the stress placed on Internet services and credit card platforms around the world. Since the pandemic began, cash-based transactions have almost entirely been eliminated. This has caused a large increase in volume across the entire credit card association ecosystem. Two specific instances outside of CPS affected the entire card processing platform, including CPS users. In the first instance, the main credit card backbone of TSYS went down (May 30th outage) and the second instance was when Microsoft and their Azure platform went down (April 1st outage). Previously, we would have considered these “once-in-a-decade” events due to their past performance and robust setups. However, both systems went down in the span of 60 days. Each of these companies have made large improvements in their infrastructure that make another occurrence extremely unlikely. These are unprecedented times! (For perspective, facilities using CPS software processed around 1,400,000 credit card transactions in the month of April, an increase of 240% over two years ago).
Other outages are a result of our own setup, server infrastructure, and system architecture. With that in mind, we wanted to share some of the improvements we’re making at Club Prophet:
- Improved Hardware Settings – On June 5th, we upgraded our credit card server to have 4-times more RAM and doubled the CPUs on the PCI Credit Card server to help handle the increased load. The prior settings were more than acceptable for several years, but during times of high-volume, we would occasionally have spikes in performance that would cause a slow-down or crash.
- Failover Servers with Load Balancing – We are adding additional servers that will be controlled with a Load Balancer. This will double again (from the item above) our capacity and have the added benefit of routing all traffic (should one server crash or go offline) to the second server. This will allow the ecosystem to continue operating while the server issue is addressed. The tech staff is currently working on an implementation timeline with contingency plans for bringing this online.
- Software Improvement – We addressed two bugs that were causing system performance issues, adjusted the app pool settings that would cause performance spikes, and eliminated automatic server reboots.
- Targeted Communication of Outages/Issues – We are now putting in 24-hour surveillance and server monitoring, as well as targeted communication to key people at each facility in case of future issues. This will also include text messaging for major outages and corrective measures.
- Future Credit Card Enhancements - Later this summer we will be rolling out a new architecture for the credit card gateway. This new design will remove the current local credit card service removing an entire layer of complexity and using the same load balance and fail-over features noted above. This new platform is currently going through a full PCI-DSS Level-1 audit.
If you continue to experience issues, please let us know as there are always unique conditions in each environment (Internet Service Providers, routers, firewalls, etc.). We are proud of the work our team is doing in the face of these challenges, and we will continue to make improvements to deliver the best credit card infrastructure in the industry.
Thank you for your patience and continued support!
- Tom and Rick Robshaw
