production.songselectapi.com (legacy SongSelect API)
This incident appears to be resolved.
The extra servers are helping and the API is usually responding, but performance is still poor as we are in the heaviest of Sunday morning load. Once again, this only impacts the legacy SS API, and the new API is performing well. Customers may download resources directly from SongSelect or upgrade partner software if it yet supports the new API.
This only affects the legacy SS API (the new API is working well). The fix we had put in place for the legacy SS API last week is working, but has apparently increased the need for system resources so we are increasing server instances and monitoring as they come online.
We are considering this incident resolved. However, we will continue to monitor closely; especially Sunday under heavy load.
We have deployed a fix to the legacy SongSelect API. This appears to have resolved the partial outages that have occurred over the past week. We will be closely monitoring. This fix was extensive and we have worked diligently to minimize the chances of impacting partner software. For partners, if you believe this is causing a breaking change in your software, please email details to firstname.lastname@example.org. Once again, this only affects our legacy SongSelect API which is being retired soon, and customers can always go directly to SongSelect to download CCLI resources until partner software fully supports our new API.
Within the next few hours, we will be deploying a fix to the legacy SongSelect API and then will update this status. This will resolve the partial outages that have occurred over the past week. This fix was extensive and we have worked diligently to minimize the chances of negatively impacting partner software.
This ONLY affects partners on our legacy SongSelect API. A known issue started last Monday with our infrastructure for our legacy SongSelect API which is failing. Services are automatically restarted when they go down, but this solution can fail entirely under heavy load. We continue to anticipate partial outages, but we are working on a complete fix which we hope to deploy Tuesday Mar 28. Users can always go direct to SongSelect to download their resources, and then import them to partner software. Or, if a partner software upgrade is available that uses the new CCLI API, then customers can upgrade to bypass the problem entirely.
This incident has been resolved.
We implemented a fix and are currently monitoring the result.
This is a stubborn issue, and we are continuing to work on a fix for this incident. Again, this only impacts the legacy SongSelect API so if you are on our newer API you are not affected by this outage. We are hoping to have a temporary fix in place by end-of-day today so it will be up and working.
We are currently investigating this incident. This ONLY affects the legacy SongSelect API. For partners who have converted over to our new API, there are no issues.
This incident appears to have been resolved by Microsoft and CCLI solutions are operational.
Microsoft is working on the issue with their Azure cloud service. The problem appear to have stopped and CCLI systems appear to be working again. We will continue to monitor.
Some of our systems are intermittently returning errors. Our cloud service, Microsoft Azure, is reporting that they have identified an issue and are working on a hotfix to resolve it.
We are aware of the problem and implementing a solution now.
We just resolved the issue!
We implemented a fix and currently monitoring the result.
We are currently investigating this incident.
Feb 2023 to Apr 2023