We are considering this incident resolved. However, we will continue to monitor closely; especially Sunday under heavy load.
Monitoring
Update
We have deployed a fix to the legacy SongSelect API. This appears to have resolved the partial outages that have occurred over the past week. We will be closely monitoring. This fix was extensive and we have worked diligently to minimize the chances of impacting partner software. For partners, if you believe this is causing a breaking change in your software, please email details to partner@ccli.com. Once again, this only affects our legacy SongSelect API which is being retired soon, and customers can always go directly to SongSelect to download CCLI resources until partner software fully supports our new API.
Monitoring
Update
Within the next few hours, we will be deploying a fix to the legacy SongSelect API and then will update this status. This will resolve the partial outages that have occurred over the past week. This fix was extensive and we have worked diligently to minimize the chances of negatively impacting partner software.
Monitoring
Monitoring
This ONLY affects partners on our legacy SongSelect API. A known issue started last Monday with our infrastructure for our legacy SongSelect API which is failing. Services are automatically restarted when they go down, but this solution can fail entirely under heavy load. We continue to anticipate partial outages, but we are working on a complete fix which we hope to deploy Tuesday Mar 28. Users can always go direct to SongSelect to download their resources, and then import them to partner software. Or, if a partner software upgrade is available that uses the new CCLI API, then customers can upgrade to bypass the problem entirely.
