Authentication Experience
Incident Report for YSoft SAFEQ Cloud
Resolved
We are pleased to announce that the incident affecting our services has been resolved. Our technical team has successfully implemented a solution, and all systems are now operating as expected.

We apologize for any inconvenience this may have caused and appreciate your patience and understanding during this time.

A Root Cause Analysis (RCA) will be conducted to determine the underlying causes of the incident and prevent similar occurrences in the future. We will keep you informed of any further developments or findings.
Posted Feb 09, 2024 - 09:46 CET
Update
The bugfix has been applied to the system, and the solution is now operational and stable.

We will continue to monitor throughout the day and update later today.
Posted Feb 08, 2024 - 12:57 CET
Update
We are going to apply a bugfix to the servers in the EU Data Center.

While we expect minimal impact during this upgrade, customers may experience brief interruptions in service, while connected clients and secondary gateway access points locally reconnect.
Posted Feb 08, 2024 - 12:14 CET
Update
The system is operational.

We will schedule a maintenance window later this evening to apply additional bug fixes.
Posted Feb 08, 2024 - 11:57 CET
Update
Our team has identified an issue with some performance changes that were implemented yesterday.
To recover, we are rolling back these changes.

In the meantime, we are investigating the root cause of the issue and working on a fix.

We will keep you updated on our progress. Thank you for your patience and understanding.
Posted Feb 08, 2024 - 10:52 CET
Update
Our team has noticed that in rare cases, the PC Client does not sync print queues after login. We are currently investigating the issue and working towards a resolution
Posted Feb 08, 2024 - 10:13 CET
Update
The system is operational.

We have applied a hotfix this evening to mitigate the system outages and will continue to monitor.
Posted Feb 08, 2024 - 00:07 CET
Update
We are currently in the process of restarting services and will send a further update when service is restored.
Posted Feb 07, 2024 - 21:34 CET
Update
The system is fully operational.

We have identified the root cause of the recent system crashes.

It appears that custom certificate management is causing the system outages. Our team are working on addressing this issue with a hotfix that will be released in the coming days.
In the meantime, we ask that if you need to add, modify, or delete a custom certificate within your tenancy, please do not do so via the admin console. Instead, contact your partner, or Y Soft for assistance.

We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve this ongoing issue.
Posted Feb 07, 2024 - 17:50 CET
Update
We are currently in the process of restarting services and will send a further update when service is restored.
Posted Feb 07, 2024 - 16:51 CET
Update
At this time, we see stability across the platform. We are tracing the cause of the recent service disruptions and working on a bugfix. We will provide further updates later.

We apologize for any inconvenience this may have caused and appreciate your patience as we work to resolve the issue.
Posted Feb 07, 2024 - 14:55 CET
Update
At this moment, the system is operational.

However, we have not yet fully identified the trigger for the issue, so the system may have intermittent issues with connectivity
Posted Feb 07, 2024 - 12:22 CET
Update
The system is operational. However, a small number of clients may experience slow authentication.

Our team is continuing to monitor the situation.
Posted Feb 07, 2024 - 10:48 CET
Update
We have identified the cause of the recent issues, including the ongoing degraded service we are experiencing.

It appears that some external activity, request via API or IPP is causing the crash of the web service, which in turn causes an infinite loop of web service reloads.

This results in the unavailability of all services and, in some cases, memory exhaustion and system outage.
Our team is working on fixing this issue and providing a hotfix to prevent it from happening again. In the meantime, the system is getting back to operational, and a further update will be posted within the next 60 minutes.

We apologize for any inconvenience this may cause and appreciate your patience as we work to resolve the issue.
Posted Feb 07, 2024 - 10:04 CET
Update
We have detected an increase in alerts similar to the issue that occurred yesterday.
We are taking measures to resolve this immediately following working procedure, so expect the system to be fully operational shortly.

We continue to investigate root cause, and will post another update shortly.
Posted Feb 07, 2024 - 09:24 CET
Update
The system is operational.
We are deploying a hotfix later today and we will continue to monitor the situation closely
Posted Feb 06, 2024 - 20:16 CET
Monitoring
The system is operational. However, a small number of clients may experience slow authentication.

Our team is continuing to monitor the situation.
Posted Feb 06, 2024 - 19:07 CET
Update
The EU Cluster performance has stabilized, we no longer see specific errors in the system. However, we still have slower authentication, which we are investigating.
Posted Feb 06, 2024 - 17:40 CET
Update
During the recovery process, we experienced performance constraints for a number of customers. Our team is aware of the issue and is working diligently to address it.
Posted Feb 06, 2024 - 16:07 CET
Update
We are currently in the process of recovering services.

Our team is working diligently to resolve the issue and restore normal operations as soon as possible. We apologize for any inconvenience this may cause and appreciate your patience while we work to resolve the issue. We will provide further updates as more information becomes available.
Posted Feb 06, 2024 - 13:05 CET
Identified
We are currently experiencing longer authentication times within our service.
Posted Feb 05, 2024 - 14:19 CET
Investigating
We are receiving an increased number of alerts on our cloud environment. The print or authentication services are affected. We are introducing measures to improve customer experience with high priority.

Updates will follow on this channel.
Posted Feb 05, 2024 - 14:18 CET
This incident affected: EU data center.