The developers of Sourcehut have published a report on an incident that caused the service to be disrupted for 7 days due to a prolonged DDOS attack, for which the project infrastructure was unprepared. While basic services were restored on the third day, some services were inaccessible from January 10 to 17. At the initial stage of the attack, the developers were unable to react in time and address the issue on their own servers, resulting in all traffic to Sourcehut servers being completely blocked by the higher provider.
In order to ensure the operation of SourceHut, servers in three datacenters were utilized. The first was in working order, the second for backup, and the third for experimenting with the migration of infrastructure for a more scalable and fault-tolerant implementation of the service. This was part of the development of the next generation Sourcehut.
In an attempt to regain access to their servers, it took approximately 9 hours for the main datacenter to unblock after being blocked. However, the developers were unable to take any action as the attack intensified in the morning and began to cover the entire subnet, causing the provider to once again redirect traffic to the NULL interface. Consequently, the developers were forced to urgently deploy SourceHut infrastructure in another datacenter using reserve copies, but it took 2 days to obtain a temporary subnet for access to the main servers.
To protect against DDOS attacks, it was deemed optimal to place an intermediate server in the network of the OVH cloud provider, which offers DDOS protection. All requests were sent to this server before being routed to the working infrastructure. However, during the migration, there were unavoidable errors such as incorrect recovery using the RSYNC utility, network setup mistakes, and issues with traffic redirection. The servers were broadcasting the DDOS attack traffic, causing the DDOS protection system to view the server’s receiving requests as a source of attack.
The developers also reached out to Cloudflare and other DDOS protection services, but the cost of protection was prohibitively high. Eventually, Cloudflare agreed to provide free protection to the SourceHut project as a sponsorship, but the developers declined the offer as they had already made significant progress in resolving the issue on their own by that time.
The implementation of the new SourceHut infrastructure and the transfer of the project to servers in another datacenter was originally planned to take at least one year. However, due to the current circumstances, the migration had to be carried out urgently within 7 days. As of now, all SourceHut services have been successfully transferred to another datacenter and the platform is fully operational once again.