DDoS Botnet Aisuru Blankets US ISPs in Record DDoS

108 points 85 comments 7 hours ago
martinald

This really is a function of two things:

1) (Mainly) the huge increase in upstream capacity of residential broadband connections with FTTH. It's not uncommon for homes to have 2gbit/sec up now and certainly 1gbit/sec is fairly commonplace, which is an enormous amount of bandwidth compared to many interconnects. 10, 40 and 100gbit/sec are the most common and a handful of users can totally saturate these.

2) Many more powerful IoT devices that can handle this level of attack outbound. A $1 SoC can easily handle this these days.

3) Less importantly, CGNAT is a growing problem. If you have 10k (say) users on CGNAT that are compromised, it's likely that there's at least 1 on each CGNAT IP. This means you can't just null route compromised IPs as you are effectively null routing the entire ISP.

I think we probably need more government regulation of these IoT devices. For example, having a "hardware" limit of (say) 10mbit/sec or less for all networking unless otherwise required. 99% all of them don't need more than this.

toast0

> 3) Less importantly, CGNAT is a growing problem. If you have 10k (say) users on CGNAT that are compromised, it's likely that there's at least 1 on each CGNAT IP. This means you can't just null route compromised IPs as you are effectively null routing the entire ISP.

Null routing is usually applied to the targets of the attack, not the sources. If one of your IPs is getting attacked, you null route it, so upstream routers drop traffic instead of sending it to you.

martinald

Sorry, late here. You are right. I mean filter the IP in question.

bsder

> If you have 10k (say) users on CGNAT that are compromised, it's likely that there's at least 1 on each CGNAT IP. This means you can't just null route compromised IPs as you are effectively null routing the entire ISP.

How about we actually finally roll out IPv6 and bury CGNAT in the graveyard where it belongs?

Suddenly, everybody (ISPs, carriers, end users) can blackhole a compromised IP and/or IP range without affecting non-compromised endpoints.

And DDoS goes poof. And, as a bonus, we get the end to end nature of the internet back again.

lgeek

From having worked on DDoS mitigation, there's pretty much no difference between CGNAT and IPv6. Block or rate limit an IPv4 address and you might block some legitimate traffic if it's a NAT address. Block a single IPv6 address... And you might discover that the user controls an entire /64 or whatever prefix. So if you're in a situation where you can't filter out attack trafic by stateless signature (which is pretty bad already), you'll probably err on the side of blocking larger prefixes anyway, which potentially affect other users, the same as with CGNAT.

Insofar as it makes a difference for DDoS mitigation, the scarcity of IPv4 is more of a feature than a bug.

zamadatix

(Having also worked on DDoS mitigation services) That "entire /64" is already hell of a lot more granular than a single CG-NAT range serving everyone on an ISP though. Most often in these types of attacks it's a single subnet of a single home connection. You'll need to block more total prefixes, sure, but only because you actually know you're only blocking actively attacking source subnets, not entire ISPs. You'll probably still want something signature based for the detection of what to blackhole though, but it does scale farther in a combo on the same amount of DDoS mitigation hardware.

spongebobstoes

you can heuristically block ipv6 prefixes on a big enough attack by blocking a prefix once a probabilistic % of nodes under it are themselves blocked, I think it should work fairly well, as long as attacking traffic has a signature.

consider simple counters "ips with non-malicious traffic" and "ips with malicious traffic" to probabilistically identify the cost/benefit of blocking a prefix.

you do need to be able to support huge block lists, but there isn't the same issue as cgnat where many non-malicious users are definitely getting blocked.

bsder

This DDoS is claimed to be the result of <300,000 compromised routers.

That would be really easy to block if we were on IPv6. And it would be pretty easy to propagate upstream. And you could probabilistically unblock in an automated way and see if a node was still compromised. etc.

nine_k

Is there any money an ISP would make, or save, by sinking money and effort on switching to IPv6? If there's none, why would they act? If there is some, where?

For instance, mobile phone operators, which had to turn ISPs a decade or two ago, had a natural incentive to switch to IPv6, especially as they grew. Would old ISPs make enough from selling some of their IPv4 pools?

ROBLOX_MOMENTS

They already lease them out. TELUS in Canada traditional old ISP rents large portion of their space to a mostly used for Chinese GFW VPN server provider in LA „Psychz“

rendaw

Presumably they'd lose money when a DDoS originating from their network causes all their ips to get blocked.

beeflet

less expensive IP space, more efficient hardware, and lower complexity if you can eliminate NAT.

ralnivar

I am a bit split this topic. There is some privacy concerns with using ipv6. https://www.rfc-editor.org/rfc/rfc7721.html#page-6

Some time ago I decided for our site to not roll out ipv6 due to these concerns. (a couple of million visitors per month) We have meta ads reps constantly encourage us to enable it which also do not sit right with me.

Although I belive fingerprinting is sofisticated enough to work without using ip's so the impact of using ipv6 might not be a meaningful difference.

miyuru

its hilarious that you have privacy concerns while at the same time using meta ads.

rectang

Is it advantageous to be someone who supports IPv6 on a day like today?

idiotsecant

Haha that last part is pretty wild. rather than worrying about systemic problems in the entire internet let's just make mandates crippling devices that China, where all these devices are made, will defffinitely 100% listen to. Sure, seems reasonable. Systems that rely on the goodwill of the entire world to function are generally pretty robust, after all.

saagarjha

If they don’t then the devices are not sold in the United States. It’s quite simple.

dylan604

Great to know that smuggling hardware into the US has been completely stopped.

morsch

If the analysis above is accurate, a few smuggled devices would not be an issue, as long as the zillions of devices sold at Walmart are compliant.

gjsman-1000

> I think we probably need more government regulation of these IoT devices. For example, having a "hardware" limit of (say) 10mbit/sec or less for all networking unless otherwise required. 99% all of them don't need more than this.

What about DDoSs that come from sideloaded, unofficial, buggy, or poorly written apps? That's what IoT manufacturers will point to, and where most attacks historically come from. They'll point to whether your Mac really needs more than 100mbps.

The government is far more likely to figure it out along EU lines: Signed firmware, occasional reboots, no default passwords, mandatory security updates for a long-term period, all other applicable "common sense" security measures. Signed firmware and the sideloading ID requirements on Android also helps to prevent stalkerware, which is a growing threat far scarier than some occasional sideloaded virus or DDoS attack. Never assume sideloading is consensual.

ShowalkKama

>What about DDoSs that come from sideloaded, unofficial, buggy, or poorly written apps? That's what IoT manufacturers will point to, and where most attacks historically come from.

any source for this claim? Outside of very specific scenarios which differ significantly for the current botnet market (like manjaro sending too many requests to the aur or an android application embedding an url to a wikipedia image) I cannot remember one occourence of such a bug being versatile enough to create a new whole cybercrime market segment.

>They'll point to whether your Mac really needs more than 100mbps.

it does, because sometimes my computer bursts up to 1gbps for a sustained amount of time, unlike the average iot device that has a predictable communication pattern.

>Signed firmware and the sideloading ID requirements on Android also helps to prevent stalkerware, which is a growing threat far scarier than some occasional sideloaded virus or DDoS attack. Never assume sideloading is consensual.

if someone can unlock your phone, go into the settings, enable installation of apps for an application (ex. a browser), download an apk and install it then they can do quite literally anything, from enabling adb to exfiltrating all your files.

gjsman-1000

Historically, it was called Windows XP and Vista about 15 years ago (Blaster, Sasser, MyDoom, Stuxnet, Conficker?). Microsoft clamped down, hard, across the board, but everyone outside of Big Tech is still catching up.

Despite Microsoft's efforts, 911 S5 was roughly 19 million Windows PCs in 2024, in news that went mostly under the radar. It spread almost entirely through dangerous "free VPN" apps that people installed all over the place. (Why is sideloading under attack so much lately? 19 million people thought it would make them more secure, and instead it turned their home internet into criminal gateways with police visits. I strongly suspect this incident, and how it spread among well-meaning security-minded people, was the invisible turning point in Big Tech against software freedom lately.)

https://www.fbi.gov/investigate/cyber/how-to-identify-and-re...

> if someone can unlock your phone, go into the settings, enable installation of apps for an application (ex. a browser), download an apk and install it then they can do quite literally anything, from enabling adb to exfiltrating all your files.

Which is more important, and a growing threat? Dump all her photos once; or install a disguised app that pretends to be a boring stock app nobody uses, that provides ongoing access for years, with everything in real-time up to the minute? Increasingly it's the latter. She'll never suspect the "Samsung Battery Optimizer" or even realize it came from an APK. No amount of sandboxing and permissions can detect an app with a deliberately false identity.

devwastaken

1gb upload is extraordinarily rare.

saagarjha

It’s not; most places that give you gigabit fiber will give you a symmetric connection.

typpilol

Yup. Spectrum is Michigan will give you up to 2gbps down but not anything more than 200mbps up

dylan604

Is Spectrum fiber or DOCSIS? I didn't realize anyone was pushing these kinds of numbers for fiber. What's the point other than screwing the users?

vitaflo

Most places do not have fiber.

Dylan16807

We know. The problem is that the above comment said "extraordinarily rare" which is a very different and incorrect threshold.

dylan604

But for those that do...symmetric is the norm. The number of fiber connections is only going up.

ls612

This is probably technically true but very misleading. Fiber penetration in the US has been consistently rising for over a decade now and it is not at all uncommon to have either Google Fiber, Fios, or a local fiber provider available to you in a big city. I bet within the next decade most places will have gigabit fiber available.

_carbyau_

The US is a big place. But the world is bigger. The internet works across the whole world.

There's a long way to go before fibre is commonplace across the world.

nick32661123

Seems more likely that residential modems will be required to use ISP-provided equipment that has government mandated chips, firmware, etc to filter outbound traffic for DDoS prevention.

DaSHacka

Why should they be required to have hardware in their own network to filter that out when the ISP is obviously receiving all of their traffic anyway?

lgeek

This is very challenging, in about one year the biggest recorded DDoS attack has increased from 5 Tbps to almost 30.

Almost all of the DDoS mitigation providers have been struggling for a few weeks because they just don't have enough edge capacity.

And normal hosting companies that are not focused on DDoS mitigation also seem to have had issues, but with less impact to other customers as they'll just blackhole addresses under larger attacks. For example, I've seen all connections to / from some of my services at Hetzner time out way more frequently than usual, and some at OVH too. Then one of my smaller hosting providers got hit with an attack of at least 1 Tbps which saturated a bunch of their transit links.

Cloudflare and maybe a couple of the other enterprise providers (Gcore?) operate at a large enough scale to handle these attacks, but all the smaller ones (who tend to have more affordable rates and more application-specific filters for sensitive applications that can't deal with much leakage) seem to be in quite a bad spot right now. Cloudflare Magic Transit pricing supposedly starts at around $4k / month, and it would really suck if that became the floor for being able to run a non-HTTP service online.

Something like Team Cymru's UTRS service (with Flowspec support) could potentially help to mitigate attacks at the source, but residential ISPs and maybe the T1s would need to join it, and I don't see that happening anytime soon.

rectang

> “The problem is, even if those infected IoT devices are rebooted and cleaned up, they will still get re-compromised by something else generally within minutes of being plugged back in.”

In the year 2025, we should understand that such devices are defective. They should become bricks and companies that continue to sell such defective merchandise should fail.

dylan604

Just wait 6 months, and the IoT vendor will go out of business and shut down the cloud servers which will effectively brick the device.

rectang

That may brick it for its intended use, but nothing's stopping a botnet from repurposing it.

fukka42

Absolutely not. They should be patched and fixed.

rectang

So they get to ship a dangerous device that harms innocent third parties because they cut corners, but we’re supposed to reward them by doing the work to secure the devices they couldn’t be bothered with?

danparsonson

Patched and fixed by the manufacturer, was surely what GP was suggesting.

Bricking the devices doesn't punish the supplier as much as it punishes the consumers.

rectang

Many of the companies that sold these devices have already bitten the dust, so they aren't even around to fix anything.

bikelang

Are there any practical ways to monitor my home network traffic and detect if any devices on my network are compromised?

tylerflick

Not sure about monitoring, but I always put any device I don’t trust on a jailed LAN/AP.

bentcorner

A lot of home routers will give you a traffic graph - if yours doesn't you can either find one that does or flash/build one.

I currently run opnsense which has an ok graph out of the box, I haven't fiddled with it to see if there's something fancy I could do here.

I also used to use IPFire which was slightly clunkier but had a nicer usage graph.

userbinator

Your ISP should give you a bandwidth usage meter.

skinner927

Haha. My ISP barely gives me an Internet connection

dylan604

You can rest assured your ISP can produce the graph. If your graph reaches a certain shape, they'll start shaping your traffic for you to something they prefer all while charging you the normal rate.

userbinator

I'd rather there be periodic DDoS attacks, than a locked-down highly-regulated internet. Don't forget that infamous Franklin quote, and what Stallman has been warning us about for the past few decades.

I can already see the authoritarians salivating every time something like this happens.

dylan604

> I can already see the authoritarians salivating every time something like this happens.

Tinfoil hat theory says they do this intentionally so that the users demand stricter access willingly. Always better to have someone think it is their idea

koakuma-chan

Big botnet has nothing better to do than DDoS Minecraft servers?

thenthenthen

…to sell ddos protection to the minecraft server admins, basically extortion.

dylan604

what about as an demonstration of their capabilities for someone else?

shermantanktop

This is what I wonder. Must be fascinating to engineer such a massive distributed system, but at some point there’s no added value from another bazillion hosts in the network.

crummy

I guess if they go after bigger targets they draw unwanted attention? Seemed odd to me too.

spatley

Seems pretty clear that the US needs strict regulation on any device connecting to the internet.

* no default password * * no login if not on the local wifi or wired ethernet *

DaSHacka

Many manufacturers are already moving there of their own accord. I really don't think we'd need some legislation to fix this problem.

Eji1700

Ehhh I can see it. The right attack at the right time could directly or indirectly kill people, and that’s ignoring the fact it can cause economic havoc.

Having the entire internet function on a “pay or be nuked” threshold that could easily get much worse if companies like cloudflare become less ethical (not that they’re saints).

dehrmann

I'd rather the industry standardizes on some sort of guest network and proxy/hub. It could even ship with hardware from ISPs. Separating the network buys you a lot of security, and running everything through a proxy makes it easier to inspect data and creates a standard hook for using abandonware.

kylecarbs

9gigsofram was prolific in the "Minecraft Server era" (2010-2016).

Source: Server Owner's Chat

Groxx

I'm honestly kinda curious why nobody's blocking these IPs from sending data near the source.

Like, I can come up with plenty of possible reasons, and reasons why it could potentially be very bad if ISPs started cracking down on this, but I don't actually know any reasons.

Are any talking about why / why not? It seems like this whole insecure-IoT-device thing would probably dry up pretty quickly if people's internet was cut off when one was detected. They can then turn around and lambast / sue / etc the company that sold it, putting pressure on the source of the problem. Right now there's no reason for sellers to do anything at all to ensure security, afaict.

So... not actually arguing in favor of it, but definitely curious about any stated ISP / core networking system's stated reasons.

Mindless2112

> “The outbound and cross-bound DDoS attacks can be just as disruptive as the inbound stuff,” Dobbin said. “We’re now in a situation where ISPs are routinely seeing terabit-per-second plus outbound attacks from their networks that can cause operational problems.”

ISPs are starting to feel the pain, so perhaps in the near future they will do something about it.

dloy

Perhaps, or perhaps not. Maybe if we held them accountable they would?

MartijnBraam

This does happen, but it seems to depend on the ISP. In the Netherlands I've seen ISPs block the internet connectivity when they've detected infected devices, sometimes they send a letter before blocking and some ISPs seem to dump your internet connection in a captive portal. In all these cases it's been enough to call the ISP after finding the problem and you're connected again minutes later.

kibbel

A large part of the article is dedicated to this, noting how disruptive it is to other services and customers, and listing a few countermeasures (detection and blocking at the ISP level, detection and blocking at the router level, and educating customers on not buying vulnerable IoT trash).

Groxx

Not really? At best it's "DDOS prevention sellers are having trouble" and "ISPs say they're doing fine". The vast majority of the article is talking about the various kinds of malware causing this, and how some have been "fixed" by stopping the individuals running it (which clearly doesn't work very well, new ones just fill the void).

Or this:

>“The crying need for effective and universal outbound DDoS attack suppression is something that is really being highlighted by these recent attacks,” Dobbins continued. “A lot of network operators are learning that lesson now, and there’s going to be a period ahead where there’s some scrambling and potential disruption going on.”

Uh. No. That's gross negligence if they are only starting to think about it now - the trend has been clear for over a decade, and the IoT threat has been obvious since day 1 and even blasted over public news for the past few years. Their status is pretty much only one of: incompetent, malicious, or they have had plans but haven't acted on them fast enough or strongly enough for [some reason], and that reason isn't something I've seen. Surprises happen, prevention costs money and time, and there are plenty of reasons why everyone isn't already prepared for everything, so I think "incompetent or malicious" is pretty rare.... but what are those reasons?

bombcar

There's no economic incentive for YOU (as the proximate ISP) to do anything about it, it would cost money, and cost you customers.

Any idea why they don't fix it?

Groxx

Yes, you generally see this kind of thing start from the pain-feelers and move up the chain to the pain-causers.

So why hasn't that happened? These are clearly damaging to many, and ISPs are apparently doing next to nothing to prevent it, and it has been extremely clear for a while now that it's going to just become a bigger and bigger problem.

userbinator

Are there ISPs that don't charge customers for the amount of bandwidth they consume? Even "unlimited" has been ruled by courts to not really mean "unlimited", after all.

fukka42

Yes, most if not all of them. Is it different in the US?

bombcar

Most non-mobile ISPs will let you get pretty high on bandwidth usage before they flag you; and since DDoS are almost always relatively low-levels of bandwidth (on the source) it's unlikely you'd get flagged.

martinald

Of course there is. If you've got all your internet egress tied up with DDoS attacks from your network it is a big problem.

toast0

Most eyeball networks have a lot of inbound traffic and not very much outbound, but interconnections with other networks are almost always symmetric, so there's a lot of room for excess egress before it causes pain to the ISP.

When I ran a large web site that attracted lots of DDoS, it didn't really seem worthwhile to track down the source and try to contact ISPs. I had done a lot of trying to track and stop people sending phishing mail under our name, and it's simply too much work to write a reasonable abuse report that is unlikely to be followed up on. With email, mostly people seem to accept the Received headers are probably true; with DDoS, you'd be sending them pcaps, and they'd be telling you it's probably spoofed, and unless I've got lots of peering, I'm not going to be able to get captures that are convincing... so just do my best to manage the inbound and call it a day.

bombcar

I think we’re just starting to see attacks that big - which might start some practical mitigations (or they’ll just upgrade transit).

TZubiri

> They can then turn around and lambast / sue / etc the company that sold it, putting pressure on the source of the problem

Or just unplug the culprit. But the key seems to be that the device continues working. Ideally you would just shutdown or disconnect the device. If fridge is infected, the fridge can still fridge, but it no longer has internet privileges.

quantummagic

Any device that participates in a DDOS needs to be recalled by the manufacturer, mandated by law. Make it potentially economically crippling to sell a vulnerable device, and security will be taken very seriously. Frivolous uses of tech, won't be worth the risk.

DaSHacka

This just in: every computer manufacturer forced to recall every single computer model they've ever sold because some users use weak passwords.

I can't wait for all of them to switch to IOS-ified devices incapable of installing alternative operating systems or programs, as that would be the inevitable end solution for all these manufacturers if this was implemented.

quantummagic

Maybe that's a good thing; relying on users to choose good passwords is a cop-out. Systems should be safe-by-default. And owners losing their system if it participates in a DDOS, would add to the incentives to stop the nonsense. It persists because perpetrators, and those who unwittingly abet them, feel no consequences.

DaSHacka

At that point, you should force the pain on the individual themselves. Why should all of us be handicapped because there's a couple morons that can't set decent passwords and connect their devices directly to the internet?

Even if the device removed the capability for passwords and used key based authentication, connecting it directly to the internet means if there's ever a vulnerability, all that was for naught anyway.

pabs3

This is the way, there should be no access by default, then on first access the user has to setup their desired authentication details, and if they want passwords, then they get a randomly generated one, not one they choose. There should also be a factory reset button too.

bee_rider

Does a weak user password have to provide remote access by default?

sreekanth850

No evidence, just a wild guess, Aisuru might be tied to DDoS mitigation folks.

cynicalsecurity

Isn't OVH known for offering relatively good anti-DDoS services? Why did they boot them?

old_bayes

"A spokesperson for Comcast responded, 'Currently our network is not experiencing impacts and we are able to handle the traffic.'"

In other words, attacks coming from our own IP's are not our problem ¯\_(ツ)_/¯

zdw

This doesn't seem very loving.

qiqitori

Cool, can they try aiming it at twitter or truth please?

Made by @calebRussel