Warding off hackersPosted: February 14, 2011
Configuring Dude’s Syslog was probably the most beneficial thing that I’ve done in my spare time. Ben came into my office this morning and asked about the numerous attempts to log in to one of the branch office’s Internet routers. I went to The Dude to investigate.
First, I thought that we disabled HTTP from all of our routers. We paid special attention to the ones that are connected directly to the Internet – or so I thought. Second, I was surprised that we didn’t get paged about these failed attempts while they were happening. Third, I wanted to be sure that this was a valid threat. So, I took a counter-intuitive troubleshooting approach and addressed my concerns in reverse order.
Was this a valid threat? Aside from the obvious failed attempts at logging in, I wanted to be sure the IP address attempting to get in wasn’t local. So, I attempted a reverse-lookup:
X:\>ping -a 188.8.131.52 Pinging 184.108.40.206 with 32 bytes of data: Reply from 220.127.116.11: bytes=32 time=273ms TTL=109
What I expected was a name. This IP doesn’t have an in-addr-arpa reverse lookup. That’s odd. Usually I can get enough information about the geographical source of the IP from the reverse-lookup response. Being that there wasn’t one, I decided to trace the IP. Initially, I traced using the Windows default of 30 maximum hops. Seeing that there were more than 30, I added “-h 40” to extend the max 10 additional hops; however, in the end, the intruder was 30 hops away.
X:\>tracert -h 40 18.104.22.168 Tracing route to 22.214.171.124 over a maximum of 40 hops 1 1 ms <1 ms <1 ms [private] 2 12 ms 7 ms 10 ms [private] 3 9 ms 7 ms 7 ms [private] 4 9 ms 9 ms 9 ms [private] 5 11 ms 12 ms 11 ms [private] 6 15 ms 16 ms 13 ms [private] 7 24 ms 15 ms 16 ms te-8-4.car1.Phoenix1.Level3.net [126.96.36.199] 8 16 ms 16 ms 16 ms ae-2-5.bar1.Phoenix1.Level3.net [188.8.131.52] 9 14 ms 15 ms 18 ms ae-0-11.bar2.Phoenix1.Level3.net [184.108.40.206] 10 28 ms 25 ms 24 ms ae-4-4.ebr2.LosAngeles1.Level3.net [220.127.116.11] 11 42 ms 35 ms 42 ms ae-6-6.ebr2.sanjose5.level3.net [18.104.22.168] 12 39 ms 35 ms 36 ms ae-5-5.ebr4.sanjose1.level3.net [22.214.171.124] 13 51 ms 62 ms 42 ms ae-94-94.csw4.sanjose1.level3.net [126.96.36.199] 14 39 ms 42 ms 43 ms ae-43-99.car3.sanjose1.level3.net [188.8.131.52] 15 87 ms 99 ms 77 ms cogent-comm.car3.sanjose1.level3.net [184.108.40.206] 16 60 ms 58 ms 57 ms te2-4.mpd01.sjc01.atlas.cogentco.com [220.127.116.11] 17 59 ms 59 ms 62 ms te0-0-0-4.mpd22.sfo01.atlas.cogentco.com [18.104.22.168] 18 105 ms 104 ms 102 ms te0-3-0-3.mpd22.mci01.atlas.cogentco.com [22.214.171.124] 19 110 ms 109 ms 107 ms te0-1-0-2.mpd22.ord01.atlas.cogentco.com [126.96.36.199] 20 214 ms 212 ms 213 ms te0-3-0-3.mpd22.yyz02.atlas.cogentco.com [188.8.131.52] 21 210 ms 210 ms 211 ms te0-3-0-6.ccr21.ymq02.atlas.cogentco.com [184.108.40.206] 22 215 ms 215 ms 220 ms te0-2-0-6.ccr21.lpl01.atlas.cogentco.com [220.127.116.11] 23 230 ms 227 ms 213 ms te0-3-0-7.mpd21.ams03.atlas.cogentco.com [18.104.22.168] 24 239 ms 234 ms 233 ms te0-1-0-5.mpd21.par01.atlas.cogentco.com [22.214.171.124] 25 246 ms 234 ms 235 ms te0-1-0-0.mpd21.fra03.atlas.cogentco.com [126.96.36.199] 26 235 ms 234 ms 234 ms te0-2-0-4.ccr21.ham01.atlas.cogentco.com [188.8.131.52] 27 247 ms 269 ms 265 ms te8-3.ccr01.bts01.atlas.cogentco.com [184.108.40.206] 28 267 ms 272 ms 261 ms te2-1.ccr01.bud01.atlas.cogentco.com [220.127.116.11] 29 * 253 ms 253 ms te1-1.ccr01.bud03.atlas.cogentco.com [18.104.22.168] 30 239 ms 239 ms 238 ms 22.214.171.124 Trace complete.
Well, this is very interesting. I see that I went from my ISP (private) then to my ISP’s peer in Phoenix. After I peer with Cogentco, I start hopping around California, Kansas, Chicago, before I started to get lost at around the 20th hop. I just happened to recognize the MCI-ORD-AMS-FRA as airport codes that I’ve flown to several times in the past. Many ISPs use abbreviations for cities to ID their routes; apparently, Cogentco uses airport codes:
SFO – San Francisco
MCI – Kansas City
ORD – Chicago O’Hare
YYZ – Toronto, ON
YMQ – Montreal, QC
LPL – Liverpool, UK
AMS – Amsterdam, NL
PAR – Paris, FR
FRA – Frankfurt, DE
HAM – Hamburg, DE
BTS – Bratislava, Slovakia
BUD – Budapest
So, the pest is coming out of Budapest. This looks like it just may be a valid threat.
Why no pages?
Puzzled as to why I didn’t get paged numerous times at 6:30AM, I looked at the Dude’s Syslog regex setting to look for any matches on failed login attempts. The only alert to be sent was when someone/something successfully logs into a piece of network gear and enters enable mode. Logging in, enabling and checking the running config on the branch office router confirmed this was the rule – I got a page telling me that my connection entered the ENABLE mode. I then looked at the notes for the alarm and found that I set it up this way on purpose. The number of failed attempts during an attack would cripple our alerting systems. So, the note was a reminder that we decided only to alert when someone successfully logs into and enables a router. This is fine; I’ll keep it this way.
The fact is, that these were only attempts. Each attempt was logged and we did find out about the attempts within an hour. Granted, these logs are glanced at throughout the day; however, it is a rule that two admins look at them first-thing every morning. Also, alerting is more of an art than a science. You don’t want your alerts to be a bunch of noise. Also, we can tweak these settings on-the-fly and for each user/admin wanting to get the page if they want to monitor anything interesting – like a login attempt from a user in Budapest.
Check the router
While I was logged into the router, I checked to see the running config. As a general rule, we disable telnet, http, and https access from the router. SSH and console is the only valid means of configuring them and these are firewalled by standard access lists and all failed attempts are logged. To my surprise, both of the web management options were enabled in the config:
no ip tftp server no ip tftp server overwrite ip http server ip http secure-server no ip snmp agent no ip ftp server no ip scp server no ip sntp server
I negated the two with a ‘no’ and continued reading the config. I confirmed that telnet access was disabled with the following:
line telnet 0 4 login shutdown
Next, I checked the ssh config:
line ssh 0 4 login local-userlist no shutdown access-class management in
Just to be safe, I checked my access list to be sure nothing was added that I didn’t recognize. The standard access list had all of my allowed ranges of public IPs that could SSH into the router. The last entry was a “deny any log” meaning that all other IP addresses that attempt to connect will be sent to the syslog. Everything checked out. Next, I tested remote connecting on an IP not in the access list. When I fired up putty to perform the test, the connection failed – as expected; however, there was no log sent to syslog about the failed attempt. Standard acls also provide logging, don’t they?
I decided to open a support ticket with Adtran to find out why I’m not getting logs for failed attempts.
The https and http management features on the routers were disabled. The ssh access list was tested. Everything checked out except for the log that is supposed to be sent when there is a denied attempt to connect to the ssh port of my router. I have a ticket open for this one.
An interesting start to the week so far…
Update! @ 15:29
The following is some scrubbed tech support e-mails you might be interested in:
Jay: Ian, the log command implements a specific counter that can be viewed with another command, but it does not log ACL hits to syslog. Unfortunately there is currently no way to be notified every time and ACL is matched. Please let me know if you have any additional questions. Thanks
Ian: When I type “show access-lists” while I am connected directly from an allowed range using ssh, no matches show up here either. This is the same for denied traffic. No acl logging appears to be functioning.
Jay: Ian, where is that access-list applied?
Ian: The access-list is applied as an access-class to inbound “line ssh 0 4” management interface. I had a Romanian hacker attempt to access this router this morning. We usually disable the http/https management interfaces, but for this router we did not. Syslog shows thousands of attempted logins on http. We disabled the http/s management interface. Thinking that it was odd that no logs shown any attempts at telnet or ssh, we did a test of the logging function of the access-class standard access-list. There are NO logs for attempts, success, or failures when using ssh. This is reason for this ticket. More information can be found in my blog.
Jay: Ian, what AOS version are you running?
Ian: ADTRAN, Inc. OS version 15.12.00 Boot ROM version 12.02.00 Platform: NetVanta 3205 AC, part number 1202870L1
Jay: Ian, it appears that functionality works in a 3rd gen unit, but it doesn’t work in the 2nd gen 3200. I will log a ticket with engineering about getting that functionality corrected. Unfortunately I can’t provide a timeline as to how long that will take. Please let me know if you have any questions in the meantime. Thanks.