Search

Warding off hackers

Hacker?

Configuring Dude’s Syslog was probably the most beneficial thing that I’ve done in my spare time.  Ben came into my office this morning and asked about the numerous attempts to log in to one of the branch office’s Internet routers.  I went to The Dude to investigate.

Well, well... I see numerous problems here

First, I thought that we disabled HTTP from all of our routers.  We paid special attention to the ones that are connected directly to the Internet – or so I thought.  Second, I was surprised that we didn’t get paged about these failed attempts while they were happening.  Third, I wanted to be sure that this was a valid threat.  So, I took a counter-intuitive troubleshooting approach and addressed my concerns in reverse order.

Threat?

Was this a valid threat?  Aside from the obvious failed attempts at logging in, I wanted to be sure the IP address attempting to get in wasn’t local.  So, I attempted a reverse-lookup:

X:\>ping -a 86.107.157.219
Pinging 86.107.157.219 with 32 bytes of data:
Reply from 86.107.157.219: bytes=32 time=273ms TTL=109

What I expected was a name.  This IP doesn’t have an in-addr-arpa reverse lookup.  That’s odd.  Usually I can get enough information about the geographical source of the IP from the reverse-lookup response.  Being that there wasn’t one, I decided to trace the IP.  Initially, I traced using the Windows default of 30 maximum hops.  Seeing that there were more than 30, I added “-h 40” to extend the max 10 additional hops; however, in the end, the intruder was 30 hops away.

X:\>tracert -h 40 86.107.157.219

Tracing route to 86.107.157.219 over a maximum of 40 hops

 1     1 ms    <1 ms    <1 ms [private]
 2    12 ms     7 ms    10 ms [private]
 3     9 ms     7 ms     7 ms [private]
 4     9 ms     9 ms     9 ms [private]
 5    11 ms    12 ms    11 ms [private]
 6    15 ms    16 ms    13 ms [private]
 7    24 ms    15 ms    16 ms  te-8-4.car1.Phoenix1.Level3.net [4.53.104.145]
 8    16 ms    16 ms    16 ms  ae-2-5.bar1.Phoenix1.Level3.net [4.69.148.118]
 9    14 ms    15 ms    18 ms  ae-0-11.bar2.Phoenix1.Level3.net [4.69.148.114]
 10    28 ms    25 ms    24 ms  ae-4-4.ebr2.LosAngeles1.Level3.net [4.69.133.38]
 11    42 ms    35 ms    42 ms  ae-6-6.ebr2.sanjose5.level3.net [4.69.148.202]
 12    39 ms    35 ms    36 ms  ae-5-5.ebr4.sanjose1.level3.net [4.69.148.142]
 13    51 ms    62 ms    42 ms  ae-94-94.csw4.sanjose1.level3.net [4.69.134.254]
 14    39 ms    42 ms    43 ms  ae-43-99.car3.sanjose1.level3.net [4.68.18.197]
 15    87 ms    99 ms    77 ms  cogent-comm.car3.sanjose1.level3.net [4.68.110.138]
 16    60 ms    58 ms    57 ms  te2-4.mpd01.sjc01.atlas.cogentco.com [154.54.41.201]
 17    59 ms    59 ms    62 ms  te0-0-0-4.mpd22.sfo01.atlas.cogentco.com [66.28.4.149]
 18   105 ms   104 ms   102 ms  te0-3-0-3.mpd22.mci01.atlas.cogentco.com [154.54.7.225]
 19   110 ms   109 ms   107 ms  te0-1-0-2.mpd22.ord01.atlas.cogentco.com [154.54.3.201]
 20   214 ms   212 ms   213 ms  te0-3-0-3.mpd22.yyz02.atlas.cogentco.com [154.54.42.14]
 21   210 ms   210 ms   211 ms  te0-3-0-6.ccr21.ymq02.atlas.cogentco.com [154.54.44.121]
 22   215 ms   215 ms   220 ms  te0-2-0-6.ccr21.lpl01.atlas.cogentco.com [154.54.0.69]
 23   230 ms   227 ms   213 ms  te0-3-0-7.mpd21.ams03.atlas.cogentco.com [154.54.37.77]
 24   239 ms   234 ms   233 ms  te0-1-0-5.mpd21.par01.atlas.cogentco.com [130.117.51.62]
 25   246 ms   234 ms   235 ms  te0-1-0-0.mpd21.fra03.atlas.cogentco.com [130.117.0.101]
 26   235 ms   234 ms   234 ms  te0-2-0-4.ccr21.ham01.atlas.cogentco.com [130.117.49.210]
 27   247 ms   269 ms   265 ms  te8-3.ccr01.bts01.atlas.cogentco.com [154.54.36.198]
 28   267 ms   272 ms   261 ms  te2-1.ccr01.bud01.atlas.cogentco.com [130.117.3.58]
 29     *      253 ms   253 ms  te1-1.ccr01.bud03.atlas.cogentco.com [130.117.3.74]
 30   239 ms   239 ms   238 ms  149.6.182.34

Trace complete.

Well, this is very interesting.  I see that I went from my ISP (private) then to my ISP’s peer in Phoenix.  After I peer with Cogentco, I start hopping around California, Kansas, Chicago, before I started to get lost at around the 20th hop.  I just happened to recognize the MCI-ORD-AMS-FRA as airport codes that I’ve flown to several times in the past.  Many ISPs use abbreviations for cities to ID their routes; apparently, Cogentco uses airport codes:

SFO – San Francisco
MCI – Kansas City
ORD – Chicago O’Hare
YYZ – Toronto, ON
YMQ – Montreal, QC
LPL – Liverpool, UK
AMS – Amsterdam, NL
PAR – Paris, FR
FRA – Frankfurt, DE
HAM – Hamburg, DE
BTS – Bratislava, Slovakia
BUD – Budapest

So, the pest is coming out of Budapest.  This looks like it just may be a valid threat.

Why no pages?

Puzzled as to why I didn’t get paged numerous times at 6:30AM, I looked at the Dude’s Syslog regex setting to look for any matches on failed login attempts. The only alert to be sent was when someone/something successfully logs into a piece of network gear and enters enable mode.  Logging in, enabling and checking the running config on the branch office router confirmed this was the rule – I got a page telling me that my connection entered the ENABLE mode.  I then looked at the notes for the alarm and found that I set it up this way on purpose.  The number of failed attempts during an attack would cripple our alerting systems.  So, the note was a reminder that we decided only to alert when someone successfully logs into and enables a router.  This is fine; I’ll keep it this way.

The fact is, that these were only attempts.  Each attempt was logged and we did find out about the attempts within an hour.  Granted, these logs are glanced at throughout the day; however, it is a rule that two admins look at them first-thing every morning.  Also, alerting is more of an art than a science.  You don’t want your alerts to be a bunch of noise.  Also, we can tweak these settings on-the-fly and for each user/admin wanting to get the page if they want to monitor anything interesting – like a login attempt from a user in Budapest.

Check the router

While I was logged into the router, I checked to see the running config.  As a general rule, we disable telnet, http, and https access from the router.  SSH and console is the only valid means of configuring them and these are firewalled by standard access lists and all failed attempts are logged.  To my surprise, both of the web management options were enabled in the config:

no ip tftp server
no ip tftp server overwrite
ip http server
ip http secure-server
no ip snmp agent
no ip ftp server
no ip scp server
no ip sntp server

I negated the two with a ‘no’ and continued reading the config.  I confirmed that telnet access was disabled with the following:

line telnet 0 4
 login
 shutdown

Next, I checked the ssh config:

line ssh 0 4
 login local-userlist
 no shutdown
 access-class management in

Just to be safe, I checked my access list to be sure nothing was added that I didn’t recognize. The standard access list had all of my allowed ranges of public IPs that could SSH into the router.  The last entry was a “deny any log” meaning that all other IP addresses that attempt to connect will be sent to the syslog.  Everything checked out.  Next, I tested remote connecting on an IP not in the access list.  When I fired up putty to perform the test, the connection failed – as expected; however, there was no log sent to syslog about the failed attempt. Standard acls also provide logging, don’t they?

I decided to open a support ticket with Adtran to find out why I’m not getting logs for failed attempts.

Summary

The https and http management features on the routers were disabled.  The ssh access list was tested.  Everything checked out except for the log that is supposed to be sent when there is a denied attempt to connect to the ssh port of my router.  I have a ticket open for this one.

An interesting start to the week so far…

Update! @ 15:29

The following is some scrubbed tech support e-mails you might be interested in:

Jay:  Ian, the log command implements a specific counter that can be viewed with another command, but it does not log ACL hits to syslog. Unfortunately there is currently no way to be notified every time and ACL is matched. Please let me know if you have any additional questions. Thanks

Ian: When I type “show access-lists” while I am connected directly from an allowed range using ssh, no matches show up here either.  This is the same for denied traffic.  No acl logging appears to be functioning.

Jay: Ian, where is that access-list applied?

Ian: The access-list is applied as an access-class to inbound “line ssh 0 4” management interface. I had a Romanian hacker attempt to access this router this morning.  We usually disable the http/https management interfaces, but for this router we did not.  Syslog shows thousands of attempted logins on http.  We disabled the http/s management interface. Thinking that it was odd that no logs shown any attempts at telnet or ssh, we did a test of the logging function of the access-class standard access-list.  There are NO logs for attempts, success, or failures when using ssh.  This is reason for this ticket.  More information can be found in my blog.

Jay: Ian, what AOS version are you running?

Ian: ADTRAN, Inc. OS version 15.12.00 Boot ROM version 12.02.00 Platform: NetVanta 3205 AC, part number 1202870L1

Jay: Ian, it appears that functionality works in a 3rd gen unit, but it doesn’t work in the 2nd gen 3200. I will log a ticket with engineering about getting that functionality corrected. Unfortunately I can’t provide a timeline as to how long that will take. Please let me know if you have any questions in the meantime. Thanks.

Advertisements

3 Comments on “Warding off hackers”

  1. itcoop says:

    I got an e-mail from Jay this afternoon:

    “Ian, you can download an engineering build with the fix from here:

    ftp://ftp.adtran.com/outgoing/NV3200A-15-12-01N8001.biz

    Keep in mind that this is an engineering build and it has not gone through the same testing as a normal build.”

    Sweet. I’m uploading it to all my routers now. This should take care of it.

  2. itcoop says:

    After checking the results of the load above, I found no change with the new firmware. I e-mailed adtran and they made another fix for me:

    Ian,

    Here is a new version of code that does resolve the issue in my testing:
    ftp://ftp.adtran.com/outgoing/NV3200A-15-12-01N8001-2.biz

  3. itcoop says:

    I uploaded the new firmware and rebooted the routers. There are counters for matches showing up on the access-list now; however, I am confused as to the significance of the word log has on the “deny any any log” sentence.

    My intention is to get specific information from the packets that match the deny statement. Otherwise, would the implicit “deny any any” catch it and give counters anyway? I assumed that placing an explicit deny statement at the end of the access-list was intended to log details about the denied traffic.

    Bottom line: I need to know if someone is attempting to access our routers using ssh who are not part of the acl. Obtaining a rate from the access-list counter would be fine if I can access it remotely without having to log into the router every couple of seconds to retrieve it. My NMS can use syslog regexp or SNMP. Is there another way to accomplish this?

    Reply from Adtran:

    Ian,

    The significance of the “log” keyword is to cause any hits on that line to display in a “debug access-list “. The debug will not display the IP address that created the denied traffic, just state that the entry was hit X number of times. It will also not generate a syslog message, unfortunately.

    There are also no IEEE access-list MIBs, and I am not aware of any that Adtran has created for pulling those kinds of statistics. Unfortunately, I am not aware of any way to inform the user of the offending traffic when an access-group or access-class blocks it.

    If you instead enable the firewall, and reduce the policy-log threshold to 1, then you will see a message each time the firewall blocks an incoming packet with the message “No Access Policy matched, dropping packet”; I believe that does provide the packet details along with the message and is syslogged at an INFO level.

    Using that type of a configuration, your setup would look as follows:

    ip firewall
    ip firewall policy-log threshold 1
    !
    ip access-list extended MatchAll
    permit ip any any
    !
    ip access-list extended AdminAccess
    permit ip allow.ed.ip.here 0.0.0.7 any
    !
    ip policy-class FilterAdminOnly
    allow list AdminAccess self
    allow list MatchAll policy FilterAdminOnly
    !

    This setup will allow all traffic, statelessly if you prefer, through the unit, but still firewall admin traffic (destination policy “self”). If anything is denied, it should create a syslog message.


Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s