FACTS: Keeping out the hackers
Posted on Thursday, June 02 @ 12:57:26 EDT by admin
|
|
If you have a dedicated machine with ensim installed, please read more for important information concerning hackers, how to check and protect your machine:
+ Check MRTG +
The easiest way to determine if
your box is being used to
distribute illegal warez or porn,
would be to check your bandwidth
usage.if you have
MRTG installed on your machine,
Login to your MRTG
area, click on Monitoring,
followed by Bandwidth Monitoring,
then "Click To See Bandwidth
Graphs."
Take a close look at your daily
graph - do not glimpse - remember
that MRTG makes all it's graphs
proportional to the average
bandwidth usage. Constant high
usage will not be clearly evident
without reading your axis. Check
your left axis (Bits per Second),
if you are constantly peaking over
about 2M/sec you could be in
trouble. Very high usage (in our
case) would show as much as
9.6M/sec.
+ Check Disk Space +
While logged into your shell
account, type:
df
Results will show similar to this:
code:-----------------------------
----------------------------------
-----------------
Filesystem 1k-blocks
Used Available Use% Mounted on
/dev/hda3 56388752
10733264 42791100 21% /
/dev/hda1 295564
11314 268990 5% /boot
none 514104
0 514104 0% /dev/shm
----------------------------------
----------------------------------
------------
The items that I have bolded show
used diskspace. These numbers are
the ones you want to look at. If
you are using ALOT more (we're
talking Gigs here) then you have
allocated to your clients. There
is another possibility you're in
trouble.
+ Run Chkrootkit +
Be on alert for infected files, in
my case, Chkrootkit returned:
"Warning: Bogus unix lines
detected."
Don't panic if it returns infected
files, google any of your results
and you're sure to find answers.
At this point, if you've had
suspicious results in the last
three checks, you most definitely
need to continue.
+ Upgrade Upgrade Upgrade +
The machine that I cleaned out was
running Ensim 3.1 - no upgrades
had been made to it what-so-ever.
I encourage ALL Ensim users to
upgrade to Ensim 3.1.1 - it is the
easiest way to upgrade the
majority of outdated software on
your box.
Follow the instuctions on Ensim's
site to upgrade flawlessly. I have
upgraded two Ensim machines now,
both went very smoothly.
+ Search and Destroy +
Your next step here is to find
suspicious shell users and groups.
We will scan through three files.
But before we continue, PLEASE do
NOT deleted (we're using less to
view them, but we may have to vi
them later) anything out of these
files without confirming they are
indeed not supposed to be there.
There are some users that look
funky, but they are supposed to be
there. That said, let's continue
(as root):
less /etc/passwd
The first column represents users
on the system. In my case, the
hackers created a few users:
ADM1N, mysqi, vgodz, and noone.
I'd recommend not deleted anything
out of these files without asking
people here on the forum if they
also exist on their box.
Next:
less /etc/shadow
Looks similar to /etc/passwd.
Again, look for suspicious users.
Next:
less /etc/groups
Look for groups that do not
belong, in my case there were a
couple: mysqi and vgodz.
Now, if you have CONFIRMED that
there are malicous users, go back
to /etc/passwd and take a good
look at the default homes of each
suspected user. Copy these down
for later reference - you will
need them to continue your quest.
After you've got all the
information you need,
vi/pico/useyourfavoriteeditor to
remove their accounts COMPLETELY.
Remove the entire line.
The hackers often make backups of
these files, look for similar
filenames in /etc/ - there's a
good chance they were made by the
hacker.
+ Locating Fingerprints +
Each of these users may have left
behind records of their doings,
these histories will GREATLY aid
you in determining what has been
done to your system. In my case,
the hackers did not pass
elementary school - and left
behind records of all of their
modifications. Let's take a look:
find '/' -iname .bash_history
Once you've listed all of the
histories, make copies of them and
do NOT lose them. View the
contents of each of the suspected
hacker's history. More than likely
you will find that they have
created a sudo account for
themselves. This gave them root
access to modify whatever they
pleased.
By default Ensim does not come
with a sudoer file, so let's check
to see if there was one created.
cd /etc/
ls -la | grep sudoer
Chances are, if the hacker managed
to create users on your box, they
managed to put themselves in this
sudoer file, and if the sudoer
file exists, and you did not put
it there, you have a very good
chance that you're box has been
comprimised.
If you have confirmed the sudoer
file is not yours, and you do not
need it, delete it.
rm /etc/sudoers
To prevent further possibilities
of 0wnership, you can chmod 000
/usr/bin/sudo.
+ Laying the Smackdown +
At this point, we're ready to pull
up to the hackers house and beat
them with wet noodles till they
bleed, but we're all civilized
people and have better ways of
pissing them off.
Let's review what we've done:
- Removed all malicious shell
accounts
- Removed the groups the shell
accounts belonged to
- Removed sudoer configuration
- Took notes
There is a high likelyhood that
your box is being used to either:
- Distribute files via FTP
- Distribute files via DCC (irc
term: direct connect)
My experience showed that the
hackers attemped to do both. Using
my notes, I discovered a file
nested in the server that was an
attempted download of a DCC
daemon. It was never successfully
installed.
The hacker then attempted to
install two copies of glftpd (an
ftp daemon), I am still unsure
whether this software was actually
used, but it certainly was left
scattered all over the machine.
Let's look for it:
find '/' -iname *glftp*
If results are found, take the
necessary measures to view all of
its log files, take notes and then
delete it.
Also check your crontab for
malicous entires, I discovered one
that reset glftpd:
crontab -e
The next step in our hacker's plan
was the modification of proftpd
config files. Before continuing, I
would recommend that you do not
give users anonymous FTP access
(...you do have AnonFTP disabled
in your control panel...right...?)
cd /etc/proftpd
ls -l *.anonftp
Assuming you have anonftp off, all
of these files should have file
size of 0 bytes. In my case, the
hacker did attempt to modify one
of the site's and it was about 217
bytes. If you have confirmed these
files were modified malicously,
delete their content but NOT the
actual file.
Next:
ls -l
And confirm that all site# files
have a size of 37 bytes. Anything
large could have been a
modification made by the hacker.
Do futher investigation and take
necessary steps to fix this.
The hacker, in my instance, made a
small modification to
/etc/proftpd.conf:
less /etc/proftpd.conf
Look for the addition of a
DefaultRoot line. This line, if
you have not discovered through
your research in the bash
histories, this will give you a
pretty good (if not completely
accurate) idea of where the hacker
is storing their files. If you
have confirmed this line was added
malicously, delete it.
Your next step would be to remove
these files created by the hacker.
Once you have back-tracked EVERY
single step the hacker took to
modify your server, go back and
rerun Chkrootkit. It should return
a Not Infected message where it
previously did.
+ Finishing It Up +
Once everything has been cleaned
up, restart proftpd and other
services that modifications were
made to:
/etc/rc.d/init.d/proftpd restart
I failed to mention a few things
in my above post, the information
found here is equally informative.
There is a VERY high likelyhood
that the way the hackers go in is
through an exploit that they
placed in /tmp/ check for folders
in /tmp that were created around
the same data as the other events
occured. Remove those suspicious
folders if possible.
You may also detect large files in
the following manner:
find '/' -size +50000k
This will find files > than 50MB.
Bootleg movies are often
compressed in .rar format. Search
for files ending in .rar:
find '/' -iname *.rar
You may also want to monitor
/var/log/messages for login
attempts after you've removed all
traces of hack-itis.
less /var/log/messages | grep
proftpd
You'll see entries like this:
Nov 9 05:00:07 ensim
proftpd[21141]: ensim.domain.com
(p50832E46.dip.t-dialin.net[80.131
.46.70]) - FTP session opened.
Nov 9 05:00:09 ensim
proftpd[21141]: ensim.domain.com
(p50832E46.dip.t-dialin.net[80.131
.46.70]) - no such user
'anonymous'
Nov 9 05:00:14 ensim
proftpd[21141]: ensim.domain.com
(p50832E46.dip.t-dialin.net[80.131
.46.70]) - FTP session closed.
Nov 9 06:12:41 ensim
proftpd[24994]: ensim.domain.com
(ool-18bba13b.dyn.optonline.net[24
.187.161.59]) - FTP session
opened.
Nov 9 06:12:41 ensim
proftpd[24994]: ensim.domain.com
(ool-18bba13b.dyn.optonline.net[24
.187.161.59]) - no such user
'vgodz'
Nov 9 06:12:41 ensim
proftpd[24994]: ensim.domain.com
(ool-18bba13b.dyn.optonline.net[24
.187.161.59]) - FTP session
closed.
Booyah...Denied!
+ Are your loggers running? You
better catch them! +
Lots of crackers/hackers will turn
off loggers to ensure they don't
get caught. Syslog is one of these
logs that should always be
running. Check if it is running by
doing:
ps -aux | grep syslogd
If it isnt, you better find out
why . Restart it with:
/etc/rc.d/init.d/syslog restart
+ Check out "last" +
Linux features a command called
"last" which comes in handy when
needing to see who logged into
your box, and under what account
they did so. If your box has been
comprimised, this tool may prove
incredibly useful...
At the command prompt, simply type
"last" (no quotes). It will begin
to list all the user IPs/Hostnames
and what local user they logged in
as (it also shows the time, so you
can very accurately pinpoint the
time your machine was rooted).
You may have many, many logins -
in which case, you can pipe the
results of last into less, and
scroll through it at your own
pace...
last | less
+ Change Root E-Mail +
If you want to recieve e-mails
intended to go to the root user on
your machine, simply:
vi /etc/aliases
Scroll down to the bottom of the
file and change the line that
begins with "root:" (it may be
commented out), below is an
example of how you should make
this line look:
root: you@yoursite.com
+ Use LogWatch +
Logwatch is a very nice program
that will scan through your logs
and send you suspecious lines that
might be errors or warning
generated by internet service
(mail, httpd, ssh, myslq etc.). It
sends you a daily summary of logs
to your email address.
See: http://www.logwatch.org/ for
more info.
+ Use SIM +
Our very own rfxn (a member here
on the RS forum) created an
insanely useful script known as
SIM.
It monitors all services in your
box and restarts them for you when
they are down. It also monitors
the load of the machine using a
cron job. Doing so, it gives you
the option that everytime the load
is up, to run a script. In this
script I usually add a bunch of
commands that basically collect
logs, process info, and netstat
info. That get's stored somewhen
on the server and also gets
emailed to me. This way, when the
load is high, I am able to figure
out the problem by just looking at
that data. The benefit of this is
that you don't need to monitor
your box 24/7 yourself, let sim do
it for you.
See:
http://www.rfxnetworks.com/sim.php
for more info.
Forward your root mail to another
email address you use
vi /etc/aliases
change
root: my@email.com
I don't don't know about you guys,
but sometimes I am busy with other
things and I don't log in to my
box for days. Having all mail of
root sent to me, it is nice, I am
informed of any problem.
LogWatch
Logwatch is a very nice program
that will scan through your logs
and send you suspecious lines that
might be errors or warning
generated by internet service
(mail, httpd, ssh, myslq etc.). It
sends you a daily summary of logs
to your email address.
SIM
Sim is also another excellent
script I use and I got it from
this forum. I monitors all
services in your box and restarts
them for you when they are down.
It also monitors the load of the
machine using a cron job. Doing
so, it gives you the option that
everytime the load is up, to run a
script. In this script I usually
add a bunch of commands that
basically collect logs, process
info, and netstat info. That get's
stored somewhen on the server and
also gets emailed to me. This way,
when the load is high, I am able
to figure out the problem by just
looking at that data. The benefit
of this is that you don't need to
monitor your box 24/7 yourself,
let sim do it for you.
|
| |
|
Info
Support
Extras
Content
Off Site
