I just took over full ownership of the dedicated server where I host most of my client websites.
It is definitely a learning experience for me as I have limited knowledge of linux and ssh, etc.
However, as I am trying to familiarize myself with the full admin access to the server, I noticed on the desktop a link to the Mail Queue. So, I clicked.
What I found was that there were 12,000 e-mails sitting in the remote queue. And, most of them were from email@example.com. Also, there were other, legitimate e-mails stuck in various sections between the spam messages. So, I started to use the Mail manager to delete these messages, 100 at a time. Have I gotten any real work done in the past 6 weeks? I would say NOT!
I started doing some house cleaning looking at all the applications that were being used for all the sites on the server, who has ftp access, etc. What applications can be moved to somewhere else? The first one I decided to move off was the php WebCalendar. Google’s is just as good and I had already started to move client’s calendars to use Google’s already, so I moved everyone else off.
Still, spam filled up the mail queue.
So, as I have been wanting to do for years, I started moving client’s e-mail off the server, moving them over to Google Apps. A lot of the incoming spam stopped, of course, but the outgoing continued.
At this point, I had figured out how to ssh into the server and use qmail to delete all e-mails with a certain subject. Until about 20,000 showed up with no subject. This resulted in me having to delete the entire queue (which I did over the weekend). I was still receiving mail so incoming was coming in. I waited until incoming dropped down to zero, then blasted all the remote away.
Up to this point, I had already instructed clients to not use the server for outgoing e-mail, but use their ISP. So, I knew that there had to be some vulnerability on the server that was allowing e-mail to be sent from the server.
I started cleaning out every domain account, going through all the pages to make sure they were in use, deleting all orphans, deleting all old php forms no longer in use (as I had determined some time ago that these were not secure).
After the server filled up with 140,000 outgoing e-mails, I had enough. I finally got some relief from my friend who helps me out in my business. She was looking at the spam this week, and saw a reference to a file on a client site. My second clue (first is below).
I immediately clicked on the file, and to my horror, what had been links.php, with a list of links to legitimate websites, was now a form!
Earlier on I had found a clue in one spam that referenced a client’s php WebCalendar, and this changed file was in the same account.
I immediately got rid of the offending form, and the mail queue is back to normal. However, I decided to search on the web to see how many others have this form or something like it, and you would be amazed. I looked up the links to the website in the form and the webhost is one of the top 25 spammers in the world.
In addition, I found where a security advisory was issued back in April about the php Webcalendar. The version I was using supposedly had these fixed, but I have obviously found otherwise.
As I finish cleaning up what I can with this server, I am planning to upgrade to a newer server and find someone who knows how to manage it, as this is only what one novice has been able to do. Yes, the web gets you coming and going. In order to have the flexibility you need to work in this arena, you really need to have your own server, but then you get stuck with all the headaches along the way. Sometimes I am not certain the flexibility is worth the trouble!