Hacking Doug « PREV | NEXT »: Collected Stories

April 29, 2010

UPDATE BELOW: How Google Is Killing An Arts Site (A Twist On "Too Big To Fail")

On the web, Google gives the words "too big to fail" new meaning. The company's dominance in directing the flow of information and people around the web is so big that if Google decides to block information, then there's not much you can do about it. And once you get caught up in its gears, it's difficult to extricate yourself without getting ground up in the process.

I run a small news site called ArtsJournal. Every day we look at a thousand or so arts stories from all over the world and aggregate the best of them into a daily digest of arts and culture news. I started the site in 1999, and was one of the first wave of aggregator sites of this sort. In those days I had to hand-code all the pages and paste each story into the html. No content management systems then.

Over the past 10 years we've become the leading digest of arts news, and cultural leaders and arts journalists all over the world use us to keep up with cultural news. We're also home to almost 60 arts bloggers, including some of the leading arts journalists on the web. And from time to time we host conversations about issues of the day. We don't get Andrew Sullivan numbers or Boing Boing numbers or Gizmodo or Mashable or Gawker numbers, but we reach a very specific niche. What Romenesko does for journalism news, we do for arts news.

What Happened

Monday night about 10:30 I had just sent out our daily newsletters when I got an email from a professor at the University of Oregon that he was getting blocked trying to get to ArtsJournal and that a notice had come up saying ArtsJournal was a "Reported Attack Page." The notice is red and scary-looking. We had been hacked.

malware warning reduced.jpegNow, I'm a journalist, not a techie, but I have picked up enough over the years doing ArtsJournal that I can usually figure out the technical side. I followed the directions Google pointed to for how to scrub a site of malware. I made sure the software powering the site was latest-issue, and went through all the pages Google had flagged as being infected. When I couldn't see any of the code they referenced, I went back to the Google Webmaster page and submitted the site for review. A few hours later Google reported that we were still infected.

I redoubled my efforts and discovered that

the hackers had found their way into our OpenX ad server and appended a script to be delivered every time an ad was called on the site. The reason I couldn't see the code when I examined the pages initially was because it wasn't being delivered until the ad software called it up. Tricky buggers, and quite ingenious. You actually couldn't even see the rogue script in the ad server initially. It was set up so the code only briefly flashed when you opened the admin for the ads, then appeared blank. But that flash was suspicious, and I followed its trail.

I removed all the malware scripts, then disabled the ad server, then shut down access to the ad server altogether. I went to every blog and took the ad call scripts out. Okay, way too much detail here, I realize, but I wanted to give at least a bit of a sense of the methodical work I went through to get rid of the malware.

After I was sure everything was fixed about 3 PM on Tuesday, I tried to resubmit ArtsJournal for Google review, but was told on the site that I had to wait "several hours" before resubmitting. Okay -- I get that there has to be a limit, otherwise people would be resubmitting over and over again. So I kept checking back every hour, and it wasn't until 3 AM Wednesday that Google would allow resubmission. Great, I thought. I resubmitted and thought we'd be back up in a few hours.

  Google malwar notice reduced.jpegAll day the Google page said the review was "being processed" or "pending." The Google report page on AJ said that malware hadn't been detected since Tuesday morning. Review still "pending." By mid-afternoon, the site still wasn't unblocked and I went to the review page and asked what was up. By late evening, still "pending." This morning 3:30 AM still pending. All morning Thursday, still "pending" or "being processed."

But A Larger Issue?

Maybe this is a good place to say how amazing I think Google is (no, I'm not trying to suck up, as you'll see in a moment). The company has transformed my life. I use Gmail to manage my email. Google Calendar to manage my appointments. Google Reader has been a godsend in helping to collect stories for ArtsJournal. Google Analytics helps me keep track of how ArtsJournal works. Google Voice helps manage my calls. And of course the various Google Search services feed my seemingly endless need for information. And did I mention YouTube? Or the indispensible Google Toolbar? Or Google Sync? In short, Google has become nearly impossible to do my work without. I spend more time using Google on the web every day than anything else.

And therein may lie the problem. "Too big to fail" when you're talking about banks means a bank so big that if it fails it might bring down the whole financial system. In the information business, "too big to fail" means that our free access to information is too dependent on the actions (benign or not) of a single company. You want to talk media consolidation? Google has become all media consolidated. I don't in any way think Google is singling me out. I don't think Google is being malicious. Indeed, I'm glad Google blocks malware (bastards!) to help contain it and protect others.

And why shouldn't Google try to be as big and powerful as it can be?

But should any one company have this much ultimate control over what we have access to? In my case (and probably for many others), the Google system for getting unblocked from the rest of the web sucks. Google offers low information about your status once you've been blocked and no information after you've done what they've suggested. For a news site that depends on constant updating, three days being blocked from most of the web is devastating.

That most of us are now tied into Google's ability to sort and deliver information is testament to the amazing things Google has been able to create. That most of us are now dependent on Google for our access to information is maybe not such a good thing. Has any one company ever had such control over the world's information? (And I haven't even mentioned Google's book project.)

Yes, you can see ArtsJournal on Internet Explorer. And a third of our visitors use IE, so they can still see the site as we continue to update it. But do a search for ArtsJournal and you get a nasty warning. And many of our regular visitors have been scared off by the big ol' nasty red "Attack Site" warning.  

It's 10:47 AM PDT (Day 3) and I continue to wait.

UPDATE: As of 6:56 AM Friday morning, Google seems to have lifted the block. (the Google Review page still says review of the site is "pending"). So no official word, but we appear to be back.

April 29, 2010 10:47 AM | | Comments (16)

16 Comments

Thank you for posting this commentary. I was dismayed at the messages I was receiving as Arts Journal is my home page. Glad to hear the site will be up at some point and appreciate your compilation of art news that might otherwise be unreachable.

As much aggravation as we all have with "Too big to respond in person", and as much as I miss access to your site, I can only guess that with so many infections to deal with, much of Googles response is on automatic.

In any event, thank you for providing such an excellent service. Sometimes we are not aware of how much we rely on something until it is not available.

I think Facebook is becoming just as important as Google in site referrals. If I had to choose between the two, Google is the better company. That being said, Google Webmaster tools are notoriously slow in updating the information.

To be brutally honest I think it's a small price to pay. Part of the way that google prevents further infections is by displaying the warning. People who visit your site without using google are more than likely infected.

It's your responsibility to keep your software up to date. And it's law enforcements responsibility to go after criminals. Its not Google's problem, what they're doing is just trying to slow infection for the good of the internet. They could simply be an impartial search engine and pass traffic through to the end page regardless.

There are parties at fault here. But Google isn't one of them.

We had some problems with OpenX as did many, many sites. I would recommend going with the hosted version, or switching your sites to Google Ad Manager.

you also come up in the google reader, where i follow you and, although it's mildly frustrating not to be able to read more than the first paragraph of anything since i can't click through, it's that reader that will enable me to go back and check the things i need when i can again. and it was the googlesearch i did on "artsjournal" that led me to your twitter page that linked to this information. the whole world is more vulnerable--because more accessible--for adapting to these megatechs but there's no going back, and i would say all in all that was a pretty quick workaround! see you on the flip side.

This is an awful story, especially for a reader such as myself who would have no idea how to go about "scrubbing malware."

Google needs to get on top of human/customer relations in addition to search algorithms, but it sounds like a lot of your ire should be directed at this OpenX outfit. Are they addressing their vulnerabilities?

I am just so happy that you will be back up soon. I have ArtsJournal as my homepage and I can't wait to read it again. The service and information you provide is invaluable!

Actually, your excellent synopsis just tells all of us how awesome YOU are! I realize it is frustrating when malware causes blockages, and I missed the newsletter, but it is a small price to pay for continued safety. No problemo!

You and the ArtsJournal are much appreciated.

Hi Doug and Arts Journal Readers,
Thank you for this discussion. It is very important to all of us. You don't realize that this can happen to you and then when it does it is really awful. We had a similar hacking at VoiceofDance.com almost a year ago. We are still recovering. We were blacklisted from Google for almost a week. Our traffic suffered tremendously and we have never regained our position on Google search.
It was very disheartening to be hacked but then also to be labeled and in a sense to be black listed by Google even long after the problem had been addressed.
Thank you Doug for your thoughts on this matter. I am glad to see you are back up!
best,
Lori Sparrow
CEO

Thanks Lori. Sorry this happened to you too. Amazing how many websites I've heard from in the past week that have been hacked. Bad enough. Then they find Google's inattention in the recovery process to be even worse punishment than the original hack. Erg.

Thank you for the post. I actually agree with Tim. As a consumer, I would rather choose a bit of delay than no warning combined with the risk of infection. But I also think updates can be faster, considering 20000+ employees and $$ they are making.
Good to see you back.

Wow. This is amazing that it took that long to get indexed again. I would love to see a follow up story on long term issues this causes you. For example, do you lose your rankings? What kind of traffic loss are you going to see for this week? Will you be losing visitors because they saw the malware once and decided to never come back?

Glad to see you are back up, I appreciate the explanation you have provided here.

Three days without your site and I was getting the DTs. Thank you for providing us with your astounding range of arts information.

Y'll are missed!

Leave a comment

















Archives

Creative Commons License
This weblog is licensed under a Creative Commons License.


About

    ARTicles ARTicles is a project of 
    the National Arts Journalism Program, an association of some 500 journalists in the United States. Our group blog is a place for arts and cultural journalists to share ideas and information, to celebrate what we do, and to make the case for its continuing value. ARTicles is edited by Laura Collins-Hughes. To contact her, click here.
    more

    ARTicles Bloggers Meet our bloggers: Sasha Anawalt, MJ Andersen, Alicia Anstead, Laura Bleiberg, Larry Blumenfeld, Jeanne Carstensen, Robert Christgau, Laura Collins-Hughes, Thomas Conner, Lily Tung Crystal, Richard Goldstein, Patti Hartigan, Glenn Kenny, Wendy Lesser, Ruth Lopez, Nancy Malitz, Douglas McLennan, Tom Moon, Abe Peck, Peter Plagens, John Rockwell, Werner Trieschmann, Lesley Valdes and Douglas Wolk. more

    NAJP NAJP is America's largest organization dedicated to the advancement of arts and cultural journalism. The NAJP has produced research, publications and discussions and works to bring together journalists, artists, news executives, cultural organization administrators, funders and others concerned with arts and culture in America today. more

    Join NAJP Join America's largest organization of arts journalists. Here's how more

see all archives

Contact: articles@najp.org

Recent Comments