Competitive Intelligence: Know Thy Competitor Well

I have just secured myself a power outlet for the rest of the day. There aren’t words for how excited that makes me. I am a nerd. A nerd with power!

Okay, time to get all secret squirrel. Bruce Clay will do his best to moderate Jake Baille, Andy Beal, Larry Mersman and Will Achison. [I secretly wants to run on stage and give Bruce a hug. I wonder if he'd have security come and take me away? :)]

Bruce gets right to it and introduces Andy Beal. Andy Beal calls him out for saying he’s old. Now, now, boys.

He’s going to give us some of his favorite tools for spying on your competition.

1. Domaintools.com: Takes the WHOIS info you usually get and takes it to a whole new level. You can find out who owns a site, what other sites they own, what sites are hosted on the same server, what directories they’re listed in, etc.
2. ranks.nl/tools/spider.html: Check out keyword densities for your competitors. Breaks it up into 2-word, 3-word and 4-word combinations
3. sitexplorer.search.yahoo.com: Check backlinks for your competition. Best one-stop tool for backlinks. Very useful since Yahoo puts the most important backlinks first.
4. seomoz.org/tools: They have a bunch of free and paid tools.  He’s not a member but he values it a lot. [I'm not a member, but I give the SEOmoz tools props too.]
5. soloseo.com/tools/indexrank.html: Takes Google’s data and puts it in an easy-to-use dashboard so you can compare your site to your competitors. You can compare over the past year, 6 months, or 2 weeks for a site. You can see how strong the site is.
6. copernic.com: Paid site, only works on a PC. Track site changes. If text or a new image changes, you get an alert. You can monitor anything [Watchthatpage.com is a free alternative.]
7. Technorati.com: Find out who’s talking about your competition. And you!
8. google.com/alerts: Use Google News to monitor references. Alerts come via email. Pretty good way to keep a look on your brand name. [I have so many Google Alerts registered it's insane.]
9. searchanalytics.compete.com: You type in the domain name and Compete will give you an approximation as to what key phrases are bringing traffic to your competitors Web site.
10. touchgraph.com: Helps you find your competitors’ hubs. Visually shows you where their links are coming from. Find a place to get links. See what sites are attracting attention.
11. google.brand.edgar-online.com: You can sign up to get alerts of public companies’ FCC filings. Keep an eye on new product developments or changes to management
12. seekingalpha.com/transcripts: Scan through transcripts for public companies.
13. google.com/patents: Use it to spy on the patent filings of your competitors.
14. oodle.com: Keep an eye out on whether or not your competitors are hiring and where.
15. Spy on employees: Find out who your competitors employees are and find out if they’re on social media. It’s amazing the amount of information they’ll disclose, especially if they’re on Twitter.

Next is Larry Mersmen. He says he’s a tool. He means he works for a company that HAS a tool. The speakers are punchy today.

The definition of competitive intelligence can mean many things depending on the channel we are dealing with. Sources of info: newspapers, blogs, financial info, traffic patterns, etc.

Without going into the log files, info can be gathered by different means using Hitwise, Trellian and comScore. Information can be collected several ways and from many sources. The most typical data pools are Internet Service Providers, user panels and Web site Search history.

You already know who your competitors are. Some have a strong Web presence and others don’t. Find out who’s competing with you for certain phrases.

Target Relevant Keywords

Who is optimizing around your target keyword? Find out how they got there, and where their traffic is coming from. Who is sending them the most traffic? What keywords are actually being clicked on to get the user to your competitor? Your competitors traffic can be coming from the search engines, articles, blogs, affiliate links, articles, etc.

Knowing what keywords your competitor is targeting is important, but knowing which keywords are getting clicked on by the user to get to their site is key. Reviewing keywords reports that list frequency or “performance” of a term gives you the ability to see what’s working well and what’s not. Knowing a keywords performance, both paid and organic, will help you optimize your site around proven data, possibly streamline your spending and increase your ROI.

Many companies will optimize their site around the keywords they think their site will be found under, or where the end users will find a link to their site. In reality, its the end user that makes the choices that drive the traffic to you site. Or your competitors. Do you guess? Or do you use the data that you know has been successful?

Bill Atchison is up next.

Stop Snooping Competitors. He talks about LinkScape and who he figured out how to stop it from indexing his site.

Why let competors use your hard work and paid research as low hanging fruit to launch their business? Competitors want the path of least resistance to encroach on your online business. SEOs want to make easy money, helping competitors rank by leveraging your information. SEO tool vendors like SEOmoz want to sell your data for a profit and help those competing with you.

Who else gathers intelligence?

• Competitive shopping sites
• Intelligence gather spybots - Copyright compliance, branding compliance, corporate security monitoring, etc.
• Data aggregators
• Lawyers
• And many more!

How is Info Collected?

• Google, Yahoo, MSN
• Internet Archives
• Whois, Domaintools, etc
• SEO tools that directly crawl your site.

Noarchive bans cache. Eliminates the engine cache to covert researchers from gathering data on your meta tags, internal anchor text and outbound links.

Ban the Interent Archive. It’s used to covertly gather both historical and recent site data. Block it in your robots.txt. Also block it in the .htaccess file, as they still tend to crawl and results show up in Alexa.

Private WHOIS information: Remove clues about you, your administrative, technical contacts, and how many domains you own. The WHOIS data is easily blocked.

Whitelist Robots.txt: Use robots.txt to tell well-behaved bots whether they’re allowed to crawl or not.

Verify Spider Identity: Make sure the search engines are who they claim to be by using full trip reverse DNS checking, avoid spoofing.

Next we have Jake Baille. He wants his presentation to be an open discussion. That means there’s no PowerPoint. He’s just going to talk.

He asks how many people now think Bill is crazy? Hee. Not many. One of the interesting things is that SEO is changing. Social media is changing. But this competitve intelligence stuff hasn’t changed all that much.

Google is still the best competitive intelligence tool on the planet. At the end of the day, you want to rank particular pages for particular keyword. Find pages getting inbound links for the keywords you’re going after. Google’s link operator tool is still one of the best. At the end of the day, your competitors are everyone trying to rank for your terms.

Do you trust your ISP? He doesn’t. He asks who uses Quickbooks. About 7 people. Would you put your QB files on a server you didn’t own? No. ISPs are a weak point in the competitive intelligence landscape.  There have been instances where people have bribed the 16 year old tech support guy to get information. [Seriously? Those people need lives.]

Jake says we’ve been seeing a lot of fake Googlebots lately, so don’t trust that. Do the round trip DNS lookup and do a WHOIS lookup on the IP address. He uses CompleteWHOIS.com.

He talks about social engineering. People will write all sorts of things on Twitter that they shouldn’t. It’s amusing. You can also call the marketing department or SEO firms and hear them talk about their clients. Call the SEO, tell them you’re in a similar industry to one of their clients and ask what they’ve done for their clients. Hee! Talk to disgruntled employees and the disgruntled exwives of business people.

He asks who uses robot.txt to block development sites. Don’t do that. It’s bad. You can serve robots.txt differently based on who’s asking for. He recommends password protection for those sites. Use strong passwords.  Jake says he can guess the majority of our passwords.

Okay, last year Jake kind of scared me with this presentation. This year I think he’s kind of awesome and hysterical. I wonder who changed — me or Jake.

Question and Answer

For ecommerce sites, is there software that monitors pricing?

Jake doesn’t know of any unless you manually scrape Google Base. He’s writen crawlers for it inhouse.

Andy rementions Copernic. You can tell it what content you want to watch. You can export the data.

Can you find out how popular a certain page is?

Andy asks if he wants a legit way or a Jake way. Hee. Jake says any of the competitive intelligence tool will do that. Bill mentions Hitwise.

Jake says if you manage to acquire your competitors log files…

I use SEO for FireFox, SEOquake and I find that our stats show up stronger than some of the othe sites in the SERP that rank above us. Does that mean that all we need to do is better onpage SEO?

Jake says no.  Without knowing the source of that data, who knows.

Lunch!