If you build it, maybe they will come. Then again, maybe they will not. By most accounts there are approximately 20 billion web pages on the Internet today. Search engines like Google and Yahoo only index approximately 50% of those pages, at best. Search engine optimization (SEO) should be a key component of any interactive marketing strategy. SEO increases the quantity and quality of visitors to a website.

Companies spend thousands—or millions—of dollars to build state of the art websites and develop a digital brand. Yet, it is surprising how many fail to properly optimize their websites for search. While many companies focus on optimizing for large search engines, they must also pay attention to niche or topic-specific search engines. So, how exactly do search engines work? How can companies optimize their websites to make sure they are quickly and easily found by the right people?

SEARCH ENGINES

The first step in building a solid SEO strategy is to understand exactly how search engines work. There are three types of search engines: crawler-based, human-powered or directories, and hybrid.

Web Crawlers
The best example of a crawler-based search engine is Google. Contrary to common belief, when one searches for a key term on Google, it does not initiate a search of the Internet for matching web pages. Instead, Google searches its own indexed database of web pages that have been previously collected and that match the keyword(s) being searched. Google’s search process is both simple and complex. It follows three simple steps to get results: crawling, indexing and searching. The methods used at each step are a bit more complicated.

Everyday Google uses a software agent known as a spider, crawler or bot to “crawl” the web in search on new pages to add to its massive index. Googlebot, Google’s spider, has the ability to visit a page, read the text and find links to other pages. If appropriate, Googlebot will add a partial or full copy of the visited page to Google’s database. It will then follow the links found on that page to other pages and repeat the process.

Google’s database is not located in a single location but rather it is scattered across hundreds of computers. This makes sense since more machines lead to faster search results. Furthermore, scanning every word on every page for every search would be tedious and very slow. Google solves this problem by assigning numbers to pages. For example, the word “advertising” could be found in pages 1, 3, 5, and 8. The term “agency” could be found in documents 2, 3, 4, 7, 8 and 9. If you do a search for “advertising agency,” Google will serve pages 3 and 8. Of course, this is just a simplified example for explanation purposes. Google has indexed billions of web pages into its database.

So, let’s continue with our indexing example. If you do a search for “advertising agency” how will Google know which page to serve first? Page 3 or page 8? Google will use algorithms to determine which one gets a higher ranking. An algorithm is a mathematical rule or procedure used to solve a problem. If the terms “advertising” and “agency” appear together next to each other on page 3, but separately on page 8, it will know to give page 3 a higher ranking.

Google’s main algorithm, PageRank, uses content relevancy and page popularity to rank pages. Page popularity is determined by the number and quality of incoming links to a website. For instance, if both page 3 and 8 display the terms “advertising agency” together but page 3 has inbound links from three blogs and page 8 from five, then page 8 would rank higher. However, if page 3 has been linked to by just one site and that site is high traffic reputable site such as businessweek.com, then page 3 would move up. Google also uses another algorithm, Hilltop, to determine popularity and link value. Hilltop takes into account topic specific relevancy when rating links. For an auto parts website, a link from motortrend.com will have a higher value than amazon.com. More advanced algorithms take into account previous searches made in order to improve search results. Yahoo! saves every search you make and uses that data to optimize future searches.

Directories or Human-Powered Engines
At the dawn of the Internet age, in the early 90s, there were only two ways to find a website. One could either type in the domain name address or go to an existing web directory. One of the first Internet directories was Yahoo!. To get listed on this directory all one had to do was submit a company name, a domain address and a short description. Yahoo! was organized in a hierarchical manner with sites listed by categories in a alphabetical order. To make it to the top of a given list a company would have to change its name to ABC or better yet 123. Unlike a true search engine, the directory did not display any part of a web page. It only displayed the description submitted. So changing the content of a web page did not affect the listing or the ranking.

Much has changed since then and with few exceptions most web directories today are small and topic specific. Nevertheless, they are as important to most companies as any of the search engines. For example, it is as important for a university to be listed on collegeboard.com, the number one university directory on the Internet, as it is to get a high ranking on Google or MSN.

Hybrid
Today, most search sites are hybrid. That is, they are search engines complemented by an Internet directory. As the Internet grew it became almost impossible for major directories to keep up with the number of websites being submitted everyday. Eventually they were forced to partner with, buy or develop their own search engines. In 2001, Yahoo! partnered with Google to support its Internet directory with a search engine. By 2004, Yahoo! had put in place its own search engine. Likewise, Google now partners with the Open Directory Project (ODP), the Internet’s largest directory. ODP is maintained and edited by volunteers and sponsor by major search engines—including AOL, Netscape and Alexa in addition to Google.

SEARCH OPTIMIZATION

Trying to get your website to rank high on search engines is like playing a game of poker. The only difference is that there are millions of players and what constitutes a winning hand is defined partially by the person doing the search and partially by the search engine’s secret algorithms. However, there a ways to stack the odds in your favor. Now that we understand how search engines and their algorithms work, we can build an effective SEO strategy.

Now, we are only going to focus on natural search engine optimization. That is, a strategy that excludes paid search engine advertising or search engine marketing (SEM). Furthermore, while there are dozens of search engines we are only going to focus on the two engines that cover the most searches: Google with 45% of online searches and Yahoo with 30%. It is worth to note that two up and coming search engines that are likely to take market share from both Google and Yahoo!. They are Microsoft’s live.com and IAC’s ask.com. Submitting your pages to all four search engines should be your first step in trying to get your website indexed and ranked.

Know Your Target
Page ranking will be determined to a large extent by the key words a person enters into a search. It is extremely important to identify who your website is trying to target. Furthermore, determine what words or phrases your target will likely enter when searching for your company or products. Once you have built a solid list, enter the key terms in both Yahoo! and Google to see which pages rank first. If the pages are relevant and the competition is included, then you know you have good list. The words or phrases that do not return relevant results should be deleted from the list. Still unsure about your list? There are three great tools you can use to hone in your keyword list: Overture, Adwords and Word Tracker.

Know Your Competition
No matter what business you are in, it is likely that the competition will be in the top results of search engines. If they are not, you are in luck. For the rest of us it is imperative that an in-depth analysis of the competition’s web pages be conducted to answer some key questions. Why are they at the top of the results? Who are they linked to and who links to them? What key words are found throughout their pages? META tags? Title tags? Also, how much traffic is the competition’s website getting? Answering these questions is key if you want to develop a strategy that will counter theirs. Remember the card game metaphor; you will only win if your hand is better than theirs.

Text is King
The right text in the right places, that is. As we know, spiders work by analyzing the content of web pages and following their links. However, spiders cannot read graphic files such as pictures, videos or flash. They can only read text. Text that can be found in the body of the page as well as in titles, headings, file names, alt tags, comment tags and META tags. Your list of keywords should appear in all these, as well as sprinkled throughout the body of your web page. Make sure to be selective on the keywords used. Only use those words that your target is likely to search for. Do not limit the use of keywords to your homepage. Make sure they are properly placed throughout your website. However, make sure that META tags throughout your website do not have the same description. Search engines might assume that they all have the same content. Make sure that your META tags are not longer than 200 characters. Finally, do not overuse graphics or flash. Algorithms tend to negatively rate web pages that have a high graphics-to-text ratio.

SEO is a Popularity Contest
In addition to relevant words, search engine algorithms rank a web page in accordance of the quantity and quality of websites linking to that page. Make sure that you get as many inbound links from quality sites as possible. The higher the relationship between these links and your keywords the better. A quality website is one with a high search ranking and high traffic volume. You can use PageRank Checker and Alexa to assess website quality. Submit your website to Internet and industry specific directories. Commenting on blogs as well as submitting to association and publication websites related to your line of business will help direct traffic to your site.

Validate Your Code
Make sure that your website works well on all web browsers, does not return any error messages and loads quickly. If your website takes too long to upload, your target is likely to quickly move on to your competitor’s site. The same goes for pages that are difficult to read in some browsers. Not validating your code could lead to a decrease in traffic and thus a decrease in ranking. In addition to validating your code, make sure that you check for broken links and missing images. There are several tools that you can use to validate your pages—among them ScrubTheWeb, Fire Bug, Web Developer and RankQuest.

Create a Sitemap Page
Make your website is spider friendly by creating a sitemap page. A sitemap will allow the spider to swiftly find other pages in your website. This is especially important for large websites with dozens or hundreds of pages. The fewer number of clicks that it takes a spider to access pages in your website, the more likely it is to get a higher rank. Some search engines offer sitemap tools to help with this task. For instance, Google offers a free downloadable program named Google Sitemap.

Do Not Get Banned
Yes, search engines do ban websites. Websites are usually banned for behavior the search engines consider to be shady or unscrupulous. They include:

• Using hidden text or links for the sole purpose of improving page ranking
• Cloaking—that is, creating pages with highly desirable keywords and then redirecting traffic to a different website
• Using link farms or feed based websites that use spamming to improve your ranking
• Spamming your pages with meaningless keywords to boost ranking (i.e. Britney Spears, American Idol, etc)

A FINAL WORD

Search engine optimization requires that websites be frequently maintained and optimized. For every action that you take to improve your ranking, you are likely to get an equal and opposite reaction from your competitors. Furthermore, search engines are continuously working on improving their algorithms—that by the way, are kept secret. Search engine optimization leads to higher rankings and traffic, which in turn leads to higher revenues and brand equity.