web counter

Let Internet Enhancement help you build a stronger, more prominent business!

Continental U.S. Only

English Language Only

How do I crawl webpages?  

Internet Enhancement Services 




Do not forget to click on the links below to view our other Free SEO Information pages

If you have trouble purchasing while we are doing a payment system upgrade, please call us.  715.284.7879
We will gladly help and take the payment over the phone.

Phone:  715.284.7879

How a web crawler works.  As a web crawler or Googlebot I am likened to a caterpillar with every hair on my body being a receptor for information.  I analyze millions of web pages to find the best website for a given search.

I first search for web pages having page titles that have the words that are searched.  I then search the actual pages for those words and even the adverbial words in the search phrase.  If you change one word in the searched words I will likely return different results.  My job is to return the most relevant results for the words searched.  My results are pretty good but not perfect.

If the page title matches the search phrase but the content does not have those words I scream S-P-A-M !!!.  If those searched words are not near the beginning of the web page I say “Low relevance”.  Even little words like any, all, every, etc are important if they are in the searched phrase.  I look for these early in the content too.  I often produce those words in bold even if they are not used alongside the keywords.  I take into account how popular a website is, its load speed and its domain age.

I pay little attention to meta descriptions as I am more interested in finding actual content that matches.  In the absence of a meta description I go directly to the content to find the matching words.  I do pay attention to domain names that have keywords in them but not as much attention as I used to.  I do notice emphasized words such as bolded, underlined, etc as long as it is not common throughout the page.  I’m okay with pictures but I can’t see them.  I look for keywords in the picture description.  I give more points for a website having multiple pages or lots of incoming links from websites with common keywords.  I subtract points when there  are  lots of links from sites that do not have keywords in common.  I cry F O U L if there are links to pornographic sites.  I substitute synonyms where I think appropriate but you will get different results for “buy a house” vs. “purchase a house”.  I like fresh or recently changed content.  I will also visit those sites more often.  I show where I found the words in the content by making them bold.  Sometimes it is not logical where I pull a word or two so again I’m not perfect.

Search “furnace” and I will provide different results than if you search “furnaces”.   This means you need to include both words if you deal with furnaces.  Search “service any furnace” (or brand) vs. “service all furnaces” (or brands) and I will provide different results.  This means you should say we service any and all furnace brands in your content.  Changing one or two words in your content can significantly improve your exposure.

Google and others periodically tweak how I should see things (brought about by someone trying to game the system) but my mission is always clear.  To return the most relevant web sites for the search requested.  Those who say content is king are right unfortunately a special word here or there can make a difference.

                                                                                BY:  Marvin Matejka

Marvin Matejka is the CEO of Internet Enhancement at www.internetenhancement.com which offers free information pages and low cost mentoring to small business website owners.

If you like our page, like us on Facebook.

How do I Crawl Webpages?