A comparative analysis of search engine indexing time
Zuze H, Weideman M
Proceedings of the 13th Annual Conference on World Wide Web Applications
Zuze, H. & Weideman, M. 2011. A comparative analysis of search engine indexing time. Proceedings of the 13th Annual Conference on World Wide Web Applications. 14-16 September. Johannesburg, South Africa. Online: www.zaw3.co.za
Internet usage is increasing dramatically daily, as is Web development which is enhanced by the emergence of new Web technologies. Social networks are continuously dominating our Web interactions through Facebook, Twitter, LinkedIn and MySpace, drawing a considerable number of Internet users. However, the dominance of the Web as the epicentre of both important and useless information has affected competition in industries like e-business. It takes some time for a website to be indexed and appear on the search engine result page. The indexing process involves the reading and recording of the weight-carrying words in a search format to an index file by a crawler. This results in the webpage being discovered by users on a search engine results page. A user can also submit a website manually for indexing. Crawlers should then visit this website, record all the words on the pages, and note links to other sites. The index file is updated regularly, either by human editors or by these crawlers. After an extensive literature survey, empirical evidence indicated that the indexing period for webpages is not fixed. In a recent two phase experiment, five websites were monitored to determine their indexing time for Google, Yahoo! and Bing. The experiment proved that there is a relationship between keyword density and indexing time. Yahoo! and Bing seem to favour sites with high keyword density when indexing. During the first experiment the shortest indexing waiting time was five days and the longest, 33 days. For the second one, the waiting time varied between 19 and 29 days. However, according to this study, a period of approximately 19 days is a reasonable average waiting time. It is possible that the Google sandbox effect plays a role in these experiments, and this was also investigated.
- Benczur, A.A., Erdelyi, M., Masanes, J., Siklosia, D. 2009. Web Spam
Challenge Proposal for Filtering in Archives. In: Proceedings of the 5th
International Workshop on Adversarial Information Retrieval on the Web
(AIRWeb2009), April 2009. Spain: Madrid: 438-444.
- Borglum, K. 2009. Getting your website to show up in search ranking
(Practice Management Q&A). Medical Economics, GALE, 86(14):30.
- Castle, R. 2011. Google Sandbox Effect Sucks? Overcome the Google
Sandbox and Earn TrustRank! Available WWW:
http://www.roncastle.com/google-sandbox-aging-delay.htm (accessed 31 May
- Chen, L. 2010. Using a two-stage technique to design a keyword suggestion
system. Information Research, 15(1). Available WWW:
http://informationr.net/ir/15-1/paper425.html (accessed 10 April 2011).
- Egele, M., Kolbitsch, C., Platzer, C. 2009. Removing web spam links from
search engine results. Journal in Computer Virology, 7(1):51-62.
- Eisenberg, B., Quarto-vonTivadar, J., Davis, L.T., Crosby, B. 2008. Always be
testing: The complete guide to Google website optimizer. Sybex: Indianapolis.
- Erdelyi, M., Garzo, A., Benczur, A.A. 2011. Web spam classification: a few
features worth more. In: Joint WICOW/AIRWeb Workshop on Web Quality
(WebQuality 2011) In conjunction with the 20th International World Wide Web
Conference, March 2011. Hyderabad: India: 27-35.
- Flosi, L.S. 2011. ComScore Releases April 2011 U.S. Search Engine
Rankings. Available WWW:
Releases_April_2011_U.S._Search_Engine_Rankings (accessed 26 May
- Fox, W., Bayat, S.M. 2007. A guide to Managing Research. Cape Town: Juta
& Co Ltd.
- Kritzinger, W.T., Weideman, M. 2007. Key word placing in Webpage body text
to increase visibility to search engines. South African Journal of Information
Management, 9(1). Available WWW: http://www.sajim.co.za (16 August
- Malaga, R.A. 2009. Web 2.0 Techniques for search engine optimization: Two
case studies. Review of Business Research, 9(1):132-139.
- Mathews, J. 2011. Get on Google front page: 2011 SEO tips. Jason Mathews.
e&q&f=false (accessed 09 March 2011).
- Nade, J. 2010. Evidence of Google Sandbox Effect & Does a Drop from
Pagerank 3 to Pagerank 0 Equal a Google Penalty? Available WWW:
0dfdf9&hl=en (accessed 31 May 2011).
- Parhizkar, M. 2010. Critical Analysis of Web Crawlers’ Algorithms. Available
(accessed 26 May 2011).
- Ron, B., Zsolt, K. 2011. The Role of Search Engine Optimization in Search
Marketing. Social Science Research Network. Available WWW:
http://ssrn.com/abstract=1745644 (accessed 15 April 2011).
- Snack, K. 2011. Search Engine Market Share (April 2011). Available WWW:
07 April 2011).
- SPSS Manual. 2007. SPSS Advanced Statistics 17.0: Kaplan-Meier Survival
Analysis. Available WWW:
S%20Advanced%20Statistics%2017.0.pdf (accessed 18 March 2011).
- Visser, E.B., Weideman, M. 2011. An empirical study on website usability
elements and how they affect search engine optimisation. South African
Journal of Information Management 13(1). Available WWW:
http://www.sajim.co.za (accessed 07 April 2011).
- Weideman, M. 2009. Website Visibility: The theory and practice of improving
rankings. Oxford: Woodhead Publishing Limited.
- Weideman, M. 2008. Internet Searching and other Research Challenges:
Publish or Perish. Cape Town: Cape Peninsula University of Technology.
- Weideman, M. 2004. Empirical evaluation of one of the relationships between
the user, search engines, metadata and websites in three-letter .com
websites. South African Journal of Information Management, 6(3). Available
WWW: http://www.sajim.co.za (accessed 16 June 2010).
- Zahorsky, R. M. 2010. A Web trick catches a venerable law directory. ABA
- Zhang, J., Dimitroff, A. 2005. The impact of webpage content characteristics
on webpage visibility in search engine results (Part I). Information Processing
and Management, 41:665-690.