A comparative analysis of search engine indexing time

Zuze H, Weideman M

Proceedings of the 13th Annual Conference on World Wide Web Applications

Zuze, H. & Weideman, M. 2011. A comparative analysis of search engine indexing time. Proceedings of the 13th Annual Conference on World Wide Web Applications. 14-16 September. Johannesburg, South Africa. Online: www.zaw3.co.za

Internet usage is increasing dramatically daily, as is Web development which is enhanced by the emergence of new Web technologies. Social networks are continuously dominating our Web interactions through Facebook, Twitter, LinkedIn and MySpace, drawing a considerable number of Internet users. However, the dominance of the Web as the epicentre of both important and useless information has affected competition in industries like e-business. It takes some time for a website to be indexed and appear on the search engine result page. The indexing process involves the reading and recording of the weight-carrying words in a search format to an index file by a crawler. This results in the webpage being discovered by users on a search engine results page. A user can also submit a website manually for indexing. Crawlers should then visit this website, record all the words on the pages, and note links to other sites. The index file is updated regularly, either by human editors or by these crawlers. After an extensive literature survey, empirical evidence indicated that the indexing period for webpages is not fixed. In a recent two phase experiment, five websites were monitored to determine their indexing time for Google, Yahoo! and Bing. The experiment proved that there is a relationship between keyword density and indexing time. Yahoo! and Bing seem to favour sites with high keyword density when indexing. During the first experiment the shortest indexing waiting time was five days and the longest, 33 days. For the second one, the waiting time varied between 19 and 29 days. However, according to this study, a period of approximately 19 days is a reasonable average waiting time. It is possible that the Google sandbox effect plays a role in these experiments, and this was also investigated.
  1. Benczur, A.A., Erdelyi, M., Masanes, J., Siklosia, D. 2009. Web Spam Challenge Proposal for Filtering in Archives. In: Proceedings of the 5th International Workshop on Adversarial Information Retrieval on the Web (AIRWeb2009), April 2009. Spain: Madrid: 438-444.
  2. Borglum, K. 2009. Getting your website to show up in search ranking (Practice Management Q&A). Medical Economics, GALE, 86(14):30.
  3. Castle, R. 2011. Google Sandbox Effect Sucks? Overcome the Google Sandbox and Earn TrustRank! Available WWW: http://www.roncastle.com/google-sandbox-aging-delay.htm (accessed 31 May 2011).
  4. Chen, L. 2010. Using a two-stage technique to design a keyword suggestion system. Information Research, 15(1). Available WWW: http://informationr.net/ir/15-1/paper425.html (accessed 10 April 2011).
  5. Egele, M., Kolbitsch, C., Platzer, C. 2009. Removing web spam links from search engine results. Journal in Computer Virology, 7(1):51-62.
  6. Eisenberg, B., Quarto-vonTivadar, J., Davis, L.T., Crosby, B. 2008. Always be testing: The complete guide to Google website optimizer. Sybex: Indianapolis.
  7. Erdelyi, M., Garzo, A., Benczur, A.A. 2011. Web spam classification: a few features worth more. In: Joint WICOW/AIRWeb Workshop on Web Quality (WebQuality 2011) In conjunction with the 20th International World Wide Web Conference, March 2011. Hyderabad: India: 27-35.
  8. Flosi, L.S. 2011. ComScore Releases April 2011 U.S. Search Engine Rankings. Available WWW: http://www.comscore.com/Press_Events/Press_Releases/2011/5/comScore_ Releases_April_2011_U.S._Search_Engine_Rankings (accessed 26 May 2011).
  9. Fox, W., Bayat, S.M. 2007. A guide to Managing Research. Cape Town: Juta & Co Ltd.
  10. Kritzinger, W.T., Weideman, M. 2007. Key word placing in Webpage body text to increase visibility to search engines. South African Journal of Information Management, 9(1). Available WWW: http://www.sajim.co.za (16 August 2010).
  11. Malaga, R.A. 2009. Web 2.0 Techniques for search engine optimization: Two case studies. Review of Business Research, 9(1):132-139.
  12. Mathews, J. 2011. Get on Google front page: 2011 SEO tips. Jason Mathews. Available WWW: http://books.google.co.za/books?id=6n70oyVmgAQC&pg=PT7&dq=how+long +does+it+take+to+index+a+website&hl=en&ei=f1R3TYymIpC38QPhto2gDA& sa=X&oi=book_result&ct=result&resnum=8&ved=0CE8Q6AEwBw#v=onepag e&q&f=false (accessed 09 March 2011).
  13. Nade, J. 2010. Evidence of Google Sandbox Effect & Does a Drop from Pagerank 3 to Pagerank 0 Equal a Google Penalty? Available WWW: http://www.google.com/support/forum/p/Webmasters/thread?tid=0eb1ec7537 0dfdf9&hl=en (accessed 31 May 2011).
  14. Parhizkar, M. 2010. Critical Analysis of Web Crawlers’ Algorithms. Available WWW: http://skincarefreesamples.info/critical-analysis-of-web-crawlersalgorithms/ (accessed 26 May 2011).
  15. Ron, B., Zsolt, K. 2011. The Role of Search Engine Optimization in Search Marketing. Social Science Research Network. Available WWW: http://ssrn.com/abstract=1745644 (accessed 15 April 2011).
  16. Snack, K. 2011. Search Engine Market Share (April 2011). Available WWW: http://www.karmasnack.com/about/search-engine-market-share/ (accessed 07 April 2011).
  17. SPSS Manual. 2007. SPSS Advanced Statistics 17.0: Kaplan-Meier Survival Analysis. Available WWW: http://www.hks.harvard.edu/fs/pnorris/Classes/A%20SPSS%20Manuals/SPS S%20Advanced%20Statistics%2017.0.pdf (accessed 18 March 2011).
  18. Visser, E.B., Weideman, M. 2011. An empirical study on website usability elements and how they affect search engine optimisation. South African Journal of Information Management 13(1). Available WWW: http://www.sajim.co.za (accessed 07 April 2011).
  19. Weideman, M. 2009. Website Visibility: The theory and practice of improving rankings. Oxford: Woodhead Publishing Limited.
  20. Weideman, M. 2008. Internet Searching and other Research Challenges: Publish or Perish. Cape Town: Cape Peninsula University of Technology. Inaugural Speech.
  21. Weideman, M. 2004. Empirical evaluation of one of the relationships between the user, search engines, metadata and websites in three-letter .com websites. South African Journal of Information Management, 6(3). Available WWW: http://www.sajim.co.za (accessed 16 June 2010).
  22. Zahorsky, R. M. 2010. A Web trick catches a venerable law directory. ABA Journal. 96(2):32-33.
  23. Zhang, J., Dimitroff, A. 2005. The impact of webpage content characteristics on webpage visibility in search engine results (Part I). Information Processing and Management, 41:665-690.
Full text of Conference Proceedings No 0008: A comparative analysis of search engine indexing time

Digital Library with full-text of academic publications on website visibility, usability, search engines, information retrieval

Back to Home page