How to Speed up Search Engine Indexing

Image via Wikipedia

It’s a common knowledge that nowadays users don’t only search for trusted sources of information but also for fresh content. That’s why the last couple of years, the Search engines have been working on how to speed up their indexing process. Few months ago, Google has announced the completion of their new indexing system called Caffeine which promises fresher results and faster indexation.

The truth is that comparing to the past, the indexing process has became much faster. Nevertheless lots of webmasters still face indexing problems either when they launch a new website or when they add new pages. In this article we will discuss 5 simple SEO techniques that can help you speed up the indexation of your website.

1. Add links on high traffic websites

The best thing you can do in such situations is to increase the number of links that point to your homepage or to the page that you want to index. The number of incoming links and the PageRank of the domain, affect directly both the total number of indexed pages of the website and the speed of indexation.

As a result by adding links from high traffic websites you can reduce the indexing time. This is because the more links a page receives, the greater the probabilities are to be indexed. So if you face indexing problems make sure you add your link in your blog, post a thread in a relevant forum, write press releases or articles that contain the link and submit them to several websites. Additionally social media can be handy tools in such situation, despite the fact that in most of the cases their links are nofollowed. Have in mind that even if the major search engines claim that they do not follow the nofollowed links, experiments have shown that not only they do follow them but also that they index the pages faster (Note that the fact that they follow them does not mean that they pass any link juice to them).

2. Use XML and HTML sitemaps

Theoretically Search Engines are able to extract the links of a page and follow them without needing your help. Nevertheless it is highly recommended to use XML or HTML sitemaps since it is proven that they can help the indexation process. After creating the XML sitemaps make sure you submit them to the Webmaster Consoles of the various search engines and include them in robots.txt. So make sure you keep your sitemaps up-to-date and resubmit them when you have major changes in your website.

3. Work on your Link Structure

As we saw in previous articles, link structure is extremely important for SEO because it can affect your rankings, the PageRank distribution and the indexation. Thus if you face indexing problems check your link structure and ensure that the not-indexed pages are linked properly from webpages that are as close as possible to the root (homepage). Also make sure that your site does not have duplicate content problems that could affect both the number of pages that get indexed and the average crawl period.

A good method to achieve the faster indexation of a new page is to add a link directly from your homepage. Finally if you want to increase the number of indexed pages, make sure you have a tree-like link structure in your website and that your important pages are no more than 3 clicks away from the home page (Three-click rule).

4. Change the crawl rate

Another way to decrease the indexing time in Google is to change the crawl rate from the Google Webmaster Tools Console. Setting the crawl rate to “faster” will allow Googlebot to crawl more pages but unfortunately it will also increase the generated traffic on your server. Of course since the maximum allowed crawl rate that you can set is roughly 1 request every 3-4 seconds (actually 0.5 requests per second + 2 seconds pause between requests), this should not cause serious problems for your server.

 

 

 

 

5. Use the available tools

The major search engines provide various tools that can help you manage your website. Bing provides you with the Bing Toolbox, Google supports the Google Webmaster Tools and Yahoo offers the Yahoo Site Explorer. In all the above consoles you can manage the indexation settings of your website and your submitted sitemaps. Make sure that you use all of them and that you regularly monitor your websites for warnings and errors. Also resubmit or ping search engine sitemap services when you make a significant amount of changes on your website. A good tool that can help you speed up this pinging process is the Site Submitter, nevertheless it is highly recommended that you use also the official tools of every search engine.

If you follow all the above tips and you still face indexing problems then you should check whether your website is banned from the search engines, if it is developed with search engine friendly techniques, whether you have enough domain authority to index the particular amount of pages or if you have made a serious SEO mistake (for example block the search engines by using robots.txt or meta-robots etc). A good way to detect such mistakes is to use the Web SEO Analysis tool which provides detailed diagnostics.  Finally most of the major search engines have special groups and forums where you can seek for help, so make sure you visit them and post your questions.

If you like this article, use the bookmark buttons on the top of the page and share it with other users.

Posted by bbriniotis on 30th of August, 2010 at 11:17

Google Optimization: Using Search Operators

Image by rustybrick via Flickr

The underlying premise of SEO suggests that you understand the task at hand when it comes to outranking the other 999 entrants for any given keyword.
Google stops indexing a particular keyword after 1,000 results when assessing the aggregate relevance score to determine which results are spawned. By truly understanding this, you can discover a great deal from using a few basic Google search operators to determine what type of foothold a competitor has for a given keyword or niche.

Basic Competitive Analysis Metrics

1. Start with the keyword you are interested in researching. Place the keyword "in quotes" in a Google search box.
For example "SEO" returns 262,000,000 competing pages with the chronological order of the strongest sites first.
Then look to the right and determine the number of competing pages you are up against "for that keyword". It will say results 1 of 10 of (the number of competing pages).
This allows you to assess the competitive landscape with one brief metric. The extent of what you consider a competitive keyword depends on the website. For example, most websites can acquire a keyword under 50,000 competing pages with ease and competitive keywords start above 100,000 results and ascend into the millions (pages in index / divided by the top 1,000 results).

The next few metrics will allow you to understand where your SEO ceiling is (what threshold your website has for keyword benchmarks). Our blog for example can devour a keyword with up to 1,000,000 competing pages just from one post of mentioning those keywords (without backlinks).
So, all the talk about building website authority does have a place when you understand the implications to rank with less effort. Authority sites have the ability to zero in on a keyword and skip over hundreds of other websites and reach the top 10 results by the merit of trust and internal link weight and dynamism they possess. In keeping with the topic at hand, let's move to the next metric.

2. Evaluate your competitors domain and determine the amount of pages they have by using this search command in Google. You can use the #1 site and the #10 site to gauge an average of pages required to capture the keyword or, if you want you can use the #1st, 2nd and 3rd site that rank for the selected keyword to see which formulas they are entrenched in.
site:competitorsite.com (this shows you how many pages they have indexed in Google)

3. Next, determine how saturated their website is with the keyword in question.
site:competitorsite.com keyword
This shows you how many pages are indexed that include the keyword within their website. If the site in the top 10 is an authority domain, it can rank from one keyword alone in the title tag, description tag or having the keyword in the body text (or any combination of these three metrics).
While most websites do not have that luxury, often dozens or hundreds of pages are required to cross the tipping point of co-occurrence for that keyword within the website and acquire a top ranking. However each keyword has a threshold which is going to vary depending on the unique metrics of each website (which is why you need to look at more than one site for evaluation).

4. Now that you know that your competitor's site contains Y amount of pages and X amount of those pages are dedicated to a specific keyword, you can go the the most relevant listing returned from their site and look at the off page factors (which means finding out how many backlinks are linking to that page). To do so, use Yahoo Site Explorer and type the specific URL in and look at the inlinks tab to see how many pages are linking to that page.

For example, if the homepage is returned as the top ranking result for the keyword using the competitorsite.com keyword search command, ignore it and look for an actual page that has a title, or relevant shingle with the keyword (in the title, URL or description).
If they targeted a keyword using a broad match method (which means it was not necessarily the objective, but their site acquired the ranking based on ambient factors, then you will only see a sparse mention of the keyword). The point being, the homepage is a catch all and will not provide you with the same amount of depth when attempting to data mine deep links from your competitors.

The idea is, you want to know (a) how many pages they have indexed (b) how many pages contain the keyword (c) how many deep links (how many links just to that page) the top ranking page has (from outside the site) as well as (d) how well the site in internally linked (for that keyword).
We can determine criteria a-c with simple search commands, and you can also determine if the site is treated as an authority based on the keywords that appear in bold when using the site: command, websites start transforming into authority sites through topical relevance after 200-300 pages are developed around a topic (if they are linked and optimized properly).

5. Crunch the numbers and assess the competitive landscape of the keyword in question.
For example, if you know that the top 3 sites all have an average of 1000 pages and out of those 1000 pages 50% or more of them contain the keyword in question and your site has 20 pages, then you are not being realistic with your ranking objectives.
I am not suggesting to go add 1000 pages overnight (as that would not be natural) but rather, start chipping away at the keyword using a variety of SEO tactics.

6. Check the allintitle, allintext and allinanchor thresholds for the selected competitors' sites. This means finding where they rank in Google (in the top 1,000 results before the results get obscured / redundant) using the following search operators.
allintitle:keyword (who has the highest occurrence of keyword in title)
allintext:keyword (who has the highest occurrence of keyword in their body text)
allinanchor:keyword (who has the highest occurrence of anchor text / links with this keyword)
Using Google again, you can look at the competitor's on page and off page metrics, instead of breaking them out individually, you can just use NicheWatch instead, or our Ultimate SEO Toolkit, to perform this function.

The Conclusion
SEO is only limited by your imagination when it comes to determining the extent of how you use tactics for discovery and analysis. We covered a few simple metrics using Google search operators above that allow you to isolate co-occurrence and determine the global keyword density for a site.
This does provide a preliminary analysis to at least let you know what your up against (qualifying a competitor or your domain to a keyword). If you reverse engineer the averages, you can find the tipping point for essentially any keyword and craft a plan of action to acquire it.
For example 1,000 pages indexed, 900 have the keyword in exact match and the main landing page has 50 inbound links from Page Rank 4 pages. Now you have a threshold to exceed. Although this is a preliminary method, sometimes looking at basic metrics such as these can provide an immense amount of insight and determine the next competitive threshold you target for analysis.

About The Author:
Jeffrey Smith is an active internet marketing optimization strategist, consultant and the founder of Seo Design Solutions Seo Company. He has actively been involved in internet marketing since 1995 and brings a wealth of collective experiences and fresh marketing strategies to individuals involved in online business.

Ten Ways to Use LinkedIn | Guy Kawasaki

Image via CrunchBase

by:Mario Sundar
When Guy Kawasaki blogged about the Ten Ways to Use LinkedIn, LinkedIn had 8.5 million users in 130 industries. Since then we’ve grown to over 12 million users covering 147 industries, but many of Guy’s suggestions on using LinkedIn (see below), still remain a great way for professionals to strengthen their online brand reputation and leverage their professional network. Happy Reading!

1. Increase your visibility.
   By adding connections, you increase the likelihood that people will see your profile first when they’re searching for someone to hire or do business with. In addition to appearing at the top of search results  (which is a major plus if you’re one of the 52,000 product managers on LinkedIn), people would much rather work with people who their friends know and trust.
   
2. Improve your connectability.
   Most new users put only their current company in their profile. By doing so, they severely limit their ability to connect with people. You should fill out your profile like it’s an executive bio, so include
   past companies, education, affiliations, and activities.
   You can also include a link to your profile as part of an email signature. The added benefit is that the link enables people to see all your credentials, which would be awkward if not downright strange, as
   an attachment.
   
3. Improve your Google PageRank.
   LinkedIn allows you to make your profile information available for search engines to index. Since LinkedIn profiles receive a fairly high PageRank in Google, this is a good way to influence what people see when they search for you.
   To do this, create a public profile and select “Full View.” Also, instead of using the default URL, customize your public profile’s URL to be your actual name. To strengthen the visibility of this page in
   search engines, use this link in various places on the web> For example, when you comment in a blog, include a link to your profile in your signature.
   
4. Enhance your search engine results.
   In addition to your name, you can also promote your blog or website to search engines like Google and Yahoo! Your LinkedIn profile allows you to publicize websites. There are a few pre-selected categories like “My Website,” “My Company,” etc.
   If you select “Other” you can modify the name of the link. If you’re linking to your personal blog, include your name or descriptive terms in the link, and voila! instant search-engine optimization for your site. To make this work, be sure your public profile setting is set to “Full View.”
   
5. Perform blind, “reverse,” and company reference checks.
   LinkedIn’s reference check tool to input a company name and the years the person worked at the company to search for references. Your search will find the people who worked at the company during the same time period. Since references provided by a candidate will generally be glowing, this is a good way to get more balanced data.
   Companies will typically check your references before hiring you, but have you ever thought of checking your prospective manager’s references? Most interviewees don’t have the audacity to ask a
   potential boss for references, but with LinkedIn you have a way to scope her out.
   You can also check up on the company itself by finding the person who used to have the job that you’re interviewing for. Do this by searching for job title and company, but be sure to uncheck “Current titles only.” By contacting people who used to hold the position, you can get the inside scoop on the job, manager and growth potential.
   By the way, if using LinkedIn in these ways becomes a common practice, we’re apt to see more truthful resumes. There’s nothing more amusing than to find out that the candidate who claims to have caused some huge success was a total bozo who was just along for the ride.
   
6. Increase the relevancy of your job search.
   Use LinkedIn’s advanced search to find people with educational and work experience like yours to see where they work. For example, a programmer would use search keywords such as “Ruby on Rails,” “C++,” “Python,” “Java,” and “evangelist” to find out where other programmers with these skills work.
   
7. Make your interview go smoother.
   You can use LinkedIn to find the people that you’re meeting. Knowing that you went to the same school, plays hockey, or shares acquaintances is a lot better than an awkward silence after, “I’m doing fine, thank you.”
   
8. Gauge the health of a company.
   Perform an advanced search for company name and uncheck the “Current Companies Only” box. This will enable you to scrutinize the rate of turnover and whether key people are abandoning ship. Former employees usually give more candid opinions about a company’s prospects than someone who’s still on board.
   
9. Gauge the health of an industry.
   If you’re thinking of investing or working in a sector, use LinkedIn to find people who worked for competitors—or even better, companies who failed. For example, suppose you wanted to build a next generation online pet store, you’d probably learn a lot from speaking with former Pets.com or WebVan employees.
   
10. Track startups.
    You can see people in your network who are initiating new startups by doing an advanced search for a range of keywords such as “stealth” or “new startup.” Apply the “Sort By” filter to “Degrees away from you” in order to see the people closest to you first. [Republished from: Ten Ways to Use LinkedIn via "How to Change the World" -- Guy Kawasaki's blog]
    

Guy is known for bringing the concept of technology evangelism to the masses through his pioneering work at Apple’s Macintosh group. In addition to blogging the above post earlier this year, he has also blogged on topics ranging from LinkedIn Profile Extreme Makeover to LinkedIn and the Art of avoiding an Asshole Boss. [Source: Guy Kawasaki's weblog -- How to Change the World]