Search Engine Terminology

Algorithms
Crawler-based search engines determine the relevancy of hundreds of millions of web pages by following a set of rules, known as an algorithm. Exactly how a particular search engine's algorithm works is a closely-kept trade secret.

Backup & Other Results
Most search engines where the main results come from crawling the web will also provide human-powered "directory" results.

For search engines where the main results come from human work , it's common for them to have a "backup" partnership with a crawler-based search engine.

Blocking the Robots
The robots.txt file is a specially formatted file that webmasters use to indicate (to visiting search engine robots/spiders) which parts of their site should not be visited by the robot.

Boolean Query
This is a search query using the Boolean operators, AND, OR, and NOT, and parentheses to construct a complex condition from simpler criteria.

Categorical Search
A categorical search is simply a keyword or phrase entered on a directory while making a search. The resultant page will attempt to find the correct category for your search.

Crawlers or Spiders
Crawler-based search engines automatically visit website pages and categorize the information contained in the website to compile their listings. They send out ‘spiders', robots or bots, and follow links. Good links pointing at your web site enable the crawlers to find and include your pages, often resulting in high ranking on search engines.

Deep Crawl
All crawler-based search engines will find pages to add to their indeces, even if those pages have never been submitted to them. Some crawlers will "deep crawl" and gather many pages from your website.

The larger a search engine's index is, the more likely it will list many pages per website.

Different Results for the same search
The same search on different search engines produces different results. Some search engines index more web pages than others; some index web pages more often than others and some penalize pages or exclude them from the index if they detect search engine "spamming.

The result is that no search engine has the exact same collection of web pages to search through.

Directories
Search Engine Directories are compiled by humans, tend to be smaller than search engine databases, and typically index only the home page or top level pages of a site. Directories are organised hierarchically into browsable subject categories and sub-categories, similar to the Yellow Pages.

There are general directories, academic directories, commercial directories, portals and vortals.

 Note:
Most subject directories have partnered with search engines to query their databases and search the web for additional sources, while search engines are acquiring subject directories or creating their own.

Directory Submission
Directory submission means getting your homepage listed in organized, categorized listings of Web sites. Web directories are compiled by human reviewers and if your site meets their criteria, that person will then decide which category your site ‘fits'.

Engine
A search engine is the software that collates, indexes, sorts and provides listings from billions of web pages on the internet, based on the search term (keywords) used. There are hundreds of search engines, each with their own criteria and algorithms.

Frames Support
Search engines have a tough time with frames. Using frames either prevents them from finding pages within a web site, or it causes them to send visitors into a site without the proper frame "context" being established. A typical message seen if you select a website using frames is:

“Sorry! You need a frames-browser to view this site”

 Discover how to Avoid Search Engine Stumbling Blocks such as frames.

Full Body Text
All the major search engines say they index the full visible body text of a page, though some will not index stop words or exclude copy deemed to be spam. Google generally does not index past the first 101K of long HTML pages.

Invisible Web
There is a large portion of the Web ( 60 to 80 percent of existing Web material ) that search engine spiders cannot, or may not, index.

Called the "Invisible Web" or the "Deep Web" this includes password-protected sites, documents behind firewalls, archived material, interactive tools, and the contents of certain databases. Much of this information isn't static, but assembled dynamically in response to specific queries.

 Find out more about the Invisible Web

Keywords and Search Engines
Here are some terms used when optimising web pages for search engines:

•  Keyword : refers to any word the user might search for in order to find a website or a page. The plural form of the word is treated like a separate keyword.

•  Key phrase : are combinations of keywords describing web page content that a typical web searcher may use when searching.

•  Keyword frequency : is how often a keyword appears on a page and in specific areas on a page. The more times a keyword is included, the more relevant the page to the search. But if the keyword appears too many times this will be classed as search engine spamming.

•  Keyword placement : refers to the many areas on a web page that a search engine looks for data, and includes the page title, heading, body text as well as link text and comment tags.

•  Keyword Prominence : is how close to the start of the page content that the keyword appears. In general, a keyword that appears closer to the top of the page or area will be more relevant to some search engines.

•  Keyword weight :  is simply a factor in percentage form of a keyword or phrase in relation to all other words on a page.

•  Total words : this is a count of the words in an area on a page, excluding HTML tags. Some search engines rank pages more favorably if they have a certain number of words on the page.

Library Gateways
Library gateways are collections of databases and informational sites, arranged by subject, that have been assembled, reviewed and recommended by specialists, usually librarians. These gateway collections support research and reference needs by identifying and pointing to recommended, academically-oriented pages on the Web.

They are good sources for direct links to database information stored on the "Invisible Web."

Link Analysis/Site Popularity
Every major search engine uses link analysis/site popularity as part of their ranking algorithms. The number, relevance and quality of inbound links to a website are very important factors for ranking/positioning in search engine listings. Google will not list a site with no inbound links.

Listings
Search engine listings are the report or result of an engine or directory search, usually 10 to 20 per page. The position that websites appear in the listing is the 'rank' of that page on that search engine.

Location/Frequency
One of the main rules in a ranking algorithm involves the location and frequency of keywords on a web page.

Search engines check for search keywords appearing near the top of a web page, such as in the headline or in the first few paragraphs of text. They assume that any page relevant to the topic will mention those words right from the beginning.

Frequency is the other major factor in how search engines determine relevancy. A search engine will analyze how often keywords appear in relation to other words in a web page. Those with a higher frequency are often deemed more relevant than other web pages.

Manual submission
For the best possible chance of success submit to search engines manually. A automated submission may actually harm your position because search engines penalise sites submitted by software.

Off The Page Factors
All major search engines now also make use of "off the page" ranking criteria. Off the page factors are those that a webmasters cannot easily influence. These include:-

• Link analysis : By analyzing how pages link to each other, a search engine can both determine what a page is about and whether that page is deemed to be "important" and thus deserving of a ranking boost.

Optimization
This refers to the act of altering your site so that it may rank well for particular search terms, especially with crawler-based search engines.

Paid Results Every major search engine has paid listings that are also presented alongside its editorial results. Examples are:

• Paid Inclusion: Search engines offer programs where you can pay to guarantee that your pages will be included in its index.

Query Expansion
Query expansion is the process of a search engine adding search terms to a user's weighted search. The intent is to improve precision and/or recall. The additional terms may be taken from a thesaurus. For example a search for "car" may be expanded to: car cars auto autos automobile automobiles.

The additional terms may also be taken from documents that the user has specified as being relevant; this is the basis for the "more like this" feature of some search engines.

Results
Search engine main results are obtained from crawlers, human-powered directories and paid listings. Many search engines have alliances with third-party search providers for their results.

Search Engine Marketing or Search Engine Promotion
Both terms refer to the overall process of marketing a site on search engines. This includes submission, optimization, managing paid listings and more.

These terms also highlight the fact that doing well with search engines is not just about submitting right, optimizing well or getting a good rank for a particular term. It's about the overall job of improving how your site interacts with search engines, so that the audience you seek can find you.

Search engine Placement, Search Engine Positioning and Search Engine Ranking
These terms refer to a site actually doing well for particular search terms or a range of terms at search engines. To get that "top ten" ranking for a particular keyword or search term is the ultimate goal for many people.

Search Engine Submission
The act of supplying a URL to a search engine in an attempt to make a search engine aware of a site or page is called search engine submission or search engine registration. Getting listed does not mean that you will necessarily rank well for particular terms. It simply means that the search engine knows your pages exist.

Search Term
A search term is an element of a search or query. A search term is the basic building block of a Boolean search or a weighted search. In a search engine a search term is typically a word, phrase, or pattern match expression. For example: cosmonaut or "space travel" or astronaut*

In a database a term is typically the comparison of a column with a constant or with another column. For example: last_name like 'Smith%'

Spy
A search engine spy is a perpetually refreshing page that provides a real-time view of actual Web searches. Search engine spies come in filtered and unfiltered varieties. Filtered services attempt to remove "objectionable" searches. Some spy services only offer one version and some offer both so you can choose.

Stop Words
Some search engines either leave out words when they index a page or may not search for these words during a query. These extremely common words (the, and, a, etc) are excluded as a way to save storage space or to speed searches.

Stumbling Blocks
These are the things that can prevent your website being found, listed or even penalised by the search engines.

 Learn more about Search Engine Stumbling Blocks

Tags for Search Engines
There are very important tags which most search engine spiders are programmed to find on web pages:

• Title Tag: this META/HTML tag contains the page title, which should be determined by the contents of the page.

Vortals/Subject-Specific Databases
Subject-specific databases, or vortals (i.e., "vertical portals") are databases devoted to a single subject, created by professors, researchers, experts, governmental agencies, business interests, and other subject specialists and/or individuals who have a deep interest in, and professional knowledge of, a particular field and have accumulated information and data about it.

They are good sources for direct links to database information stored on the "Invisible Web."

Weighted Search
A weighted search is a search based on frequencies of the search terms in the documents being searched, and is often used by search engines.

It produces a numerical score for each possible document. A document's score depends on the frequency of each search term in that document compared with the overall frequency of that term in the entire corpus of documents.

Inverse document frequency means the more documents a term appears in, the less important the term is.

A simple weighted search is just a list of search terms, for example: car automobile.

Back