Almost all domainers are interested in keyword frequencies appearing in different contexts. One way to valuate domain name, for example, is to obtain Google Search Results Count for phrase in domain name. I am interested particularly in connection to algorithms for domain name splitting in compound words and appraisal methodologies.

There are some well known word frequency lists:

• Wikipedia http://en.wiktionary.org/wiki/Wiktionary:Frequency_lists
• Google 1 trillion tokens: Web 1T 5-Gram
• BNC: ftp://ftp.itri.bton.ac.uk/bnc/all.num.o5 (Large file!)

I decided to create my own list of words and associated frequences based on all articles that are in the English version of Wikipedia.

Wikipedia is HUGE. Only the English part is 21GB in XML format. It takes a 5h to parse entire file and extract statistics for all tokens that looks like a word.

Some statistics:

• Total tokens (words, no numbers): 1,570,455,731
• Unique tokens (words, no numbers): 5,800,280

If you know what a Zipf distribution is, I created a chart of all words in log/log scale, just to observe later that the same chart can be seen here.

Wikipedia Words Frequency List

The chart can be divided on four parts:

• Rank(1-50)  Count(86M-3M) Examples(the, of, and, in, to, a, is) Words that are stop words.
• Rank(51-3K) Count(2.4M-56K) Examples(university, January, tea, sharp) Words form the  “core” of the English dictionary — words that are most frequently used.
• Rank(3K-200K) Count(56K-118) Examples(officiates, polytonality, neoligism) Words that can be found in some large and comprehensive dictionaries (above rank 50K are mostly Long Tail words)
• Rank(200K-5.8M) Count(117-1) Examples(euprosthenops, eurotrochilus, lokottaravada) Terms from obscure niches, misspelled words, transliterated words from other languages, new words  and “not words at all”

Google study shows that there are 14M one word and 315M two word phrases (bigrams). Currently I have no plans to extract two words phrases due to their large number, but it is interesting to analyze them in context of two words domain names.

Technical info

The process of extracting all words and counting them is not an easy task:

• Download a copy of Wikipedia. I used version dumped in XML format.
• Write parser to extract text from <title> and <text> tags.
• Wikipedia uses its own markup language. Write parser to extract all data from markup language and filter-out some unnecessary parts.
• Filter out numbers, special characters.
• Tokenize.
• Collect useful statistics.

The good news is that Wikipedia is much clean and organized then the rest of the web.

Some selected words and associated counts:

• Google  197920
• Twitter 894
• domain  111850
• domainer 22
• Wikipedia 3226237
• Wiki    176827
• Obama   22941
• Oprah   3885
• Moniker 4974
• GoDaddy 228

I am developing algorithm for keywords detection in domain names. Thanks to Jeff at inforum.in I obtained a copy of .IN zone file (from Feb. 09) and decided to analyze words distribution in it. Keywords splitting is not easy task. Some problems I am facing are:

• Keywords are from different languages. I use English dictionary with 100K words and this statistics is only for domains that seems to be composed of English words. There are lot of Indian proper names that are not in the dictionary. If domain name contain proper name or other unrecognizable part and English part, then only English part is included in final statistics. Numbers in domains are not counted in statistics.
• Most domain names can be slitted in different ways. My approach splits the names in all possible ways and choose one by some heuristic but more accurate approach can involve a statistical methods for keywords co-occurrence.
• Some domains can’t be splitted by humans too.
• Some domains are intentionally misspelled and I do not use algorithm that detects misspelled word variants.

Some statistics:

• Total number of domains: 315K
• Number of domains that are splitted: 175K (numeric domains only are not counted)
• Total number of keywords detected: 366K or 29K unique.

Following is the first 300 keywords sorted by frequency. As expected a lot of “skip words” appear at the top.

Domain length distribution is shown on the next chart. It is based on all domains in zone file.

Domains Length distribution

There are some names with the maximum size. It is really difficult to me to understand why some people register such names like:

“141592653589793238462643383279502884197169399375105820974944592.in” – this seems to be π after the dot e.g 3.141592…

“1i1i1ii1iii1i1i1ii1i1iii1i1i11i11i1i1i1ii1i1i1ii1i1ii1i1i1i1i1i.in” or

“angles-channels-beams-sections-structural-steels-exports-mumbai.in”  

Next chart shows distribution of lengths of detected keywords.

Keywords Length Distribution

It is all for now. I will try to update this post if some new statistics is produced. Also, some work is underway to collect similar statistics in .com and .net zone files.

Following chart shows average price dependence of name length for different types of major extensions (TLD-s). I am working on more robust estimator for average price then simple mean that should give more smooth chart curves. For same combinations of extensions and length, sample is small and resulting point is inaccurate (at some very long names).

Domain name price vs length (for TLDs).

Note: price is in logarithmic scale.

Today I am studying how hyphens and numbers affect domain name price. It is wide spread story that domain names containing dashes and numbers are of lower value. This comparison will be done by calculation of average price of domains with similar characteristics (domain length). Statistics is on .com only. Following chart shows the price versus domain length in logarithmic scale. I have very small sample of domains with hyphens + numbers and associated line is not very reliable (purple one).

Prices of domains with hyphens and numbers

As we expected, domains without numbers and hyphens are of higher value. There is evidence that  domains with numbers are more valuable then domains with hyphens for lengths below 12 characters and in reverse for longer names.  These are just general observations  over a distribution of prices with big variation and not a rule for concrete domain valuation.

Sedo published new market study for 2008.

PDF: http://www.sedo.com/press/domainmarketstudy2008-us.pdf

The data presented is interesting and seems to be in conformance of my observations and data presented in previous posts. I skipped a chart before that presents domain sales for each day of the year that is comparable to chart “Domain Sales per Quarter” in this report. Also the shown trend that .DE is one of the most traded ccTLD-s is observation that seems to be true. SEDO have access to bigger sales database and more accurate statistics.

Domain Sales per day

In previous post I charted the number of domain names traded vs domain length. In this post I would like to examine more deeply this relation. This time i used more comprehensive database that cover sales data for the past year to date. It contains data for different TLD-s and .COM is, as we expect, most traded TLD followed by .NET .DE .ORG etc. I observed that there is strong peak for domains with length 4 that are most intensely traded (2-6 times more then other lengths). Following charts also examine this relation in other TLD-s.

Count of domain names vs domain length (.com)

Logarithm of Prices vs Domain length (.com)

We can see that .net and .org TLD-s have similar peaks but for .net 3 and 4 letter domain names have maximum and .org have maximum only at 3 letter names.

Count of domain names vs length (.net .org)

Other TLD-s have no peak at 4 letter domains and some small at 3 letter. Also, .uk domain names seems to have no peak at small lengths and just a general bell curve.

Count of domain names vs length (.de .uk. eu)

Markets are always moving and TLD-s other then .com are following its behaviour. My predictions are that at some point in the future .de .uk .eu (others?) TLD-s will have strong peak at 4 and probably .COM at 5 as interest to domains with length of 5 increases.

Just a quick statistics extracted from Domain Names database. It is based on (1565  .COM only) sold domains list for the last 4 months (Oct 08, Jan 09). Some domain names contain dashes and numbers. Last column in table is average price for given name length.

Chart shows this dependence in logarithmic price scale.

Log Domain Name price dependence of name length

Next chart shows domain sales by size. It seems that 4 character domains are most traded at the moment.

Domain sales by name length

Fig. 1 Distribution of words length.

In this article I would like to focus on quantitative factors that form the base for domain name price estimation and particularly the number of characters and number of words in domain name. It is well known that the less characters and words in the name, the more desirable it is.

Simple questions that I want to answer are:

• How many domain names can be generated with fixed length and arbitrary combination of characters?
• How many domain names can be generated that are combination of 1, 2, 3 or more dictionary words?

For this study I have compiled a dictionary of US and UK English words (totally 101,930 words and names) . I will not estimate domain names that contain words from other languages. Distribution of words length is shown on Fig.1.

It is easy to count number of all possible domain names with letters “a…z” (see table below). I will omit numbers and “-” for now.

Notes:

• The numbers are just estimates. They dependent from dictionary size and especially the number of small words.
• Some domain names can be split in words in multiple ways. Famous example is “who represents” and “whore presents”.
• Not all words combinations create meaningful domain names, most of them not.

Work in progress …

Domain Name Appraisal is a process of price estimation of given domain name. This estimation is based on quantitative factors as well as subjective ones.

For example, if the domain name contains words and phrases that have meaning in some language and that words are desirable by multiple participants in domain market, estimated price is higher. Other factors that attract potential domain name buyers (and increase price) are the size (in terms of words count and characters count) of domain name.

Domains can be classified in given category (news, finance, business, etc.). Some categories are more profitable for domain owners then other and prices of domain names in given category are higher. There are a lot of subjective factors (people can like or dislike some names but can’t give explanation why) that are difficult to be quantitatively estimated.

Domain markets are not still, domain prices change due to economics conditions and buzz of given words and phrases. Because the given domain name is unique its value to domain market participants is also unique.

The best, one can do, is to quantitatively estimate “distance” between given domain name and domain names sold on auctions (actually sold domains prices, not desired prices) and based on this distance measure to price this domain accordingly. This measure can be estimated in the space of chars count, words count, links to domain, counts in search engines and news for this phrase, etc… Producing accurate model for domain name appraisal is possible with domain market experience and usage of robust and accurate statistical methods.