A Fundamental Scale of Descriptions for Analyzing Information Content of Communication Systems

Abstract

The complexity of the description of a system is a function of the entropy of its symbolic description. Prior to computing the entropy of the system’s description, an observation scale has to be assumed. In texts written in artificial and natural languages, typical scales are binary, characters, and words. However, considering languages as structures built around certain preconceived set of symbols, like words or characters, limits the level of complexity that can be revealed analytically. This study introduces the notion of the fundamental description scale to analyze the essence of the structure of a language. The concept of Fundamental Scale is tested for English and musical instrument digital interface (MIDI) music texts using an algorithm developed to split a text in a collection of sets of symbols that minimizes the observed entropy of the system. This Fundamental Scale reflects more details of the complexity of the language than using bits, characters or words. Results show that this Fundamental Scale allows to compare completely different languages, such as English and MIDI coded music regarding its structural entropy. This comparative power facilitates the study of the complexity of the structure of different communication systems.

1. Introduction

The understanding of systems and their complexity requires accounting for their entropy. The emergence of information upon the scale of observation has become a topic of discussion since it reveals much of the systems’ nature and structure. Bar Yam [1] and Bar-Yam et al. [2] have proposed the concept of complexity profile as a useful tool to study systems at different scales. Among others, Lopez-Ruiz et al. [3], and Prokopenko [4] focus on the change of the balance between the system disorder and self-organization for different scales of observation. In a different approach, Gell-Mann [5] considers complexity as a property associated to the irregularities of the physical system. But Gell-Mann sees both randomness and order as manifestations of regularity, and therefore quantities that offer the possibility for reducing the length of a description and hence the computed complexity of a system.

These complexity concepts are all evaluated using arbitrarily selected symbol scales. The selected observation scale depends on the communication system used in the description; for example, systems described with human natural languages are prone to be analyzed with the characters and words scales because they hold the most meaning for humans. When the analysis of information is in the context of its transmission, it is common to find binary codes as the base of study. A possible consequence of this preselected scale of observation is the possible inclusion of our assumptions about the system’s structure, which skews our interpretation about system properties.

Many studies have evaluated the entropy of descriptions based on a preconceived scale; in 1997 Kontoyiannis [6] evaluated the description entropies at the scale of characters; in 2002 Montemurro and Zanette [7] studied the entropy as a function of the word-role; more recently Savoy [8], and Febres et al. [9,10] have studied the impact of the style of writing over entropy speeches using the word as the unit of the scale. In 2009 Piasecki and Plastino [10] studied entropy as a function of a 2-dimensional domain. They explored the effects of multivariate distributions and calculate the entropy associated to several 2D patterns. All these studies share the same direction; assume a space for a domain and a scale and compute the entropy. The strategy of the present study is to set the same problem in a reversed fashion: given an entropy descriptor of a multivariate distribution defined for some domain space, what would be the best way to segment that domain space in order to reproduce the known entropy descriptor? The answer to this question would have a twofold value: (a) an indication to the scale that best represents the system expression as the distribution of sizes of the space segments, and (b) an approximation to the algorithmic complexity of the description.

Algorithmic complexity as a concept does not consider the observation scale [5,11]. Algorithmic complexity―also called Kolmogorov’s complexity―is the length of the shortest string that completely describes a system. Since the shortest string is a characteristic impossible to guarantee, algorithmic complexity has been regarded as an unreachable figure. Nevertheless, estimating complexity by searching for a nearly uncompressible description of a system, would have the advantage of being independent of the observation scale. In fact, a method to search for a nearly uncompressible description could be achieved by adjusting the observation scale until the process discovers the scale that best comprises the original description. The result would lead to an approximation to the algorithmic complexity of the system.

While these previous studies assume symbols as characters or words, in our present study we leave freedom to group adjacent characters, to form symbols in order to comply with a higher hierarchy criterion, as is the minimization of the entropy. This study develops a series of algorithms to recognize the set of symbols that, according to their frequency, leads to a minimum entropy description. The method developed in this study mimics a simplified communication system’s evolution process. The proposed algorithm is tested with short example of English text, and two descriptions, the first is an English text and the second, a sound musical instrument digital interface (MIDI) file. This representation of the components may convey a description of a system and its structural essence.

2. A Quantitative Description of a Communication System

A version of Shannon’s entropy formula, generalized for communication systems comprised of D symbols, is used to compute quantity of information in a descriptive text. To determine the symbols that make up the sequential text, a group of algorithms were developed. These algorithms are capable of recognizing the set of symbols which form the language used in the textual description. The number of symbols D represents not only the diversity of the language but also the fundamental scale used for the system description.

2.1. Quantity of Information for a D’nary Communication System

We refer to language as the set of symbols used to construct a written message. The number of different symbols in a language will be referred as the diversity D.

To compute the entropy h of a language, that is, the entropy of the set of D different symbols, used with a probability p_i to form a written message, we use the Shannon’s entropy expression, normalized to produce values between zero and one:

$$h=-{\displaystyle \sum }_D^{i=1}{p}_i·{\log }_D{p}_i,$$

(1)

Note that the base of the logarithm is equal to the language’s diversity D, whereas classical Shannon’s expression uses 2 as the base of the logarithm; also equal to the diversity of the binary language that he studied. Researchers such as Zipf [12], Kirby [13], Kontoyiannis [6], Gelbukh and Sidorov [14], Montemurro and Zanette [7], Savoy [15], Febres, Jaffe and Gershenson [9] and Febres and Jaffe [16], among others, have studied the relationship between the structure of some human and artificial languages, and the symbol probability distribution corresponding to written expressions of each type of language.

All these studies assume symbols as characters or words, in our present study we leave freedom to group adjacent characters, to form symbols in order to comply with the minimization of the entropy h as expressed in Equation (1). In the following sections we explain this optimization problem, and our approach to find a solution reasonably close to the set of symbols that produce an absolute minimum entropy.

2.2. Scale and Resolution

We propose a quantitative concept of scale: the scale of a system equals the diversity of the language used for its description. Thus, for example, if a picture is made with all available colors in an 8-bit-color map of pixels, then the diversity of the color language of the picture would equal 2⁸, and the scale of the picture description, considering each color as a symbol, would be also 2⁸. Another example would be a binary language, a scale 2 communication system made up of only two symbols. Notice we have used the term “communication system” to refer to the media used to code information.

Interestingly, the system’s description scale is determined, in first place, by the observer, and in a much smaller degree by the system itself. The presumably high complexity of a system, functioning with the actions and reactions of a large number of tiny pieces, simply dissipates if (a) the observer or the describer fails to see the details, (b) the observer or describer is not interested the details, and prefers to focus on the macroscopic interactions that regulate the whole system’s behavior, or (c) the system does not have sufficient different components, which play the role of symbols here, to refer to each type of piece. It is clear that any observed system scale implies the use of a certain number of symbols. It is also clear that the number of different symbols used in a description is linked with our intuitive idea of scale. There being no other known quantitative meaning of the word scale, we suggest its use as a descriptor of languages by specifying the number of symbols forming them.

Resolution specifies the maximum accuracy of observation and defines the smallest observable piece of information. In the computer coded files we used to interpret descriptions, we consider the character as the smallest observable and non-divisible piece of information.

Let E denote the physical space that a symbol or a character occupies, and let the sub-index signal the object being referred to. Thus, considering a written message M, constructed using D_M different symbols Y as $\mathit{M}=\{Y_1,Y_2,\dots ,Y_{D_{\mathit{M}}}\}$, we would say the message M occupies the space E_M and each symbol Y_i occupies the space $E_{Y_i}$. We define the length of all characters equal one. Therefore $E_{C_i}\equiv 1$ for any i. Finally, if the number of characters in a message is N, each symbol Y_i appears $F_{Y_i}$ times within the message, and the symbol diversity is D_M, we can write the following constraints over the number of characters, symbols and the space they occupy:

$$E_{\mathit{M}}={\displaystyle \sum }_{D_{\mathit{M}}}^{i=1}{F}_{Y_i}\cdot E_{Y_i}={\displaystyle \sum }_N^{i=1}{E}_{C_i}=N$$

(2)

2.3. Looking for a Proper Language Scale

We see the scale of a language as the set of finite symbols that “best” serves to represent a written message. The qualification “best” refers to the capacity of the set of symbols to convey the message with precision in the most effective way.

Take for example the western natural languages. Among their alphabets, there are only minor differences; too few differences to explain how far from each other those languages are. As Newman [17] observes, some letters may be the basic units of a language, but there are other units formed by groups of letters.

Chomsky’s syntactic structures [18], later called context-free grammar (CFG) [19] offers another representation of natural language structure. The CFG describes rules for the proper connections among words according to their specific function within the text. Thus, CFG is a grammar generator useful to study the structure of sentences. Chomsky himself treats a language as an infinite or finite set of sentences. CFG works at a much larger scale than the one we are looking for in this study.

Regarding natural languages it is common to think that a word is the group of characters within a leading and a trailing blank-space. At some time a meaning was assigned to that word, and thereafter the word’s meaning, as well as its writing, evolves and adopts a shape that works fine for us, the users of that language. Zipf’s principle of least effort [14] and Flesch’s reading ease score [20] certainly give indications about the mechanisms guiding words, as written symbols, to reduce the number of characters needed to be represented.

From a quantitative linguistics perspective, this widely accepted method for recognizing words offers limited applicability. Punctuation signs, for example, have a very precise meaning and use. The frequency of their appearance in any western natural language compete with the most common words in English and Spanish [21]. However, punctuation signs are very seldom preceded by a blank-space and are normally written with just a single character, which promotes the false idea that they function like letters from the alphabet; they do not. They have meaning as well as common words have. Another situation revealing the inconvenience of this natural but too rigid conception of words, is the English contraction when using the apostrophe. It is difficult to count the number of words in the expression “they’re”. How many words are there, one or two? See Febres et al. [21] for a detailed explanation on English and Spanish word recognition and treatment for quantification purposes.

Intuitively the symbols forming a description written using some language, should be those driving the whole message to low entropy when computed as the function of the symbols frequency. In this situation the message is fixed as fixed is also the text and the quantity of information it conveys. Then, there appears to be a conflict: while the information is constant because the message is invariant, any change to the set of symbols considered as basic units, alters the computed message entropy, as if the information had changed; it has not. To solve this paradox, we return to the question asked at the beginning of this section about the meaning of “best” in the context of this discussion. From the point of view of the message emitter, the term “best” considers the efficiency to transmit an idea. This is what Shannon’s work was intended for: to determine the amount of information, estimated as entropy, needed to transmit an idea. From the reader’s point of view the economy of the problem works different. The reader’s problem is to interpret the message received to maximize the information extracted. In other words, the reader focuses on the symbols which turn the script as an organized, and therefore easier to interpret message. If the reader is a human and there are words in the message, the focused symbols are most likely words because those are the symbols that add meaning for this kind of reader. But if there existed the possibility to select another set of symbols which makes the message look even more organized, the reader would rather use this set of symbols because it would require less effort to read.

In conclusion, what the reader considers “best” is the set of symbols that maximizes the organization of the message while for the sender the “best” means the set of symbols needed to minimize the disorder of the message and thus the quantity of information processed. These statements are expressed as objective functions in Equations (3) where the best set of symbols is named B, the message is M, the message entopy is h_M and the message organization is (1−h_M):

$$\begin{array}{l}\text{Senders’ objective}:\underset{\mathit{B}}{\min }{h}_{\mathit{M}}\\ \text{Receiver’s objective}:\underset{\mathit{B}}{\max }\left(1-h_{\mathit{M}}\right)=\underset{\mathit{B}}{\min }{h}_{\mathit{M}}\end{array}$$

(3)

Following this reasoning, “best” means the same for both sides of the communication process. This may have important implications when considering languages as living organisms or colonies of organisms. Both parts of the communication process push the language to evolve in the same direction: augmenting self-organization and the reducing of entropy of the messages. Both come together. Self-organization can be seen as one of the evolving directions of languages. Thus, self-organization is an indirect way to measure how deeply evolved a language is and what its capacity is to convey complex ideas or sensations. Finally, an objective function to search the most effective set of symbols―the set with minimal entropy―to describe a language has been found. It will be used to recognize the set of symbols that best describes a language used to write a description.

2.4. Language Recognition

Consider a description consisting of a message M built up with a sequence of N characters or elementary symbols. The message M can be treated as an ordered set of characters C_i as:

$$\mathit{M}=\{C_1,C_2,\dots ,C_N\}.$$

(4)

No restriction is imposed over the possibility of repeating characters. Consider also the language B, consisting of a set of D_B different symbols Y_i, each formed with a sequence of $E_{Y_i}$ consecutive characters found with probability $P(Y_i)>0$ in message M. Thus:

$$\mathit{B}=\{Y_1,Y_2,\dots ,Y_{D_B},P(Y_i)\}.$$

(5)

$$Y_i=C_j,C_{j+1},\dots ,C_{j+E_{Y_i}-1}\},1\le i\le D_{\mathit{B}},1\le j\le N-E_{Y_i}+1.$$

(6)

The symbol probability distribution P(Y_i) can be obtained dividing the frequency distribution f_i by the total number of symbols N in the message:

$$P\left(Y_i\right)=\frac{f_i}{N}$$

(7)

Language B, used to convey the message M, can now be specified as the set of D_B different symbols and the probability density function P(Y_i) which establishes the relative frequencies of appearance of the symbols Y_i. Each symbol Y_i is constructed with a sequence of contiguous characters as indicated in Equation (6). The set of symbols that describes the message M with the least entropy comes after the solution of the following optimization problem:

$$\begin{array}{l}\underset{B}{\min }-{\displaystyle \sum }_{D_{\mathit{B}}}^{i=1}\frac{F_{Y_i}\cdot E_{Y_i}}{N}·{\log }_{D_{\mathit{B}}}\frac{F_{Y_i}\cdot E_{Y_i}}{N},\\ \text{Subject to}:\mathit{B}=\left\{Y_1,Y_2,\dots Y_i\dots ,Y_{D_B},\mathit{P}\left(Y_i\right)\right\},\text{for}i=1,2,\dots ,D_{\mathit{B}}\\ Y_i=\left\{C_j,C_{j+1},\dots ,C_{j+E_{Y_i}-1}\right\},\text{for}i=1,2,\dots ,D_{\mathit{B}}\mathrm{and}j=1,2,\dots N-E_{Y_i}+1,\\ {\sum }_{i=1}^{D_B}{F}_{Y_i}\cdot E_{Y_i}=N,\\ F_{Y_i}\ge 1,E_{Y_i}\ge 1,\mathrm{for}i=1,2,3,\dots ,D_{\mathit{B}}.\end{array}$$

(8)

The resulting language will be the best in the sense that it is the set of symbols that offers a maximum organization of the message. The symbol lengths will range from a minimum to a maximum defining a distribution of symbol lengths characteristic of this scale of observation which is referred to as the Fundamental Scale.

3. The Algorithm

The optimization problem (8) is highly nonlinear and restrictions are coupled. A strategy for finding a solution has been devised. It is a computerized process compound of text-strings processing, entropy calculations, text-symbol ordering and genetic algorithms. Given a description consisting of a text of N characters, the purpose of the algorithm is to build a set of symbols B whose entropy is close to a minimum. The process forms symbols by joining as many as V adjacent characters in the text. A loop where V is kept constant, controls the size of the symbols being incorporated to language B. The process ends when the maximum symbol length of V_mx characters is considered to form symbols. We add a sub-index to language B_V to indicate the symbol size V considered at each stage of its construction. We have defined several sections of the algorithm and we named them according to their similarity with a system where each symbol appears and ends up being part of a language, only if it survives the competence it must stand against other symbols. A pseudo-code of the fundamental scale algorithm is included in Appendix A.

3.1. Base Language Construction

In the first stage, the message M is separated into single characters. The resulting set of characters along with their frequency distribution constitute the first attempt to obtain a good language and it will be denoted as B₁. The sub-index indicates the maximum length that any symbol can achieve.

3.2. Prospective Symbol Detection

The prospective symbol detection consists of scanning the text looking for strings of exactly V characters. All V-long strings are considered as prospective symbols to join the previously constructed language B_V−1 made of strings of up to V−1 characters. The idea is to find all possible different V-long strings present in the message M, which after complying with some entropy reduction criteria, would complement language B_V−1 to form language B_V.

To cover all possibilities of character sequences forming symbols of length equal to V, several passes are done over the text. The difference from one pass to another is the character where the initial symbol starts, which will be called the phase of the pass. Figure 2 illustrates how the strategy covers all possibilities of symbol instances for any symbol size specification V.

3.3. Symbol Birth Process

Prospective Symbols detected in the previous stage whose likelihood to be an entropy reducer symbol is presumed too low, are discarded and never inserted as part of the language. Interpreting entropy Equation (1) as the summation of contributions of the uncertainty due to each symbol, we can intuit that minimum total uncertainty―minimum entropy―occurs when each symbol uncertainty contribution is about the same. Thus, any Prospective Symbol must be close to the average uncertainty per symbol in order to have some opportunity to actually reduce the entropy after its insertion. The average contribution of the uncertainty u_i for symbol i can be estimated as:

$$u_i=-p_i{\log }_{D_{{\mathit{B}}_V}}{p}_i=\frac{h}{D_{{\mathit{B}}_V}},$$

(9)

This leads us to look for symbols complying with condition shown in Equation (10), and save processing time whenever a prospective symbol is not within a 2λ-width band of around the average uncertainty value:

$$\frac{h}{D_{{\mathit{B}}_V}}-\mathrm{\lambda }<u_i<\frac{h}{D_{{\mathit{B}}_V}}+\mathrm{\lambda }.$$

(10)

Parameter λ can be adjusted to avoid improperly rejecting entropy reducer symbols or to operate in the safe side at the expense on processing time.

3.4. Conservation of Symbolic Quantity

The inclusion of prospective symbols into the arrays of symbols representing the language B, is performed to avoid the overlap of the newly inserted symbols and the previous language existing symbols. Therefore, every time a prospective symbol is inserted into the stack of symbols, the instances of former symbols occupying the space the new symbols, must be released. Sometimes this freed string is only a fraction of a previously existing symbol. Thus, the insertion of a symbol may produce a break up of other symbols, generating empty spaces for which recovered symbols must be reinserted in order to keep the original text intact.

3.5. Symbol Survival Process

A final calculation is performed to confirm the entropy reduction achieved after the insertion of a symbol into the language being formed. Those symbols not producing an entropy reduction, are rejected and the Language B is reverted to its condition prior to the last insertion of a symbol.

3.6. Controlling Computational Complexity

The computational complexity of this algorithm is far beyond polynomial. A rough estimation sets the number of steps required above the factorial of the diversity of the language treated. Thus, segmenting the message into shorter pieces, allows the algorithm to find a feasible solution and to keep affordable processing times for large texts. This strategy is in fact a sort of parallel processing which significantly reduces the algorithm’s computational complexity down to becoming an applicable tool. A complex system software platform has been developed along with this study to deal with the complexities of this algorithm, and the structure needed to maintain record of every symbol of each description within a core of very many texts. This experimental software, is named Monet and a brief description of it can be found in [21].

The noise introduced when cutting the original description in pieces, is limited. At most two symbols may be fractured for each segment. Very low compared to the number of symbols making each segment. The algorithm calculates the entropy of each description chunk. But, as Grabchak et al. [22] explain, the estimation of the description’s entropy must consider the bias introduced when short text samples are evaluated. Taking advantage of the extensive list of symbols and frequencies available and organized by means of the software Monet, we used the alternative of calculating the description entropy using the joint sets of symbols for each description partition, an then forming the whole description. As a result, no bias has to be corrected.

4. Tests and Results

In order to compare the differences obtained when observing a written message at the scales of characters, words and the fundamental scale, we designed an Example Text.

Table 1. Results of the analysis of the Example Text at the three scales studied.

Example Text: symbol sets at different scales.
-What is an adverb? An adverb is a word or set of words that modifies verbs, adjectives, or other adverbs. An adverb answers how, when, where, or to what extent, how often or how much (e.g., daily, completely). Rule 1. Many adverbs end with the letters “ly”, but many do not. An adverb is a word that changes or simplifies the meaning of a verb, adjective, other adverb, clause, or sentence expressing manner, place, time, or degree. Adverbs typically answer questions such as how?, in what why?, when?, where?, and to what extent?. Adverbs should never be confused with verbs. While verbs are used to describe actions, adverbs are used describe the way verbs are executed. Some adverbs can also modify adjectives as well as other adverbs.
FY = Frequency, EY = Space occupied, N = Message length, ø =space, Vmx = max. symb. length = 13
Char scale			Word scale						Fundamental scale
D = 38 h = 0.8080			Diversity D = 82 Entropy h = 0.9033						Diversity D = 80 h = 0.7628
d = 0.0486 N = 782			Specific diversity d = 82 Length N = 171						Specific diversity d = 1384 Length N = 578
Idx.	Symbol	FY	Idx.	Symbol	FY	Idx.	Symbol	FY	Idx.	Symbol	FY	EY	Idx.	Symbol	FY	EY
1	ø	169	1	,	21	41	completely	1	1	ø	100	1	41	ul	2	2
2	e	86	2	.	11	42	)	1	2	e	70	1	42	wi	2	2
3	a	45	3	or	7	43	Rule	1	3	a	40	1	43	io	2	2
4	s	44	4	adverbs	7	44	1	1	4	s	36	1	44	ie	2	2
5	r	44	5	?	6	45	end	1	5	t	36	1	45	im	2	2
6	t	39	6	adverb	5	46	letters	1	6	r	33	1	46	whe	2	3
7	o	34	7	verbs	4	47	ly	1	7	o	22	1	47	øan	2	3
8	d	32	8	how	4	48	but	1	8	n	21	1	48	dif	2	3
9	n	30	9	an	4	49	do	1	9	,	18	1	49	uch	2	3
10	h	28	10	what	4	50	not	1	10	h	17	1	50	,øc	2	3
11	i	25	11	is	3	51	changes	1	11	b	12	1	51	anyø	2	4
12	v	21	12	a	3	52	simplifies	1	12	dv	10	2	52	wordø	2	5
13	b	21	13	other	3	53	meaning	1	13	d	9	1	53	describ	2	7
14	w	21	14	to	3	54	verb	1	14	c	8	1	54	.øAdverb	2	8
15	,	21	15	the	3	55	adjective	1	15	u	7	1	55	ød	1	2
16	c	17	16	as	3	56	clause	1	16	l	6	1	56	øv	1	2
17	l	16	17	are	3	57	sentence	1	17	?	6	1	57	word	1	4
18	.	11	18	word	2	58	expressing	1	18	wh	6	2	58	yø	1	2
19	u	11	19	of	2	59	manner	1	19	w	5	1	59	ma	1	2
20	m	10	20	that	2	60	place	1	20	i	5	1	60	f	1	1
21	y	10	21	adjectives	2	61	time	1	21	.	4	2	61	ns	1	2
22	f	7	22	when	2	62	degree	1	22	g	4	1	62	An	1	2
23	?	6	23	where	2	63	typically	1	23	x	4	1	63	w	1	2
24	A	5	24	extent	2	64	answer	1	24	ly	4	2	64	b,	1	2
25	g	5	25	with	2	65	questions	1	25	m	4	1	65	v	1	1
26	p	5	26	“	2	66	such	1	26	verbs	4	5	66	-	1	1
27	x	4	27	used	2	67	in	1	27	y	3	1	67	(	1	1
28	j	3	28	describe	2	68	why	1	28	p	3	1	68	)	1	1
29	W	2	29	many	2	69	and	1	29	dj	3	2	69	R	1	1
30	"	2	30	-	1	70	should	1	30	øof	3	3	70	1	1	1
31	-	1	31	set	1	71	never	1	31	ctiv	3	4	71	M	1	1
32	(	1	32	words	1	72	be	1	32	.øA	2	3	72	q	1	1
33	)	1	33	modifies	1	73	confused	1	33	.	2	1	73	S	1	1
34	R	1	34	answers	1	74	While	1	34	W	2	1	74	ho	1	2
35	1	1	35	often	1	75	actions	1	35	"	2	1	75	øm	1	2
36	M	1	36	much	1	76	way	1	36	ow	2	2	76	ng	1	2
37	q	1	37	€	1	77	executed	1	37	me	2	2	77	if	1	2
38	S	1	38	e	1	78	Some	1	38	le	2	2	78	in	1	2
			39	g	1	79	can	1	39	øi	2	2	79	on	1	2
			40	daily	1	80	also	1	40	pl	2	2	80	si	1	2
						81	modify	1
						82	well	1

shows the symbols obtained after the analysis of the Example Text at the three observation scales used in this study. The entropies calculated at the scales of characters and words were 0.81 and 0.90 respectively, the entropy at the fundamental scale was 0.76; an important reduction of the information required to describe the same message.

These results also get along with our intuition. Clearly, the selection of a certain character-string as a fundamental symbol, is favored by the frequency of appearance of the string of characters. As a result, the “space character” (represented as ø in the table) is recognized as the most frequent fundamental symbol. It indeed is an important structural piece in any English text, since it defines the beginning and the end of natural words. The length of the string of characters also favors the survival of the symbol in its competence with other prospective symbols. The string “describ”, for example, appears twice in the Example Text and the algorithm recognized it as a symbol. On the other hand, the 11-char long string “An adverb” also appears two times, but the algorithm found it more effective in reducing the overall entropy, to break that phrase apart and increase the appearances of other symbols. A similar case is that of the word “adverb”, which appears in nine instances (not including those written with the first capital letter) on the Example Text. But the entropy minimization problem found a more important entropy reduction by splitting the word “adverb” in shorter and more frequent symbols as “dv” (10 times), or the characters as “e” (70 times), “a” (40 times),), “r” (33 times), and “b”(12 times).

In another experiment, we contrasted two different types of communication systems by performing tests over full real messages. The first test is based on a text description written in English and the second in test based on the text file associated to music coded using the MIDI format. The English text is a speech by Bertrand Russell given in 1950 during the Nobel Prize ceremony. The MIDI music is a version of the 4th movement of Beethoven’s ninth symphony. The sizes of these descriptions are near the limit of applicability of the algorithm. English descriptions of 1300 words or less can be processed in short times of less than a minute. Larger English texts have to be segmented using the control computational complexity criteria mentioned in Section 3.6 to reach reasonable working times. Bertrand Russell’s speech was fractioned in seven pieces. For MIDI music files, the processing times show an attitude of sharp increase starting for music pieces lasting about 3 min. The version of 4th movement of Beethoven’s ninth symphony used, is a 25 min long piece. It was necessary to process it by fractioning in 20 segments.

To reveal the differences of descriptions when observed at different scales, symbol frequency distributions were produced. For the English text, characters, words and the fundamental scale were applied. For the MIDI music text distributions at character and fundamental scale were constructed. Words do not exist as scale for music. The corresponding detailed set of fundamental symbols can be seen in Appendix B. The frequency distributions were ordered upon the frequency rank of the symbols, thus the obtained were Zipf’s profiles.

Table 2. Details of two descriptions used to test the fundamental scale method.

		Name of scale
		Characters			Fundamental			Words
Text tag	Communication System	Length N	Diversity D	Entropy h	Length N	Diversity D	Entropy h	Length N	Diversity D	Entropy h
.Bertrand Russell 1950.NobelLecture	English	32,621	68	0.7051	26,080	1227	0.5178	6476	1590	0.8215
Beethoven. Symphony9.Mov4	MIDI Music	103,564	160	0.6464	84,645	2824	0.4658	not defined

shows the length N, the diversity D and the entropy h obtained for these two descriptions analyzed at several scales and Figure 3 shows the corresponding Zipf’s profiles for Bertrand Russell’s speech English speech and Beethoven’s 9th Symphony’s 4th movement. Both descriptions’ profiles are presented at the scales they were analyzed: character-scale and the fundamental scale for both, English and music, and the word-scale only for English.

In Figures 3a and 3b, the character scale exhibit the smallest diversity range. Taking only the characters as allowable symbols, leaves out any possibility of combination to form more elaborated symbols and excluding any possibility of representing how the describing information of a system arranges to create what could be loosely called the “language genotype”. Allowing the composition of symbols as the conjunction of several successive characters, dramatically increases the diversity of symbols.

The selection of the symbols to build an observation scale holding the criteria of minimizing the resulting frequency distribution entropy, bounds the final symbolic diversity in a scale while capturing a variety of symbols that represents the way characters are organized to represent the language structure. The fundamental scale appears as the most effective scale, since with it, the original message can be represented with the most compressed information, expressed as the lowest entropy measured for all scales in both communication systems evaluated.

Any scale of observation has a correspondence with the size of the symbols focused at that scale. When that size is the same for all symbols, the scale can be regarded as a regular scale and specified indicating its size. If on the contrary, the scale does not correspond to a constant symbol size, then a symbol frequency distribution based on the sizes is a valid depiction of the scale. That is the case of the scales of words for English texts and the fundamental scale for our two examples. Figures 4 and 5 show those distributions and are useful to interpret the fundamental scales of both examples.

5. Discussion

The results clearly showed the calculus of the entropy content of a communication system varies in important ways, depending on the scale of analysis. Looking at a language at the scale of characters provides a different picture than examining it at the level of words, or at the here described fundamental scale. Thus, in order to compare different communication systems, we need to use a similar scale applicable to each communication system. We showed that the fundamental scale presented here is applicable to very different communication systems, such as music, computer programs, and natural languages. This allows us to perform comparative studies regarding the systems entropy and thus to infer about the relative complexity of different communication systems.

In both examples analyzed, the profiles at the scale of characters and the fundamental scale run close to each other, within the range of the most frequent symbols to the symbols with a rank placed near the mid logarithmic scale. For points with lower ranking, the fundamental-scale profile extends its tail toward the region of low symbol frequencies. The closeness of fundamental and character scaled profiles in the high frequency region, indicates that the character-scaled language B₁ is a subset of the fundamental scale language. The language at fundamental scale, having a greater symbolic diversity and therefore more degrees of freedom, finds a way to generate a symbol frequency distribution with a lower entropy as compared to the minimal entropy distribution when the description is viewed at the scale of words. Focusing in the fundamental scale profiles, the symbols located in the lower rank region―the tail of the profile―tend to be longer symbols formed by more than one character. These multi-character symbols, which cannot exist at the character scale, are formed at the expense of instances of single character symbols typically located in the profile’s head. This explains the nearly constant gap between the two profiles in the profiles’ heads.

The English description, observed at the scale of words, produces a symbol profile incapable of showing short symbols―fragments of a word―which would represent important aspects of a spoken language as syllabus and other typical fundamental language sounds. On the opposite extreme, by observing at the character scale, the profile forbids considering strings of characters as symbols, thus meaningful words or structures cannot appear at this scale, missing important information about the structure of the described system.

The fundamental scale, on the other hand, appears as an intermediate scale capable of capturing the essence of the most elementary structure of a language, as its alphabet, as well as larger structures which represent the result of language evolution in its way to form more specialized and complex symbols. The same applies for music MIDI representation. There is no word scale for music, but clearly the character scale does not capture the richness that undoubtedly is present in this type of language.

Another difference between the fundamental scale, and other scales is the sensitivity to the order of the symbols as they appear in the text. At the scale of words or the scale of characters, the symbol frequency profile does not vary as the symbol order. The profiles depend only on the number of appearances of each symbol, word or character, depending on the subject scale. The profile built at the fundamental scale does change as the symbol order is altered, not because of the symbol order itself, but because the symbol set recognized as fundamental, changes when the order or words or characters are modified. As a consequence, the character and word scales do not have any sense of grammar. The fundamental scale and its corresponding profile, on the other hand, is affected by the order in which words are organized—or disorganized—and is therefore sensitive to the rules of grammar. Other communication systems may not have words, but they must have some rules or the equivalence of a grammar. Assuming rigid rules as symbol size or symbol delimiters seems to be a barrier when studying the structure of system descriptions.

In the search for symbols, the fundamental scale method accounts for frequent sequences of strings which result from grammar rules. The string “ing”, for example appears at the end of words representing verbs or actions. Moreover, it normally comes followed by a space character (“ ”). As the sequence appears with noticeable frequency, the fundamental scale method recognizes the char sequence “ing” (ending with a space) as an entropy reducer token and therefore an important descriptive piece of English as a language. The observation of a description at its fundamental scale is therefore, sensitive to the order in which char-strings appear within the description. The fundamental scale method detects the internal grammar which has been ignored when analyzing Zipf’s profiles at the scale of words in many previous studies.

Despite the concept of fundamental scale being applicable to descriptions built over multidimensional spaces, the fundamental scale method and the algorithm developed is devised for 1-dimensional descriptions. The symbol search process implemented scans the description along the writing dimension of the text file being analyzed. This means that the fundamental symbols constituting 2D descriptions like pictures, photographs or plain data tables cannot be discovered with the algorithm as developed. To extend the fundamental scale algorithm to descriptions of more than one dimension, the restriction (8c) must be modified or complemented, to incorporate the sense of indivisible information unit―as has been the character in the development of this study―and the allowed symbol boundary shape in the description-space considered. This adjustment is a difficult task to accomplish because establishing criteria for the shapes of the boundaries becomes a hard to solve topology problem, especially in higher dimensional spaces.

There are other limitations for the analysis of descriptions of one dimension. Some punctuation signs which belong more to the writing system than to the language itself, work in pairs. Parenthesis, quotes, admiration and question marks are some of the written punctuation signs which work in couples. Intuition indicates that each one of them is a half-symbol belonging to one symbol. In these cases, not considering each half as part of the same symbol most likely increases the entropy associated to the set of symbols discovered, thus becoming a deviation of the ideal application of the method. Nevertheless, for English, Spanish and human natural languages, in general, the characters which work in couples, appear infrequently as compared to the rest of characters. Thus the minimal entropy distortion introduced by this effect is small.

Practical use of the algorithm is feasible up to some description lengths. The actual limit depends on the nature of the language used in the description. For syllabic human natural languages the algorithm can be directly applied to texts of 40,000 characters or less. Longer texts, however, can be analyzed by partitioning. Thus the application limit for texts expressed in human natural languages, covers most needs. For the analysis of music, the use of the algorithm is limited to the MIDI format, result in large processing times even for powerful computers available today. The problem of scanning all possible sets of symbols in a sequence of characters grows as a combinatorial number. The Problem rapidly gets too complex in the computational sense, and its practical application is only feasible for representations of music in reduced sets of digitized symbols like the MIDI coding. Using more comprehensive formats like .MP3, a compressing technology capable of reducing the size of a music pack while keeping reasonably good sound quality, would be enough to locate the solution of the problem beyond our possibilities of performing experiments with large sets of musical pieces. Yet, the fundamental scale method provides new possibilities for discovering the most representative dimension of small sized textual descriptions, allowing us to advance in our understanding of languages.

The Fundamental Scale, as a concept and as a method to find a quantitative approximation to the description of communication systems promises to be fruitful in further research. Tackling the barriers of the algorithm by finding ways to reduce the number of loops and augmenting the assertiveness of the criteria used, may extend the space of practical use of the notion of a description’s fundamental scale. Here we showed that the method reveals structural properties of languages and other communication systems, offering a path for comparative studies of the complexity of communication.