Choosing Natural Adjective Ladders

By Fred Hicks, Lee Valentine, John Morrow, and Ian McDonald

This entire text is presented as Open Gaming Content under the terms of the Open Gaming License v 1.0 (see end of document).

Summary

Several role-playing games use adjectives instead of numbers to rate attribute or skill levels. In order to help game designers pick adjectives which are easy to remember, we surveyed English-speakers in the UK and US, including non-native speakers. We present our survey results, and a list with as many items as possible for which our respondents have a consensus order. Our suggested list is "Abysmal, Awful, Bad, Poor, Mediocre, Fair, Good, Great, Excellent, Amazing, Phenomenal".  

How we did it

We selected 24 words describing generic quality, starting with the thesaurus and winnowing away words which only applied in a particular context. We then surveyed 35 people, asking them to rank the words. We did this by handing out paper forms to friends, colleagues, and at role-playing groups; the sample included role-players and non-roleplayers, speakers of multiple dialects of English and people who had learnt English as a second language. We also recorded the respondent's first language, enabling us to track differences between American and British English, and notice which words were differentiated poorly in non-native speakers.

The words are: Abysmal, Amazing, Awful, Bad, Decent, Deficient, Dreadful, Excellent, Exceptional, Fair, Fine, Good, Great, Incredible, Inferior, Limited, Mediocre, Middling, Phenomenal, Poor, Superior, Terrible, Weak, and World-Class.

In the first part of the form, respondents were asked "How would you rate an adjective driver?" and given a scale from 1 to 10 on which they could circle a number. Hereafter, we call this one the scale.

In the second part, respondents were given the 24 words in a jumbled order, and asked to re-order them from best to worst either by rewriting them in order, or by putting "1" by the highest, so on down to "24" by the lowest. Hereafer, we call this one ordered.

Results in detail:

Rated on a scale from 1 to 10

Below is the mean and standard deviation of the ratings words were given out of ten. All 35 respondents rated words, except for "middling" (30 out of 35), abysmal, and fair (34 out of 35). They are grouped into words which have been ranked close to each other, and thus might be easily confused.

Adjective	mean	SD
Phenomenal	9.5	1.2
World-Class	9.5	0.6
Incredible	9.0	0.9
Amazing	8.9	1.0
Exceptional	8.7	1.1
Excellent	8.3	1.0
Superior	8.2	1.0
Great	8.0	1.1
Good	6.9	1.1
Fine	6.6	1.2
Decent	6.2	0.9
Fair	5.4	0.9
Middling	5.0	0.8
Mediocre	4.4	0.9
Limited	3.8	1.0
Weak	3.4	0.9
Deficient	3.2	0.9
Inferior	3.1	1.0
Poor	2.9	1.1
Bad	2.6	1.0
Awful	1.9	0.9
Terrible	1.8	1.3
Dreadful	1.9	0.9
Abysmal	1.3	0.8

Table 1: Scale Ratings

We also used these data to generate word orders for each person and discover how much agreement there is about the order of each one. We can calculate the "approval percentage" (the percentage agreeing with that order minus the percentage who would prefer the reverse, out of everyone who rated both words) for each combination of words (100% approval is abbreviated to "•"). The table below shows this for each combination - each word on the right names both a column and a row, and the intersection between the row of one world and the column of the less "positive" word shows the approval percentage for putting those two words in that order.:

94	94	94	94	94	94	94	94	94	•	94	93	94	94	94	88	80	77	77	57	48	45	17	phenomenal
•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	•	80	68	74	48	37	31	world-class
97	97	94	97	97	97	97	•	97	•	•	96	94	97	97	91	71	59	54	28	14	incredible
•	•	94	•	•	•	•	•	•	•	•	•	94	•	94	97	57	42	54	11	exceptional
97	97	94	97	97	97	97	•	97	•	•	96	94	97	88	85	57	28	37	amazing
•	•	94	•	•	•	•	•	•	•	•	•	91	94	80	74	8	-17	superior
97	97	94	97	97	97	97	•	97	•	•	96	94	97	94	85	25	excellent
97	97	94	97	97	97	97	•	97	•	•	96	94	91	82	74	great
97	97	94	97	97	97	97	•	97	•	94	90	76	60	5	good
94	94	94	94	94	94	94	97	94	97	91	86	70	37	fine
97	97	94	97	97	97	97	•	97	•	94	86	55	decent
•	•	94	•	•	•	96	94	97	97	67	31	fair
96	93	86	93	90	93	96	90	83	60	36	middling
94	91	91	91	88	80	70	68	65	48	mediocre
91	74	88	85	71	60	44	34	22	limited
97	82	82	85	71	40	17	14	weak
91	68	77	74	54	25	11	deficient
78	58	64	58	35	20	inferior
85	60	68	62	25	poor
76	54	62	54	bad
47	-2	14	awful
38	-11	terrible
47	dreadful
abysmal

Table 2: Scale Approval Ratings

An approval rating can be negative. For example, the approval rating for the belief that superior is greater than excellent, rather than the other way round, is "-17%", showing that most respondents rate "excellent" more highly, despite the fact that the mean value assigned to superior is higher. This discrepancy is because whilst more people rate "excellent" higher, the ones who rated "superior" higher did so by a bigger margin, and the difference in mean value is tiny (0.013).

Ordered into a list

32 out of the 35 respondents who returned survey forms answered the longer second question and ordered the adjectives into a list. This figure shows the same order as above, with the approval rating of every combination. We can see that asking people to rank every word leads to higher approval ratings; this is expected, because respondents might have given words the same mark out of ten for question one, but ranked one higher for question two.

•	•	•	•	•	•	•	•	•	•	•	•	96	96	96	96	84	84	84	72	60	68	21	phenomenal
•	•	•	•	•	•	•	•	•	•	•	•	96	96	96	96	90	90	81	51	42	53	world-class
•	•	•	•	•	•	•	•	•	•	•	•	96	96	96	96	84	75	50	28	9	incredible
•	•	•	•	•	•	•	•	•	•	•	•	96	96	96	96	75	45	57	3	exceptional
•	•	•	•	•	•	•	•	•	•	•	•	96	96	96	96	75	60	60	amazing
•	•	•	•	•	•	•	•	•	•	•	•	96	93	90	93	33	12	superior
•	•	•	•	•	•	•	•	•	•	•	•	96	96	96	96	57	excellent
•	•	•	•	•	•	•	•	•	•	•	•	96	96	90	96	great
•	•	•	•	•	•	•	•	•	•	96	•	84	46	21	good
•	•	•	•	•	•	•	•	•	96	93	92	75	28	fine
•	•	•	•	•	•	•	•	•	96	90	•	68	decent
•	•	•	•	•	•	•	96	•	78	70	57	fair
•	•	•	•	96	96	•	92	96	84	25	middling
•	•	•	•	93	80	75	75	74	33	mediocre
•	•	•	•	93	75	83	80	59	limited
•	96	90	•	66	40	32	19	weak
•	96	87	93	58	36	31	deficient
•	96	93	96	54	10	inferior
•	96	93	93	59	poor
93	90	81	75	bad
75	33	15	awful
62	21	terrible
71	dreadful
abysmal

Table 3: Ordered Approval Ratings

The next two tables supply much more information, by rating for each combination of words:

• the number of respondents agreeing with that word order
  
• the number of respondents who think that the words should be the other way around
• the number of respondents who think they should share a space on the word order

These are also the raw data used to generate the table above. To better fit the space, the full dataset is split into two tables, the top two-thirds, and the bottom two-thirds.

•	•	•	•	32 0 1	31 0 1	32 0 1	32 0 1	28 0 5	29 1 3	29 1 3	25 1 7	24 4 5	24 2 6	16 9 8	phenomenal
•	•	•	•	32 0 1	31 0 1	32 0 1	32 0 1	30 0 3	30 0 3	29 2 2	22 5 6	21 7 5	22 5 5	world-class
•	•	•	•	31 0 1	30 0 1	31 0 1	31 0 1	27 0 5	26 2 4	21 5 6	16 7 9	12 9 11	incredible
•	•	•	•	32 0 1	31 0 1	32 0 1	32 0 1	27 2 4	21 6 6	22 3 8	14 13 6	exceptional
•	•	•	•	32 0 1	31 0 1	32 0 1	32 0 1	25 0 8	23 3 7	24 4 5	amazing
•	•	•	•	32 0 1	30 0 2	31 1 1	31 0 2	18 7 8	15 11 7	superior
•	•	•	•	32 0 1	31 0 1	32 0 1	32 0 1	22 3 8	excellent
•	•	•	•	32 0 1	31 0 1	31 1 1	32 0 1	great
•	•	30 0 1	•	29 1 3	19 4 9	15 8 10	good
•	31 0 1	29 0 2	24 0 2	26 1 6	17 8 7	fine
•	30 0 1	27 0 3	•	24 2 6	decent
•	28 3 1	25 3 3	18 3 5	fair
25 0 1	22 1 2	11 5 8	middling
27 4 0	17 7 6	mediocre
23 4 5	limited
weak

 

•	•	•	•	•	•	•	•	•	31 0 1	29 0 2	24 0 2	26 1 6	17 8 7	fine
•	•	•	•	•	•	•	•	•	30 0 1	27 0 3	•	24 2 6	decent
•	•	•	•	•	•	•	30 0 1	•	28 3 1	25 3 3	18 3 5	fair
•	•	•	•	25 0 1	24 0 1	•	24 1 0	25 0 1	22 1 2	11 5 8	middling
•	•	•	•	29 0 2	26 2 2	25 3 1	25 3 1	27 4 0	17 7 6	mediocre
•	•	•	•	31 1 0	27 3 2	26 1 3	25 1 4	23 4 5	limited
•	32 0 1	31 1 1	•	25 3 5	17 4 11	17 7 7	17 11 3	weak
•	30 0 1	29 2 0	29 0 2	23 5 3	19 8 3	15 6 8	deficient
•	30 0 1	30 1 0	30 0 1	21 4 6	14 11 5	inferior
•	31 0 1	30 0 2	31 1 0	22 3 7	poor
31 1 0	30 0 3	28 1 4	27 2 4	bad
25 1 6	16 5 12	11 6 16	awful
23 3 6	14 7 12	terrible
25 2 5	dreadful
abysmal

Table 4: Detailed Ordering Data

Choosing the Ladder

Criteria

Our sample is clearly not large enough to do sophisticated analysis of the entire data set, but does show broad perceptions of words and permit trend predicting. This means we can make reasonable guesses about which combinations of words will be easiest for players to use without being confused about which word is higher.

Choosing a set of adjectives to use in a role-playing game is not solely a question of which combinations rank themselves the most clearly; it is also a question of which words fit naturally into the context of ability scores, outcomes, and challenge levels. We need to reject some because they fit less comfortably into the kind of sentences which turn up in role playing games. ("She is an adjective driver; that was an adjective performance; this is an adjective longbow." - perhaps even "That is an adjective challenge").

Our aim is to include as many as possible whilst retaining a consensus order.

Discussing the Survey Data

"Middling" did not poll well; a seventh of our respondents, and a quarter of our US respondents , did not even rate it. All 35 respondents rated almost all other words, the only three exceptions being one person failing to rate each of "Inferior", "Fair", and "Abysmal".

The split between "Great" and "Good" has an ordered approval rating of 96%, the clearest of two adjacent words, so those should be included. Going down from "Good", the next clearly approved rating is "Fair" (ordered approval 84%), and the one after that is "Mediocre" (ordered approval 70%). These straightforward choices are also the middle of the scale used by the leading "adjective scale" role-playing game. (Given this, it would have been interesting to see how "Superb" polled.)

"Decent" had enough separation that there might, if trends continue, be enough of a distinction between "Fair" and "Good" to slip it between the two if needed. We had expected differences with dialect, but the low approval of the ordering that "Good is better than Decent" was the same (at 46% ordered approval rating) for the British as for the English-speaking world at large.

Similarly to "Good" and "Great", "Bad" and "Awful" are an obvious choice (75% ordered approval for the gap), but only "Abysmal" can be used as a level below "Awful" without confusion.

The data are not as clear regarding what to use between "Bad" and "Mediocre", or for which words to use above "Great".

Between "Bad" and "Mediocre", "Limited" is too close to "Mediocre" (only 33% ordered approval, 48% scale approval) but the other words are close enough to "Bad" to confuse some users (54-66% approval). The highest approval belongs to "Weak". "Poor" has two fewer respondents agreeing with the ordering (out of 32), but the same number (3) disagreeing. So either "Weak" or "Poor" would be suitable between "Bad" and "Mediocre".

Above Great, the mean scale ratings group the adjectives into three sets:

• Excellent & Superior
• Exceptional
• Incredible & Amazing
• Phenomenal & World-Class

For Excellent & Superior, the mean values on the scale (Table 1) and the consensus ordering (Table 3) disagree with each other. As ordering "Superior" (subjectively, a relative term begging the question "superior to what?") above "Great" only has a 33% ordered approval rating, we cannot use it. "Excellent", on the other hand, might remain an option, depending what words are used above it.

At the high end of the scale, "Phenomenal" gives most room for fitting other words beneath it, so we used it.

Exceptional comes between Incredible and Amazing when we ask people to place words in order, so we should examine these as a group of three words. Looking at Table 3 and comparing these three words to "Great", "Excellent", and "Phenomenal", we see at least 60% ordered approval for all comparisons except "Exceptional is higher than Excellent". In fact, checking the ordering relative to "Great" and "Phenomenal", "Exceptional" is always the least certain comparison. This makes subjective sense - after all, an exception can be bad as well as good, so we should expect this word to be interpreted more ambiguously.

Final Choices

The survey data are quiet on some points:

• Whether to use Weak or Poor
• Whether to use Incredible or Amazing
• Whether to use Excellent at all

Because "Weak" has specific associations with "Strength", we chose "Poor" to be part of the ladder.

Whilst deciding between "Incredible" and "Amazing", we examined the literal meanings of "Amazing" (as something which amazes) and "Incredible" (as something one cannot believe in), considering not just which literal meaning fitted in better with using the word in play, but also which word was most likely to be used literally. We finally chose "Amazing" on those grounds.

We had long discussions about whether to include "Excellent". Two thirds of users actively agreed that "Excellent" was better than "Good", yet none disagreed that "Amazing" or "Incredible" were. We finally decided that including an eleventh word in the scale -allowing it to run from zero to ten - was worth the inconvenience to the one user in 11 who disagreed.

Using Our Suggested Ladder

This is the best representation of a set of “ranked” terms we could determine, based on the results of our survey. After much discussion, we have boiled the list down to a comfortable eleven — small enough to aid easy memorization, big enough to cover a wide range of description with manageable detail. Here is the list, ordered from “most positive”, to “most negative”:

Phenomenal

Amazing

Excellent

Great

Good

Fair

Mediocre

Poor

Bad

Awful

Abysmal

Table 5: The Adjective Ladder

The most basic aspect of this ladder is its most obvious: starting at the bottom, the terms indicate progressive levels of improvement. These adjectives may be applied however you like, and are most often used to rate abilities, stats, skills, and in some cases, difficulties or outcomes, in your own game. This ladder does not dictate the numerical meaning (if any) that these terms might have in a game, so long as the order of the terms, and the terms themselves, are kept constant. That said, they offer three obvious possibilities:
 

	Centre-weight	Range	Out of ten
Phenomenal	+5	20-21	10
Amazing	+4	18-19	9
Excellent	+3	16-17	8
Great	+2	14-15	7
Good	+1	12-13	6
Fair	+0	10-11	5
Mediocre	-1	8-9	4
Poor	-2	6-7	3
Bad	-3	4-5	2
Awful	-4	2-3	1
Abysmal	-5	0-1	0

Table 6: Ways to use the Adjective Ladder

Centre-Weight: Because the ladder has eleven terms, one can easily choose Fair as the middle-point. This method gives a range of bonuses and penalties running from +5 to -5, which neatly fits the bonuses and penalties of any number of other systems, and as such can be used as a general guide for converting material from those systems.

Range: This example's numbers best fit a 3d6, 2d10, or 1d20 based distribution, but in general terms, a range-based translation of the adjective terms is gives you the opportunity for some finer grain within each rung on the ladder, which may be attractive. This is also obviously compatible with a wealth of OGL material already available.

Out of ten: Because the ladder has eleven terms, it is also very easy to link it to an intuitive "Score out of ten", which is almost as natural language as the adjectives themselves, and a common choice in RPG systems.

However you choose to use this ladder or our survey results, we hope that you will be able to use natural language in a way that is easy to remember and flows smoothly in your games.

Acknowledgements

Carl Davis for helping with the survey and discussions, Reimer Behrends, and Rob Donoghue for discussions, members of "Dragons on the Hill RPG Club" and other survey respondents for taking the time to complete our survey. Naturally, this article is written as a consensus after much discussion and we do not all agree with every part of it.

Open Gaming License

OPEN GAME LICENSE Version 1.0a

The following text is the property of Wizards of the Coast, Inc. and is Copyright 2000 Wizards of the Coast, Inc ("Wizards"). All Rights Reserved.

1. Definitions: (a)"Contributors" means the copyright and/or trademark owners who have contributed Open Game Content; (b)"Derivative Material" means copyrighted material including derivative works and translations (including into other computer languages), potation, modification, correction, addition, extension, upgrade, improvement, compilation, abridgment or other form in which an existing work may be recast, transformed or adapted; (c) "Distribute" means to reproduce, license, rent, lease, sell, broadcast, publicly display, transmit or otherwise distribute; (d)"Open Game Content" means the game mechanic and includes the methods, procedures, processes and routines to the extent such content does not embody the Product Identity and is an enhancement over the prior art and any additional content clearly identified as Open Game Content by the Contributor, and means any work covered by this License, including translations and derivative works under copyright law, but specifically excludes Product Identity. (e) "Product Identity" means product and product line names, logos and identifying marks including trade dress; artifacts; creatures characters; stories, storylines, plots, thematic elements, dialogue, incidents, language, artwork, symbols, designs, depictions, likenesses, formats, poses, concepts, themes and graphic, photographic and other visual or audio representations; names and descriptions of characters, spells, enchantments, personalities, teams, personas, likenesses and special abilities; places, locations, environments, creatures, equipment, magical or supernatural abilities or effects, logos, symbols, or graphic designs; and any other trademark or registered trademark clearly identified as Product identity by the owner of the Product Identity, and which specifically excludes the Open Game Content; (f) "Trademark" means the logos, names, mark, sign, motto, designs that are used by a Contributor to identify itself or its products or the associated products contributed to the Open Game License by the Contributor (g) "Use", "Used" or "Using" means to use, Distribute, copy, edit, format, modify, translate and otherwise create Derivative Material of Open Game Content. (h) "You" or "Your" means the licensee in terms of this agreement.

2. The License: This License applies to any Open Game Content that contains a notice indicating that the Open Game Content may only be Used under and in terms of this License. You must affix such a notice to any Open Game Content that you Use. No terms may be added to or subtracted from this License except as described by the License itself. No other terms or conditions may be applied to any Open Game Content distributed using this License.

3.Offer and Acceptance: By Using the Open Game Content You indicate Your acceptance of the terms of this License.

4. Grant and Consideration: In consideration for agreeing to use this License, the Contributors grant You a perpetual, worldwide, royalty-free, non-exclusive license with the exact terms of this License to Use, the Open Game Content.

5.Representation of Authority to Contribute: If You are contributing original material as Open Game Content, You represent that Your Contributions are Your original creation and/or You have sufficient rights to grant the rights conveyed by this License.

6.Notice of License Copyright: You must update the COPYRIGHT NOTICE portion of this License to include the exact text of the COPYRIGHT NOTICE of any Open Game Content You are copying, modifying or distributing, and You must add the title, the copyright date, and the copyright holder's name to the COPYRIGHT NOTICE of any original Open Game Content you Distribute.

7. Use of Product Identity: You agree not to Use any Product Identity, including as an indication as to compatibility, except as expressly licensed in another, independent Agreement with the owner of each element of that Product Identity. You agree not to indicate compatibility or co-adaptability with any Trademark or Registered Trademark in conjunction with a work containing Open Game Content except as expressly licensed in another, independent Agreement with the owner of such Trademark or Registered Trademark. The use of any Product Identity in Open Game Content does not constitute a challenge to the ownership of that Product Identity. The owner of any Product Identity used in Open Game Content shall retain all rights, title and interest in and to that Product Identity.

8. Identification: If you distribute Open Game Content You must clearly indicate which portions of the work that you are distributing are Open Game Content.

9. Updating the License: Wizards or its designated Agents may publish updated versions of this License. You may use any authorized version of this License to copy, modify and distribute any Open Game Content originally distributed under any version of this License.

10 Copy of this License: You MUST include a copy of this License with every copy of the Open Game Content You Distribute.

11. Use of Contributor Credits: You may not market or advertise the Open Game Content using the name of any Contributor unless You have written permission from the Contributor to do so.

12 Inability to Comply: If it is impossible for You to comply with any of the terms of this License with respect to some or all of the Open Game Content due to statute, judicial order, or governmental regulation then You may not Use any Open Game Material so affected.

13 Termination: This License will terminate automatically if You fail to comply with all terms herein and fail to cure such breach within 30 days of becoming aware of the breach. All sublicenses shall survive the termination of this License.

14 Reformation: If any provision of this License is held to be unenforceable, such provision shall be reformed only to the extent necessary to make it enforceable.

15 COPYRIGHT NOTICE
Open Game License v 1.0 Copyright 2000, Wizards of the Coast, Inc.
Chooing Natural Adjective Ladders Copyright 2006, Fred Hicks, Lee Valentine, John Morrow, and Ian McDonald.