tola

Synopsis

A user interface element which allows a user to input a finite string of text (through speech, touch or gesture) in natural language, to be interpreted by the computer as a command to carry out an action.

Rationale

This design concept is the application of natural language processing to the established concept of a "command line interface". The aim is to create a versatile input method by which a user can command a computer to perform an action in natural language. This will allow the user to tell the computer what they wish it to do, perhaps as an alternative to using graphical design metaphors.

Features

• A method of indicating the boundaries of the string of text e.g. :
• Holding down a button while while the command is being input
• Pressing a button at the beginning and the end of the command
• Speaking a key word out loud before issuing a command. Leaving a pause once the command is finished.
• Clicking a text box to gain focus, pressing "enter" when the command is finished
• A means for inputting a string of text e.g. :
• Speaking into a microphone, speech recognition changing the speech into ASCII text
• Pressing buttons which represent letters, numbers and symbols
• Making gestures or sign language signs which are interpreted using computer vision and changed into ASCII text
• The processing of a string of text to attempt to extract its meaning or the user's intention e.g. :
• Parsing a sentence of natural language
• Recognising certain string formats such as URIs, sums and unit conversions
• Optionally, the ability to allow the user to confirm or reject the computer's interpretation of their command
• The ability to execute an action based on the user's command

Use Cases

Example commands listed here include:

• Direct commands beginning with a verb which instruct an action
• Questions beginning with a question word such as what, where or who which require an answer
• String patterns such as algebra or URIs which need a particular type of action or response, based on the type of pattern

A string which does not follow any defined patterns could be interpreted simply as a request to search for more information or a list of possible actions, based on the text entered.

Social Software:

• Email laura
• Call laura
• Text laura
• Schedule dinner tomorrow at five with Laura

Home Automation:

• Turn the kitchen light off
• Turn the heating down
• Close the curtains
• Close all the downstairs curtains
• Dim the lights

Multimedia:

• Play some Jazz music
• Tell me what's on TV
• Read me today's national news headlines

Search:

• Search the web for trees
• Search wikipedia.com for trees
• Where is laura?
• Locate my slippers

Misc:

• Calculate 63 percent of nine hundred and five
• Calculate six multiplied by seven
• Calculate how many metres there are in 10 miles
• Calculate ninety seven dollars in pounds sterling
• Add milk to the shopping list
• Define tree

Patterns:

(150+175)/2^2
• metres in 3 miles
• www.google.com