Monthly Archives: January 2015

languagengine – Blog – Type Checking in JavaScript

Type Checking in JavaScript

posted by Darryl on 16 Jan 2015

I’d like to follow up to my previous blog post on implementing a simply typed lambda calculus in JavaScript by demonstrating a more complete, more natural type system. If you haven’t read it already, you should do so. The previous type system was a little unnatural because it had no type inference mechanism, and so all of the relevant type information had to be provided in the programs themselves. This time, however, we’ll build a type checker that has inference, so we can write more natural programs.

One thing to note regarding type inference like this is that well-designed type programming languages, with properly defined type systems, can infer a lot in the way of types. One often hears that having to write a type signature is just a pain in the butt, and therefore strictly typed programming languages aren’t as pleasant to use as untyped or duck-typed ones. But with type inference, you frequently don’t need to write type signatures at all. In a language like Haskell, for instance, type signatures can very frequently be omitted.

The GitHub repo for this type checker can be found here.

To begin, let’s define the type theory we’d like to embody. We’ll again have just pair types and function types, as well as three primitive types FooBar, and Baz which will serve as place holders until we implement some other types in a later post. Again we’ll start with the type formation judgment A type that tells us when something is a type or not:

A type

-------- Foo Formation      -------- Bar Formation
Foo type                    Bar type

            -------- Baz Formation
            Baz type

A type    B type                  A type    B type
---------------- * Formation      ---------------- -> Formation
    A*B type                        A -> B type

The JavaScript implementation will be the same as before:

var Foo = { tag: "Foo" };
var Bar = { tag: "Bar" };
var Baz = { tag: "Baz" };

function prod(a,b) {
    return { tag: "*", left: a, right: b };

function arr(a,b) {
    return { tag: "->", arg: a, ret: b };

function judgment_type(a) {
  if ("Foo" == a.tag || "Bar" == a.tag || "Baz" == a.tag) {
      return true;
  } else if ("*" == a.tag) {
      return judgment_type(a.left) && judgment_type(a.right);
  } else if ("->" == a.tag) {
      return judgment_type(a.arg) && judgment_type(a.ret);
  } else {
      return false;

As before, we’ll use a snoc linked-list for encoding variable contexts:

var empty = { tag: "<>" }

function snoc(g,x,a) {
    return { tag: ",:", rest: g, name: x, type: a };

We’ll have new intro and elim rules for the judgment G !- M : A which defines well-typed-ness. These will define programs that have less annotation for types, and which are therefore more natural and easier to use.

G !- M : A

G !- M : A    G !- N : B
------------------------ * Intro
   G !- (M,N) : A*B

G !- P : A*B    G, x : A, y : B !- M : C
---------------------------------------- * Elim
     G !- split P as (x,y) in M : C

This time, our split elim, which does pattern matching for pairs, is not annotated for the types of x and y. If you look at the stuff above the inference line, you see Aand B, but these don’t appear below the line. If we wanted to simply type check, we’d need to invent these out of thin air, and there are a lot of ways to do that. So instead we can do inference on P, to get back it’s type, which had better be of the form A*B, and proceed from there. So, we’ll have the old checking function from before, but also a new inferring function:

function pair(m,n) {
  return { tag: "(,)", first: m, second: n };

function split(p, x, y, m) {
    return { tag: "split",
             pair: p,
             name_x: x, name_y: y,
             body: m };

// judgment_check will be modified shortly
function judgment_check(g, m, a) {
    if ("(,)" == m.tag && "*" == a.tag) {
        return judgment_check(g, m.first, a.left) &&
               judgment_check(g, m.second, a.right);
    } else if ("split" == m.tag) {
        var inferred_pair = judgment_infer(g, m);
        if (!inferred_pair || "*" != inferred_pair.tag) {
            return false;
        return judgment_check(snoc(snoc(g, m.name_x, inferred_pair.left),
                                   m.name_y, inferred_pair.right),
    } else {
        return false;

This much is basically the same as before, but we’ll also define now a function judgment_infer:

// judgment_infer will also be modified shortly
function judgment_infer(g, m) {
    if ("(,)" == m.tag) {
        var inferred_left = judgment_infer(g, m.first);
        var inferred_right = judgment_infer(g, m.second);
        if (!inferred_left || !inferred_right) {
            return null;
        return prod(inferred_left, inferred_right);
    } else if ("split" == m.tag) {
        var inferred_pair = judgment_infer(g, m.pair);
        if (!inferred_pair || "*" != inferred_pair.tag) {
            return null;
        return judgment_infer(snoc(snoc(g, m.name_x, inferred_pair.left),
                                   m.name_y, inferred_pair.right),
    } else {
        return null;

This new function has some obvious similarities to judgment_check, but instead of checking if a term has the appropriate type, it computes the correct type and returns that.

Our definitions for function types are very similar to what we had before as well:

 G, x : A !- M : B
-------------------- -> Intro
G !- \x:A.M : A -> B

G !- M : A -> B    G !- N : A
----------------------------- -> Elim
        G !- M N : B

Unlike before, however, we now tag lambdas with their argument type. This particular type inference algorithm isn’t smart enough to solve that for us (to do that would require a unification-based algorithm).

And the implementation of judgment_check is only slightly modified, including the syntax for function application. Again, tho, we use judgment_infer to recover the missing content of the elim rule.

function lam(x,a,m) {
    return { tag: "lam", name: x, arg_type: a, body: m };

function app(m,n) {
    return { tag: "app", fun: m, arg: n };

// the new version
function judgment_check(g, m, a) {
    ... else if ("lam" == m.tag && "->" == a.tag) {
        return type_equality(m.arg_type, a.arg) &&
               judgment_check(snoc(g,, a.arg), m.body, a.ret);
    } else if ("app" == m.tag) {
        var inferred_arg = judgment_infer(g, m.arg);
        if (!inferred_arg) {
            return false;
        return judgment_check(g,, arr(inferred_arg, a));
    } ...

And now the modification to judgment_infer:

// judgment_infer will also be modified shortly
function judgment_infer(g, m) {
    ... else if ("lam" == m.tag) {
        var inferred_body = judgment_infer(snoc(g,, m.arg_type),
        if (!inferred_body) { return null; }
        return arr(m.arg_type, inferred_body);
    } else if ("app" == m.tag) {
        var inferred_fun = judgment_infer(g,;
        if (!inferred_fun || "->" != inferred_fun.tag ||
            !judgment_check(g, m.arg, inferred_fun.arg)) {
            return null;
        return inferred_fun.ret;
    } ...

And of course we have the variable rule as before:

function v(n) {
    return { tag: "variable", name: n };

function judgment_check(g, m, a) {
    ... else if ("variable" == m.tag) {
        return type_equality(type_lookup(g,, a);
    } ...

To infer the type of a variable, we’ll need to look it up in the context:

function type_lookup(g, n) {
    if ("<>" == g.tag) {
        return null;
    } else if (n == {
        return g.type
    } else {
        return type_lookup(, n);

But with that, inference is easy:

function judgment_infer(g, m) {
    ... else if ("variable" == m.tag) {
        return type_lookup(g,;
    } ...

Lastly, type equality, which is unchanged:

function type_equality(a,b) {
    if (("Foo" == a.tag && "Foo" == b.tag) ||
        ("Bar" == a.tag && "Bar" == b.tag) ||
        ("Baz" == a.tag && "Baz" == b.tag)) {
        return true;
    } else if ("*" == a.tag && "*" == b.tag) {
        return type_equality(a.left, b.left) &&
               type_equality(a.right, b.right);
    } else if ("->" == a.tag && "->" == b.tag) {
        return type_equality(a.arg, b.arg) &&
               type_equality(a.ret, b.ret);
    } else {
        return false;

For convenience in inspecting what’s going on with these functions, it’ll help to have a function to convert types to strings, so let’s define one:

function show(a) {
    if ("Foo" == a.tag) { return "Foo"; }
    if ("Bar" == a.tag) { return "Bar"; }
    if ("Baz" == a.tag) { return "Baz"; }
    if ("*" == a.tag) {
        return "(" + show(a.left) + "*" + show(a.right) + ")";
    if ("->" == a.tag) {
        return "(" + show(a.arg) + " -> " + show(a.ret) + ")";
    return "Unknown";

If we now infer the types for some common functions, we’ll get exactly what we expect:

// \x : Foo. x
// infers to  Foo -> Foo
judgment_infer(empty, lam("x",Foo, v("x")));

// \x : Foo. \f : Foo -> Bar. f x
// infers to  Foo -> (Foo -> Bar) -> Bar
judgment_infer(empty, lam("x",Foo,
                          app(v("f"), v("x")))));

// \p : Foo*Bar. split p as (x,y) in x
// infers to  Foo*Bar -> Foo
judgment_infer(empty, lam("p",prod(Foo,Bar),
                        split(v("p"),"x","y", v("x"))));

If you have comments or questions, get it touch. I’m @psygnisfive on Twitter, augur on freenode (in #languagengine and #haskell). Here’s the HN thread if you prefer that mode, and also the Reddit thread.

event – FoundersSpace – Big Data: Everything You Need to Know!

Please review and share.

Charles Jo 650.906.2600 Twitter @charlesjo

Begin forwarded message:

It would be fantastic if you could let the people know about our upcoming events:

Big Data: Everything You Need to Know!
February 18th at 6 pm – 9 pm in Palo Alto
25% Discount = 25OFF

Startup Party: 300+ Entrepreneurs, Angels, VCs & More!
February 19th at 8 pm – 12 am in Silicon Valley
25% Discount = 25OFF


networking – for those interested – Welcome to the “1k-beta-testers” mailing list​

Please share with your friends!

We are trying to recruit 1000 beta testers for my network of startup founders… Please share with your friends

Charles Jo 650.906.2600 Twitter @charlesjo

Begin forwarded message:

On Thursday, Jan 15, 2015 at 12:08 PM, <1k-beta-testers-request>, wrote:

Welcome to the mailing list!

To post to this list, send your message to:

General information about the mailing list is at:

languagengine – Blog – NLP for AI Needs Structured Meaning

NLP for AI Needs Structured Meaning

posted by Darryl on 14 Jan 2015

In my previous blog post, I mentioned that new school NLP techniques for handling meaning can’t really cope with the complexities of actual meaning in language, and typically are pretty primitive as a result. In this blog post, I’m going to discuss a number of aspects of natural language semantics which pose a problem for primitive, unstructured approaches, all of which are relevant for bigger AI projects. Hopefully you’ll get a sense of why I think more structure is better, or at least you’ll have an idea of what to try to solve with new school techniques.

There are five major phenomena related to natural language meanings which I’m going to focus on:

– Linear Order
– Grouping
– Non-uniform Composition
– Inference Patterns
– Invisible Meanings

Don’t worry if you don’t know what these are, they’re just presented here so you know the route we’re going to take in this post.

Linear Order

Linear order matters. Not just to grammaticality, making “John saw Susan” grammatical in English but not “saw John Susan”, but also to meaning. Treating a sentence as a bag of words doesn’t suffice to capture meaning, because then the sentences “John saw Susan” and “Susan saw John” are the same. They have the exact same bag of words representation, so the exact same meaning. But of course they have very different meanings.

Even simple order information isn’t quite enough. Even tho “John” and “Susan” are in the same order in the sentences “John saw Susan” and “John was seen by Susan”, they mean different things because of the verbal morphology. Topicalization in English makes things even worse, because it has no morphological effects, so “JOHN Susan saw” doesn’t mean the same thing as “John saw Susan”. (If you don’t quite like this sentence, try “Susan I don’t know, but John I know”.)

If we have order together with some kind of constructional analysis (active vs. passive, topicalized vs. not) we’d be able to sort out some of this, perhaps mapping to a single meaning for all of these. This would constitute a richer class of semantic representations, and a considerably richer mapping from the syntactic form to the semantic form.


Not only does linear order matter, but there are also grouping or association effects. That is to say, the meanings that correspond to different parts of a sentence “go together” in ways that seem to depend on how the parts of the sentence themselves go together. Multi-clause sentences are a perfect example of this.

Consider the sentence “John saw Susan before Michael met Stephen”. As a bag of words we’re going to miss out on a lot of what’s going on here, but even if we have some consideration of word order as described above, we probably don’t want to view this sentence as one big whole. That is to say, we probably don’t want to look at this as having four noun phrases — John, Susan, Michael, and Stephen — in some construction.

It’d be better to view it as two clauses, “John saw Susan” and “Michael met Stephen” combined with the word “before”, so that John and Susan “go with” the seeing, and Michael and Stephen “go with” the meeting, not vice versa. The fact that we can reverse the clauses and change only the temporal order, not the goings-with — “Michael met Stephen before John saw Susan” — emphasizes this.

Another example of how things associate in non-trivial ways is ambiguity. Consider “the man read the book on the table”. We can say this to describe two situations, one where we’re talking about a man sitting on table while reading a book, and another where the man is sitting in a chair reading a book, which happens to be currently (at the time of speech) on the table. The second interpretation is especially drawn out in this conversation:

Q: Which book did he read? A: The one on the table.

There is also no way to get a third interpretation where the guy who read to book is currently sitting on the table, even tho he read it while sitting in a chair. So this conversation does NOTcohere with that sentence:

Q: Who read the book? A: The guy on the table.

Any solution we give to this problem, so that these meanings are all distinguished, will require more structure than just a bag of words or even linear order.

Non-uniform Composition

Another aspect of meaning which requires structure is the way meanings compose to form larger meanings. Some meanings can go together to form larger meanings, while others can’t. For instance, while both dogs and ideas can be interesting, only dogs can be brown. A brown idea is kind of nonsense (or at the least, we don’t know how to judge when an idea is brown).

Even when meanings can compose, they sometimes compose differently. So while a brown dog and a brown elephant are brown in more or less the same way — it’s kind of conjunctive or intersective, in that a brown dog is both brown and a dog. On the other hand, a small dog and a small elephant are small in different ways. A small dog is on the whole pretty minute, but a small elephant isn’t small, it’s still about as big as some cars! When the meanings of words like “small” (called subsective adjectives) combine with noun meanings, they produce a meaning that is relative to the noun meaning — small for an elephant, not just small and an elephant.

Inference Patterns

Inferences are another place where structure reveals itself. One thing we do with meanings is reason about them, and figure out what other things are true in virtue of others being true. For example, if someone says to me, “John is tall and Susan is short”, then I can go tell someone else, “John is tall”. That is to say, the sorts of meanings we have support inferences of the form

if I know A and B then I know A

Other things, such as adjectives, have similar inferences, so if I know “John ate a Margherita pizza”, then I know “John ate a pizza”, because after all, a Margherita pizza is just a kind of pizza. If I negate the sentence, however, this inference is not valid: if I know “John didn’t eat a Margherita pizza”, that does not mean I know “John didn’t eat a pizza”. After all, he might have eaten a Hawaiian pizza instead. However, if I swap the sentences, it’s valid: if I know “John didn’t eat a pizza”, then I know “John didn’t eat a Margherita pizza”.

These entailment patterns aren’t just found with the word “not”, but with all sorts of other words. Quantifiers, such as “every”, “few”, and “no”, all impose entailment patterns, and sometimes not the same ones in all places. Consider the following examples, where  means “entails”, and means “does not entail”:

every dog barked ⊢ every brown dog barked every brown dog barked ⊬ every dog barked every dog barked ⊬ every dog barked loudly every dog barked loudly ⊢ every dog barked few dogs barked ⊬ few brown dogs barked few brown dogs barked ⊬ few dogs barked few dogs barked ⊢ few dogs barked loudly few dogs barked loudly ⊬ few dogs barked no dogs barked ⊢ no brown dogs barked no brown dogs barked ⊬ no dogs barked no dogs barked ⊢ no dogs barked loudly no dogs barked loudly ⊬ no dogs barked

The basic pattern for these sentences is “Q dog(s) barked”, either with or without “brown” and “loudly”. But these words have completely different patterns of entailment! Here’s a table summarizing the patterns and the data:

A = Q dog(s) barked B = Q brown dog(s) barked C = Q dog(s) barked loudly Q \ Direction | A ⊢ B | B ⊢ A | A ⊢ C | C ⊢ A --------------------------------------------------------- every | Y | N | N | Y few | N | N | Y | N no | Y | N | Y | N

Invisible Meanings

Lastly, some words or constructions seem to have more meaning than is obvious from just the obvious parts of the sentence. Consider the verb “copy”. It has a subject, the copier, and an object, the original being copied. There’s no way to express the duplicate object being created, however. You can’t say “John copied the paper the page” to mean he copied the paper and out from the copying machine came the copied page, or something. There’s just no way to use that verb and have a noun phrase in there that represents the duplicate.

But the duplicate is still crucially part of the meaning, and in fact can be referred to later on. In the absence of a previously mentioned duplicate, it’s weird to say “he put the duplicate in his book”. Which duplicate? You didn’t mention any duplicate. But if you first say “he copied the paper”, then you know exactly which duplicate, even tho there’s no phrase in that sentence that mentions it explicitly. It’s implicit to the verb meaning, but still accessible.

Wrap Up

Building an AI requires grasping meaning, and hopefully this post has helped to lay out some of the reasons why structure is necessary for meaning and therefore for NLP for AI. There are solutions to all of these that are pretty straight forward, but which are vastly richer in structure than the kind of things you find in a new school NLP approach. I think new school algorithms can be combined with these richer structures, tho, so the future looks bright. If you think there might be a pure new school solution that doesn’t require richer structure, let me know! I’d be interested in alternatives.

If you have comments or questions, get it touch. I’m @psygnisfive on Twitter, augur on freenode (in #languagengine and #haskell). Here’s the HN thread if you prefer that mode, and also the Reddit thread.

languagengine – Blog – The Conversational AI Problem Landscape by @psygnisfive

Original post:


The Conversational AI
Problem Landscape

posted by Darryl on 10 Jan 2015

In this blog post, I’m going to discuss some of the major problems that I think conversational AI faces in achieving truly human-like performance. The issues I have in mind are

– Syntax / Grammar
– Non-idiomatic Meaning
– Idiomatic Meaning
– Metaphors / Analogies
– Speaker’s Intentions and Goals

Some of them are, in my opinion, relatively easy to solve, at least in terms of building the necessary programs/tools (data and learning is a separate issue), but others, especially the last one, are substantially harder. But hopefully I can convince you that at least the first four are something we can seriously consider working on.

Syntax / Grammar

Syntax/grammar is the way words combine to form larger units like phrases, sentences, etc. A great deal is known about syntax from decades of scientific investigation, despite what you might conclude from just looking at popular NLP.

One major reason NLP doesn’t care much about scientific insights is that most NLP tasks don’t need to use syntax. Spam detection, for instance, or simple sentiment analysis, can be done without any consideration to syntax (tho sentiment analysis does seem to benefit from some primitive syntax).

Additionally, learning grammars (rather than hand-building them) is hard. CFGs are generally learned from pre-existing parsed corpuses built by linguists, like the Penn Treebank. As a result, NLPers tend to prefer simpler, but drastically less capable, regular grammars (in the form of Markov models/n-gram models/etc.), which require nothing but string inputs.

Non-idiomatic Meaning

By non-idiomatic meaning I simply mean the meaning that an expression has in virtue of the meanings of its parts and the way those parts are put together, often called compositional meaning. For example, “brown cow” means something approximately like “is brown” and “is a cow” at the same time. Something is a brown cow, non-idiomatically, if it’s brown, and if it’s a cow.

In a very real sense, non-idiomatic meaning is extremely well understood, far more than mainstream NLP would indicate, just like syntax. Higher-order logics can capture the compositional aspects tidily, while the core meanings of words/predicates can be pushed out to the non-linguistic systems (e.g. its a non-linguistic fact that the word “JPEG” classifies the things it classifies the way it does).

Idiomatic Meaning

Idiomatic meaning is just meaning that is not compositional in the above sense. So while “brown cow” non-idiomatically or compositionally means a thing which is both brown and a cow, idiomatically it’s the name of a kind of drink. Similarly, “kick the bucket” non-idiomatically/compositionally means just some act of punting some bucket, idiomatically it means to die.

Idioms are also pretty easy. Typically, meaning is assigned to syntactic structures as follows: words get assigned values, while local regions of a parse tree are given compositional rules (for instance, subject + verb phrase is interpreted by function application, applying the meaning of the verb phrase to the meaning of the subject).

Idioms are just another way of assigning meanings to a parse tree, e.g. the tree for “kick the bucket” is assigned a second meaning “die” along side its normal compositional meaning. There are relatively few commonly used idioms, so enumerating them in a language system isn’t too huge a task.

Metaphors / Analogies

Metaphorical or analogical language is considerable harder than idiomatic meaning. Unlike idioms, metaphors and analogies tend to be invented on the fly. There are certain conventional metaphors, especially conceptual metaphors for thinking, such as the change-of-state-is-motion metaphor, but we can invent and pick up new metaphors pretty easily just by encountering them.

The typical structure of metaphors/analogies is that we’re thinking about one domain of knowledge by way of another domain of knowledge. For instance, thinking of proving a theory as a kind of game, or programs as a kind of recipe. These work by identifying key overlaps in the features and structures of both domains, and then ignoring the differences. Good metaphors and analogies also have mappings back and forth associating the structure of the domains so that translation is possible.

Take, for instance, the linguistic metaphor “a river of umbrellas”, as one might describe that one scene from Minority Report outside the mall. The logic behind the metaphor is: the concept ‘river’ is linked to the word “river”, and rivers are long, narrow, and consist of many individual bits moving in roughly the same direction. This is also what the sidewalk looked like — long, narrow, with many individual bits moving in roughly the same direction, only instead of the bits being bits of water, they were people holding umbrellas. So we can think of this sidewalk full of people like we think of rivers, and therefore use the words for rivers to talk about the sidewalk and the people on it with their umbrellas.

The most likely scalable solution to metaphorical/analogical thinking is identifying the core components that are readily available. Geometric, spatial, kinematic, and numeric features tend to be very easy to pick out as features for making analogies. As for identifying the use of metaphors in language, typically metaphorical usages are highly atypical. The earlier metaphor of a river of hats is unusual in that rivers aren’t made of hats, and this mismatch expectation and usage can be used to cue a search for the metaphorical mapping that underlies the expression.

Fortunately, there is also quite a bit of work on conceptual metaphor that can be drawn upon, including a great deal of work in cognitive linguistics such as Lakoff and Johnson’s Metaphors we Live By, Lakoff’s Women, Fire, and Dangerous Things, and Fauconnier’s Mental Spaces, just to name a few classic texts.

Speaker’s Intentions and Goals

Possibly the hardest aspect of interacting conversationally is knowing how a speakers intentions and goals in the conversation guide the plausible moves the AI can make in a conversation. All sorts of Gricean principles become important, most of which can probably be seen as non-linguistic principles governing how humans interact more broadly.

Consider for example a situation where a person’s car runs out of gass on the side of the road, and a stranger walks by. The driver might say to the stranger, “do you know where I can get some gas?” and the passer-by might say “there’s a gas station just up the road a bit”. Now, any geek will know that the _literal_ answer to the drivers question is “yes” and that’s it — it was a yes/no question, after all. But the passer-by instead gives the location of a gas station. Why? Well probably a good explanation is that the passer-by can infer the speaker’s goal — getting some gas — and the information required to achieve that goal — the location of a gas station — and provides as much of that information as possible.

The driver’s question was the first step in getting to that goal, perhaps with a literal conversation going something like this:

Driver: Do you know what I can get some gas?
Passer-by: Yes.
Driver: Please tell me.
Passer-by: At the gas station up the road a bit.

But the passer-by can see this conversation play out, and immediately jumps to the crucial information — where the gas station is located.

This kind of interaction is especially hard because it requires, firstly, the ability to identify the speakers goals and intentions (in this case, getting gas), and figuring out appropriate responses. A cooperative principle — be helpful, help the person achieve their goals — is a good start for an AI (better than a principle of ‘avoid humans they suck’) — but something more is still necessary.

A possible solution is some inference tools, especially frame-based. The situation that the driver was in ought to be highly-biasing towards the driver’s goal being to get gas even without the driver saying a thing, because the functions and purposes of various things involved, plus typical failure modes, would strongly push towards getting gas as important. However this now requires large-scale world knowledge (tho projects like Cyc and OpenCog might be able to be relied-on?).

If you have comments or questions, get it touch. I’m @psygnisfive on Twitter, augur on freenode (in #languagengine and #haskell). Here’s the HN thread if you prefer that mode, and also the Reddit threads.

Teamstory startup community

Another startup community to check out.

Charles Jo 650.906.2600 Twitter @charlesjo

Begin forwarded message:

On Tuesday, Jan 6, 2015 at 2:04 PM, Kevin <kevin>, wrote:

HeyCharles Jo!

Kevin from Teamstory here :) Hope you had a great 2014 and off to an awesome start in 2015. I am just checking in to see if everything was going well with you and your projects.

Teamstory has been growing as a unique, supportive community since the last time you checked it out. Founders and startups are sharing some amazing moments, thoughts, and questions to spark conversations. There were also some neat changes and improvements in the product to make Teamstory a great product for entrepreneurs (such as improved thought posts, 1-on-1 chat system etc.)

Would love to see what you and your team’s moments & thoughts on Teamstory soon!
Check it out, you’ll want to use it everyday 😉

Check out Teamstory AngelList | Twitter | Facebook

Kevin from Teamstory 6 Jan 2015
Powered by Intercom
Unsubscribe from our emails


Launching KingForADay v2.0

To check out…


Launching KingForADay v2.0

Announcing KingForADay v2.0
View this email in your browser

KingForADay V2.0 has launched!

Group Chats, Profile Prompts, Refined UI

Eight weeks have passed since we introduced KingForADay. What a response we have had from the community! 57 Kings, 1.5k chats initiated, and 24k messages sent later, we have cultivated a strong, passionate community around social discovery.

We are introducing some great new features with v2.0 including group chat, and profile prompts, all to enhance the experience of interacting with the King and with others. See the press release.

Download Now

Group Conversations

Invite others in to your conversation with the King. Create groups to chat with future Kings.

Discover more than just a bio

Profile prompts allow for a more free-form, easily-consumable method of providing the hook for engagement.

color-facebook-48.png Share
color-twitter-48.png Tweet
color-forwardtofriend-48.png Forward
Copyright © 2015 King For A Day, All rights reserved.
You are receiving this email because you registered in the KingForADay app.

Our mailing address is:
King For A Day16817 Liggett St
North Hills, CA 91343

Add us to your address book

unsubscribe from this list update subscription preferences

Email Marketing Powered by MailChimp


languagengine – Blog – Deep Learning Won’t Cure The Language Cold

Interesting tech blog from Darryl McAdams’ startup.

Charles Jo 650.906.2600

Deep Learning Won’t Cure the Language Cold

posted by Darryl on 5 Jan 2015

Every once in a while I get asked about what sort of machine learning I used to build Language Engine, and when I say I didn’t use any, the response is something like “But deep learning is so big these days!” While I’m tempted to just pull a Peter Thiel and say this buzzword-y nonsense is just fashion, and suggest you avoid fashions like the plague, I want to explain a bit instead. So in this post, I’m going to elaborate on how deep learning relates to Language Engine, and why we don’t use it, or any other kind of machine learning, and why future versions will only use minimal amounts.

Deep Learning, Roughly

Before jumping into that, however, it’ll be good to discuss deep learning from a birds-eye view. So, what is deep learning, exactly? In a way, deep learning is a way of building a bucket sort program, only the algorithm discovers the buckets for you. Given a bunch of input data, you train a deep learning neural network on the data, and out comes the buckets that work best to group the input data. Later, during the usage of the network, you feed novel data in, and out comes a bucket identifier.

For example, you might train your deep learning neural network on a collection of images of cats and dogs, and it will automatically discover, on its own, that there are two useful buckets it can categorize things into: “cat” and “dog” (or maybe something else, who knows!). Later, you take a picture you’ve never seen before, show it to the network, and it will tell you either “cat” or “dog”, depending on which choice it considers most probable.

Let me repeat the main point for emphasis: deep learning sorts things into buckets that it discovers automatically.

Natural Language

Now, the fundamental problem that Language Engine solves is: how can we extract useful structured representations of meaning from natural language input, so that host applications can use this to respond appropriately via actions, etc. With this in mind, how might we use deep learning? One option would be to use the whole sentence as input, and use deep learning to categorize the sentences based on some kind of “intent”. Maybe it’s a “turn-on-the-lights” intent, or a “send-a-tweet” intent. Regardless, this is a very coarse-grained approach.

How many intents are there? How many meanings of English are there? As many as there are sentences: infinitely many. So if we wanted to use such a coarse-grained approach, the results would be useful only up to a point. Any aspects of the meaning that isn’t as granular as that will be lost. We only have so many buckets. We can keep increasing the number of buckets to get more and more detail, but this becomes increasingly hard to use: eventually we get one intent, one “meaning”, for each sentence, and they’re all different! Infinitely many buckets is just as useless as very few buckets.

So what’s the right solution? Structured meaning. It’s not enough to know that “John saw Susan” goes in Bucket A, and “Stephen saw Michael” goes in Bucket B. It’s not even enough to know that Bucket A and Bucket B are similar to one another. What you really need, to properly understand what this sentence means, is what the structure of the meaning is: what are the parts of the sentence, their meanings, and how do these meanings combine to form a cohesive whole?

A good analogy for programmers is, well, programs. The parts of a program mean things, and the meaning of the whole program comes from the meanings of the parts and the way they’re put together. You can categorize programs into buckets like “sorting function” or “web server”, but that’s not enough to understand any given program, nor is it enough to run a program. If we want to “run” sentences of natural language as if they’re little programs telling a computer what to do, we need the same structural richness that programs have, hence Language Engine.

But isn’t there some way to…

All of this is not to say that there is no use for machine learning at all. Quite the contrary, there’s plenty of use, especially down the road! Right now, the best application of machine learning, including deep learning, to natural language involves using the bucketing tools to determine most-likely parses. Natural language, you may have heard, has lots of ambiguity. That is to say, there might be many parses for the same string of words, so which is the “right” parse? Work by Socher, Bauer, Manning, and Ng (2013) (here) uses deep learning for precisely this purpose, and perhaps similar techniques can be used on semantic representations to make it even more powerful.

But that’s down the road. Before Socher et al. could bolt deep learning onto context free grammars, someone had to invent the idea of context free grammars, and explain how to use them to represent sentence structures. So before you can apply your deep learning algos to meaning, you have to have a structured meaning to use, and that’s what Language Engine provides.

If you have comments or questions, get it touch. I’m @psygnisfive on Twitter, augur on freenode (in #languagengine and #haskell). Here’s the HN thread if you prefer that mode, and also the Reddit thread.


StartupStudyGroup mailing list