Analytics with MongoDB (or.. who sharded??)
Hello there!
Previously, I told you about how much we @LocalResponse love Node.js. Well, last night we had out first NYC Node.js Meetup and it totally rocked! It’s great to see the energy and the passion of developers for new technologies. Today, I want to delve a bit deeper into another technology that we love… MongoDB.
First a super quick primer on MongoDB and how it differs from typical relational databases. Mongo is Schema-less, which means that the database is not confined a strict Schema like those imposed on MySQL DB’s. As a concrete example, consider your standard MySQL DB that has several columns, each with a type and rows to fill those columns. If you’ve defined a column named amount as an int, then whatever data you put in there must always be an int or else bad, bad things happen. With Mongo, there is no such constraint. Within one collection (the Mongo equivalent of a MySQL table), you can have completely different documents (MySQL’ers call these rows) with wildly differing data sets! I won’t get into more detail than that except to say there are TONS of awesome resources out there to learn more about MongoDB:
So one problem that we have is: how do we access LOTS of data quickly? We threw around a lot of ideas and the one that we settled on (for now) is to split one large collection into thousands and thousands of smaller collections (not that these smaller collections are that small - each one can grow to ~50,000 documents). The way MongoDB shards, it allows these different collections to be stored on different machines and we don’t need to worry about those technical details. Go Mongo!
We are now able to perform (near) real-time analytics on thousands (and thousands and thousands) of data points with minimal load on our system.
Share your thoughts, community! You can tweet me: @shamoons and let’s the conversation going. I’d love to hear how others have solved similar problems.
Till next time,
Shamoon Siddiqui, Director of Technical Magic
Welcome to LocalResponse Technology!
Hello there from the land of LocalResponse Technology. We’re doing some pretty incredible things and this is the place for you to hear about them. So first, we’re big into Node.js - we love it. For those that don’t know about it, I’ll get into what it’s all about below. Second, we love MongoDB! With the kind of data we’re dealing with, we need a system that’s fast, scalable, reliable and… fun. What can I say - we love awesome technology.
For this post, I want to talk a bit about Node.js and why we love it. Node.js is Server-side JavaScript so there’s no new language to learn. It’s event-driven and non-blocking which is great for running lots and lots of operations without waiting for a result. As an analogy, think about a standard language (PHP, Python, Ruby, etc) as if it’s a DMV. Think about Node.js as if it’s a restaurant.
With the DMV, we have X number of windows (let’s say 5) each with one employee. There’s one massively long line of people with different needs - someone needs a new license, someone has to pay a bill, etc - and each of these needs takes a different amount of time. Once a request gets to a window, it will take however long it takes to process that request while everyone else waits. Ordinarily, we scale this using horizontal scaling by simply adding more windows. This works since adding windows (or in a web example: servers) is relatively cheap, but fundamentally, you keep throwing resources at the problem.
Node.js is more like a restaurant. The diners sit and enjoy their meal without caring much about the operations of the restaurant. A waiter comes by and takes an order and then immediately leaves to do something else with that order (this is akin to a callback). The order then goes to the chef so that the food can get made. All the while, the waiter is free to take care of other patrons! Once the food is ready (the event is emitted), the food is brought to the patron and all is right in the world.
This is a highly simplified example and of course, it’s not a perfect analogy. But we’re so excited about Node.js that we’re hosting our own Meetup: NYC Node JS. So if you’re in the New York City area, come on by. We’ll form other Meetups as time goes on to discuss even more awesome technology, so stay tuned.
Oh, one more thing.. who am I??? I’m Shamoon Siddiqui the Director of Technical Magic @LocalResponse. You can find me on twitter: Follow @shamoons. I’m a technology leader with ample experience with all sorts of cutting edge technology. I’m eager to help, so definitely reach out to me with any questions you might have (about anything). I’m also a card carrying member of the Society of American Magicians. You can catch a quick video below:
Two Ways the Web is FAILING Local Businesses
Local business owners are frustrated with the web. Yet it holds so much promise. Whats wrong here? Generally, there is a common theme: everyone.com and their mother is trying to separate a business from their customers, rather than bring them together. Isn’t this the exact opposite of what the web is supposed to do? Let me explain. For years, online dot-coms have built portals for customers to talk to each other, talk to strangers, talk to experts, all to get information about businesses. All this communication, but as soon as the businesses get involved, suddenly there is this awkward middleman. He talks on behalf of your business, tells you what you can say, when & where you can say it, and to whom. What just happened? Whose customers are they? Consider if you owned a business and you heard these pitches:
Scenario 1 - Borrowing customers & renting out your business model
“Hey! Want to run a daily deal? We’ll email a list of our customers (consider who has the credit card numbers) and resell them your services, LocalBiz. We can guarantee hundreds of our customers will use your services! The cost? All you have to do is let us use your business model at a 75% (!) discount. Assuming you dont have 90% margins, your job is then to convince MY daily deal customers to come back, to become YOUR regular old customers. Of course when you do, you’ll be charging them twice as much for the exact same thing.” —No wonder 400+ daily deal sites can afford to call business owners every day. Their customers pay them a lot to do so.
Scenario 2 - Cross promotion, to brand dilution, to appreciation fraud
We have a hot new app that all the kids are ‘checking in’ to your business on. Wanna run a deal for free? No problem, LocalBiz! Just tell your customers to become my customers- to ‘download SocialBizFriend.com!’ Put signs up in your store, in fact, make it look like a Nascar for dot-coms, because as it turns out there are hundreds of apps like mine. If you do, we’ll let you engage with… wait for it …your (now our) customers. People you brought me. Go ahead and offer them a deal. When you do, I’ll tell ‘em for you & fraudulently take the credit. “This deal brought to you by SocialBizFriend.com!”
—Local Businesses are somehow left running ads for dot-coms, for free… not the other way around.
What?
This is the crux of the problem with both scenarios: Businesses deserve to own that customer relationship. To talk to their customers directly. When they get a sales call that mentions the word “customers”, they should ask the question - “will I be able to contact this audience directly?”. If the answer is ‘no’, it means they arent your customers at all, and there is almost definitely a ‘catch’, no matter how hidden.
The good news? This will change. The bad news? A lot has to change before we get there. Its our vision at LocalResponse. But we’ll be the first to admit that we are just one fish in the largest of ponds. Local Businesses need to demand they own the customer relationship, and support the businesses that help them do so (there’s our shameless plug).
Local businesses get online-savvier every day. When enough are asking for Local 2.0, the floodgates will come crashing down.
—Michael Muse
Announcing LocalResponse
They say your next moment is the culmination of all previous ones. We feel like we have been waiting for this moment for a very long time.
It’s taken us a cumulative 35+ years of online advertising, mobile marketing, and location-based services experience to put this all together. From ipsh!, Admob, Omnicom, WPP, Media6degrees and most recently buzzd, this team has certainly seen a lot. Taking advantage of a recent but pervasive phenomena — people posting their location/presence to the public (aka the “check-in”) — there is finally enough data for scalable behavioral and location-based targeting, the golden key we’ve been seeking for some time.
Although we believe our platform has wide-ranging implications, we’ve picked three related products to help articulate immediately its market potential. We launch today, with the first— a self-service platform for local businesses to identify and respond to their best customers. We’ve noticed that small business owners have quickly become savvy marketers, educated by a multitude of local sales teams. Yet there still wasn’t one easy, turnkey solution for these businesses to quickly identify their customers across all public data, and interact with them on a one-to-one basis, at scale. That’s where LocalResponse comes in. We hope you enjoy what we’ve created and will join us on this ride to define a brand new marketing medium.
Cheers,
Nihal, Michael & Kathy
Co-Founders, LocalResponse
Response vs. voice
Our team is launching a tool called LocalResponse - a service that helps merchants and brands interact with public conversations about their location. If you’ll pardon the pun, the tool itself is really a response to the advent/growth of people creating public content via mobile devices. This data is really valuable, and its just sitting there begging for a response.
Our goal is to find all these pieces of content, and make it easy for merchants and brands to promote loyalty across platforms by participating in public conversation. However, it is a young channel. There are many dangers in going out with a thoughtless approach. We have done tons of research, surveys, as well as collecting behavioral data and both merchant/brand and customer feedback about communications between business entities and customers over this medium. This is our whitepaper on the best practices for responding to the conversation.
—-
The public channel
In our tool, we find and organize public updates about places. A customer publicly mentions a place, and our tool helps that place monitor that activity, and in response, publicly mention that user. In essence, a user talks about a place and that place talks about the user. The easiest way to do this is via Twitter, with an @mention. Currently, user behavior is such that people monitor their @mentions, so as to know who is talking about them. What this means is that currently there is a large audience of people that care if you talk about them publicly – they are listening.
Soft opt-in
If I am watching who is talking about me and another Twitter user is watching who is talking about him, then we can actually converse by watching each other talk about each other. This was perhaps the greatest innovation presented by the Twitter platform and really key in early viral growth. People could interact without having to establish permissions in advance. This type of communication is what we call a ‘soft opt in’. The users haven’t agreed to allow messaging, but since both are electing to pay attention to public mentions of themselves they can replicate the experience of a conversation, all through the public channel. And many users do.
While @mentions are in fact, mentions, and not direct messages (which currently are private only on Twitter) – user behavior commonly involves checking these mentions, which leads people to use the channel as an ‘inbox’ of sorts. While it is not, Twitter is obviously very focused on supporting this user behavior, so they protect the channel from uses that degrade this experience. To do this, Twitter is watching for users that abuse the channel. They state in their terms various conditions that would lead a user to be blocked from the service, but indiscriminate or robotic messaging is the primary concern. This channel, then is protected by a ‘soft opt in’ – meaning you can use the channel as long as you play by the rules – use it as a place for ‘conversations’ and not indiscriminate messaging. Alternatively, if a user’s @mentions are spammy, then what they are doing is not scalable and Twitter will eventually block the user through a variety of methods.
Context and content
What makes an @mention spammy? Unfortunately, there is no distinct definition. The better place to start is: what makes an @mention NOT spammy. This is particularly delicate in conversations between businesses and consumers, who until recently have not typically conversed. Social media has changed this paradigm. This document aims to present guidelines and best practices for this conversation. In our experience running beta trials with merchants and users – there are two main factors – context and content.
Context can be established in a variety of ways. If a user has recently been to a place, and talked about it publicly via Twitter, there is a substantial amount of context in getting an @mention by the venue in response. This is why we generate a link and a landing page in our mentions that frame the context of the “conversation”.
This is our prototypical case, and it works great for loyalty programs, some of which can achieve 100% CTRs. Context can also be established in other creative ways, but it needs to be intuitive and expressible in ~110 characters.
Content involves the hook within the body of a tweet. They initially notice the tweet because of the mention, but the rest of the tweet needs to deliver value. Think about the nature of your relationship with this person. What would you say if you were face to face with them in your restaurant and wanted to thank them for spreading the word to their followers? Surely you wouldn’t just tell them about an ongoing special. The ‘message’ should not be informational – it should be a special reward designed to make the customer feel a personal connection of some sort with the merchant. For example, in early trials, a free item (appetizer, pitcher) performs much better than a percentage off your bill, even if the former item is often of lower value than the percentage discount. Twitter users don’t want to do math to understand the value of that tweet. We aim to templatize the types of messaging that perform the best. This will allow merchants to find the very best content – the stuff that’s gonna bring customers back, convert them to followers, or even retweet merchant updates to their social graph.
All things in moderation
People love to feel special. They increasingly initiate a social relationship with the merchants and brands they frequent – talking about them or with them directly. However, there is a limit to their attention span. All the best context and content in the world starts to get old if its all promotional. There is no perfect rule of thumb as to a limit, so have established conservative limits. Some users will be annoyed if a merchant mentions them even once, even if they mention the merchant all the time. More often, our data shows that people love the interaction, but not in large quantities. This means that you need to avoid fatiguing the special nature of this interaction. We have developed some best practices for messaging in this channel:
1. Rate limiting – We limit the number of mentions a single user can receive in a day, from all merchants. We have initially made this 1 mention/user/day to be on the conservative side. This way, a user will never get more than one mention from a merchant per day, no matter how many merchants in our system believe they have the context and content to warrant doing so.
2. Frequency capping – This is a limit imposed on the quantity of mentions that a single user will get the same merchant. A common mistake in social media is leaving the loudspeaker on. If a user sees a mention every single day from even their favorite merchant, they would likely find it annoying. We limit each merchant to 1 mention/week/customer. This keeps the interaction casual and special, as opposed to repetitive and mundane. If a user stops listening (clicking) on your updates, even at one per week, this will also be frequency capped, and interactions will be freezed for a time long enough to allow the content and context to refresh.
3. Opt out – If a user doesn’t like being mentioned in the channel – they need to be able to prevent it. In some edge cases, consumers may feel it is ok to mention the businesses without being mentioned in response. In this case, the consumer will be given an option to opt out of communications through the channel. While we cannot block all usage in the @mention channel, we can and will block mentions that come from our system. We will also allow a user to block mentions from a single business and not our entire channel, if they so choose, but the default will be a network-wide opt out.
The big picture
Brand presence 1.0 on Twitter has been as follows: write updates about events and interesting offers at your location, and put up physical signage and hope customers will remember to follow you next time they log into their social media profiles. That’s a pretty weak value proposition – as a customer – there isn’t really any reason I need to adopt the behavior. Merchants need to recruit followership. Within the public channel, they need to demonstrate why a customer would want to do so.
The minor tipping point of establishing a connection over @mention with context and content is actually a major leap forward – its CRM 2.0 on this channel. Acquiring that person to follow you (subscribe to your marketing effort) and recruit their social graph to do the same, all on a public channel, is a very, very powerful thing. At LocalResponse, we build the tools that help you optimize and organize checkins -and respond with meaningful context and content. We will help you organize all the different check in services, show you which posts are public to the world and actionable, help you respond to those customers , and give you the analytics and benchmarks to measure engagement and effectiveness.
-Michael Muse
