Posts Tagged ‘data’

Ensuring the Best Data For Our Customers

Sysomos Twitter CommunitiesAt Sysomos, we pride ourselves on the quality of our data and of our data science. For the last eight years we have enabled our clients to leverage social intelligence to help power their business decisions and keep them aware of trends, insights and discoveries.

We’re also proud to partner with some of the most exciting companies to provide access to real-time conversations as they hit social networks.

One of our valued data partners is Twitter.

With a recently announced change to Twitter’s API, we wanted to ensure our customers that it’s business as usual for Sysomos.

Through our work with Twitter, we have been and continue to be a Twitter Certified Partner. This gives us complete and direct access to Twitter data.

As a part of our ongoing relationship with Twitter, Sysomos is able to leverage the benefits afforded to Twitter Certified Partners including:

  • Insights into Twitter’s product direction
  • The ability to work with new products
  • An open dialogue with and direct communication to the Twitter team

Upcoming Changes to the Facebook API

In addition to Twitter, Facebook is also changing the way it provides data to the marketplace. As of April 30, Facebook is making changes to the data they offer to analytics companies to help protect the privacy of their users. (See important update below)

It is important to note that this deprecation will affect all social listening platforms that rely on this data. Only public Facebook data will be affected – Facebook fanpage data will still be available in Sysomos.

While public status updates will no longer be available from Facebook, Fanpage data continues to be accessible. Post April 30th Sysomos customers will now be able to search this data within Sysomos. Data collected from Heartbeat via existing integrated Fanpages will also remain in your Heartbeat and will continue to collect mentions without change.

Update Apr 30, 2015: We have extended our access to Facebook public posts until at least June 19th. Customers will continue to see Facebook data in MAP and Heartbeat for the time being.

Engineering a new solution

In the wake of this Facebook API news, we’re taking a two-pronged approach to ensure the best data for our customers. First, we’ve held ongoing discussions with Facebook and are working on new ways to use Facebook’s available API’s within our platform.

Also, through our longtime partnership with Datasift (a preferred data provider that is working with Facebook) we are investigating integrations including using a new data set called Topic Data.

Our Partner and Data Science teams are very excited about exploring future ways to ensure the most complete data sources for our customers.

Should you have any questions, please contact your Sysomos representative.

What Exactly Is Social Intelligence?

It’s not uncommon for people in the tech world to create a new term when they feel like an old one has lost its lustre.

But in the case of “social intelligence”, we really are referring to the next generation of how social data informs your enterprise far beyond “brand watching”, listening or monitoring.

The Social Intelligence Continuum

What’s so different?

We all know how social media has revolutionized the web. In fact, you could argue that social media now is the web, since the construct of social pervades nearly every corner. The static web is all but extinct.

So we’ve taken the unfiltered, dynamic voices of the web and amplified them exponentially.

The difference between “now” and “then” is a function of two important things: 1) scale; and 2) customer relevance.

How Data Has Grown Up

When people talk “Big Data”, what they’re really referring to is the huge masses of unstructured data that comes from the complex social interactions that make up the web today. It’s impossible to just run that kind of data through a spreadsheet.

Sysomos processes several terabytes of data in a day, and that adds up to several petabytes of data in a year, and it is ever-increasing as the universe of the web continues to expand.

Applying the sophisticated math and science to make sense of that data and deliver something meaningful and actionable in a dashboard is no small feat, and Sysomos is one of the few companies in the world that is working with data at this scale.

So when we talk about Intelligence, we truly mean moving the needle from simply finding “hits” on a mention of a keyword to understanding the historic patterns in the data and using that information to not only predict what could happen next through predictive analytics, but get in front of future events.

Why is being predictive so darned important?

Customer needs have evolved far beyond just needing a response to a tweet (though we still have work to do to make that a mainstream best practice, but that’s for another post).

Frankly, the real promise of social – the one we all talked about back in 2006 or so – hasn’t really been fulfilled.

We promised that social would create for customers a more personalized, relevant, real-time world.

Now, data science like we’re doing at Sysomos is finally allowing us to deliver on that promise by observing and analyzing patterns in information, behaviours, preferences…and helping companies deliver exactly the right content, offer, or product in the moment when someone needs it, where it’s most appropriate.

It’s not just about listening anymore.

Social intelligence brings about an age of understanding influence, measuring community dynamics, predicting future outcomes, and applying those learnings to create mass relevance through products, experiences, and business operations in real-time.

We can not only become the early warning system for business threats and opportunities, but the engine for customer advocacy inside the business.

We can pinpoint the trends AND use that information to help access the “dark” side of social, including the silent majority of people that never publish content but use the web to inform their buying decisions and lifestyles.

And most importantly of all, delivering this level of customer experience and relevance means that we help future-proof companies. We are arming them with the information they need to constantly skate to where the puck is moving, make smart pivots, stay the course when warranted, and shape a business dynamically around the ever-more sophisticated and connected consumer.

That’s a heck of a lot more than the “show me when someone mentions my brand” stuff we started with.

The future is here, and it’s a socially intelligent one.

Are you ready to meet it?

If you’d like to get a taste of what Sysomos’ social intelligence solutions like MAP and Heartbeat can do to move beyond brand mentions and create relevance for your customers, sign up for a demo of the smart, new Sysomos today

5 Takeaways From The PreCommerce Summit You Can Act On Now

W2O's 5th Annual PreCommerce SummitLast week we had the extreme pleasure of sponsoring The W2O Group’s 5th Annual PreCommerce Summit. This is a gathering of some very smart people to talk about what’s driving markets and people today. The event was a whole day of speakers from across a wide variety of industries that had something interesting to teach everyone in the audience. Topics for the day ranged from data across all industires to biases in our minds and workplaces to to the internet of things to telling better stories and even a fun fireside chat with Al Roker.

While everyone that spoke at this event had something interesting to say or teach the crowd, we picked out 5 things that we thought everyone in any industry should learn and can think about. In no particular order, here they are:

Try doing different things” – Mike Marinello, Head of Global Communications, Bloomberg

Mike Marinello’s talk at the PreCommerce Summit was actually about the need to be able to measure everything they do so they can show the value of it. That said though, Marinello said that his company isn’t afraid to try new and different things to attain their goals. When they know exactly what they’re measuring towards, they can try interesting new things because they will know which of them works and which don’t. Not everything is going to work in terms of helping to achieve your goals, but you’ll never know what else will if you’re not willing to at least try new and different things.

Use your data to tell an important story” – Chuck Hemann, Global Analytics Manager, Intel

As the Global Analytics Manager, Chuck Hemann sees a lot of data every day. He knows though that not all of it can be fully used all the time. That’s why he tells his team to pick out the one or two points of data that they think are most important for what the company is trying to achieve and tell that data’s story in a way that everyone will understand. Why is that data important? What does it mean to us right now? What does it mean we should be doing in the future? Data is very important to companies today, but you can sometimes have way to much data. Know what data is important and don’t just show numbers to people, use that data to tell a story and make people understand why it’s so important.

Engage or die” – Ray Kerins, SVP, Head of Communications and Government Relations, Bayer

Ray Kerins works in the world of pharma, which is notorious for having a lot of restrictions around what they can and can’t say. However, in a world where everyone and anyone has a voice, Kerins knows that if him and his team aren’t joining the conversation along with those other voices, whatever they say will be the only thing heard. Kerins talked about how he believes engagement is the way to get your story heard by everyone, especially those talking about you. If you don’t engage with those people that are talking about you then those people will own your compny’s story, not your company. So, engage or die.

Tech isn’t the big disruptor, business models are” – Mike Edelheart, CEO, Pivotcon

Too often we hear “this new app/software is going to change the game.” But how often is that actually the case? During his talk at the PreCommerce Summit, Mike Edelheart was quick to point out that tech can sometimes help to change games, but really it’s the business model behind what the tech is doing that’s the driving factor. Yes, Netflix was a technology, but it was really it’s streaming movies and TV shows anytime anywhere business model that changed the game. Same with Uber. Uber is an app, but it completely changed the business model for how people can get from point A to point B quickly and efficiently. So if you really want your company to be the next big industry disruptor, maybe don’t think about that app you have to build and think about how you can change the way you’re doing business to do it better.

Stop thinking of marketing like war and instead think of it like a garden” – Daina Middleton, Head of Global Business, Twitter

Daina Middleton started her talk by pointing out the fact that a lot of marketing terms are very similar to war terminology. Marketers talk about their stratagies and tactics to get their messages out. But in a world where marketing has turned into cultivating relationships and communities, maybe we need to think of marketing more like a garden. Relationships don’t just happen because you have a good marketing campaign. They happen because you need to put time and effort into slowly gaining trust and then nurturing that so that you can form an actual relationship with people. It’s very much like a garden. Your garden doesn’t grow overnight because you found the best soil or water. It grows over time because you put time and effort into it. Middleton actually has a formula for participation that says Discovery + Engagement + Connection = Participation, which is what makes up a real community. And all of these things take time and effort, but are much more worth it in the long term.


These are just a few things that we learned at the 5th Annual PreCommerce Summit, but they are also things that we think every company can start thinking about and act on right now. For a full run down of what every speaker had to say during the PreCommerce Summit, check out the W2O blog.

Sysomos Partners With Tickr To Bring Business Intelligence To A Whole New Level

Today we’re both proud and excited to announce our new partnership with Tickr, a pioneer in the real-time visualization of enterprise-level data.

As of today, Sysomos will be known as a ‘Preferred Social Intelligence Partner’ in the already expansive Tickr ecosystem.

But what is Tickr and why would this be useful to you?

Before Tickr, marketers, analysts, mangers and others would have to log into dozens of websites so that they could see what is going on with their web analytics, track media, see sales information and, of course, view what is going on with their social media. But Tickr helps to simplify all of that by gathering data from thousands of different sources — including Oracle, SAP and Salesforce, not to mention social networks —and then presents the information in a single custom dashboard or “Command Center” that offers a complete picture of brand and business performance in real time.

“We want to partner with a leader in the social data provider industry, and Sysomos is that leader,” says Tyler Peppel, CEO and Founder of Tickr. “Sysomos enables brands to build towards the next generation of social business intelligence, and we recognize that by working together, we can visualize the enterprise like no one else can.”

As a ‘Preferred Social Intelligence Partner,’ Sysomos will deliver not only the social data that populates the stunning Tickr user interface, but also the insights that power better decision-making at the enterprise level. To put it plainly, the real-time visualization and monitoring of social activity alongside other key performance metrics will create a comprehensive intelligence platform that we feel is unmatched in the industry.

Our CEO, Jim Delaney says, “As a powerful bridge between social data and enterprise metrics, Tickr is an ideal partner for us. Tickr is an extension of Sysomos’ data capabilities, with a customizable user interface that allows us to offer customers rich information in a compelling visual format that businesses and marketers can quickly digest.”

We’re incredibly enthusiatic about this new partnership and think that you should be as well.

If you’d like to learn more about how Sysomos and Tickr can work together to help you improve your analysis, decision making and ultimately, your business, please feel free to contact us and we’re happy to get the conversation started.

For more information about this new partnership, see the official press release here.

How To Present Data So That It Sticks

It was four years ago at SXSW that I first met Eric Swayne. At the time, Eric was a Sysomos client through the agency he was currently working and wanted to meet someone from our team. I gladly agreed and we sat down over lunch and had a very interesting conversation.

I quickly realized that Eric was a pretty smart cookie. Ever since then we’ve kept in touch via emails and Twitter and we always meet up every year in Austin when we return for SXSW. Also, if I’m ever at an event where Eric is speaking, I make sure that I get to his session so I can hear what he has to say.

This year’s SXSW Interactive was no different. I had casually bumped into Eric at a BBQ event and he told me that would be giving a talk two days later. While it wasn’t on my original schedule (simply due to a mistake of not seeing he’d be speaking), I quickly changed my plans for Monday afternoon to make sure I was in attendance.

What I saw was a fantastic presentation entitled Science to Storyteller: Enter the Data Narrator. In this presentation Eric talked about how you present data really matters and how to do it better. I thought this would be quite an interesting topic, especially to people who read this blog, so I took the tweets that came out during his presentation and turned them into a Storfiy story.

Below you’ll find the Storify of Eric’s presentation with highlights that include great tips on presenting data, such as:

  • Seeing something in data and getting others to understand are two completely different things
  • What a data insight is
  • How to find an insight and then make it stick with the people you’re presenting it to
  • Show just the right amount of data
  • And, how to deal with insights and scheduled or automated reports

Here’s what people picked out and tweeted during the presentation:

If you’d like to see the actual presentation, head over to Eric Swayne’s website and see the Prezi presentation in full.

Do you have any tips on presenting data? Leave them in the comments for everyone to see.


200 Billion And Counting

Everyday the world of social media grows exponentially. New people get online. More people discover a social network that they love. People upload more and more media. And of course, the flow of content, whether it’s a tweet, an article, a blog post, a status update or a video, never stops.

You may remember that a mere 8 months ago our social media monitoring and analytics software powered by the powerful Sysomos engine indexed its 100 billionth piece of content. Well, if only to highlight to the world the quick and massive growth of social media use, on Tuesday we indexed our 200 billionth piece of content. That’s 200,000,000,000 written out in numbers.

This means that our customers now have access to over 200 billion social media conversations that they can analyze in mere seconds.

To demonstrate how quickly the rate of social media content grows I decided to conduct a little experiment. I took a bunch of common words (it, its, and, the, what, why, I, a, to, too, or, if, you, your) and looked them up in our MAP software to see how many times they appeared yesterday (May 29, 2013), a year ago (May 30, 2012) and the date we hit 100 billion (September 19, 2012).

The results I found were actually quite interesting and help to demonstrate my point quite nicely.

One year ago, I found 125 million conversations between blogs, online news, forums and Twitter containing my list of common words. By the time September 19th rolled around those same words generated 127 million results. That’s an increase of 2 million posts per day in almost 4 months. Then 8 months later, yesterday, those same words appeared in an astounding 139 million posts. That’s a jump in 12 million pieces of content.

May 30, 2012

September 19, 2012

May 29, 2013

Granted, my list of common words is far from covering the full gambit of what’s out there in social media and the use of these specific words could vary from day to day. However, for illustration purposes, it works well.

As time goes on, more social networks and channels will appear and more people will realize the magic of social media and being able to connect with people around the world. And as that happens, we’re going to keep on capturing and indexing all those conversations to give our customers the largest and most complete sets of social media data.

Using Data To Delight Your Community

Every company or brand out there has its fans and naysayers. It’s just a part of business. One great example where we’ve seen both types prominently over the past few years is the company formerly known as RIM (now known as just BlackBerry).

The company was at one point the leader in smartphone technology. In fact, they were probably the first real smartphone makers in the market. But then other companies like Apple and Google entered the market and some people felt that BlackBerry had been left behind. Fast forward a few years and BlackBerry has made a stunning reemergence in the field with their fully redesigned operating system known as BlackBerry 10.

When BlackBerry announced that it was completely revamping itself from the ground up, it was again met with its fair share of vocal fans and naysayers. For example, take a look at this tweet below that highlights one of the naysayers being countered by one of the very vocal fans:

One thing is for sure; whether it was from a naysayer or a fan, there was a lot of talk leading up to the launch of BB10. Check out this popularity chart below for mentions of BB10 over the past 6 months leading up to the launch. There were over 19 million tweets during this period.

Now, this is where the story gets very cool:

TELUS, a large telecom in Canada, was just as excited about launching the BB10 line on their network as some of the super fans out there were. They also knew how excited a lot of their customers were for the new BlackBerry devices. That’s why they decided to reward a lucky customer who was the most eager (and persistent) for the big release.

Using our MAP platform, TELUS was able to analyze millions of conversations about BlackBerry and BB10 from across Canada to find the people that were talking the most positively about their excitement for the new smartphone. By cross-referencing the top BB10 anticipators with their client records, they were able to grant one lucky customer’s wish of being one of the first people in the world to own a BlackBerryZ10 device.

This is a great example of how companies can use big data (both from social media and from their own databases) to show their customers and fan base that they’re listening and that they care what they think.

Dan Fricker, TELUS’s Social Media Community Manager, had this to say:

“Social media’s one of our many ways of connecting and actually having conversations with customers. What Sysomos offers is an incredible way to listen to those conversations, from different people all over the country. Beyond engaging in real-time interactions, we can also go back and see what people have been anticipating most about the launch of BlackBerry 10, for example, or who’s been talking about this new device the longest. That’s the case with @Im_Sure_ who’s been tweeting with @TELUS about the BB10 for weeks. With tools like Sysomos and the power of social we can engage in customer conversations like Matt’s, arguably the BB10’s #BestFan. Given Matt’s such a big BlackBerry fan, we surprised him with his very own Z10 today.”

Data, Data and More Data

This blog post is first in the “Engineering” series by Sysomos’ co-founder and CTO Nilesh Bansal. As part of this series, Nilesh will share experiences in engineering Sysomos’ social media platform.

One question that is frequently asked is: What’s the biggest challenge I face? The simple answer is: data.

As I write this sentence, in less than a minute, our crawlers have collected tens of thousands of new conversations happening online. Within the same minute, each of these conversations was discovered, retrieved, cleaned, analyzed, and stored on our servers. Now, that is a lot of data.

We store billions of documents on our servers. Every hour, millions of them are read and analyzed by users. In the last few years, our team has experimented with a variety of options to get a better understanding of the black art of data management. I’ll share some of them in this post.

There are two main components of the storage layer: hardware and software. As well, there’s a third option: outsourcing by using Amazon’s cloud infrastructure, S3 and EC2. Cloud storage is a convenient option, and, if used properly, even economical. But convenience comes at the price of flexibility.

While Amazon has steadily added more customization options and features, there still isn’t enough flexibility to meet our needs. The lack of flexibility also limits our options to innovate such as our plans to start using solid-state drives. As a result, we have stayed with conventional on-premise solution.

Storage Hardware Disks and storage bays are the most expensive part of a purchase order. They are also the slowest and the least reliable. This means they have to be selected carefully.

There are three main architecture options. First, network mounts and NAS obviously will not work given the low latency requirements. Second, fiber-based SAN offers flexibility in adding new disk arrays or moving them across hosts, but is significantly more expensive. If planned properly, this flexibility is not really needed.

The last option, which I prefer among the three, is internal and direct attached storage. If I had to select one configuration option, I would go with a 2U server with 12 bays containing 1TB SATA disks, 60-80GB RAM, and 16 processing cores. This provides a good balance of computing power and storage space. Adding more disks is easy by adding an external disk array connected via SAS cable.

Reliability Disks fail, and when they fail all the data is lost. RAID is used to store multiple copies of the same data on different disks to ensure reliability. RAID 5 is the most commonly used option. However, the disk sizes have increased exponentially and the bit error rates still remain at the same level, there is a non-zero chance of data loss in RAID 5. RAID 6 adds an extra disk to RAID 5 to provide higher reliability. The data write speeds in both RAID 5 and 6 is slow and not best for what we do.

We use RAID 1+0 where all data is mirrored on two different disks. Since all data is stored twice on two separate disks, it means twice the cost but it also provides the best reliability and high performance.

Storage Software As our crawlers continue to add more data every minute, and our users analyze thousands of documents every second, data storage is an important consideration. While we use a combination of different solutions from flat files and custom data structures to inverted indexes, key value pairs and relational database, the bulk of our data is handled by MySQL.

For the most part, MySQL is used as a simple key value store. Since a single instance of MySQL can’t hold all the billions of documents within Sysomos, we partition the data logically across several big, fat servers. Each server is maintained independently (as NDB cluster does not really scale) using primarily the InnoDB table format. Inside each instance, we further partition the data logically to hundreds of different tables. This partitioning let us add new data without hitting the wall.

While MySQL is good enough for basic SELECT and INSERT operations, this is all it can do. Even thinking of a JOIN or any complicated operation can make the server crash. But as long as it is used as a key-value store, MySQL can handle a lot of data and provide for all replication and backup needs.

Key-Value Stores New generation of data stores are gaining popularity. Most notable ones are Apache’s HBase, Facebook’s Cassandra, LinkedIn’s Project Voldemort and Baidu’s Hypertable. Each of these have big-name backers with a lot of hype, and are trying to do what Google does with BigTable.

But they have to become more mature before they become useful for us. For example, when HBase crashes (and it does), it prints the most uninformative error log. Hash based partitioning is used for load balancing, which provides little visibility in where the data actually is, and is often less optimal than logical partitioning when it comes to latency. There is also  very limited user base for each of these outside of there parent companies (which also means bad documentation).

Tokyo Tyrant is another option because it is simple, fast and good for specialized needs.

In summary, it’s all about data. More data means, more we can do with it (and sleep less). I will explore some more topics, including real-time indexing, sentiment analysis, and load balancing, in my next posts, so stay tuned.