Archive for the ‘Engineering’ Category

Ensuring the Best Data For Our Customers

Sysomos Twitter CommunitiesAt Sysomos, we pride ourselves on the quality of our data and of our data science. For the last eight years we have enabled our clients to leverage social intelligence to help power their business decisions and keep them aware of trends, insights and discoveries.

We’re also proud to partner with some of the most exciting companies to provide access to real-time conversations as they hit social networks.

One of our valued data partners is Twitter.

With a recently announced change to Twitter’s API, we wanted to ensure our customers that it’s business as usual for Sysomos.

Through our work with Twitter, we have been and continue to be a Twitter Certified Partner. This gives us complete and direct access to Twitter data.

As a part of our ongoing relationship with Twitter, Sysomos is able to leverage the benefits afforded to Twitter Certified Partners including:

  • Insights into Twitter’s product direction
  • The ability to work with new products
  • An open dialogue with and direct communication to the Twitter team

Upcoming Changes to the Facebook API

In addition to Twitter, Facebook is also changing the way it provides data to the marketplace. As of April 30, Facebook is making changes to the data they offer to analytics companies to help protect the privacy of their users. (See important update below)

It is important to note that this deprecation will affect all social listening platforms that rely on this data. Only public Facebook data will be affected – Facebook fanpage data will still be available in Sysomos.

While public status updates will no longer be available from Facebook, Fanpage data continues to be accessible. Post April 30th Sysomos customers will now be able to search this data within Sysomos. Data collected from Heartbeat via existing integrated Fanpages will also remain in your Heartbeat and will continue to collect mentions without change.

Update Apr 30, 2015: We have extended our access to Facebook public posts until at least June 19th. Customers will continue to see Facebook data in MAP and Heartbeat for the time being.

Engineering a new solution

In the wake of this Facebook API news, we’re taking a two-pronged approach to ensure the best data for our customers. First, we’ve held ongoing discussions with Facebook and are working on new ways to use Facebook’s available API’s within our platform.

Also, through our longtime partnership with Datasift (a preferred data provider that is working with Facebook) we are investigating integrations including using a new data set called Topic Data.

Our Partner and Data Science teams are very excited about exploring future ways to ensure the most complete data sources for our customers.

Should you have any questions, please contact your Sysomos representative.

Product Updates: Heartbeat Mobile Dashboard and More

It’s been a little while since you last heard from us about product updates, but don’t let that fool you – we’ve been working diligently behind the scenes to make Heartbeat and MAP even better – easier to use and more relevant to your work.  Today’s announcements are just a start.

Some of the tricks up our sleeves include new features like in-place query editing in MAP, the ability to “Like” Facebook posts directly from Heartbeat and, most exciting of all, the Mobile Heartbeat Dashboard.

Keep reading for a full list of our latest updates.

Heartbeat Mobile Dashboard

Heartbeat has always been accessible by any mobile device. However, with more and more professionals constantly on the go and using their mobile devices as their primary access to information, we’ve given your Heartbeat dashboard a look more suited to the medium. Now, Heartbeat users can glance at their brand’s dashboard while on the go and quickly view their important information for the past 7, 14 or 30 days in an easy-to-read-and-understand format.

To access your Heartbeat mobile dashboard, simply point your mobile browser to and login.

Heartbeat Mobile Dashboard

“Like” Posts and Comments on Your Facebook Fan Page Directly from Heartbeat

Let’s face it, not every fan post to your Facebook page or comment on one of your posts is going to require a full response. However, it’s always nice to let your fans know that you saw what they did and appreciate it. That’s why we’ve now given you the ability to “like” their actions directly from the Facebook tab in Heartbeat.

Previously, you could leave comments on a post from within Heartbeat, but now we’ve given you the ability to “like” what fans are doing as well. Right beside the post in Heartbeat you will see the thumbs-up symbol from Facebook. Just click on the icon to like the post on your Facebook Fan Page, in turn, showing your fans that you “like” them too.

Heartbeat, Powered By Sysomos - Facebook Engagement

Better Customization for Email Alerts in Heartbeat

Heartbeat’s email notifications are a great way to stay in touch and know what’s happening in social media around your brand. For those of you who use the Daily or Weekly email alerts (either under the “Your Heartbeats” – “Subscribe by Email” link on the bottom of the page OR via “Settings” – “Email Subscriptions”), we’ve given you a new way to get those alerts when you want them the most.

Previously, Daily or Weekly email notifications were only shown in Eastern Standard Time. Well, EST is no longer the standard. You can now customize Daily and Weekly email alerts to come in at the time you want them – such as in your own local time. When you set up a Daily or Weekly email alert now, select the time you’d like to receive the email and then beside it select the time zone in which you live. This way, the email alerts will always come in exactly when you want them to.

Heartbeat, Powered By Sysomos - Customize Email Alerts

Click Through to Your Top Influencers from the Heartbeat Dashboard

Ever want to know more about your brand’s top 5 influencers that you see on the right hand side of your Heartbeat dashboard? Now you can click directly on the influencer and be taken to his or her page, be it a Twitter page, blog, news site or something else.

Heartbeat, Powered By Sysomos - Top InfluencersEnhanced Edit In-Place MAP Queries

It’s never been difficult to edit a query in MAP, but we’ve made it even easier. Now instead of having to click on the “Modify” query button and enter your new query into the text box, you can simply click on the large display of your query and start editing. Just a simple enhancement to make your experience using MAP even easier than before.

MAP, Powered By Sysomos - Edit Queries

That’s all for this round of updates, but keep your eyes out for more amazing updates we have planned in the very near future.

If you’re already a client and want to learn more about any of these new features, please feel free to reach out to your account manager who will be more than happy to help you.

If you’re not already a client, but would like to learn more about these new features or our social media monitoring and analytics software, powered by Sysomos, feel free to contact us here.

Introducing Sysomos Publishing

Monitor, analyze and publish all from within Heartbeat

When we first started, Sysomos focused primarily on monitoring and analyzing everything that is said in social media. Over the years, the social media landscape has evolved beyond simply listening to conversations – people are actively participating in discussions with their customers, peers and influencers. Our users wanted a single, unified platform to do everything in one place, from monitoring to publishing. Today, we are announcing exactly that.Throughout 2012 we released 104 product updates, making Sysomos not only the most advanced social media analytics platform, but also a place where brands can engage their online communities. To deliver a truly holistic, end-to-end social platform, we are releasing our Publishing Tab and Content Calendar.Keep reading to find out all of the great features that are now available with this latest update to Heartbeat.
Content Calendar
All published and scheduled messages are displayed in a user-friendly calendar format that helps your team maintain and stay on top of a well-defined content calendar for weeks at a time. All entries in the Content Calendar are colour-coded based on their status. A list view is also available for printing and exporting.Schedule Messages to Multiple Accounts with Multimedia
You can easily schedule messages to multiple accounts, including Twitter and Facebook, at once. For Twitter, you can also attach images and for Facebook, a customizable multimedia preview is added for URLs automatically.

Customize Time Zone
Each user can select the time zone in which they would like to schedule publishing and view the content calendar. This allows globally dispersed teams to collaborate more easily.

Manage Approvers
For each social channel, including Facebook and Twitter, you can define content approvers. Once this functionality is set up, nothing will be published without first being approved. This allows your larger team to create content, but go through an approval process before actually publishing.

Publishing Stats
Heartbeat’s new publishing section also provides statistics on how much content you are publishing. You can view this report across all social channels or for specific ones individually.

Outlook Integration
The Content Calendar can be easily integrated with your personal Outlook or mobile phone. No more logging in to Heartbeat to see what is scheduled and everything will be automatically synced in the cloud.

Workflow Log
Everything you do is recorded in our Workflow module, making the entire process auditable and accountable.

Best of all, everything is fully integrated in Heartbeat’s existing monitoring and analytics user interface. This means that whether you engage from the Your Heartbeats page, reply from Twitter Engagement Central, or schedule using the Publish page, everything works in unison. For the first time, our users have a fully integrated social platform providing end-to-end functionality.


If you’re already a Sysomos subscriber and would like to learn more about our new publishing features, please feel free to reach out to your account manager.

If you’re not a Sysomos subscriber and would like to learn more about the new publishing and other great Sysomos features, feel free to contact us.


UPDATE: Check out our tutorial on how Sysomos Publishing works

Using the Publish Tab from Marketwire on Vimeo.

Learning Boolean with Sysomos

 Do you know how to search using boolean searches? Do you even know why you would want to? Well, the reason you’d want to is because it will help you get much better results when you search. If you don’t know how to do a boolean search though, don’t worry, you’re not alone. In order to help our subscribers (and the general public) understand how we focus your searches to only pull in relevant information, we put together this great new video to help teach you.

Boolean is the use of the words AND, OR and NOT (as well as a few other things) within a search to really help you focus in on what’s actually important to you when you search. Boolean is a key player in how our Sysomos software, both MAP and Heartbeat, helps you to weed out the bad or unintended information you get when searching. OR helps you expand your search to include more information., AND helps to focus your search and NOT tells your search to stay away from certain information.

In order to help you understand these concepts better, we’ve put together this handy tutorial video that explains what a boolean search is and how to use it effectively. We used our Sysomos software for the purpose of this tutorial, but the lesson learned can be applied other places as well.

As an added bonus lesson for today, we wanted to show you how our Sysomos software helps sort all of this relevant information you just brought in using a boolean query. The following video shows how the use of queries and tags in our Heartbeat software bring in and make sense of information. Queries go out into the social world and bring the information back into our system. Using a boolean query (as we just learned how to do) will help to bring in the best quality information. Tags then help you to sort and make sense of all that information by placing it into the “buckets” that make the most sense for your purposes. Watch the video below to get a much better explaination.

A big thank you to the team at Think 33 for helping us to make these great videos.

So, are you feeling smarter yet?

Data, Data and More Data

This blog post is first in the “Engineering” series by Sysomos’ co-founder and CTO Nilesh Bansal. As part of this series, Nilesh will share experiences in engineering Sysomos’ social media platform.

One question that is frequently asked is: What’s the biggest challenge I face? The simple answer is: data.

As I write this sentence, in less than a minute, our crawlers have collected tens of thousands of new conversations happening online. Within the same minute, each of these conversations was discovered, retrieved, cleaned, analyzed, and stored on our servers. Now, that is a lot of data.

We store billions of documents on our servers. Every hour, millions of them are read and analyzed by users. In the last few years, our team has experimented with a variety of options to get a better understanding of the black art of data management. I’ll share some of them in this post.

There are two main components of the storage layer: hardware and software. As well, there’s a third option: outsourcing by using Amazon’s cloud infrastructure, S3 and EC2. Cloud storage is a convenient option, and, if used properly, even economical. But convenience comes at the price of flexibility.

While Amazon has steadily added more customization options and features, there still isn’t enough flexibility to meet our needs. The lack of flexibility also limits our options to innovate such as our plans to start using solid-state drives. As a result, we have stayed with conventional on-premise solution.

Storage Hardware Disks and storage bays are the most expensive part of a purchase order. They are also the slowest and the least reliable. This means they have to be selected carefully.

There are three main architecture options. First, network mounts and NAS obviously will not work given the low latency requirements. Second, fiber-based SAN offers flexibility in adding new disk arrays or moving them across hosts, but is significantly more expensive. If planned properly, this flexibility is not really needed.

The last option, which I prefer among the three, is internal and direct attached storage. If I had to select one configuration option, I would go with a 2U server with 12 bays containing 1TB SATA disks, 60-80GB RAM, and 16 processing cores. This provides a good balance of computing power and storage space. Adding more disks is easy by adding an external disk array connected via SAS cable.

Reliability Disks fail, and when they fail all the data is lost. RAID is used to store multiple copies of the same data on different disks to ensure reliability. RAID 5 is the most commonly used option. However, the disk sizes have increased exponentially and the bit error rates still remain at the same level, there is a non-zero chance of data loss in RAID 5. RAID 6 adds an extra disk to RAID 5 to provide higher reliability. The data write speeds in both RAID 5 and 6 is slow and not best for what we do.

We use RAID 1+0 where all data is mirrored on two different disks. Since all data is stored twice on two separate disks, it means twice the cost but it also provides the best reliability and high performance.

Storage Software As our crawlers continue to add more data every minute, and our users analyze thousands of documents every second, data storage is an important consideration. While we use a combination of different solutions from flat files and custom data structures to inverted indexes, key value pairs and relational database, the bulk of our data is handled by MySQL.

For the most part, MySQL is used as a simple key value store. Since a single instance of MySQL can’t hold all the billions of documents within Sysomos, we partition the data logically across several big, fat servers. Each server is maintained independently (as NDB cluster does not really scale) using primarily the InnoDB table format. Inside each instance, we further partition the data logically to hundreds of different tables. This partitioning let us add new data without hitting the wall.

While MySQL is good enough for basic SELECT and INSERT operations, this is all it can do. Even thinking of a JOIN or any complicated operation can make the server crash. But as long as it is used as a key-value store, MySQL can handle a lot of data and provide for all replication and backup needs.

Key-Value Stores New generation of data stores are gaining popularity. Most notable ones are Apache’s HBase, Facebook’s Cassandra, LinkedIn’s Project Voldemort and Baidu’s Hypertable. Each of these have big-name backers with a lot of hype, and are trying to do what Google does with BigTable.

But they have to become more mature before they become useful for us. For example, when HBase crashes (and it does), it prints the most uninformative error log. Hash based partitioning is used for load balancing, which provides little visibility in where the data actually is, and is often less optimal than logical partitioning when it comes to latency. There is also  very limited user base for each of these outside of there parent companies (which also means bad documentation).

Tokyo Tyrant is another option because it is simple, fast and good for specialized needs.

In summary, it’s all about data. More data means, more we can do with it (and sleep less). I will explore some more topics, including real-time indexing, sentiment analysis, and load balancing, in my next posts, so stay tuned.