Amazon Kinesis Data Analytics Can Now Detect Hotspots in Real-Time Data Streams

Posted On: Mar 19, 2018

Starting today, Amazon Kinesis Data Analytics supports real-time hotspot detection, which allows you to automatically detect regions of high density in your data, like a high concentration of vehicles on a highway indicating traffic bottlenecks, surging rideshare requests in a certain area indicating a popular event, or higher sales of products within a category indicating feature similarity. Detecting hotspots like these can help you gain actionable insights quickly and react to changing customer and business needs promptly.
To get started, simply call the Kinesis Data Analytics hotspot function from your Kinesis application. The hotspot function automatically builds and trains an appropriate machine learning model to identify subsections of your data streams that need attention. It identifies and reports one or more bounding boxes of these subsections in real-time.

Kinesis Data Analytics hotspot detection is unsupervised, meaning that the function does not require you to label the data for model training. In addition, Kinesis Data Analytics automatically updates the model behind the scenes to adapt to changes in the data stream. For more information including sample code and hotspot visualization, see Detecting Hotspots on a Stream in the Kinesis Data Analytics developer guide.

Kinesis Data Analytics is the easiest way to process data streams in real time with standard SQL without having to learn new programming languages or processing frameworks. Kinesis Data Analytics is available in the US East (N. Virginia), US West (Oregon), and EU (Ireland) regions. To get started, visit the Kinesis Data Analytics management console.

Apache spark to power microsoft bigdata analytics

Microsoft is  making a serious commitment to the open source Apache Spark cluster computing framework.

After dipping its toes into the Spark ecosystem last year, the company today launched a number of Spark-based services out of preview and announced that the on-premises version of R Server for Hadoop (which uses the increasingly popular open source R language for big data analytics and modeling) is now powered by Spark.
In addition, Microsoft announced that R Server for HDInsight (essentially the cloud-based version of R Server) is coming out of preview later this summer and Spark for Azure HDInsight is now generally available with support for managed Spark services from Hortonworks. Power BI, Microsoft’s suite of business intelligence tools, will now also support Spark Streaming to allow users to push real-time data from Spark right into Power BI.

Microsoft Shops prepare for next generation features in SQL Server 2016

Microsoft announces SQL Server 2016, an intelligent platform for a mobile first, cloud first world.  The next major release of Microsoft’s flagship database and analytics platform provides breakthrough performance for mission critical applications and deeper insights on your data across on-premises and cloud. Top capabilities for the release include: Always Encrypted – a new capability that protects data at rest and in motion, Stretch Database – new technology that lets you dynamically stretch your warm and cold transactional data to Microsoft Azure, enhancements to our industry-leading in-memory technologies for real-time analytics on top of breakthrough transactional performance and new in-database analytics with R integration.

Always Encrypted

Data security is top of mind, especially for mission critical applications, and SQL Server has been the enterprise database with the fewest security vulnerabilities six years running.*  To help customers with data security and compliance when using SQL Server on-premises or in the cloud, we are introducing Always Encrypted. Always Encrypted, based on technology from Microsoft Research, protects data at rest and in motion. With Always Encrypted, SQL Server can perform operations on encrypted data and best of all, the encryption key resides with the application in the customers trusted environment. Encryption and decryption of data happens transparently inside the application which minimizes the changes that have to be made to existing applications.

Stretch Database

Today, in the Ignite keynote, we showcased how you can gain the benefits of hyper-scale cloud in the box with new hybrid scenarios including Stretch Database. As core transactional tables grow in size, you may need to archive historical data to lower cost and to maintain fast performance. This unique technology allows you to dynamically stretch your warm and cold transactional data to Microsoft Azure, so your operational data is always at hand, no matter the size, and you benefit from the low cost of using Microsoft Azure.  You can use Always Encrypted with Stretch Database to extend your data in a more secure manner for greater peace of mind.

Real-time Operational Analytics & In-Memory OLTP

Building on our industry leading and proven in-memory technologies, customers will benefit from the combination of real-time operational analytics with blazing fast transactional performance – a first among enterprise vendors.  For In-Memory OLTP, which customers today are using for up to 30x faster transactions than disk based systems, you will now be able to apply this technology tuned for transactional performance to a significantly greater number of applications as well as benefit from increased concurrency.  With these enhancements, we also introduce the unique capability to use our in-memory columnstore delivering 100X faster queries with in-memory OLTP for in-memory performance and real-time operational analytics.

Built-in Advanced Analytics, PolyBase and Mobile BI

For deeper insights into data, SQL Server 2016 expands its scope beyond transaction processing, data warehousing and business intelligence to deliver advanced analytics as an additional workload in SQL Server with proven technology from Revolution Analytics.  We want to make advanced analytics more accessible and increase performance for your advanced analytic workloads by bringing R processing closer to the data and building advanced analytic capabilities right into SQL Server.  Additionally, we are building PolyBase into SQL Server, expanding the power to extract value from unstructured and structured data using your existing T-SQL skills. With this wave, you can then gain faster insights through rich visualizations on many devices including mobile applications on Windows, iOS and Android.

Additional capabilities in SQL Server 2016 include:

  • Additional security enhancements for Row-level Security and Dynamic Data Masking to round out our security investments with Always Encrypted.
  • Improvements to AlwaysOn for more robust availability and disaster recovery with multiple synchronous replicas and secondary load balancing.
  • Native JSON support to offer better performance and support for your many types of your data.
  • SQL Server Enterprise Information Management (EIM) tools and Analysis Services get an upgrade in performance, usability and scalability.
  • Faster hybrid backups, high availability and disaster recovery scenarios to backup and restore your on-premises databases to Azure and place your SQL Server AlwaysOn secondaries in Azure.

In addition, there are many more capabilities coming with SQL Server 2016 that deliver mission critical performance, deeper insights on your data and allow you to reap the benefits of hyper-scale cloud.

Last week at Build we announced exciting innovations to support our mission of making it easier to work with your data, no matter how big or complex.  We also shared how we are bringing capabilities to the cloud first in Azure SQL Database as with such as Row-level security and Dynamic Data Masking and then bringing the capabilities, as well as the learnings from running these at hyper-scale, back to SQL Server to improve our on-premises offering.  Thus, all our customers benefit from our investments and learnings in Microsoft Azure.  In addition to our hybrid cloud scenarios and investments in running SQL Server 2016 in Azure Virtual Machine, SQL Server delivers a complete database platform for hybrid cloud, enabling you to more easily build, deploy and manage solutions that span on-premises and cloud.

As the foundation of our end-to-end data platform, with this release of SQL Server we continue to make it easier for customers to maximize your data dividends. With SQL Server 2016 you can capture, transform, and analyze any data, of any size, at any scale, in its native format —using the tools, languages and frameworks you know and want in a trusted environment on-premises and in the cloud.

Be sure to visit the SQL Server 2016 preview page to read about the capabilities of SQL Server 2016 and sign-up to be notified once the public preview is available.