Dec 21, 2015
D2iQ
D2iQ
5 min read
While the Internet of Things (IoT) is reinventing the way consumers interact with the physical world, it's also reinventing the way that application developers and product engineers do their jobs.
It's no longer enough to just design a good-looking and functional product. Consumers want intelligent products that can create, access and analyze data, and then feed the results back to them as a means of optimizing the user experience. Better gas mileage, cleaner teeth, more weight loss, voice-controlled televisions—it is digital data that powers all of these end-user outcomes.
Samsung realized this shift years ago, and developed a platform called SAMI to help developers build better user experiences for connected devices through smart data analysis. The platform can accept data from pretty much any source, and SAMI-powered applications can fuse data from multiple sources without being bound by silos. Aside from data access, SAMI also abstracts the complexity of device connectivity, cloud storage and processing, security and privacy.
Samsung launched SAMI in November 2014, and it has attracted broad adoption among early adopters inside startups and large enterprises alike. Samsung expects IoT will recreate users' interactions with technology and the company is investing heavily in IoT—and SAMI as an integral part of that vision—in the years to come, said Jerome Dubreuil, Senior Director of Engineering for SAMI. For example, Samsung is working on a collection of embedded IoT modules called ARTIK, and SAMI is the default backend for processing the data they generate out in the field.
"We come with the full stack, from hardware to software," Dubreuil explained. "The hardware being ARTIK and the software, or the cloud, being SAMI. Then the only thing you have to do is package that module in your device and build your application on top of SAMI.
Moving to Mesos
SAMI was built on a microservice-based architecture from the start, but early in 2015 its engineers decided the time was right to take that to the next level with containers. When the SAMI team was looking for systems that can run what Samsung DevOps Manager Niranjan Hanumegowda calls "rapidly growing number" of Docker containers in production, it decided that Mesos, Marathon and Chronos—three core components of Mesosphere's Datacenter Operating System (DCOS)—would provide SAMI with the strongest foundation.
The main goal of the move was to design a highly scalable platform-as-a-service (PaaS) to support a growing number of microservices and, in the process, improve internal continuous delivery and integration workflows on SAMI. Samsung wanted to enable other groups (such as QA and developers) be able to push code to production without worrying about provisioning the underlying infrastructure resources. One of the earlier successful projects was running the Jenkins CI/CD platform on Mesos, allowing developers push dozens of code releases into production daily with little friction, all while maintaining high quality.
The SAMI team first began investigating the Mesos stack in February 2015, and SAMI was running it in production by May.
Because data is the focal point of Samsung's IoT strategy, SAMI's architecture naturally puts an emphasis on data-processing and storage systems. In the first of those blog posts by Hanumegowda, he notes that the platform consists of more than 40 different backend services, including "NoSQL datastores, message brokers, service registries, configuration stores, graphing databases, HDFS, big data processors, in-memory caches and traditional SQL databases."
However, SAMI's persistent datastores are not yet running on Mesos, but the plan is to move them there over time. While the SAMI team has plenty of experience running distributed data system at scale, Hanumegowda said the team would love to alleviate that pain by deploying them on Mesos, as well. Mesosphere, as well as the broader Mesos community, continues to harden support for existing frameworks such as Kafka, Cassandra, HDFS and Elasticsearch, and also to add new integrations.
An efficient, and growing, cluster
Nonetheless, Samsung's Mesos environment is large and still growing. It is running more than 50 microservices as Marathon jobs, and manages batch jobs using Chronos. Jenkins is now running as its own Mesos framework, as is Apache Spark, which the SAMI team uses to run ad hoc analytics jobs against large datasets. Avoiding the need to deploy and manage separate Spark master nodes was a minor victory in its own right.
All of this—from stateless applications to build jobs to MapReduce jobs—is running on a shared cluster housed in the public cloud. The production SAMI cluster presently consists of about 500 CPU cores and 800 gigabytes of RAM, and runs roughly 400 containers at any given time. Its pre-production cluster consists of about 300 CPU cores and 500 gigabytes of RAM, and is running somewhere north of 250 containers at any given time.
All in all, Mesos and Marathon have saved the SAMI team about 60 percent on its infrastructure costs, because scaling is so much more cost-effective. Samsung has a better grasp on its total resource levels, can better predict when it will need to scale and saves resource when it does scale by adding containers rather than whole machines.
"What we see is that the real value of the Internet of Things is in the data, not in the fact that the devices are connected," Dubreuil said. "We think SAMI is going to power the next revolution in IT. If a new technology, like Mesos, can help us build a better platform so developers can build better data-driven applications, we're going to use it."
