Scale

Data Processing

Description

Scale enables on-demand, near real-time, automated processing of large datasets (satellite, medical, audio, video, ...) using a dynamic bank of algorithms. Algorithm execution is seamlessly distributed across thousands of CPU cores. Docker provides algorithm containerization. Apache Mesos enables optimum resource utilization.

Release Notes: https://github.com/ngageoint/scale/releases/tag/5.4.0

Documentation: https://ngageoint.github.io/scale/

Issues: https://github.com/ngageoint/scale/issues/

New User Tutorial: https://github.com/dcos/examples/tree/master/scale

Pre-Install Notes

This DC/OS Service is currently in preview. We recommend a minimum of three nodes with at least 4 CPUs and 6GBs of RAM available for the Scale services and running Scale jobs. By default, the Elasticsearch package *must* be running within your DCOS cluster. If you wish to use an externally hosted Elasticsearch cluster, specify one or more of the nodes in comma delimited format in the SCALE_ELASTICSEARCH_URLS variable. For quick-start purposes, Scale is bootstrapped with a Postgres database and RabbitMQ broker. These should *NEVER* be used for production purposes as they offer no underlying storage persistence. The database can be replaced with an externally hosted Postgres by setting db-host and associated settings appropriately in the db section of configuration. An externally hosted RabbitMQ or Amazon SQS may be used by setting broker-url in the messaging section appropriately. If you are running DCOS in a security mode other than disabled, you will need to set the service-token in the dcos section of the Advanced Install. This value can be found within the dcos.toml file under in the dcos_acs_token value on a system with an authenticated DCOS CLI.

Tutorial documentation for Scale can be found here: https://github.com/dcos/examples/tree/master/scale

Post-Install Notes

The Scale DCOS Service has been successfully installed!

New User Tutorial: https://github.com/dcos/examples/tree/master/scale

Disclaimer

The software listed above is solely subject to the license(s) listed here, as between you and the creator of the software. Mesosphere is not responsible for, and disclaims any indemnification, warranty of any kind either express or implied, or (unless described in a mutually executed written support agreement) support, with respect to the software listed here.

Developer

scale@nga.mil

Ready to get started?