Hyperparameter Optimization as a service with Celestial-AutoML

Celestial (www.celestial-automl.com) is a free web service for hyperparameter optimization of machine learning algorithms. It makes getting started very easy by providing a web user interface for setting up the optimization and tracking results. Since Celestial is just an API to get parameter configurations and send back loss values, you can easily scale up. On top of that, the web UI gives you the power to visualize your results and go back to older studies to review and learn from them.

www.celestial-automl.com

Why use Celestial

Most hyperparameter optimization tools are built as a Python library. The user downloads them from PyPi or Github, imports them, and uses them directly in the code. One major disadvantage of this approach is that it is difficult to decouple the hyperparameter optimization from the infrastructure that runs it. This becomes apparent as soon as there is a requirement for parallel evaluation of parameter settings. Some hyperparameter optimization libraries attempt to tackle parallelism themselves. However, parallelization can quickly become a big task and that is why entire companies are concerned with the issue (e.g. Anyscale). A way to sidestep the infra issue is to set hyperparameter optimization up as a service. This is exactly what Celestial does. Celestial leaves it up to the user how they want to evaluate their trials. All they have to do is to add the line of code that retrieves the parameter configuration and the one that sends results back. Whether they want to run in parallel using a bash script, a grid, Amazon EC2 instances, or an army of Ray workers, it’s up to them. At the same time the user doesn’t have to go through complicated installation. Setting up Celestial is amazingly simple as you will see below.

Example of optimizing a neural network with Celestial

Creating an account

To get started, sign up with an email address and a password. This is so that only you have access to your hyperparameter optimization runs. The email is needed in case you need to reset your password.

Setting up the hyperparameter optimization

Once you have signed up and logged in, click “Create Study” in the top right corner of the web page. This will prompt you to enter a name, description (optional), and definitions of parameters. For the parameter definitions: Name — the name you’ll use to refer to this parameter in code, Type — “Continuous” float values, “Discrete” integers, or “Choice” a list of comma separated strings. Domain — comma separated low and high for Continuous/Discrete; comma separated list of strings for Choice.

Study setup.
Empty dashboard just after creating the study.

The training script

Now that we have created the study in the web service, we need our machine learning algorithm to receive parameter settings from Celestial and send back loss values. For that we use the Celestial Client.

pip install celestial-client
import celestial
# ...

number_of_trials = 100
for _ in range(number_of_trials):
# ...use your study id to retrieve parameters...
trial = celestial.Trial(study_id=<your study id>)

# ...train with trial.parameters...
# ...assign resulting loss to loss and submit...
trial.submit_result(loss=loss)
The study ID can be found in the URL, here 6.
Console output when running the MLP example.
Dashboard at the end of the study.

A few more details

As optimization algorithm Celestial uses random search. We chose random search as a starting point because it is robust and works well across most problems. We may also implement Bayesian optimization in the future.

Future Work

We have three items in mind for future work. First, we are considering different optimization backends, mostly in the area of Bayesian optimization. Second, we would like to provide more ways to explore the results via the dashboard. Finally, we would like to make study results shareable so that collaborators or reviewers can view them. Feel free to reach out for feedback and we hope Celestial can help you optimize your machine learning algorithm.

I am a machine learning engineer with background in statistics. I like building stuff :)

Get the Medium app

A button that says 'Download on the App Store', and if clicked it will lead you to the iOS App store
A button that says 'Get it on, Google Play', and if clicked it will lead you to the Google Play store