Tech News

MLflow Project Joins the Linux Foundation

Databricks created MLflow (https://mlflow.org/) in 2018 as an open source platform for the management of machine learning lifecycles. The platform was created in response to the complicated process of machine learning model development. Unlike traditional software development, machine learning models must not only track code versions, but versions of data sets, model parameters, and algorithms. Because of this, the variables that must be tracked and managed can grow exponentially.

MLflow prevents the process from overwhelming scientists, educators, developers, and other users by providing a platform to fully manage end-to-end machine learning development lifecycles, which includes data prep, deployment, experiment tracking, creating reproducible runs, model sharing, and collaboration.

Matei Zaharia, the original creator of Apache Spark and creator of MLflow, said of his platform, "MLflow has become the open source standard for machine learning platforms because of the community of contributors, which consists of hundreds of engineers from over a hundred companies." Zaharia concludes with, "Machine learning is transforming all major industries and driving billions of decisions in retail, finance, and health care. Our move to contribute MLflow to the Linux Foundation is an invitation to the machine learning community to incorporate the best practices for ML engineering into a standard platform that is open, collaborative, and end-to-end."

The Linux Foundation will create a vendor neutral home that includes an open governance model to aid in the broadening of MLflow adoption.

Currently, MFflow is downloaded over 2 million times per month and enjoys over 200 contributors.

Original announcement: https://www.linuxfoundation.org/press-release/2020/06/the-mlflow-project-joins-linux-foundation/

Hetzner Cloud Provider Adds Load Balancers

Hetzner (https://www.hetzner.com/) is an Internet hosting provider, based in Gunzenhausen, Germany. One of the more popular offerings is cost-effective cloud hosting. That low-cost solution is about to gain a new feature – load balancers.

With the addition of these new load balancers, Hetzner customers will be able to better scale their apps and ensure they are always available. On top of that high availability, customer cloud apps will see better performance, as the Hetzner Load Balancers distribute incoming traffic to background infrastructure.

The main features of the load balancers include quick deployment, ability to process all TCP requests and both HTTP/HTTPS traffic, configurable health checks, integration into customer Hetzner Cloud Networks, TLS termination, private networks, and supported proxy protocol. The Hetzner Cloud Load Balancers can also be automated. For more information on automation, read the API documentation (https://docs.hetzner.cloud/).

At the moment customers are now able to beta test the load balancer feature for free. Once the beta test period is over, the fee to use the load balancers will be Euro 4.90 a month or Euro 0.008 an hour and will include 20TB of traffic. Each customer will have an initial default limit for the amount of load balancers they can create. To increase that limit, customers must send a request to the Hetzner support team via a support ticket.

For more information about the Hetzner Load Balancers, read the official overview (https://www.hetzner.com/cloud/load-balancer).

Linux 5.8 Kernel Gains a Notification Queue

Back in the days of the Linux 5.5 kernel, Red Hat was working on what they called the "general notification queue." This feature was an offshoot of the Linux kernel pipe code and would serve to notify user-space applications and services of events like key and keyring changes, block layer events, and USB subsystem events. This would happen in such a way to keep those user-space applications and services from having to constantly poll the kernel interfaces.

However, Linus Torvalds put the kibosh on the notification queue, because he said it wasn't ready at the time. That was then, this is now, and the feature is set for inclusion in the 5.8 kernel.

According to this comment (https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=6c3297841472b4e53e22e53826eea9e483d993e5) (by Red Hat's David Howells), the notification queue will also "…have the ability to cache your kerberos tickets in the session, user or persistent keyring so that it isn't left around on disk across a reboot or logout." Howells continues to say, "Keyrings, however, cannot currently be monitored asynchronously, so the desktop has to poll for it – not so good on a laptop. This facility will allow the desktop to avoid the need to poll."

The official overview of the notification queue (for its proposal in kernel 5.7) sums up the feature quite well (https://www.infradead.org/~mchehab/kernel_docs/watch_queue.html):

This facility appears as a pipe that is opened in a special mode. The pipe's internal ring buffer is used to hold messages that are generated by the kernel. These messages are then read out by read(). Splice and similar are disabled on such pipes due to them wanting to, under some circumstances, revert their additions to the ring – which might end up interleaved with notification messages.

This new feature was merged in Linux 5.8-rc1.