Enriched Ecosystem

The MariaDB database has been around, in one form or another, for more than two decades. It is used by some of the marquee offline and online properties and has an extremely large ecosystem. Although the development of the database mostly happens inside various organizations, it is the mandate of the MariaDB Foundation to ensure the codebase remains open source. At Percona Live Europe, I sat with Eric Herman, Chairman of the MariaDB Foundation board, to understand how the board fosters healthy collaboration in the seemingly complex and tangled ecosystem of the database.

Linux Magazine: Yesterday we were talking about how you don't feel like an outsider in this conference, although all they seem to be talking about is MySQL, not MariaDB.

Eric Herman: Do I feel like an outsider? Well, certainly I don't, and a big reason why I don't is that the community of developers for Oracle MySQL Server, for Percona Server, for MariaDB Server is a community that has a lot of overlap. When I first started at MySQL in 2004, I met many developers, most of which are still in the ecosystem at one of the various companies. There's several that are at Oracle, there's several that are at MariaDB.com, there's some that are at MariaDB.org, there's some that are Percona and Galera. So there's a community of people that have stayed in touch, and I would say, across all of the organizations, that new people have been brought in and there's a high level of competence across the field. So there's always interesting technical discussions that you can have with any of the people, and, of course, the history makes it easier to have some of these discussions and they can get quite animated. I think there's a lot of respect in there for people regardless of which of the members of the ecosystem they're working for at this moment. Because I've been with the ecosystem a long time, I certainly don't feel like an outsider.

LM: So there's no disgruntlement?

EH: When I think about some of the newer members of the MariaDB portion of the ecosystem, I can imagine, although this isn't my personal experience – this is speculation – but I can imagine that at first they might feel a bit like they're on the edge, or on the fringe, or on the outside.

But when I look at the adoption that I see happening with MariaDB, that is so inspiring. And when I think about where does adoption come from, I think that adoption doesn't usually come because somebody says "I want to install a database." Usually somebody says, "I want to accomplish something, and so maybe I start with a WordPress site, maybe I start with a small Python, Django site." They can start from anywhere and maybe it's PHP, maybe it's Ruby, but they're trying to get something done. Over time, how they accomplish that will change as they grow and scale and become amazingly more successful. Then, in five years, 10 years, they might be the people that are standing up in front of these big audiences, telling how they got their system to scale and the importance of the database scaling that was behind it.

While they are going to be making changes to their application and doing scaling things, the choice they make in database early on is a choice that, unless there's a really, really strong technical reason, they are unlikely to change.

LM: MariaDB is both a for-profit corporation and a non-profit foundation. How do you distinguish between the two when it comes to contributing to the database?

EH: MariaDB.com, the corporation, has the largest number of experienced developers contributing to the MariaDB codebase. They also have, therefore, the largest impact on what ends up being in the next version. The Foundation brings all the contributors together a couple of times a year, and everybody presents the things that they're putting forward. So somebody from Alibaba or Tencent will talk about the work that they're doing and what they're hoping to contribute. But obviously the people who have the most to say and have often made the largest contributions are indeed the people from the corporation, but the Foundation can't set their roadmap. They're serving their customers and so their contributions are likely to reflect what's the most value to their most important customers.

So is there a clear distinction? I think that the distinction has become more clear over time. I think that it has not been as clear in the past, but as the community has grown, we're starting to see that there's larger contributions coming from more angles than one. That's something that I think could still use more clarity. I think having the two names [MariaDB.com and MariaDB Foundation] both have MariaDB in them creates an opportunity, especially for new members, to not see the distinction. Indeed, I think that's starting to become more visible to more people.

LM: So how has the Foundation evolved over the years?

EH: I think the Foundation has improved its execution with regards to getting its focus around its core mission – around adoption, collaboration, promoting the server. I think that, early on, there was a lot of work to ensure that there would be adoption in the standard distributions and that was the focus there. As the number of contributors that were not part of MariaDB.com or part of the Foundation – the number of outside contributors – started to rise, reviewing pull requests started to become a much bigger part of the Foundation's role. Obviously, sometimes the person that's the domain expert may not be in the Foundation, so they may have to reach out to somebody in the corporation or perhaps somebody in one of the other organizations to be able to really give proper feedback and do a good review. But the thing that I think now is that, by setting the expectation that if I put in a pull request, within 72 or so hours of when I put that request in, a human will have looked at it and given me some initial feedback. Either they are looking for the domain expert, or maybe they are the domain expert and can already start to give me some feedback right away.

I think that's the sort of professionalism [and] execution that I've seen grow in the Foundation, especially over the last couple of years. I've really appreciated the emphasis that's gone into improving the pull request review cycle.

LM: It's all hunky dory, now?

EH: I don't want to say that it's all rainbows and unicorns, but, yeah, I think the direction is really promising, and I don't mind that there's a lot of open pull requests. I think it's great if there's a lot of open pull requests. What I don't want to see is a lot of open pull requests that are getting very, very stale, they need some love, and they're not getting it. So that's the thing that I'm happy to see: that the really tricky pull requests are at least getting attention, they're getting advanced, and you're not seeing a bunch of pull requests in the state that the contributor that's opened the pull request is waiting.

There are several [pull requests] that are open, where some feedback was given to the contributors [who] probably just are busy, [so] they haven't had time to internalize that feedback, rework their patch in order to make use of that feedback. So there's a bunch that are in that state, and that's okay, because if you're a contributor and you don't have time to take feedback, right now, you're the one who needs to feature. I don't mind if you as the contributor say, "Hey, I don't have time to accept that feedback just now. I'll get to it next week." But if you're the contributor and you're waiting and you're not getting that feedback from somebody in the Foundation, then that's what I don't like to see, and, yeah, still some of that happens, but it happens a lot less now.

LM: Could you shed some light into the reason behind MariaDB switching its storage engine from Percona's XtraDB back to InnoDB?

EH: I can talk on this some, but I'm not a storage engine expert. I do make some contributions, but this isn't my area of expertise. In general terms and – broad strokes – Percona's ExtraDB had several improvements on the stock InnoDB. Over time, the Oracle engineers improved InnoDB such that a lot of that difference vanished. Unfortunately, I can't give you enough details about which those things were. I know some of it was performance, some of it was instrumentation. I really just don't know since I'm not a storage engine expert. The excellent devs at Percona had raised the bar, and that flowed into MariaDB.

Then we saw our colleagues over at Oracle working on InnoDB; they had to rise to it, and they did and improved the storage engine on that side. Once the distance between them gets small enough, then there's not really a need to maintain two. A couple of years ago, at one of the developer meetings, it was put on the table. It was discussed, I was in the room, although I didn't have anything to contribute to the conversation, where they said, "Does this still make sense? Does it still make sense to have to maintain these two when now they are, oh so close?" So the decision of the developers at that time, was that no, it doesn't – it no longer provides enough value to justify the cost of having the second one there, so they said, "let's go in that direction." I think that was a great example of the ecosystem working.

LM: So if you're not a storage engine expert, what is your expertise?

EH: I am a developer, but, as you may know, the MySQL code base is a very rich, deep code base. It has literally more than 20 years of development in it. My contributions tend to be the result of having observed a specific problem and then coming up with the solution to that specific problem. I've done some performance work and some very small feature work, but I don't know a subsystem like the storage engine layer particularly well. By trade I am a developer, I develop [in] a number of languages, but most of my development work has been on the application layer using databases: so using MySQL, using MariaDB in order to achieve a business task. When I'm inside the server code, this is almost always because something else has been a problem, so I need to uncover why it is that we're seeing this unexpected performance.

LM: It seems we're doing this interview in reverse. Please tell us a little about yourself and what you do.

EH: My name is Eric Herman. I'm employed by the Foundation for Public Code [1], which is working with municipalities and provinces and public organizations to help them develop free and open source software. When they do procurement, instead of procuring capital expenditure (CapEx) of some proprietary software, [we] try to help them move in the direction of collaborating on free and open source software: so a little bit more OpEx [operating expenses] than CapEx, reflecting more the way software development is outside of government, where we've stopped thinking of it as buying a thing and started thinking of it as more like a process that we are participating in. So that's my job, I'm the Lead Code Steward for Quality at the Foundation for Public Code.

Also, I am connected to the MariaDB part of the database ecosystem, and I sit on the board of directors for the MariaDB Foundation [2]. Obviously, I try to help the board [and] give guidance to the chief executive (Kaj Arnö), who I believe is doing a great job.