I had the opportunity to record an episode of the “Digitale Leute Podcast” with my friend Oliver Thielmann from Giant Swarm and would like to reshare it here as well:
Why industrial IoT startup Crate.io can easily do distributed databases but still had trouble finding product-market fit
Being at number one at Hacker News or winning the TechCrunch Disrupt Startup Battle might help create a hype around your startup. But it doesn’t help finding the product-market fit. Jodok Batlogg, CTO at industrial IoT startup Crate.io explains in this episode why they needed six years to finally hit product-market fit.
Digitale Leute Insights is the podcast for passionate product people. We interview product developers from around the world and take a closer look at their tools and tactics.
When Jodok Batlogg was the CTO at StudiVZ, the largest social network in Germany before Facebook got traction in Europe, the biggest problem they had was data storage. The data of 60 million users were running on about a thousand servers and Docker had not been invented yet. It was clear to Jodok that the amount of data would be growing and the problems with it.
Four years later, he founded Crate.io with a prototype of CrateDB. The open-source distributed SQL database management system used Elastic Search when that was still “a crazy guy sitting in Israel coding at a new kind of approach on how to deal with distributed computing,” as Jodok puts it in this episode.
His startup enjoyed two hypes early. The first one was a Hacker News article that resulted in the company going into the “Big Data SQL in real-time” direction. The second boost came after winning the TechCrunch Disrupt Startup Battle, which Jodok completed with a broken fibula. Although it helped to keep the company alive with bringing in investors, they were still to find product-market fit, Jodok admits. “On the product side, it was fine, but from the going-to-market side and on the monetization side, it was totally wrong.”
How to gain product-market fit
The turnaround came as late as five years after the foundation when the company did a customer survey. The result was a transformation to a more enterprise-focused company concentrating on industrial IoT. They switched the open-source model, which allowed the customers to perceive Crate.io as a product worth buying. It also helped the sales department actually to sell the product. Before that, even hiring extra sales employees had resulted in zero sales.
Today Crate.io is a remote-first company, led by Jodok Batlogg as CTO from Dornbirn, a small town in Vorarlberg, Austria. The mountainous country and Jodoks attempt to not use his Audi anymore leads him to try out all the new electric boards, bikes, and gadgets on the market. He shares this passion with our host Oliver Thylmann. This is why they close this episode by discussing electromobility and paying it with bitcoin.
About the Host Oliver Thylmann is a serial entrepreneur based in Cologne, Germany. He is the co-founder of Giant Swarm, a 35-person SaaS company providing managed microservice infrastructure to big enterprises.
Recently I came across this tweet of Steve O’Grady (he helped to found Redmonk, a research firm I really value highly).
Especially this Graph stroke me. This is what I predicted 6 years ago. NoSQL isn’t the solution. And one of the main reasons we’ve built CrateDB.
Providing a SQL interface to a NoSQL architecture was and is one of the key and core ideas of CrateDB. CrateDB is purpose-built to scale modern applications. SQL is a major component of that.
The first wave – and the reason SQL was invented – were relational databases. The start was SEQUEL, invented around 1977 by IBM for System R. SQL1 became an ANSI standard in 1986, in the 2003 Version Window functions were added and SQL:2016 got JSON support.
Glad to see that SQL is back still strong. There are many reasons why it is great and we fully rely on it. But the main point to talk about:
“The good thing about standards is that there are so many to choose from.”
Andrew S. Tanenbaum
Over the past 20+ years some of the brightes computer scientists thought about how to best work with data. SQL has 4 language subcategories:
That’s what you do most of the time. CrateDB excels when it comes to querying data. Nowadays real-time analytics usecases of require aggregations (SELECT SUM(revenue)) out of a table (FROM my_table) that is potentially large (billions of rows) at are limited to a subset of these rows by a filter clause (WHERE ts>myts AND id=myid). Benchmarks show that we consistenly outperform other query engines for these types of queries. Add JOINs to the mix and the complexity explodes – read about the complexity involved to make that really fast. Not only on standard data types, but also on modern things like geospatial data.
Correct, to be able to query data, you need to start with writing data to the database, in single rows, in batches, via bulk load, updates or from other queries). CrateDB is fully standard compliant, the only slight difference is based on the eventual consistent model (also read about optimistic concurrency control) that CrateDB uses: it might take up to the REFRESH_INTERVAL until records are fully considered in results (selecting by primary key is always consistent).
Here’s where on needs a bit more that the original SQL standard had in. We’re storing JSON data and need the possibility to define object columns (that can have a strict schema, a dynamic schema or a simply not indexed). Btw. schemas are a great idea – storing data without a schema is just postponing problems.
When it comes to scaling to large amounts of data, the concepts of sharding, partitioning and replication are key. Here’s where the knowledge of an database administrator is needed. In contrast to a traditional database, where you can change more aspects of a table after creation, you need to think of and define more settings beforehand. I don’t think this is a problem, all of you that ever were forced to do hour long schema migrations where the database was locked – will agree.
Rights and Transaction Control (DCL, DTL)
Now we’re talking about differences. Traditional OLTP databases and their workloads heavily rely on transactions and fine-grained permissions. But this comes at a high price (performance, speed, concurrency ). A price one in the IoT or Timeseries space doesn’t want to pay. Especially because the yield is low. Therefore CrateDB supports user-management, but no further functionality (yet).
SQL-99 Complete, Really
In case you want to learn more about SQL or look up some reference documentation – at Crate.io we’ve acquired the rights to republish the fantastic SQL-99 Complete, Really book. So for all oldskool SQL fanbois – here’s a teaser – World’s Longest SQL Poem (for obvious reasons)
All the snake-oil peddlers say, there's a fast and easy way,
To get your SQL program up and running,
But they're silent re the traps, that cause subtly buggy apps,
For to catch the unaware a chasm's yawning!
Date-arithmetic exceptions, auto-rollbacked disconnections,
Bit precisions, overflows, collate coercions,
And how NULL affects your summing, for to keep your DB humming,
You must know what happens in all vendors' versions!
Should this field be DOUBLE PRECISION?
Will logic rules soon see revision?
By the ANSI:Databases sub-committee?
When you DROP should you CASCADE?
How are NATURAL joins made?
Re UNIQUE-keys matching check the nitty-gritty!
Yeah the true and standard facts, you'll avoid those later hacks
That make Structured Query Language such a bore,
You'll find tips and charts aplenty, in this one-thousand-and-twenty
Four page volume (with an index), and yet more!
Undoubtfully – it’s a very handy tool. More than 1.5 billion people use it. Free cross-platform messaging, free group chat, free calls, free video calls. This was (and is) game-changing. Revolutionary 10 years ago!
Extremely impressive backend technology, written in Erlang. In 2015 WhatsApp had only 50 engineers handling 900M users. That was roughly one year after Facebook acquired WhatsApp for $19BN. I honestly admired the team as I know what it means to build a messaging service for millions of users.
Wait. $19BN for a free service?
WhatsApp is Facebooks second biggest property. Facebook is making money with the data for their users – YOUR DATA.
The price is too high. Your messages, your messaging metadata, your friends, your connections,… Guess why the WhatsApp Founder Brian Acton left Facebook 2018 – 4 years after the acquisition and left $850M on the table?
“I sold my users’ privacy… I live with that every day”
The encrypted message from the number used by Mohammed bin Salman (Saudi crown prince) is believed to have included a malicious file that infiltrated the phone of the world’s richest man, according to the results of a digital forensic analysis.
Hi, welcome to the team. I’m so glad you are here at Crate. You’re now a Cratie!
You’ll need at least one cycle to figure this place out. Maybe even two cycles. That’s totally o.k. – take your time. Uh? a cycle? What’s that? Let’s call it 1-2 quarters (of a year) for now. When entering a new environment, the first impression is really important. And we’re working hard to make the first experience a good one. But as we’re a bunch of curious, different and complex people solving complex problems this will take some time. Btw. complex is still better than complicated1. Be curious, meet everyone, look around, ask all the questions, write and talk to whomever you want. I bet there are many new acronyms, habits and rituals to learn. This will time until you’re fully productive. We expect that. In case you feel unsafe or don’t dare to ask someone, you have multiple fallbacks: your onboarding buddy, your lead person or just me.
Company culture is reflecting the people working there. What you do is who you are2. It started based on the belief of the two founders – Christian and me. Over the last 6 years we kept a lot of principles, but also carefully changed others. The following paragraphs are a user guide to navigate Crate and navigate it. It captures what you can expect out of working as part of the team, our aspirations and the leadership values that influence our culture.
My intent is to accelerate the working relationship between all of us with this document.
Our Average Week
People work from where they want and when they want. It might seem this is less efficient than having all people at one place for fixed hours. But we’ve deliberatly decided to craft a remote-friendly culture that even allows people in different timezones to be included. It makes us resilient and adaptable. It is sustainable. We’ll do as many things asynchronously and in written format as possible. People working in one of the offices have the advantage of meet at the espresso machine – and have random chats. But even if you are remote or in a different office you will be able to see and talk with your team mates at least once per day: During the daily coffee – a video call that the team does once a day. This call is about social interaction, staying connected and getting better known to each other. The teams decide themselves how to organize the rest of the work and agile methodology. One nice weekly ritual worth mentioning for the people sharing a location is the weekly team lunch – we cook something together in Dornbirn and invite you to go for lunch in other locations.
It’s up to you and your lead to agree on a schedule to set expectations, about cadence and structure/tools of 1:1s and related topics.
We have a bi-weekly all-hands meeting (attendance optional) where the leadership will present a summary about what’s going on in the different areas of the business and is also open to questions. The meeting is also recorded in case you can’t attend.
You can contact the founders 24hours a day. Don’t worry. We’re used to that and have tuned the notifications of our phones and other tools in a way that is in harmony with our private life. If we don’t want to be disturbed we’ll manage to not be disturbed.
If one of us is traveling, out of office, ill or on vacation, we update our status beforehand in Pingboard3. All our meetings still occur albeit with time zone considerations.
Some of us work now and then on weekends, evenings or early mornings. Some start later, some leave earlier. This is their choice. We do not expect that you are going to work on the weekend, evenings or early mornings. We might be sending mails and Slack messages, but unless the thing says URGENT, it can always wait until work begins for you on Monday. It’s up to you to find your work/life balance, be an integrated part of a team that delivers results. We’ll help you to do so.
We all take vacations. You should, too. Disconnected from work is when we do some of the best work.
North Star Principles
Simply Sustainable takes a major concern and common claim – simplicity – and qualifies with the ultimate payoff: sustainability, an application and a business that can scale and maintain itself ad infinitum. It’s forward looking – in a world of data overload, this is how we do more than survive: it’s how we thrive. Sustainability demands balance, playing well with others, moving forward with peace of mind. And who doesn’t want it to be simple??
Crate is a team of passionate, experienced people who’ve crafted a bold and clever database for the fast, data-intensive world, making speed, scalability super simple, open, and accessible – a solution that’s built to be sustainable for all.
Crate.io Brand Story
Clever. We’re passionate people with a love of technology and a lot of experience. But technologic breakthroughs don’t just come from knowing your stuff. They come from looking at things from different angles and flipping the situation around: they come from being clever.
Bold. We believe in being bold, in the power of having a vision and the gumption to realize it, however radical. Being bold means seeing the big picture and believing you can impact it, shape it, make it better.
Open. The best solutions come from people listening to each other as they share information, insight, and experience. Which is why we are fanatically open and honest — listening to our customers, being good citizens of the open source community, building with others to make solutions better.
Simple. It takes a deep understanding of the craft, coupled with human empathy, to render the complex simple. To make it accessible for all. In everything we do, from how we configure data and integration with existing systems to user dashboards, documentation, and support, we aim to keep it all super simple. Because data should work for everyone.
Sustainable. We believe the best solutions are sustainable, supporting communities and businesses as they grow. Which is why we believe in working together, building on each other’s advances, all in order to create open and accessible solutions that are sustainable for all.
Our System Of Values
Our Values are the guiding principles that lead to what we do on a daily basis. They are the parameters that tune our decision process that results in how we behave if no one watches. This is what we call culture. You can rely on those values, and you can demand them at any time.
Meaningfulness. At any time we offer to explain the meaningfulness and relationale of tasks, goals and measures.
Success. We strive towards and celebrate joint success.
Colleguality. We offer a work environment based on collaboration and cooperation. At the same time we respect everyone’s individuality.
Esteem. We appreciate and respect each and everyone and their opinions and provide open and honest feedback.
Challenge. It is our aspiration to provide everyone with the opportunity to grow by taking on new and challenging tasks.
Self-determination. There is as much room for self-determination as possible.
Joy. By Living Our Values above we gain happiness and satisfaction.
Expectations and Feedback Protocol
We firmly believe that the process of setting, expressing and matching expectations is at the core of building trust and respect in a team. This requires the continuous process of giving and receiving feedback.
At Crate.io, there is a formal feedback cycle which occurs twice a year. We have outlined the structure of an appraisal interview in a separate document.
Meetings take time, we take care about our mutual time. Therfor we have a couple of principles for our meetings. First of all we deliberately run our calendars publicly visible. If you have a question about any meeting on any calendar, just ask. If a meeting is private or confidential, you’ll most likely just see a placeholder. The vast majority of meetings are neither private nor confidential.
Each meeting has to include an agenda and/or intended purpose, the appropriate amount of productive attendees, and the expected contribution of the parties invited. If Craties are attending a meeting, they prefer starting on time. If Craties are running a meeting, they will start that meeting on time. If it’s not clear to a Cratie why they are in a meeting, they will ask for clarification on their attendance.
If you send a presentation deck a reasonable amount of time before a meeting, we will read it before the meeting and will have our questions ready. If we haven’t read the deck, we will tell you.
If a meeting completes its intended purpose before it’s scheduled to end, let’s give the time back to everyone. If it’s clear the intended goal won’t be achieved in the allotted time, let’s stop the meeting before time is up and determine how to finish the meeting later.
Nuances (and Errata)
Being a diverse team is a priceless advantage. Because tolerance creates creativity. And creativity is the most precious ingredient of the future. It’s essential to have different personalities in a team. We embrace this fact – but also move out of comfort zone when needed.
We don’t put people into boxes. But to concern oneself with different scientific models helps us to work better together.
We always assume positive intent for all involved. Whenever a person takes a decision, it’s the best decision that he/she can take in that moment.
If someone of us is on the phone during a meeting for more than 30 seconds, say something. It’s easy to get distracted nowadays.
This document is a first braindump of myself – speaking as one of the founders and CTO. This is not my creation, but I was privileged to work as part of a system of people that shaped our culture and my thinking. When I came across Rands in repose article „How To Rands“, I decided to take this as template, rework it and use as basis for a wider discussion within the Crate.io Team, but also the public.
I would like to update it with you, make it a shared thing all of us can underwrite, and therefor would appreciate your feedback. I‘ll turn this into a Github document in case we need to improve tooling to collaboratively work on that document.
I was thinking about quitting Facebook already for a long time. But I didn’t have the guts to do so. But now I finally quit smoking Facebook.
Today I saw the Twitter thread (german) about our wonderful Minister of Justice Alma Zadić where Facebook provided the platform to threat her, harass her. It was the last straw that broke the camels back. I spare you the details, but Business insider compiled a list of some other not so nice things.
I was friends with Facebook for 12.5 years. It was an easy way to connect with everybody (I could even poke people). By reading my timeline once a day I was able to stay up to date and receive all relevant (?) information.
Following the concept of “Digital Minimalism” by Cal Newport I’m striving to live my digital life according to my values. I only use things that provide a clear value add to me.
It’s also about how much time you spend. 60min of Facebook a day don’t provide 60x the value add of 1min.
Cal Newport recommends to do a “Digital Declutter“. A 30 day period where you avoid all digital activities that are not absolutely necessary. In this period you start doing more and more other things that you like. That will be hard – because eating cake with your aunt doesn’t provide the same kick as a shot of heroin. After that you carefully think about what digital tools you’re adding back to your life. What is the right ratio of effort and benefit.
In the case of Facebook it took me more that 30 days. First I move the App from the Homescreen of my phone to the second page (I think already 2 years ago). Then I removed the app and used only the browser version. A couple of month i logged out on all devices (and only logged in a few times).
Now I’m going to delete my account. No I’m not going to suspend it first. Maybe I’ll miss messenger?
(especially after being a long time outside in the cold)
What is love?
It’s easy to say: I love my wife, I love my job, I love Pizza or I love this music. How valid are the ancient, greek definitions family love (storge), platonic/friends love (philía), self love (philautia), romantic love (éros), guest love (xenia) and divine love (agápe) nowadays?
I recently came across the study of Tim Lomas “The flavours of love: A cross-cultural lexical analysis“. He was analyzing words for love in many different languages and based on this analysis describing different flavours (that can be combined). E.g. Koinonia – the feeling if you’re being taken away by the emotions in a group for a short time. Wikipedia has all the theory about love.
I’m not sure what category to use when I talk about what I love in business life. But sometimes it feels like taking a foam bath after an afternoon in the cold!
The feeling when puzzle pieces fall into place. You’ve started something hard, complex, distributed and you and the team have been working hard for a long time – and at some point things suddenly and quickly fit neatly together. I love that feeling.
The moment you’re exiting the “valley of tears”. You had a series of bad events/messages/pity. Like it won’t ever stop and everything is going to be over. And then something game-changing is on the horizon. You realize the streak of bad luck is over. BUT you stay just a bit longer in that bad state to extend the joy of getting out of this.
Giving trust ahead and empower people. Against all odds you decide to trust into the person – and then you’re rewarded for that decision later. Investing that social capital and earning it back later.
Having impact on people. The moments I get wet eyes in the office are rather rare. But thinking back it were the moments where people (or their spouse) said thank you for the changes/personal development I enabled or I nudged them into. I’m still touched.
Let other shine. Of course I love the moment I’m the winner and stand in the limelight in front of thousands of people. But the more longlasting and even better feeling is if other shine – not only, but also because you coached, enabled and empowered them.
Waking up early, full of energy and starting something new. Priceless!
I’ve got it! The happenings, situations, feelings in business live I really, really love are connected to my definition of entrepreneurship:
Entrepreneurship is the pursuit of opportunity without regard to resources currently controlled
Prof. Howard Stevenson, Harvard Business School
I love all the moments of it: Seeing an opportunity, pursuing it and seeing it working out – that’s where my inner Jodok is tipping his toes into the foambath after a long walk in the cold.
Each language has some special words that can’t be translated to other languages. I think this expresses a lot the culture of these countries and people.
Japanese seems to be very powerful when it comes to describe things and situations. For example Haruki Murakami – one of my favorite authors – doesn‘t like to speak on stage in his mother tongue japanese. The reason? He needs to much brain power to think about the best/correct word among the different options to use in the sentences. This prevents him getting into the real flow of telling the story.
Long story short, I wanted to shares the books I‘m reading and even more plan to read. Then I remembered that there‘s a beautiful japanese word to describe books you‘ve bought, but not yet read:
Books… written, condensed knowledge. We don’t read enough. We need to read more. Most of situations in life already happened once. And intelligent people focussed on describing these problems and solutions.
I plan to maintain the list and grow it – and as we move on also add more details/my review to the books.
If I’m linking to a book I’m not using affiliate links and I’ll link to my local books store or to the electronic version in the Apple ecosytem.
“What You Do Is Who You Are” by Ben Horowitz It’s a book about culture, using a couple successful examples from history and giving input on how to build your own company culture. So far I didn’t learn too many new things, but I’m only through by 1/4.
I (recently) read/listened to
“Handbüchlein der stoischen Moral” by Epiktet (german e-book) After listening to “The Manual” (below) I wanted to have the written, german version of Epictetus (Epiktet). It’s a bit harder to read in the original translation, but I really like it.
“The Manual” by Epictetus, Ancient Renewal and Sam Torode (Audiobook) Sam Torode did a great job to translate/transform the teachings of Epictetus into contemporary english. I out myself that like the concept of stoicism and try to live by it. Even if you say: “Oh, he’s now a philosopher” (Hint: read the book). I purchased the audio book – but I recommend the printed version as it’s really condensed information.
“Never Split the Difference” by Chris Voss (Audiobook) I love negotiations and to learn more and more about it. For a long time I thought that the Harvard negotiation technique is the best way. But unfortunately not all negotiations are based on win-win situations. That’s where this book comes in. I know for sure I’ll enroll in Chris’ online class asap!
“Playing To Win: How Strategy Really Works” by A.G. Lafley and Roger L. Martin (Hardcover) After reading that book I decided to use the tools within Crate.io as well. It totally makes sense. Highly recommended. You can also buy the tooling around it (video material, Powerpoint slides). We bought the whole package.
“Verhandeln im Grenzbereich” by Matthias Schranner 10 years ago I had the opportunity to be part of a negotiation training with Matthias. Unfortunately I had to cancel it in the last second. Since that time I wanted to learn more about his approach. Finally I saw Matthias Schranner on stage/in a workshop at bits&pretzels in Munich and decided to listen to his book afterwards.
“Hit Refresh” by Satya Nadella Earlier this year I had the opportunity for a meeting with Satya Nadella, CEO of Microsoft. And I wanted to be prepared and understand how he thinks. The book explains a lot about the “new” Microsoft.
„What I Talk About When I Talk About Running“ by Haruki Murakami 1.5 years ago I started to run. My sister gave me the book as she knew my love for Haruki Murakami. It’s really great to see how he’s connected to running.
I’ve purchased/got, but not yet read/listened to
„The Most Important Point“ by Edward Espe Brown (Hardcover) Unfortunately I missed the cooking and meditation sessions with Edward at my sisters place so far. But his book is #1 on this list.
“Die Spionin” by Paulo Coelho (german, Hardcover) Christmas present
…are hard. But if you don’t begin, you’ll never end up somewhere. This the first post on my shiny new blog. huzzah!
I’ll write down what moves me, what I’m thinking about, what I want to share, what I want to remember and what I want to say.
This is also an important step on my move to more privacy, less tracking and towards independence of the dominating privacy-killers Google, Facebook or how you name them. I’m convinced that – apart from climate related issues – privacy is one of the most important topics in the upcoming decade.
Exactly 10 years ago I was living in Berlin and working as CTO of VZ Netzwerke. We were running Europe’s most trafficked websites – the Social Networks StudiVZ, MeinVZ and SchülerVZ with 16M active users. At this time Facebook started to grow really fast and overtake VZnet which had no chance to compete. Facebook had way more engineering resources, but a main reason for growing faster their disrespect of german/european privacy laws.
We did our best to be fully compliant with the privacy laws and spent a lot to educate society about it. It did not help. People didn’t seem to care about it. Not at all.
10 years later we’re at a point where I can feel a change in trend. More and more people are aware that the (mainly) US tech giants like Google and Facebook are building millions and millions of user profiles and abuse them. I’ve made the decision to avoid feeding them and will carefully think about the online services I use.
Here’s a selection of what I’ve done so far:
Switch to Firefox as default browser on all devices. Turn enhanced privacy protection / strict mode on. I strongly suggest you download it and do the first step to protect your privacy now.
Using DuckDuckGo as default search engine. It has vastly improved over the last years – and I only rarely need to add !g to my query to fall back to Google search.
I’m logged out of Facebook and leave it dormant right now.
I’ll keep using Twitter (even use it more) and also keep somewhat trust that Apple will keep the personal data I store with them safe and private.
Let’s see how it goes with comments – Right now I’m using Nick Grossmans Discuss on Twitter plugin. You can also subscribe via E-Mail on the About page.