11 Beautiful Event Data Models

One of the most common requests that I get here at Keen is for help with data modeling. After all, you’ve got to collect the right data in order to get any value out of it. Here’s an inventory of common, well-modeled events across a variety of industries:

  1. B2B Saas (create_account, subscribe, payment, use_feature)
  2. E-Commerce (view_item, add_to_cart, purchase)
  3. Gaming (create_user, level_start, level_complete, purchase)

All of the examples are live code samples that you can run and test yourself by clicking “Edit in JSFiddle”.

B2B SaaS Event Data Models

Track what’s happening in your business so that you can make good decisions. With just a handful of key events, you have the foundation for the classic SaaS pirate metrics (AARR: Acquisition, Activation, Revenue, Retention).

Create Account Event (Acquisition)

Capture an event when someone signs up for the first time or creates an account in some other way.

https://jsfiddle.net/7dtm77nc/6/?tabs=js

Subscribe (Acquisition)

Track an event when someone subscribes to your newsletter, chatbot, etc.

https://jsfiddle.net/7dtm77nc/7/?tabs=js

Use Feature (Activation)

It’s really common for product managers and marketers to want to know who is doing what in their products, so they can make roadmap decisions and setup marketing automation. Here’s an example of a event where a feature “Subscribe to SMS alerts” has been done by the user.

By including details about the feature on the event, you can provide yourself a nice dataset for later A/B testing and analysis. (e.g. did changing the button text increase or decrease usage?).

https://jsfiddle.net/3rezjb1h/?tabs=js

Invoice Payment (Revenue & Retention)

This is a simplified example of an invoice payment event. If you use Stripe for payments, you can consume their event firehouse into Keen directly and don’t need to model it yourself.

You can see the full Stripe Invoice object here.

https://jsfiddle.net/ff9y8ppw/3/?tabs=js

Checkout more SaaS analytics uses and applications.

E-commerce Event Data Models

Track what’s happening in your store so that you can maximize sales, marketing investments, and provide detailed analytics to your vendors.

View Item / View Product Event

People checking out your goods? Good. Track it.

https://jsfiddle.net/3rezjb1h/1/?tabs=js

Add Item to Cart

Track every time someone adds a product to their cart, bag, or basket.

https://jsfiddle.net/2wb21enc/?tabs=js

Successful Checkout Event

Track an event every time an order is successfully completed. Use this event to count the total number of orders that happen on your site(s).

Use the Purchase Product Event (below) to track trends in purchases of individual items.

https://jsfiddle.net/7yqkjjsd/1/?tabs=js

Product Purchase Event

Track an individual event for each item purchased. That way you can include lots of rich details about the product and easily run trends on specific products.

https://jsfiddle.net/xxdjmkss/?tabs=js

Gaming Event Data Models

Track what’s important in your game so that you can measure activation, engagement, retention, and purchasing behavior. Checkout this related guide: Data Models & Code Samples for Freemium Gaming Analytics

New Player Event

Track every time a new player starts your game for the first time.

https://jsfiddle.net/p0csfttc/1/?tabs=js

Level Start Event

Track each time a player starts a new level in your game.

https://jsfiddle.net/ndfdeu4s/?tabs=js

Level Complete Event

Track each time a player successfully defeats a level in your game. The data model is the same as level_start, but you’ll have much fewer of these events depending on what type of game you’ve designed.

https://jsfiddle.net/yf96tyny/1/?tabs=js

In-Game Purchase Event

Track when players making purchase in your game you get that $.

https://jsfiddle.net/p0csfttc/1/?tabs=js


Twilio Partners with Keen IO to Provide Contact Center Analytics

Exciting news at the Twilio Signal Conference this week! Twilio announced they have partnered with Keen IO to provide out-of-the-box contact center reporting and analytics. Now contact centers have the essential dashboards and metrics needed to run a contact center. The add-on seamlessly integrates with Twilio TaskRouter to provide immediate visibility into contact center usage.

Interested in learning more? The Add-on includes out-of-the-box standard contact center reports, as well as an exploratory query interface and the ability to drill down into raw task data.

Login to install Keen IO Contact Center Analytics Add-on in Twilio’s marketplace, or checkout the product screenshots below!

Out of the box reporting on queues and channels
Agent performance dashboards
Historical performance heat maps
Full access to raw data and exploratory anaylsis

The Contact Center Analytics Add-on enables you to:

  • Monitor your contact center with out-of-the-box dashboards and email reporting
  • Explore deeper insights in your data with a point-and-click visual explorer — no code or knowledge of SQL required
  • Connect other data sources, like CSAT or revenue, to customize your reports
  • Share sets of metrics and unified KPIs across every team, project, and department
  • Automate workflows and build extensions on a well-documented API
Twilio Announcing the new Add-on in the SIGNAL Keynote

Happy analyzing! We can’t wait to see what you build! ❤

Login to install Keen IO Contact Center Analytics Add-on in Twilio’s marketplace.


The Unstoppable API Era

Just as organizations moved to cloud hosting & storage in mid-2000s, companies are now moving en masse to adopt cloud API platforms. This has been covered somewhat extensively by the tech press. Reinforcing the trend, two API companies IPO’d in the past year (Twilio & Apigee), with more on the way.

What do we mean by “API platforms”? This Quora answer is pretty good, but even more succinctly: APIs boil complex processes down to simple commands that magically do lots of work for you.

It’s like that feeling you had the first time you pressed a pink lyft button, and two minutes later a car was there to pick you up. Except, we’re talking about software, so the scale can be staggering. You can request a lyft a few times per day. There are literally trillions of API requests happening right now as you read this.

APIs platforms are software building blocks that are higher level than infrastructure services (e.g. AWS servers) but lower level than SaaS apps (e.g. Salesforce). They allow developers to bolt-on core pieces of application functionality with minimal effort, and in way that is seamless to end users.

Here are a few examples of API requests being handled by API platforms:

  1. Run a query on a huge dataset | Keen IO (Analytics API)
  2. Deliver an email | SendGrid (Email API)
  3. Charge a credit card | Stripe (Payments API)
  4. Text, video, or call someone | Twilio (Communications API)
  5. Run a search on a website | Algolia (Search API)
  6. Authenticate a login | Auth0 (Security API)

It’s not that these kinds of things couldn’t be done before these API platform companies existed. It’s just that an engineering team would have to build & maintain servers, services, and operational processes to do that work. Five years ago, adding any one of those functionalities to an application would have represented several months of engineering roadmap time. Now these features can be up and running in days.

The end result is that developers can build new applications in increasingly shorter timeframes, and at a far lower upfront cost and risk. Indeed, API platforms are part of the reason we’ve seen a boom in venture funding for software businesses over the past five years. Software that used to take two years to prototype can now be brought to life and tested with a small round of seed funding and a few months of work.

Take Instacart as an example. Instacart was founded in 2012. Had it been founded five years earlier, their engineering team would likely have needed to hire engineers, write code, and run servers for email delivery, text messaging, and collecting payments. But by 2012 you could simply bolt-on SendGrid, Twilio, and Stripe to handle all of that for you. The Instacart team was left to focus on making an intuitive mobile grocery shopping experience.

While some companies may have specialized requirements that make it difficult to use API platforms (e.g. Facebook’s data infrastructure must be 1000x the size of the next closest competitor), for 99% of businesses, API platforms are far more reliable, efficient, elastic, have a lower TCO, and are usable without hiring up specialized engineers.

One of Keen’s largest customers stealthily uses Keen’s APIs to white label an in-app analytics suite for their Fortune 500 customers. Their delivery manager probably made the point most succinctly: “It would have taken our team two years to deliver what we’ve done with Keen in nine weeks.”

That’s the kind of leverage we imagined when we founded Keen IO, and what inspires us about building an API business.


Architecture of Giants: Data Stacks at Facebook, Netflix, Airbnb, and Pinterest

Here at Keen IO, we believe that companies who learn to wield event data will have a competitive advantage. That certainly seems to be the case at the world’s leading tech companies. We continue to be amazed by the data engineering teams at Facebook, Amazon, Airbnb, Pinterest, and Netflix. Their work sets new standards for what software and businesses can know.

Because their products have massive adoption, these teams must continuously redefine what it means to do analytics at scale. They’ve invested millions into their data architectures, and have data teams that outnumber the entire engineering departments at most companies.

We built Keen IO so that most software engineering teams could leverage the latest large-scale event data technologies without having to set up everything from scratch. But, if you’re curious about what it would be like to be a giant, continue on for a collection of architectures from the best of them.

Netflix

With 93 million MAU, Netflix has no shortage of interactions to capture. As their engineering team describes in the Evolution of the Netflix Data Pipeline, they capture roughly 500 billion events per day, which translates to roughly 1.3 PB per day. At peak hours, they’ll record 8 million events per second. They employ over 100 people as data engineers or analysts.

Here’s a simplified view of their data architecture from the aforementioned post, showing Apache Kafka, Elastic Search, AWS S3, Apache Spark, Apache Hadoop, and EMR as major components.

Source: Evolution of Netflix Data Pipeline

Facebook

With over 1B active users, Facebook has one of the largest data warehouses in the world, storing more than 300 petabytes. The data is used for a wide range of applications, from traditional batch processing to graph analytics, machine learning, and real-time interactive analytics.

In order to do interactive querying at scale, Facebook engineering invented Presto, a custom distributed SQL query engine optimized for ad-hoc analysis. It’s used by over a thousand employees, who run more than 30,000 queries daily across a variety of pluggable backend data stores like Hive, HBase, and Scribe.

Airbnb

Airbnb supports over 100M users browsing over 2M listings, and their ability to intelligently make new travel suggestions to those users is critical to their growth. Their team runs an amazing blog AirbnbEng where they recently wrote about Data Infrastructure at Airbnb last year.

At a meetup we hosted last year, “Building a World-Class Analytics Team”, Elena Grewal, a Data Science Manager at Airbnb, mentioned that they had already scaled Airbnb’s data team to 30+ engineers. That’s a $5M+ annual investment on headcount alone.

Keen IO

Keen IO is an event data platform that my team built. It provides big data infrastructure as a service to thousands of companies. With APIs for capturing, analyzing, streaming, and embedding event data, we make it relatively easy for any developer to run world-class event data architecture, without having to staff a huge team and build a bunch of infrastructure. Our customers capture billions of events and query trillions of data points daily.

Although a typical developer using Keen would never need to know what’s happening behind the scenes when they send an event or run a query, here’s what the architecture looks like that processes their requests.

Keen IO Event Data Platform

On the top row (the ingestion side), load balancers handle billions of incoming post requests as events stream in from apps, web sites, connected devices, servers, billing systems, etc. Events are validated, queued, and optionally enriched with additional metadata like IP-to-geo lookups. This all happens within seconds.

Once safely stored in Apache Cassandra, event data is available for querying via a REST API. Our architecture (via technologies like Apache Storm, DynamoDB, Redis, and AWS lambda), supports various querying needs from real-time data exploration on the raw incoming data, to cached queries which can be instantly loaded in applications and customer-facing reports.

Pinterest

Pinterest serves over 100M MAU doing over 10B+ pageviews per month. As of 2015, they had scaled their data team to over 250 engineers. Their infrastructure relies heavily on Apache Kafka, Storm, Hadoop, HBase, and Redshift.

Pinterest Data Architecture Overview

Not only does the Pinterest team need to keep track of enormous amounts of data related to Pinterest’s customer base. Like any social platform, they also need to provide detailed analytics to their ad buyers. Tongbo Huang wrote “Behind the Pins: Building Analytics at Pinterest” about their work revamping their analytics stack to meet that need. Here’s how they used Apache Kafka, AWS S3, and HBase to do it:

Data Architecture for Pinterest Analytics for Businesses
User View of Pinterest Analytics for Businesses

Twitter / Crashlytics

In Handling 5 Billions Sessions Per Day — in Real Time, Ed Solovey describes some of the architecture built by the Crashlytics Answers team to handle billions of daily mobile device events.

Event Reception
Archival
Batch Computation
Speed Computation
Combined View

Thank You

Thank you to the collaborative data engineering community who continue to not only invent new data technology, but to open source it and write about their learnings. Our work wouldn’t be possible without the foundational work of so many engineering teams who have come before us. Nor would it be possible without those who continue to collaborate with us day in and day out. Comments and feedback welcome on this post.

Special thanks to the authors and architects of the posts mentioned above: Steven Wu at Netflix, Martin Traverso at Facebook Presto, AirbnbEng, Pinterest Engineering, and Ed Solovey at Crashlytics Answers.

Thanks also to editors Terry Horner, Dan Kador, Manu Mahajan, and Ryan Spraetz .


Building an Empire on Event Data

Photo by Joshua K Jackson

Facebook, Google, Amazon, and Netflix have built their businesses on event data. They’ve invested hundreds of millions behind data scientists and engineers, all to help them get to a deep understanding and analysis of the actions their users or customers take, to inform decisions all across their businesses.

Other companies hoping to compete in a space where event data is crucial to their success must find a way to mirror the capabilities of the market leaders with far fewer resources. They’re starting to do that with event data platforms like Keen IO.

What does “Event Data” mean?

Event data isn’t like its older counterpart, entity data, which describes objects and is stored in tables. Event data describes actions, and its structure allows many rich attributes to be recorded about the state of something at a particular point in time.

Every time someone loads a webpage, clicks an ad, pauses a song, updates a profile, or even takes a step into a retail location, their actions can be tracked and analyzed. These events span so many channels and so many types of interactions that they paint an extremely detailed picture of what captivates customers.

Event data is sufficiently unique that it demands a specialized approach, specialized architecture, and specialized access patterns.

In the early days of data analysis, it took huge teams of data scientists and specialized data engineers to process event data for companies the size of Walmart. Now, however, even a single developer can capture billions of detailed interactions and begin running queries in seconds, accessing the data programmatically and in real time. This makes it possible to build intelligent apps and services that use insights from event data, to personalize the user experience, and display information dynamically.

One Major Challenge, but Many Rewards

A few industry giants have been able to build event data powerhouses because of the incredible access they have to talent. They hire expensive, specialized teams who build their own home-grown technology stacks. In many cases, companies like Facebook end up inventing their own distributed systems technologies to handle emergent data needs.

Most other companies lack this endless flow of resources. They can’t afford to build the infrastructure and acquire the headcount needed to maintain it. Even those that have the capital are struggling against a massive shortage of talent for roles in data infrastructure and data science. New candidates won’t materialize fast enough to build and support the world-class data capabilities every company wishes they had.

However, capturing event data is extremely important. It lets companies build a new class of products and experiences, and identify patterns that otherwise would be impossible to see. It also lets them build apps that perform far more advanced, programmatic analysis, and make real-time decisions on how to engage the user — suggesting the right product, showcasing the right content, and asking for the right actions.

Just as organizations migrated en masse from on-premise servers to cloud hosting and storage in the mid-2000s, many companies are starting to adopt data platforms like Keen so they can compete in areas they couldn’t build in-house.

Keen IO: The Event Data Platform

We built Keen to let customers use our platform as the foundation for all of the powerful event data capabilities they want to build. By leaving the analytics infrastructure to Keen, any developer or company can wield the power of event data extremely well, without a specialized background in data engineering.

We help over 3,500 customers crunch trillions of data points every day, gathering data with our APIs and storing it for them to analyze with programmatic queries to fuel any metrics or tools they need to build. Once they adopt Keen, customers report huge savings in engineering and analyst resources, far better accuracy in measuring crucial app and user analytics, and the ability to infuse real-time analytics into every part of their operations.

Event data is increasingly interwoven into software. Photo by Carlos Muza.

Event Data in Action

When companies build on an event data platform, they can accelerate their businesses in ways that weren’t possible before.

  • They anticipate what users will need and take the product in the right direction, by using event data to improve the user experience and test changes to the application or hardware.
  • They show users extremely relevant content and demand higher ad revenue from top advertisers because of the engagement metrics they derive from event data.
  • They provide deep reporting and quantify ROI for their customers — when SaaS products can provide reliable and accurate reporting, they deepen customer trust, engagement, and spend.

Can Event Data Bring a Richer Future?

The ability for companies to operate like they have Facebook’s data infrastructure is a game-changer. They can scale faster, make better decisions, and create smarter, helpful products people don’t even know they need yet. Event data will inevitably shape the way almost every company grows, and those who don’t embrace it will likely lose out to the ones who do.

Comments welcome, or start a conversion with us over at Keen IO.


Lesson One

As Keen IO’s Chief Data Scientist, I’ve worked with thousands of companies over the years. Today I’m sharing one of the most impactful lessons we’ve learned in building successful analytics integrations.

Today’s lesson is: take a moment to reflect on what you are trying achieve. Once you are confident about that, your decisions about what data to collect and how to use it fall more naturally into place.

Here’s a mini-worksheet to get you started:

  1. What’s most important to my company/team right now?
  2. What are our goals?
  3. How can data help us achieve those goals?

Write out your answers. Reflect back on them whenever you feel overburdened by the seemingly infinite possibilities for your data. Re-center and drive forward. You’ve got this.

By the way, we’re pretty helpful people over here at Keen IO. Maybe we’ve already solved a problem like yours and we can help you do it faster.


Why We Built Native Analytics

Four years ago, my friends and I embarked on a quest to build a business. We were going to build an analytics API. We dreamed big. Our platform was going to be so elegant and so powerful that, basically, anyone building anything connected to the internet would want to send data there. It would help people understand. It would help them automate. It would help them explore. It would help them discover. It would be the nervous system for the next great version of the internet.

In those four years, we’ve made some progress :). We built a database in the cloud. We built an API that allows you to query it, in real time, from anywhere. We built charting libraries so that developers could paint their apps with pies and bars.

And the whole time, we watched, delighted and amazed, to see what developers would create with these building blocks we had fashioned for them. We beamed when they showed us their dashboards. We cried when a real life rocket scientist showed us data from the Mars rovers. We watched, and we listened, as we learned, as thousands of developers integrated Keen into their products and built their businesses.

One of the main patterns we noticed was that many people were using our platform not only to analyze their business data, but to display data in the applications they were building for their users.

I’ve always thought of our business as a very organic thing, a jungle with wide-reaching and thickening roots, spreading and collecting data from all over the earth (and space!). But there is so much more to it than collecting data. The most exciting part has been seeing how people have taken that data and reflected it back out into the world. To see those roots grow into trees, and branches, and a striking variety of leaves and flowers.

It was this common usage pattern, alongside the growing demand for analytics views in nearly every modern SaaS application, that inspired us to build a number of core features that make building analytics natively into your application even faster and more powerful.

Today we’re announcing Native Analytics, a core product on the Keen platform. I hope you’ll take a peek at our new landing page, our product overview, and possibly even drop a note (see chat button, bottom right!) to talk about building analytics into your product. We’ve learned a lot over the years and look forward to building and sharing with you as we continue to grow.


How to Find a Nice Photo for Your Post

If you’re writing something that you would like a lot of people to see, it’s a good idea to include a photo in your post. It will help your piece stand out in feeds like Facebook & Twitter. If you’ve ever done a search for “FREE STOCK PHOTOS” you’ll know this is easier said than done. Those sites are terrible. Or, if not terrible, then extremely expensive.

However! Just this month, in a dusty corner of the internet, I have found a secret, glittering treasure trove of wonderful photos free for the taking. (I get really excited about this sort of thing).

Turns out Flickr, the ancient photo-sharing service I stopped using over 5 years ago, is actually pretty awesome these days. Not only to they have lots of photo content from semi-pro and pro photographers, they have great search filters too.

It’s pretty much the simplest process ever:

  1. Go to flickr and search for something related to your post or that might evoke an emotion that complements your post. A bunch of photos will come up, but don’t look at them yet.
  2. There is a drop down for license type. Select Creative Commons if you’d like to find photos for your personal site or blog. Select commercial use if you’re using it for something work-related. You are now looking at an inventory of free-to-use photos. Rad!
  3. Filter on large photos, which tend to be taken with better cameras.
  4. Try the filters for “shallow depth of field” or “minimalist”. These photos tend to be a lot more professional looking.
  5. Find a photo you like.
  6. Don’t forget to credit the photographer and mention the creative commons license when you use the photo. I included an example at the bottom of this post.

Happy photo hunting :)

Example Search.
Photo by Danny Fowler, Creative Commons

Permission to Fail

People write a lot about the emotional rollercoaster of startups. Turns out, that roller coaster doesn’t slow down after year 1, 2, or 3. I know firsthand that the metaphor doesn’t go away after you hit 100 or 500 paying customers, after you’ve been backed by Sequoia Capital, or after you’ve captured millions of dollars in revenue. The roller coaster climbs higher, making the drops all the more terrifying. Slack’s CEO sums it up pretty precisely:

Now that we’re on this crazy success trajectory, the degree of stress and the degree of doubt and the degree of second-guessing hasn’t been reduced at all,” he said. “In many respects, it’s actually worse now because there’s more at stake. […] I think I wake up every day and look in the mirror and say, ‘We’ve almost certainly fucked this up completely.’ — Stewart Butterfield, CEO, Slack

One thing I’ve noticed is that it’s easy to write and to be transparent when things are going great. It’s harder when the roller coaster dips downward, when your stomach feels like it’s in freefall. Way back in January (2015), our team experienced one of those dips. Up until this point, for 24 straight months, Keen IO’s business had grown 5–15% — every month! We were embarrassingly confident, and we felt unstoppable. It was coming off of this incredible rise that reality hit and we got to experience a little more of that roller coaster everyone was talking about.

I wrote the following company-wide memo during that time, when we got to learn what it was like to fall a little bit.

On Tue, Feb 24, 2015 at 5:55 PM, Michelle Wetzler wrote:

Hey everybody — I hope you don’t mind me sharing a relatively raw piece of writing. It started out as a sort-of blog post, but then I realized it’s really a letter to you all. Feeling a bit brave right now and clicking send. Hope it’s helpful.

-Michelle


Permission to Fail

It’s been a rough few weeks for Keen. In January, a big customer churned. We didn’t close any new big deals. So, we not only missed our revenue target, our MRR and incoming event volume actually went down for the first time ever. We discovered the dedicated topology we sold to one customer actually made their user experience worse, and refunded their money. A couple of teams have been struggling to find their footing; emotions have been running high. In last week’s outage, we dropped data for the first time in over 12 months. The instability has led not only leading to worried emails from our customers, but many sleepless and stressful days & nights for our teammates. And perhaps worst of all, several folks have admitted that they’d have a hard time recommending a friend to join Keen right now, because it’s just too stressful at this time.

During a time like this, it’d be easy to say: let’s double-down, work through the weekend, push through the issues, get ‘er done, rally!

Instead, let’s give ourselves permission to fail.

Giving myself permission to fail has been one of the most liberating, stress-relieving, and rewarding things I’ve done in last year.

The only way we can become a truly great company is if we open ourselves to the possibility that we might not be.

And you know what? It’s okay if we’re not. If Keen busts, we’ll all find new grand adventures. Some us could start a new company together, or get boring jobs at big co’s, or sail around the world, who knows, the world is full of lots of amazing opportunities.

This whole thing is an experiment and has been an experiment since the beginning. We wanted to work together, to solve problems, to create something people loved, to forge our own culture. We wanted to challenge ourselves. We wanted to be happy & fulfilled in our work. We wanted to do it our own way. For the most part, we’ve done that, and done it better than we ever imagined. Who knows if it will continue to work. I have a lot of evidence and strong intuition that it will, but I can’t predict the future. It’s still an experiment.

To give yourself permission to fail, it helps to imagine the path where the absolute worst happens.

What would happen if you walked away from your job right now and never looked back? What would happen if you did your job the best you could, and you still failed completely? What would happen if the company went under?

I can imagine all sorts of things I would do if Keen went poof. At first, I felt guilty for thinking this way. Then I felt incredibly liberated. Try it. Your work becomes 10X easier when you realize you don’t have to do it. You don’t have to do this. You are incredibly talented and there are so many opportunities out there.

To give yourself permission to fail, you have to untangle your ego from your work. Having your ego tied up in your work is a handicap. You can’t think strategically or take risks when you and your personal well-being are on the line.

I used to (and sometimes still do) romanticize Keen a little too much, thinking of it as my child, a part of myself. I’ve been working hard to untangle this. Not because I plan to care any less about Keen, but because I don’t want my ego & personal fears to get in its way. Keen and Michelle are two different things, or at the very least they are less overlapping than they used to be. If Keen is struggling, it need not mean Michelle is struggling. If Keen is taking a risk, it need not mean my happiness is on the line. Even if I fail completely at my job, I will be just fine. It just means I tried something too difficult for me, or my assumptions were wrong. That’s ok too.

I’ll admit it’s a little bit awkward for me to write all of this out. It’s possible that I care about Keen more than anyone and here I am saying it’s OK if it busts. Don’t get me wrong, it’s a little bit terrifying to think about. But it’s also a huge relief, a weight off my shoulders. It would be okay. When failure is no longer scary, my work is no longer driven by fear. Paradoxically, thinking about Keen’s failure makes me more confident in our success. Not only does it become less scary, it seems incredibly unlikely.

After 24+ solid months of growth, it would be silly to conclude it might all be falling apart because we had a stagnant month, or even a few bad months. Part of the reason the platform has struggled lately is that we continue to grow like crazy. Just today we hit a new record for event volume. In the past months, two companies have approached us about deals that would double our business. Many people at Keen will tell you “it’s the best job I’ve ever had.

Still, I appreciate this time to reflect when our confidence is shaken a bit. To be honest, we were probably overdue for it.

Building Keen is the most rewarding thing I have ever done, and I love it dearly. But I’m learning to accept that fulfillment isn’t about achievement. It’s about accepting what happens. It’s about listening to what you want to become. To do that, you have to be comfortable with many possible paths, many possible outcomes. We probably won’t fail, but if we do, I’ll be okay. We’ll be just fine.

Love,
Michelle


So what happened? Did any team members walk away? And why did I just publish this on our blog??

This lesson was valuable to me personally, and I thought it might be helpful to other people. We learn some of our most important lessons from our struggles, even though they are some of the harder things to share about.

We all know that the Silicon Valley is a strange place, a bizarre land of meteoric rises and epic failures, of techno-heroes and greedy villains. But…. actually, mostly, it’s a bunch of well-meaning nerds, generally-ordinary people trying to build stuff while facing a variety of challenges along the way. Those stories are worth sharing, too.

Finally, I’ll share how my team reacted, which will explain my third reason for sharing: it was a message that we needed at the time, and perhaps that message can help some folks out there who may need it too.

Despite my suggestion to consider their options, no one quit. Almost everyone in the company responded, either in a reply-all or a private message to me:

They said:

  • “THANK YOU”.
  • “This was so uplifting”.
  • “Bad FUCKING ass”
  • This is what I signed up for”
  • “I just teared up reading this”
  • “I have something to share too…”
  • “I’m grateful for the opportunity to work with you”
  • “I wish we talked more about our fears and failures”
  • “Your thoughts about ego remind me of this Alan Watts quote
  • “Reading this made me feel more motivated to work to get us back on track”
  • “While I know everything will be fine even if Keen fails, I really really like it here and I don’t want it to end :)”
  • “This is going to be an incredible year”

And some didn’t say anything. They simply walked over and gave me a big hug.


Roles & Responsibilities of a Successful Analytics Team

After years of working with hundreds of companies, I’ve learned a thing or two about what makes analytics projects successful. I’ve also watched many projects fail.

The most common reason for failure might surprise you. It’s not a lack of data expertise or an integration mistake. It’s simply that the organization forgot to make it anyone’s responsibility to make use of their data. Far too many companies collect months and months of interesting data only to have it sitting in the corner collecting dust. That’s what compelled me to write this guide and hopefully inspire you to add a little bit of structure to your next project. Your analytics implementation should be a wellspring of business value that you tap again and again.

If you can find at least one person who is excited to take on each of the roles I’ve outlined below, I promise your project will be 1000x* more likely to succeed. Not all of these roles need to be fulltime, and one person could certainly play more than one part (or even all parts)! More detailed role descriptions & tips following.

*not scientifically proven!

Roles & Their Outputs

Project Lead → Project plan with scope & timeline

Data Architect → Data model and queries

Product Developer → Implementation of tracking

Analyst(s) → Generation of new business questions

Reporting Developer → Reports for your business

Project Lead

A single individual should be responsible for the delivery of your initial analytics implementation. You probably already know what an effective project manager does:

  • Identifies the project’s stakeholders and figures out what they need. They ask, “What are the specific business questions we want to answer?”
  • Sets & communicates the goals, scope, and timeline for the project to everyone involved.
  • Manages dependencies and identifies roadblocks to delivery.
  • Ensures that the project is actually delivered and achieves its goals (e.g. that the data answers the questions that are important to your business).
  • Makes sure everyone involved, from the engineers to the product managers, is in sync and understands what will (and won’t) be delivered as a part of the project. This part is important as it’s also very common for people to under and over-estimate what your data will be able to do.

Tips for the Project Lead:

  • You’ll get the most immediate payback on your analytics project if you focus on questions whose answers will lead to a direct change in your product or business strategy. An example question might be: are customers from our new campaigns converting to paid (should we keep investing in this channel)? -or- We’re thinking about nixing this feature — can you find out if any paying customers use it?
  • Keep the scope of your project as small as possible. Start with tracking only a few key actions that are important to your business so you can quickly answer the most pressing business questions (e.g. what is our retention like for customers who use this feature?). Your immediate, helpful results will get your organization hooked and they’ll soon come up with more questions that you can tackle in the next sprint. In other words: analytics work should be done in an agile fashion, becoming a little bit more in-depth with each iteration. If the scope of your analytics project gets too big (e.g. taking two weeks of engineer time), you risk the entire thing getting put on hold in favor of a pressing product feature or some other business priority.

Data Architect

This is a fancy title but it just means that your team needs someone somewhat technical to create your data model and understand how querying the data works. The data model can be as simple as an email that lists what key actions you’ll track and what properties you’ll include on them. The model helps define and communicate the scope of your project. The data architect helps the team assess what business questions are possible to answer vs versus those that are not. This person is typically not a data science PhD. This role is most commonly filled by one of your app developers or someone adept at modeling things in spreadsheets.

Tips for the Data Architect:

  • Take the time to have your data model reviewed by someone who has built one before using the same toolset. For example, if you are using Keen, talk to a developer who has used Keen before. You can also ask your analytics provider to review your data model with you. No matter what toolset you are using, there will be tradeoffs and parts of the solution that don’t work quite like you’d expect. Save yourself some time and talk through your plans with someone who has enough experience to see around corners for you.
  • When modeling the data, use the vocabulary of the users and the business instead of the language of the application architecture. For example, you wouldn’t track an event like “state change” since this doesn’t have any meaning to the user of the product, and won’t have any meaning to other people in your company. If you keep the language of your data business-oriented, it helps other people in the organization understand how to query and make use of it.
  • Have at least one other person in your organization review your data model and confirm it makes sense to them too. You might find that some labels that make sense to you are unclear to others. For example, something like “uuid” might mean something different to different folks in your organization.
  • Don’t reinvent the wheel. If you’re building on Keen, please check out our Data Modeling Guide and Best Practices to learn some tricks and avoid the common pitfalls. If you’re starting a business-critical implementation, pair with one of our data architects.

Product Developer

At least one of your product developers has some pretty straightforward responsibilities at the beginning of your project: get tracking set up. They will add bits of code here and there so that every time a login, purchase, upload, or other action happens that you are able to collect data. If you have many sources of events, say your mobile app and your website, then this work might be done by multiple developers (e.g. a web dev and a mobile dev). In a small organization, the developer who sets up tracking often also plays the role of Data Architect. In larger groups, your developers should work closely with a Data Architect to make sure they model data optimally and that things are tracked and labeled in a consistent format (e.g. “user.id” = “23cv42343jk88” not “user.id” = “fran@cooldomain.com”). Setting up tracking is a relatively straightforward process as most analytics services provide drop-in client libraries to greatly simplify the effort, but your team still has to do the work of deciding what to track and how to name things.

Tips for the developer implementing tracking:

  • Make sure you’re implementing tracking according to a data model that makes sense to your organization. If your team doesn’t have a data architect, take on that role yourself and sketch out a model before you get started. This will clarify your thinking and make it easier to communicate your plan with others.
  • Implement separate repositories, with separate keys, for dev, test, and prod, so that you don’t get testing and production data mixed up.
  • Once you have tracking set up in development, get someone to peer review the incoming data before you go live with it. Your analytics implementation should go through a QA process just like any other feature of your product. It’s easy to make mistakes like sending numbers as strings, naming something in a confusing way, improperly formatting your JSON, or having typos in your labels.
  • Here is an inventory of tracking SDKs that work with Keen IO.

Analyst(s)

You’re going to be collecting lots of interesting data, but it won’t be very valuable unless someone uses it! You’ll need at least one person on your team who is very curious about what that data might reveal. I call these people analysts. Very often the analyst is a developer, product manager, or someone on the product or marketing team. Not only will these folks be dying to see the results of the business questions they set out to answer, they will be continuously thinking up new questions. Analysts love digging into the data you collected in the first phase of the project and will have a lot of ideas of what new things you can collect in the next phase. In other words, you need people on your team who enjoy the practice of analytics. Don’t worry, there are lots of people out there who do :). Having a technical background will be a huge asset for this person as they will quickly learn how to build queries to get the results they need. This role is absolutely critical for your success because if you don’t have people who want to learn from your data, you won’t be able to extract any value from it.

Tips for the Analyst:

  • The results of your analysis may be super meaningful and obvious to you, but they won’t be to anyone else. That’s because you know what questions you were looking to answer when you set out to do the analysis in the first place. You know exactly what data the dataset includes and excludes. Plus you wrote the queries that ultimately produced the visualization or report you’re looking at. That’s a lot of context that you need to share in order for other people to understand what the numbers mean.
  • When sharing results of your analysis, write out the conclusions you are drawing from the data and what business actions you think should be taken as a result of the analysis (e.g. our conversion decreased with this latest release and we should roll back). Not only do other folks perhaps not have the context to interpret the data correctly, they probably don’t find it as fascinating as you do and may not have the time to derive meaning from the data.
  • Not to hammer on it too much, but communication skills are so important for this role. Around half of the analyst’s time needs to be spent on communications. It takes quite a bit of time to explain and summarize the results and conclusions you’ll draw from your data. If the results of your analysis are sleeping in people’s inboxes, you’re not doing it right. Sometimes you may be the only person in the organization who knows about a problem or opportunity, and it’s your responsibility to make sure the organization is responding appropriately to what you’ve learned. Sometimes you gotta be the squeaky wheel. Don’t underestimate the value of your work.
  • If analysis work is something you repeatedly run out of time to do, try getting it added to your official job description and dedicating a certain number of hours per week or per month to it. Block it off on your calendar.

Reporting Developer

This role is optional, but you may want to build out some reports that can be easily accessed across the team or by other stakeholders. You can greatly increase the utility of your data by incorporating it more closely into your business workflows, rather than leaving it trapped in a database where people have to remember to login to check it. A front-end developer will be able to turn queries into a reports for product managers and others across the business. As an example, you may find your results are most useful:

Tips for the developer creating reports:

  • Get the most value out of your work by making sure the consumers of your reports understand the data. One way to do this is to ask them “When you see that conversion is 5.2%, what does that mean to you? How would you guess it is calculated?”.
  • Another way to increase report literacy is to put a guide (e.g. tooltip or footnote) that explains where the data comes from and how it is calculated. For example, does the data include users from the website and your mobile app, or just one of those? Does it include test users and internal users from your company, or have those been filtered out?
  • Have fun! Watching someone’s eyes light up when they discover something new is the best part of the whole analytics project, and you’re often the one helping bring that realization to life.

Starting an important new implementation and need some help? Our team of data architects has done this lots of times and can answer any questions you have! Just drop us a note using Keen’s support channels.


From 60,000 Readers to 1

Street Art in Clarion Alley, San Francisco

When I started writing on the Keen IO blog in 2012, it opened it up a whole new world to me. One of my posts was published in an O’Reilly book. Another became part of the entrepreneurship curriculum at Harvard Business School, where I was invited to guest lecture. Several of my posts got 50k+ views. Suddenly, people I met in San Francisco already “knew” me from the blog. Folks wrote me personal emails asking for career advice. Werner Vogels, the CTO of amazon, retweeted one of my pieces to his 65,000 followers. It was incredibly rewarding.

Then… something changed.

When our company scaled from 8 people to 15, I started a draft about how hard that was, and what worked, and what didn’t. After several weeks, half of those ideas were already being disproven. We were going so fast. Defeated, I abandoned the draft and stopped blogging completely. That was a year ago.

But, looking back, it wasn’t my writing that was blocked, it was the sharing that I’d stopped doing.

In fact, after a 10+ years hiatus, I’d started writing letters again, to my now-pregnant childhood friend, back home in Illinois. Writing to her is as effortless as talking.

I told my friend about all the biggest things that were going in my life, which to be honest were 99% company-related. I told her about the teammate we knew we needed to let go, and how we were too chicken to do it. I told her about raising our Series A funding, and my mixed feelings about what that meant to me and my husband, and our company. I confessed about how surprisingly intimate it can be to work closely with people you trust & admire, and how confusing those feelings can be sometimes.

When she told me about how she was finally getting over morning sickness, I told her I couldn’t *think* about having kids anymore, because I was completely emotionally maxed out. By that point the company was 30 people and I felt personally responsible for all of them. That letter is where I finally admitted to myself how ridiculously I’d over-extended myself. It was such a relief.

When I look back, I’ve learned so much over the last year, and have a lot of regrets about not publishing any of it.

The truth of the matter is, I am terrified of sharing any of this type of stuff publicly. It’s a mix of Imposter Syndrome and outright Fear.

Here are some examples of thoughts I have:

  • Why would anyone care about what you have to say? Your story isn’t special. Keep your ego in check.
  • Your writing is mediocre at best. You just got lucky in the past because your friends up-voted your posts on Hacker News.
  • Our company has a brand now. That some people actually know about. Don’t screw it up.
  • The only reason people liked your posts was because you are a female and they were trying to be nice.
  • People already think the only reason you have this job is because your husband is the CEO. Blabbing about stuff you don’t really know that much about on the internet is not going to help your case.
  • If you write about your company, your friends and family back home are going to think you’re bragging about your success.

Taken one by one, these doubts are pretty easy to argue against, but the combination of them can feel like quicksand. With practice, we can all learn to quiet these voices, but it takes hard work.

Then there’s the straight-up Actual Risks category of anxieties. This is a special category that disproportionately affects women & minorities in the tech community. It’s the threat of the trolls, the misogynists, the racists, the rape threats. It’s accidentally getting your company DDOS’d because you complained at a conference. It’s the stalker-fan who sends you explicit messages through every channel, and knowing there’s nothing you can do about it except hope they don’t show up on your doorstep. This special kind of fear is a topic for entirely separate article, but I wanted to mention it briefly because it is a very real and true blocker for so many of us, particularly women in technology.

DESPITE all these doubts & fears. I still think it’s worth it to write.

So, this is me reminding myself:

  • of all the helpful posts I’ve read and how glad I am that people shared them. To pay it forward.
  • of the thank you notes and tweets I’ve gotten, for the stuff I wrote that really helped people.
  • that unlike a lot of other types of work, content keeps on giving long after you’ve published it, and it always seems worth it in hindsight
  • that I have a responsibility to the tech community. to do my part so that it’s not only men’s voices talking about technology and entrepreneurship.
  • that publishing has been some of the most rewarding work I have done in my career, and that it’s incredibly fulfilling. that doing it for myself might be reason enough.

Peace & Love,

Michelle


New Year, New Pricing!

Today we are announcing new pricing!

Our pricing for new large-volume accounts is increasing. In the past year we’ve learned a ton about our operating costs, the price of alternatives to using Keen IO (namely building in-house), and the market value of our services. The new prices represent a sustainable business model for us while still providing a great value to our customers.

To our existing customers: you won’t be affected as long as you stay on your current plan. Let us know if you want to discuss in more detail!

Of course, we still offer our services for free for small companies and projects sending under 50k events/mo.

Perhaps the best part about the new pricing is that we cut the price of our first paid tier (up to 100k events/mo) in half (from $40 to $20). This was based on a lot of feedback from the developer community: the $20 pricing better matches what small startups are used to paying for developer tools & services.

One more thing I’d like to mention is that we are passionate about the open source community, developer tools, and projects for social good. We often offer discounts for startups and not-for-profits in these categories — talk to us!

We’re excited about all the positive feedback and are looking forward to a great year.

Cheers,

The Keen IO Team


What CTOs Fear Most

A few months back our CTO Dan was so stressed out dealing with a scaling issue that he started to look green in the face.

He wrote a great post about that outage, and it inspired me to talk to a bunch of other CTOs and find out what is keeping them up at night. I solicited feedback through a few of our networks: Tech Stars, 500 Startups, and Heavybit, and ended up with 17 great interviews, 6 of which I’ve already posted here.

What do CTOs fear most? Turns out it’s not unexpected downtime. For the most part, CTOs are not worried about how to be great technologists. They’re thinking about how to be great leaders.

I really enjoyed these thoughtful responses. Hope you will too!


In this post:

  1. Allen Rohner, CTO & Co-founder of CircleCI
  2. Alex Haro, CTO of Life360
  3. Joe Ferris, CTO of thoughtbot
  4. Matthew Romaine, CTO & Co-founder of Gengo
  5. Rahf Noor, CTO of Markerly

image

7. Allen Rohner, CTO & Co-founder of CircleCI

What was the biggest problem you faced in the past 6 months?

When we started CircleCI it was 2 guys. Now we have employees & customers. I spend a lot of time thinking about how to scale the team and how to scale our product.

When should we take on technical debt and when not? Is a duct-tape solution OK for now? At our stage, if the technical solution is 100% stable and not being strained at all, it probably means we’ve spent too much time on it. We need to move fast to figure out what our customers need, but if we go too fast we’ll release bugs and they will have a bad experience. We’ll also build up technical debt that will bite us later. There’s always that tension between safety and agility.

On the team side, there’s a similar kind of debt you incur when you don’t invest in recruiting and hiring. Hiring is hard and takes a long time. Do we invest the time now for the long-term gain?

What is something that you’ve been worried about in the past 2 days?

My focus this week is marketing, which ended up meaning I’ve been working on our analytics backend so we can track conversions. I’ve been thinking about what things are important to track and the best ways to do that.

I think right now our most important metric is new users getting to “success” — how many of them get there and what percentage of them get there. But even once we measure that, how do we know if the number is good or bad? How do we know what is possible?

What’s different about being a CTO vs something like a lead engineer?

Understanding the business model. Making technical choices that shift the business.

My experience before was in big companies where it seemed they didn’t even want you to know the business side. Pricing is something I think about a lot now and never had to think about before.

Also, there are trade-offs between business goals and code. We can spend a day optimizing some algorithm to reduce server costs, or we can just buy more hardware.

Describe a far-out future risk or challenge you’ve been thinking about.

What the business model will look like in two years. What are the technical choices we will make that will change our market? We started in a small niche, but there are any number of features we could build that expand our addressable market. Those technical choices impact sales, marketing and hiring. What features are possible to build? Which features are good investments?

What is your biggest fear?

It feels like a juggling act. Keeping servers running. Getting new customers. Hiring. Things have to be going well in many areas or the whole things falls apart. How do you know you’re prioritizing the right things?

What’s your favorite war story (work related crisis you overcame)?

I have lots of those stories :). Thankfully they are getting less common as we’ve added more quality checks. I don’t have a particular story to share but there’s definitely a pattern. All of the interesting failures involve multiple bugs at the same time, and you find out that thing you never thought mattered actually really does matter.

One thing I’ve learned is that your success in handling a crisis is almost always more communication-related than technical. If you let your customers know you’re having issues right away, they’re very forgiving. Wait for them to find the problem on their own and their reaction will not be so nice. On the triage side, we’ve found that putting all of our team members in a chat room is crucial during an issue. Everyone knows what’s happening and you have people to double-check your thought process — -“Are you sure you want to restart that server? It’s going to impact X”. That kind of thing.

What do you like about your job?

Multiple things. I like having control of my destiny. In a big company “you are doing X this quarter”. In our company, we talk to users every day, find out new problems, and can make them happy on the spot. That’s really rewarding.

I think one reason big companies make poor decisions is that one person rarely understands the cost and benefits of a given decision. The engineer understands how to build it, the product manager sees the time it takes, the CEO sees the money it makes, and the CFO sees the costs. One of the things I love about being on a small team is that you can see all of that at once. You have a much better understanding of how your decisions will impact the business.


CircleCI gives web developers powerful Continuous Integration and Deployment with easy setup and maintenance. @circleci @arohner


image

8. Alex Haro, CTO & Co-founder of Life360

Alex, what was the biggest problem you faced in the past 6 months?

Maintaining a great company culture. When we were small, it was easy to make sure everyone was in sync and motivated to accomplish great things. As we’ve grown, maintaining that focus and energy has become a lot harder. It really requires dedication throughout the entire team to make sure that the culture scales.

One thing that has worked really well at Life360 is our designation of an official meeting day. By setting aside all major meetings for Tuesday, we now have a day when everyone is in the same place at the same time to plan, collaborate, celebrate and commiserate. It also allows everyone to be empowered to get their jobs done for the rest of the week.

What is something that you’ve been worried about in the past 2 days?

How do I hire the next 5 people?
Is our release schedule on time?
Are our servers going to stay up?
Where did that bug come from?
Priorities priorities priorities!
Focus focus focus!
How is everyone getting along? Are there any issues I need to address?
Is everyone clear on the vision of where we are going?
How can we make this simpler?
Is my inbox at zero? Where did all these emails come from?
What’s next?

Describe a long-term challenge you’ve been thinking about.

Managing a healthy work / life balance. It is a cliche for startups to say they are work-hard play-hard, but too often only the first half actually is true. I need to make sure everyone knows when to push hard and when to step back and decompress. Life does exist outside of work and it’s important to not lose sight of that.

What is your biggest fear?

Falling off the rocket ship. Right now, Life360 is in the very fortunate position of having great traction and massive scale. It is entirely in our hands not to mess up. I spend every day making sure we can all hang on to the rocket ship even longer.


Life360 is a family locator, messaging tool and communication app all in one. @life360


image

9. Joe Ferris, CTO of thoughtbot

What was the biggest problem you faced in the past 6 months?

In the past year, we grew from one office to four. When I first started as CTO, I was helping to grow our single office while maintaining and improving our processes, tools, and talent. Working hands-on with a few teams is much easier than tracking and participating in discussions spread out across the globe. It’s been a real challenge to put outlets and processes in place for this kind of communication and collaboration.

What is something that you’ve been worried about in the past 2 days?

We spend almost every Friday on “investment time.” Our developers and designers work on pet projects within the company, contribute to products, and work on open source. A lot of this effort is undirected and happens in solo, though. The past couple of days I’ve been thinking about how to push towards working in teams to improve cross-pollination and give people more of a sense of purpose with their projects.

Describe a long-term challenge you’ve been thinking about.

We’ve done a great job of developing talent and knowledge with specific tools like Rails and iOS. We’ve also streamlined a lot of our processes around those technologies. I spend a decent amount of time figuring out how to make it possible to experiment with new platforms without introducing too much risk into our current strategies.

What is your biggest fear?

I think my biggest fear is that our industry will dry up overnight because of some kind of investor crash or something. It’s always the stuff that seems totally out of your hands that’s scariest.

What’s your favorite war story (work related crisis you overcame)?

There was a period where we weren’t very focused on hiring and three of our developers left the company, mostly because they were moving away from Boston. We’re a talent company, so it was really important to replace them with other great developers. Since we hadn’t hired anyone for a while, though, our pipeline had gotten a little stale. We did a few things to get the process moving again, but my favorite was http://www.apprentice.io/, which was a new iteration of our apprenticeship program. It turned out to be an amazing way of discovering, growing, and hiring talented developers and designers. We continue to use it when growing our offices today.

What do you like about being a CTO?

It’s been a more difficult and rewarding challenge than anything else I’ve ever done. I get to work with talented people every day and discuss how we can make great things better. Being at the center of such an amazing group of developers every day is awesome. It exercises more, different areas of my brain than just doing development.


thoughtbot. Expert designers and developers who turn your idea into the right product. @thoughtbot @joeferris


image

10. Matthew Romaine, CTO & Co-founder of Gengo

What is something that you’ve been worried about in the past 2 days?

Not giving my team the space they need to flourish, because I enjoy building product so much :) The hardest work to delegate is what you enjoy doing. I don’t code as much as I used to, but there are still parts of the system that I know best and it’s hard not to get sucked in. So my most recent worry was “Am I stepping on someone’s toes?”

Describe a long-term challenge you’ve been thinking about.

Scaling up the team; it’s one thing to scale infrastructure — lots of opinions online about that — but it’s another to scale human interaction. There’s much written about the benefits of small, modular teams. But for all pieces to work together, teams need to have consistent guidelines, processes, communications, etc. I also think it’s important to have a feedback loop with customers and users as engineering operations grow. That doesn’t have to mean engineers should be speaking directly with users. Feedback via customer support, marketing, and other touch-points needs to be fed back to the engineers. As operations grow, ensuring there are no holes in the feedback loop is a challenge.

What’s different about being a CTO vs something like a VP Eng Role?

As CTO I’m now much more visible in public. I spend a lot of time sharing the fruits of the engineering team and explaining why what we’re doing is cool, as well as obtaining new tech info to feed back into team. Blogs and prose online only give you so much perspective, where as in-person feedback from personal meetings and conferences like OSCON are important because not everyone with valuable experience is an avid writer.

At Gengo, the VPE is the true team leader, responsible for day-to-day management of the team and execution of deliverables. It’s an incredibly challenging job — dealing with inter-personal and team dynamic issues, driving deadlines, selecting and solving technical issues, etc.

Together we collaborate on the bigger-picture items too — from architecting the system for long-term growth to hiring, motivating, and rewarding the team.

What is your biggest fear?

Miscalculating opportunity cost. By that I mean not adopting new technologies that would improve our system significantly at the right time due to implementation concerns, and then having a competitor zip by because of it. Or, not adopting important technologies early enough and increasing our technical debt instead. In other words, I fear missing the tipping point of when building on one technology turns into managing it as technical debt.

What do you like about being a CTO?

I have so many opportunities to meet incredibly smart and amazing people — sometimes the CTO title itself can help open doors. I love sharing stories and challenges and learning from others’ lessons, then bringing them back to the team. I also enjoy sketching and discussing new architectures with my VPE, and making big plans for the future growth of Gengo.


Gengo is the translation service behind global companies. @gengoit @quanza


image

11. Rahf Noor, CTO of Markerly

What was the biggest problem you faced in the past 6 months?

One of the biggest challenges we had was scaling our infrastructure to support our rapidly growing user base. Six months ago, we were collecting data on users, scoring them and placing them based on the interest graph we’d created. We collected dozens of interaction points on each user, and were rapidly scaling. At the time, we were at around 9 million users, and growing significantly month over month. Our infrastructure needed a complete overhaul. We moved to a schema-free database, and switched all data collection to logs. Additionally, we moved all client/user loaded assets to a CDN (content distribution network).

What is something that you’ve been worried about in the past 2 days?

We’re going to be on-boarding our first major brand resulting in significant revenue for us. We need to make sure that we deliver the data/reporting on time and have no server interruptions/hiccups.

Describe a long-term challenge you’ve been thinking about.

Eventually, we’d like to make all of our reporting real-time. Currently we are generating end-of-day-of reports, and future clients may want reporting on demand.

What’s different about being a CTO vs something like a lead engineer?

A lead engineer is responsible for their team and current project. A CTO must balance the current project and team while budgeting time and considering long-term business goals.

What is your biggest fear?

Not necessarily a fear, but something that lingers in the back of my mind — I don’t have an extensive engineering background (~ 5 years), so I wonder if we will get to a point where my experience, background, and skill set are insufficient for our company.

What do you like about being a CTO?

Being able to choose how we want to attack a problem from an engineering standpoint. There are many ways to accomplish a task, and I’ve been afforded the ability by my co-founders to make the call when need be.


Markerly is a content agency that connects your brand to influencers. @markerly @NoorRahf


Checkout these other great interviews, too!

  1. Robert Fan, CTO & Co-founder of Sharethrough
  2. Charlotte Genevier, CEO and former CTO of The Cotery
  3. Dean Shold, CTO of Stanford Hospital & Clinics
  4. JJ Zhuang, CTO & Co-founder of Acompli
  5. Dan Kador, CTO & Co-founder of Keen IO
  6. Subbu Balakrishnan, CTO and Co-founder of Good.co

Coming Next:

  1. Tony Amoyal, CTO of GetSidewalk
  2. Matti Paksula, CTO of AppGyver
  3. Florian Motlik, CTO of Codeship
  4. Mahmoud Arram, CTO of TriggerMail
  5. Tim Jenkins, CTO of SendGrid
  6. Anonymous, CTO of Mystery Company

If Developers Are the New Steelworkers…

There’s an idea buzzing around the past year or so, I’m sure you’ve heard it by now, that developers are the new steelworkers.

The analogy compares developers to steelworkers at a time when steelworkers were highly skilled, highly paid, and in very high demand.

The analogy breaks down after that — most notably because code (and developers) evolve faster than steel. Code simply doesn’t have the same physical limitations that steel did. Iteration is not capital-bound. Advancements in our modern craft spread fast, many many times in a given person’s career. The success of a given developer, unlike that of a steelworker, is, in some ways, unbounded. Steelworkers couldn’t invent and globally distribute tools to other steelworkers.

There are some problems with this steelworker analogy (which is perhaps what makes it so thought-provoking), but I think it describes some important characteristics of our time period — such as the idea that we are in the midst of a technical boomtime with widespread opportunity for technologists.

Some references where you can find the analogy (in both positive and negative lights):

  • Venkatesh Rao describes hackers as an artisan class of steelmakers and (fascinatingly) goes on to describe entrepreneurs as the new middle management.
  • James Lindenbaum founded Heroku and Heavybit betting that there is tremendous opportunity in providing tools to these developer artisans. This article describes theHeavybit manifesto, comparing the current tech economy to the early days of the automotive industry.
  • In this critique, Rebecca Solnit compares technologists to miners during the Gold Rush, describing how the present-day boomtime (and the resulting Google buses) are destroying San Francisco.

These stories got me thinking about how our business here at Keen IO fits into this picture. If developers are “the new steelworkers”, our job here at Keen IO is to provide tools and leverage for their projects.

Andrew Carnegie founded Carnegie Steel in the late 1800’s. Carnegie didn’t invent steel. He figured out how to manufacture steel rails with incredible efficiency, so that steelworkers could build railroads and infrastructure at an unprecedented pace. Carnegie profits in 1900 were $40,000,000!

How can we be like Carnegie? If the tech economy is like the early steel industry, there will be a massive number of steelworkers building new railroads, new skyscrapers, new infrastructure. In our modern example, those will be new technology platforms, new software, and new smart devices. AWS, Heroku, Twilio, SendGrid, and Stripe are great examples of how you can build a business helping developers do their jobs faster and more efficiently. Our hope is to join them and play a part in accelerating the the next generation of technology. We want to be the rails and beams developers use to build modern data products.

Industrial vibe of the Heavybit coworking space we call home.

Stairwell made out of, you guessed it, steel!


Integrate Stripe and Keen IO in 60 Seconds

For months we’ve had “Stripe integration” on our roadmap. What data could be a better measure of success than payments?? I wanted a big revenue graph next to all of our other business metrics. I also suspect there is a large overlap between Keen fans and developers who love Stripe; they’d probably find the integration useful too.

Well, last week we discovered something AWESOME! The integration is already built. Paste a Keen event collection URL into the Stripe webhooks UI and BAM, you’re collecting JSON events from Stripe. This is what happens when companies build APIs using common design standards like ReST.

The Keen URL you need to paste into Stripe looks like this:

https://api.keen.io/3.0/projects/<keen_project_id>/events/Stripe_Events?api_key=<write_key>

Enjoy!!

Notes:

1. The default integration is functional, but it isn’t perfect so we will probably release an endpoint in the future to better organize the stripe events.

  • separate collections for the various event types (invoicing, payments, etc)
  • timestamps matching Stripe stamps rather than the time the event is received (although they are pretty close)
  • guide for how to provide custom properties to enrich Stripe data so you can do end-to-end conversion funnels

2. If you want to display your revenue in dollars instead of cents, you can divide the result by 100 and add a $ prior to calling the draw method in the Keen javascript library. Find us in our Slack chat if you have any questions.