Move Fast and Learn Things

User research doesn’t need to be a headache. It doesn’t need to be expensive, or time-consuming, or complicated. But it certainly feels that way sometimes, especially with deadlines, competing priorities, or a healthy fear of stasis.

“It sounds great, but we have way too much on our plate to spend time on this right now.”

“I already know what we should do, so why waste time second-guessing myself?”

I suspect the main underlying objection to making time for user research is simply not understanding it. Let’s change that!

This post outlines how my teammate Micah Wolfe and I redesigned Keen’s first-time developer experience during a week-long user research and design sprint. We riffed on Erika Hall’s awesome Minimum Viable Ethnography and introduced a few new tricks of our own. This may not be applicable to everyone –we have an existing product and customer base to draw from– but hopefully you find some of this helpful!

Monday: Prepare

First, I installed FullStory to observe how our customers were already using the current version of our core web app immediately after signup. This part of our site gets a lot of daily traffic from active users, so I added a little logic to only run for those who signed up less than 2 weeks ago.

Next, I scheduled a two hour idea session with 4 colleagues and a whiteboard for the next morning. By the time this meeting rolls around we’ll have made a bunch of valuable user observations and insights, so I scheduled this session in advance to create time and space to process our findings. I also made sure to schedule time before lunch so everyone would be fresh, rested, and focused.

I then messaged a subset of our users, inviting them to share input on something new in a 30 minute video call on Friday. In exchange for their time I offered to throw $25 in billing credits on their account. Soon I had 5 eager participants scheduled across a 3 hour window. It’s best to schedule these sessions 15–30 minutes apart to avoid cascading delays.

Keen has a public Slack community with over 2,000 users, and a metered pricing model that makes it easy to credit customers’ upcoming bills. If this not an option for you, Amazon gift cards work wonderfully.

Next, I shifted my focus internally. Keen has an amazing Customer Success team with a deep understanding of the customer experience, from day 1 to day 100 and beyond. Their input on new product initiatives is priceless. We joined our CS team for an hour-long deep-dive into new account onboarding, asked a few open-ended questions, and took as many notes as we could manage.

Afterward, we debriefed and discussed themes in what we learned. It’s a good idea to do this while a conversation is fresh in your mind. Where are the hot-spots or critical breakdowns? Did we hear anything that surprised us? What do they think are the most important problems worth solving right now?

Tuesday: Analyze

On Tuesday morning I checked out our Fullstory dashboard and found over two dozen new recordings waiting. Each showed a new user exploring our web console in the first several minutes after signup, page by page, click by click. I watched them all and took notes.

Clear patterns emerged that highlight the painful gaps between how we thinkour web console gets used, and how it actually gets used.

Whiteboard time! I brought all of my notes and insights to the idea session that I had scheduled on Monday. In a short amount of time we captured a ton of valuable information, covering a wide range of perspectives.

We all took turns reading through the notes and transferred each point or idea to a magnetic square on our whiteboard. Post-It notes work just as well, if you have them. We mixed and sorted our notes, looking for patterns; rearranging squares into distinct underlying themes, and ultimately finding a fresh new perspective on our first-time developer experience.

 

1_IGoo6jpKJ54WB8vNsaCrNg.jpeg

Stacking and sorting, finding themes along the way

We spent the rest of the day modeling out the sequence of tasks and actions that users go through to get set up, overlaid with the key pain points and breakdowns we uncovered:

  • Most of a new user’s first session was spent on things other than installation. Simply getting installed required jumping around several different pages. Sometimes users would get lost and simply give up, or would message support for guidance.
  • Verifying that their first integration is working properly is critical step, but our interface actually makes that quite difficult. What turned out to be one of the most helpful features was buried deep in an obscure page. We also found a nasty bug that was obscuring a successful setup for nearly half of all new users – yikes!

1_3yPUVROo7k6fqViAOEEwLQ.png

Critical path diagram, by Micah Wolfe (Source)

Moqups is a great tool for this, but a whiteboard or pen and paper will work just as well. Keep it simple and use what works best for you.

Wednesday: Explore

Day 3 was spent reimagining the critical path of our first-time developer experience. We sketched and prototyped a bunch of possible approaches, refining our thinking down to a new flow that could address all of the issues we found, without being too restrictive.

1_o8MFo5GcI4tYOa-Sa85vtw.png

New Critical Path diagram, by Micah Wolfe (Source)

Thursday: Prototype

We’ve come so far, in such short time! But now, it’s time to put our new insights to work and create something that we can test in the real world. Our efforts from this point on would be based on the following design principles:

  1. Clarity: Remove all non-critical activities and information between signup and setup
  2. Focus: Resolve any reasons why a user would need to navigate away to complete setup
  3. Confirmation: Support the validation step by verifying to the user that their chosen integration method is working correctly, and then let them view the data that they just recorded

We spent the morning sketching out a variations of interfaces and interaction sequences to tie it all together. Once we felt good about the high-level details we produced a series of high-fidelity mockups. These were then loaded into InVision and turned into an interactive demo.

 

1_V9KzmKLAahnkP0h7RUX-sQ.jpeg

Friday: Interviews

Our first customer interview began at 9:30am sharp, and began with a conversation guided only by a few open-ended questions about their past experiences with Keen. I used QuickTime Player to capture each session as a screen recording.

Here are a few of the questions I used to guide our conversation:

  • “From when you first signed up, how much time passed before you felt confident that things were set up and running correctly?”
  • “Was there ever a moment when you felt stuck or unsure of what to do next? Did you ever need to contact support for anything?”
  • “If you could go back in time and give yourself one bit of advice before getting started, what would that be?”

Next, I shared the link to our InVision demo and offered minimal instruction on what to do. I simply asked them to imagine they are signing up for a Keen account for the first time, and to share out loud what they are seeing, thinking, and doing along the way.

By lunchtime we had two and a half hours of hands-on feedback. Here are a few screenshots from our sessions:

0_9uVfmmxQBhGuM0L8.png

0_n0pdUUyLswhQV2ol.png

0_ckkLkzOaLKGwfeFT.png

0_EJarmgkZPJsglb9t.png

Here’s what we learned

At a high level, our new critical path held up. Every participant had a positive, enthusiastic reaction to the flow we had laid out. The clear prioritization and focus of getting installed, the explicit “verification” step during the setup process, and the ability to view the data that had been recorded:

  • “I like this.. Hated that it took me so long. ‘Choose Integration’ page is the first page I care about.”
  • “I like that ‘Verify’ action- satisfying. Nice big green to know it’s working.”
  • “Most recent events is where I go.. Good to see all data there.”

There was also a fair amount of confusion around the choice of wording and placement of elements in our prototype:

  • “‘Choose an Integration’ is vague.. What I really want to do is send data. They’re all ‘sources’.. All ‘integrations’.. Make it really obvious.”
  • “Didn’t notice the side panel.. Seems a little detached. Thought it was part of the InVision app.”
  • “Auto Collector blurb on the JS page is out of place and confusing.”

We also heard a common reaction on the prompt to invite teammates during the signup process:

  • “Inviting teammates is day-2, I would have blown past that to start playing with data.”
  • “I normally wouldn’t invite my team until after I’m up and running. … especially wouldn’t invite people while I’m in the middle of setting things up.”

The results of these sessions were then cycled back into several more design iterations. We are now several leaps ahead of what you see in these mockups.

1_eDOioG_9M18YnQF3FGiyAw.png

Latest iteration of our Stream Setup Wizard

Outcomes

This week-long research sprint not only sharpened our understanding of the breakdowns of our existing first-time developer experience, but also allowed us to prototype and test a new approach with actual customer input. Shortly after, two project teams were formed to implement our recommendations.

  1. Stream Setup Wizard that is shown immediately for every new project, featuring a wide assortment of possible integration types, including sample data generation
  2. Stream Manager App that improves users’ visibility into their event streams while making it easier to expand and manage their usage
  3. Team invite links that can be shared in Slack or over email, allowing others to join an organization with minimal effort from the original user
  4. Communication prompts delivered as in-app messages and email, to direct users forward in their journey with helpful content and assistance

The Stream Setup Wizard alone boosted new account activation by 35%. Once the option was added to load a project with sample data, activation jumped again to 65% above our original baseline. Alexa Meyer and her project team did a fantastic job implementing this new feature, and deserve all the high-fives for the impact this has had on our business.

This research sprint only cost a week of our time and less than $100 in account credits, and has had an incredible impact not only on our bottom line, but on the first-time experience of hundreds of new developers who decide to give Keen a try every week.


Custom Cohort Analysis is now Open Source

Build your own cohort analysis tools, with total control over the logic, look, and feel.

1_yC9_z-o_34iuB0qXw0Yh1Q.png

Open Source Code for Cohort Analysis Views

Today we’re releasing a new open source tool for cohort analysis. Cohort analysis is incredibly valuable when you’re trying to understand retention and patterns of behavior over time. If you’re not familiar with retention analysis by cohort, you can read more about the fundamental concepts here.

Outside of the obvious community benefits of open source contributions, there are a few reasons we wanted to share this simple tool:

  1. Easier cohort and retention analysis on Keen IO data You’ve always been able to run retention analysis on Keen data, but this is the first time we’ve released a tool for visualizing the results.
  2. Open source analytics building blocks Keen IO’sanalytics building blocks enable super custom, powerful analysis. Looking at the source code, you can see how in looping over a simple funnel analysis you can build this detailed behavior view. What else could you build with these powerful primitives?
  3. Give developers totally customized cohort analyses based on their business If you’ve ever been frustrated that the retention views in your analytics tools don’t quite match how your business works, or didn’t quite display the information in a way that made sense to you, now you have something to fork and make your own. As with most things we build here at Keen IO, you could even white label something like this in your product.

To start running your own cohort analysis you can create a free Keen IO project and fork our open source repo. We hope you’ll find this useful, and share your feedback in the Keen IO community slack.

Happy coding.


What we Learned Dogfooding our Native Analytics Product

This week, we released Project Analytics — a brand new analytics dashboard displaying service level usage for our customers. This project is the beginning of a new effort aimed at bringing greater insight and observability of Keen IO service usage to our customers. Many of our customers rely on our platform to deliver Native Analytics to their customers, directly in their own products. Now we’re doing exactly that: analytics for your analytics. Pretty meta, huh?

0_T3HWdol4zoyBvASv

Sample Keen IO project analytics console

This project was super fun for us to build and an opportunity for us to dogfood our recently released Native Analytics product. Read on for some juicy details on how and why we built this and some of the design challenges we faced along the way.

Why did we build Project Analytics for our customers?

First and foremost, our goal with this update is to improve the product experience by bringing more information to the surface of our web console. We have a clear vision for our apps and interfaces that leans heavily on native analytics powered by our own platform. Account usage information has also long been a common customer request that would land on our support and customer success teams. Baking these metrics directly into the product reduces both support ticket volume for this topic and the time spent tracking down this information for each customer.

This was also a chance to dog-food our own Native Analytics product in ways we haven’t had to yet. The insights we’ve taken from this project have already influenced future product development in big ways, for both our API and data visualization capabilities.

What were some of the challenges we faced while building this?

The design phase of this project was pretty intense! We spent a lot of time sketching and iterating — not just what you see now, but what is coming next. This project also coincided with a substantial visual refresh, several big performance improvements behind the scenes, and an active recruiting effort by the engineering team responsible for building this project. Coordinating a successful release around so many moving pieces proved to be a bit challenging.

These obstacles probably won’t sound too surprising or unfamiliar for anyone reading this; that’s just daily life at a growing company. The experience did reaffirm how important Developer Experience is for API-based services like ours. We put a lot of thought and care into how quickly developers can get up and running with our API and SDKs. We know our customers already have a long list of obstacles and blockers to deal with to get a project out the door. Integrating with Keen and getting data flowing in and out of your app shouldn’t be one of them.

How did we decide which metrics to show?

For this first release, we really had to resist the urge to go big, or to over-commit to granularity or complexity. This is just one small piece in a roadmap that that will unfold over time, and we know we have a lot to learn along the way. We give our customers the same advice: start small and move fast; experiment, iterate, and leave room to be surprised by what you learn. This lets you develop deep understanding and momentum at the same pace. You need both. Having one without the other generally doesn’t work out too well.

What’s next for Project Analytics?

Next, we’ll be diving deeper into project-level usage stats, data visualization, and data management, bringing more information and utility into the web console. Basic usage metrics and fixed dashboards like these are just the beginning. We believe the apps and interfaces of the future will be highly self-aware, educating users about usage behavior and opening up new feedback loops that encourage better decision-making with less effort.

Log-in to your Keen IO account to check out your project analytics or create an account to get started. We’d love your feedback. Just shoot us a note on Slack.


Announcing Keen Dashboards

We have some big news to share today!

We’re excited to announce Dashboards Beta release, a brand new tool that gives anyone the power to create custom dashboards in just a few clicks (or taps!)

0_irTYI_7wgPCTV_ai.pngExample of a custom product metrics dashboard for a customer success team

Keen Dashboards can be used to:

  • Quickly create custom dashboards with a point-and-click interface
  • Share sets of metrics around KPIs across every team, project, and department
  • Give anyone on your team from analysts, marketers, to product managers the ability to query, create, and edit their own set of metrics (no need to know SQL!)

For the past few years, we’ve focused on making it super easy for developers to explore and discover the data that matters to your business with Keen’s powerful analytics platform and API. Many of you have used our Data Exploreropen source dashboard templates and visualization libraries to create beautiful and responsive displays of your most important metrics wherever you need them.

With the release of Keen Dashboards, we’ve combined the power and flexibility of a fully custom analytics solution with a simple interface for exploring and visualizing your most critical data. Now anyone across your organization can create and share dashboards, eliminating bottlenecks and making it easier than ever to follow the metrics that matter to you and your team.

How does it work?

0_RQHuZbBsX-AAci7N.gif

  1. Log in to your Keen IO account and save a few of your favorite queries!
  2. Click on the “Dashboards” tab
  3. Click the “Edit” button to activate the Editor
  4. Add rows and configure each row’s tile layout
  5. Select a saved query for each tile
  6. Save and close the Editor to see your awesome new dashboard!

0_eX3tFV69Kz6en3NG.png

Ready to create yourself a beautiful, custom dashboard unique to your business? Log-in or sign up for Keen today and start putting your most important business metrics in one place or schedule a demo!

Request a Personal Demo

ps. Want to help us design and build awesome tools for our customers like Keen Dashboards and Data Explorer? Join us!


How Doc-Driven Product Design is Helping Us Build Better Software

Ohhai, world! Dustin Larimer, here. I’m a product designer and engineer at Keen IO. My team focuses on making our analytics APIs more accessible and understandable via point-and-click interfaces, visualization tools, and SDKs.

But designing a user-experience around an API can be challenging. At an API-centric company like Keen, Stripe, Twilio, or lots of others, your product isn’t something you can see or touch so you don’t have the luxury of visual cues to help people intuit how it works.

To tackle this challenge, we’ve put together a team with a unique blend of engineering, interaction design, content strategy, and product management. This blur of disciplines has led to some pretty interesting discoveries about the design of developer-facing products and services that I’m excited to share with the world.

Form reveals function

One of the biggest design challenges for any API-based company is that your core product is effectively invisible. Nearly every other type of physical or digital product relies heavily on affordances: visual or tactile features that hint at what a thing is capable of, and how you might use it. In this way, form reveals function, making it accessible to the mind with minimal effort or education.

0_8aoI6nmlzZWC9aIt.jpg

Magnalite Aluminum Kettle, by Scott Wilson, founder of MINIMAL

This is how many well-designed products actually teach you how they work. One glance, and you just get it. When is the last time you read a how-to guide for a kettle? An API, on the other hand, is a whole different animal.

An API is invisible, but that is not to say it’s formless. The form of an API is its documentation. Unfortunately, docs are typically the last thing anyone wants to spend a lot of time on. This results in confusing mystery-meat software for the rest of us.

If we were designing any other type of product, we would almost certainly start with its form. We would sketch and explore different interfaces and elements, bend and mold our ideas about what it even is, integrate our constraints, cut out the crap, and hopefully arrive at something so crystal clear and refined that you just couldn’t imagine it any other way.

0_t0iSxz1StloO-NCD.png

Essence — “Ideation” by Chris Murray, Virginia Polytechnic Institute & State University (VT)

There’s real beauty in this degree of clarity. Why don’t we do the same with software? What might happen if we start with documentation?

Doc-driven prototyping

My team recently kicked off a new effort to break keen-js, our JavaScript SDK, into three distinct modules:

This work was inspired entirely by what we’ve learned while supporting and maintaining this codebase. The monolithic Swiss Army knife approach to JavaScript libraries has run its course, often causing more headache than whatever convenience a bundled library once offered.

One of the biggest shortcomings of a giant bundled library is documentation bloat. Despite several attempts to streamline the docs, there was just too much to cover in an attention-constrained world. Many important details and features were simply hidden in the noise.

Breaking these out into stand-alone projects let us focus on writing the most concise, widely relatable documentation possible for each module’s specific purpose. And that’s exactly what we did, before writing a single line of code.

Each project began and evolved as a sketch within a README.md file, introducing the project and installation options, outlining high-level functionality, and then demonstrating everything with detailed examples and design rationale. This approach created a wonderful new canvas for exploring the form of these tools.

0_pY7RBpVsiFN6zsNX.png

We refined behavior patterns so that a few familiar patterns could be reliably anticipated throughout the library, and established new naming patterns for easier reference and recall. In some cases, this resulted in a few minor but necessary breaking changes. We also used this opportunity to stub out exciting new functionality while experimenting with how it might be written into use.

 

0_FgA9E3GCQj_S2nui.pngAt the end of this exercise, we were only committed to a single README.md file, which could be easily shared with other teammates and customers who were eager to see the latest and share feedback. Several iterations later, we were ready to write code… which goes surprisingly fast, when you feel confident in knowing exactly what you’re building!

What do you think of doc-driven design?

It’s important to note that this is one big experiment. It’s a work-in-progress, but one that we’re super excited about, and would love to discuss with anyone who is interested.

Have you or your teammates tried doc-driven software design? Are you tackling different challenges in similar ways, or similar problems in different ways? Is this a topic you care to read more about?

Weigh in with a comment, join us and discuss on Slackshoot me an email, or kick off a conversation on Twitter.

Thanks for reading!


Introducing Geo-Explorer: An Open Source Dashboard for Tracking App Usage by Location

Need to monitor your app activity based on location? The geo-explorer dashboard is a fun and interactive way to take a different look at your data. (You might remember it as part of our open source dashboard repo that launched a few weeks ago.)

0_3Crsk38S-460_U5O.png

The dashboard repo shipped with several examples, but the geo-explorer dashboard in particular spurred a lot of questions and excitement, so we thought it might be worth more of a deep dive.

The geo-explorer makes it super easy to visualize user activity by location. Using the Mapbox JavaScript API and Keen’s own geo-filtering capabilities, you can create an interactive dashboard that updates a set of charts as you navigate the globe.

0_HNe12RjnQBHiIr52.png

What sort of things can you monitor with this functionality? What about app usage stats during geographically significant events, like concerts or festivals? Or device activations in key cities or regions corresponding to a product launch!

If you’re already tracking your data in Keen, you can enable this functionality with two quick steps:

1. Include a location with each event:

{
    "keen": {
        "timestamp": "2012-12-14T20:24:01.123000+00:00",
        "location": {
            "coordinates": [ -88.21337, 40.11041 ]
        }
    }
}

2. Include this filter with each query:

{
    "property_name" : "keen.location.coordinates",
    "operator" : "within",
    "property_value" : {
        "coordinates":[, ],
        "max_distance_miles": ,
    }
}

We spent some time streamlining both the interface and the source code for geo-explorer, making it much easier to reason about and customize this cool little app. All browsing parameters, including pan and zoom state, are persisted in the URL, making it possible to bookmark or share specific geographic snapshots.

If you’re not tracking your data in Keen, you can sign up for a free account, grab your project ID, and start tracking events right away. The sample data powering this demo includes geo coordinates (which make this sort of analysis possible), so feel free to reuse this project to build and demo apps of your own.

Like we said, the geo-explorer is part of our open source dashboards project, which just passed 3.7k stars on Github! (It even spent a few days as the #1 trending repo!) DANG! We knew this project would be helpful, but we had no idea what was coming. Thank you all so much for helping us spread the word!

0_kFjiD26-1Ncd9KFY.png

Get in touch if you have any questions about the geo-explorer, or if you have any feedback or suggestions! Happy hacking!


Charts on Grids - Responsive Dashboard Templates with Bootstrap

Building a dashboard for your customers? Need a quick and easy way to share stuff with your team? Check out the new keen/dashboards on Github and get up and running super quick with great-looking, customizable dashboard templates.

Each template is composed of a minimal set of custom styles built on top of Bootstrap v3.2, covering some of the most common dashboard use cases and layout configurations.

Here’s a template with a line graph, a pie chart, and a row of stacked column charts:

0_zJF6M9e9LHMm2dBH.png

The position and size of each chart is automatically handled by the template. Each dashboard is responsive right out-of-the-box.

There are eight different templates available today. This one’s called Hero Thirds:

0_bXdixzV8AULrxmJp.png

Look familiar? It’s the template used in the example above.

Here’s another one called Quarter Grid:

0_xe_ldb9n2yAvfDTU.png

Each container is ¼ of the horizontal space available.

A dashboard template starts out as a set of empty cells. The HTML markup makes it easy to add a chart to each one, along with a title and optional notes.

 
Cell Title

 

</div>

Notes about this chart

</div> </div>

The chart goes in the #grid-1–1 element, embedded directly in the HTML or added with JavaScript later. If this was from the Quarter Grid template, you’d have up to element #grid-1–15 to fill with graphs, visualizations, or anything else you want to show off.

Let’s look at a few more examples of the templates in action. In these examples, queries and visualizations are handled by the Keen IO JavaScript SDK, but you can use any database and charting library you’d like.

This example is just four large cells. It’s populated with a line graph, two gauges, a bar chart, and even a map:

0_3rGs_V482AP9d4IZ.png

Now check this out: A real live NASA rocket scientist loaded Mars rover data into Keen and a created a dashboard to show when photos were taken. (He even found a 9-year-old bug they’re now investigating!).

This is that dashboard. It has a long line graph showing the weekly count of rover pics, two pie charts about camera usage, and one simple metric — the total count of pics taken.

0_HOgIGuoNkPnebjYI

These templates are designed for elegance and ease. And extensibility too — everything is open source. Hack away.

Ready to get started? If you’re already tracking your data in Keen, it’s as easy as 1, 2, 3:

  1. Clone the repo to get this sweet, sweet code.
  2. Enter your Keen IO project ID and read key.
  3. Add queries and visualizations to each cell. (The Keen IO JavaScript Docswill help you figure out how to construct and visualize queries.)

If you’re not tracking your data in Keen, you can sign up for a free account, grab your project ID, and start tracking events right away.

At Keen IO, we strive to make data science and custom analytics easier for developers. We hope you find these templates helpful, and can’t wait to see what you come up with yourself!

If you have any questions about keen/dashboards, or any suggestions for stuff we should add to it, feel free to comment below!