Diederik van Houten


~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Don’t let mistakes cascade

Diederik • @jdvhouten • • Part of Day 46

You’re in the kitchen, preparing a meal. You’re in a hurry and you’re hungry. So you move fast. You grab a plate from a cupboard but you drop it and it shatters. You bend over to pick up some shards and as you get up you hit your head on the cupboard door you left open. You curse and rub your head. As you walk to the trash bin to throw away some of the broken ceramic you notice a pot starts boiling over. You rush back to the stove to turn off the heat and step on a shard you hadn’t picked up earlier. Now your foot is bleeding. You want to move the pot from the stove but there is no available counter space. You try to shove it on there anyway. A plate pushes into a cutting board that in turn pushes into a couple of glasses that were precariously placed right next to the sink. They fall in, and break.

It’s 2am and your phone buzzes. You see a notification your app is down. You’re confused and wonder if it’s a false alarm, but you look at your email and you see a bunch of angry messages. Oh crap. You’re exhausted and groggy, but you open your laptop look at some logs. 500 errors everywhere. You realize yesterday’s feature update is the problem. You revert the code but the database is now newer than the app expects, and the ORM doesn’t know how to deal with the new columns. Now your app service doesn’t start at all anymore. You decide to drop the columns you added yesterday, but in your haste you drop the wrong column from the database and now you’re in a lot of trouble. Do you restore the database from backups? How old are the backups, do you accept the data loss? How long does restoring from backups take anyway? Do you want to restore only the missing column from backups? How long will that take? How will you fix all data inconsistencies? It’s now 2:30am, you can barely think straight, everything is down, your database is broken, and all your options look terrible.

This Is Fine GIF

These are stories of cascading mistakes. With one unforced error after another even something small can turn into a major headache. But errors don’t have to compound like this. If you just take a moment to stop and think these problems almost disappear. Imagine this, instead:

You’re in the kitchen, preparing a meal. You drop a plate and it shatters. You stop and pause for a full 10 seconds. Ask yourself what your next action should be. Answer: turn off the stove. Close the cupboard. Move things out of the way you might bump into. Then slowly clean up all the shards. Then stop for 10 seconds and ask yourself if you forgot something else. You decide to free up counter space by loading up the dishwasher. Then you resume cooking. Total delay? Maybe 5 minutes. Really no big deal.

It’s 11 at night and you finish a feature you’ve been working on. You’ve tested it, and it looks OK. You decide it’s too late to push to production. Certainly too late to do a database migration. The next morning you make some coffee and launch your feature. Everything seems fine, but after a couple of minutes you notice something weird in the error logs. A customer emails asking if the service is down. Uh-oh. You think for a minute and decide not to do a full rollback — you already migrated the database after all — but decide instead to stub out the feature. You only have to change one line of code. You reply to the customer with an apology. You fire up your dev VM and fix the bug. Looks good. Push to production. Email the customer again to inform them the problem is resolved. You’re not happy about the bumpy release, but it really wasn’t so bad.

Everybody messes up sometime. We do, too. But we’ve never had significant downtime. Never lost customer data. Never had a database migration go badly wrong. In part it’s luck, but in part it’s because we try hard not to make bad things worse.

  1. when something breaks the first thing we do is stop and reflect
  2. then we diagnose
  3. then we stop and think how the fix might backfire on us
  4. then we ask ourselves if the fix is something we can roll back if need be
  5. then we stop again to think of an easier fix
  6. and only then do we apply the fix and test if it worked

Afterwards, we look for ways to eliminate the root cause of the problem. In the case above, it’s better to release the database migration and the feature separately. That way you can roll back a buggy feature without even thinking about it. Additionally, you want to feature flag complicated new features. That way you can gradually release features in production, and when trouble arises you can just turn the feature off. It takes basically no extra effort to take these precautions, and they’ll save you a lot of time and aggravation when you do something dumb like pushing to production right before you go to bed.

Some more lessons we learned the hard way about cascading problems:

  1. Don’t do routine server maintenance when you’re in a hurry, when tired or distracted. Regular maintenance should only take a few minutes, but you have to be prepared for things to go very wrong. If you do maintenance at 11pm you risk having to work throughout the night and that’s just asking for mistakes to compound. Maintenance work is easy, but you want to do it in the morning when you’ve had your coffee and you’re fresh.
  2. Don’t hit send on emails when you’re tired or annoyed. It’s OK to let a draft be a draft and you can hit send in the morning.
  3. Have local on-machine nightly backups of all config in /etc/, deployment files, and everything you might break by accident. If you do something dumb and need to roll back in a hurry nothing beats being able to restore something with cp.

    Config backups like these saved me twice: one time I deleted a bunch of files in /etc on a production server that were necessary for the system to boot. Figuring out which Debian packages corresponded to the missing files is tricky and besides, the package manager won’t run if /etc is missing. Print directory structure with find for /etc and /backup/etc. Use diff to see which files are missing. cp -arf to restore. Use the -a (archive) flag so you restore user, group, access permissions, and atime/mtime along with the files themselves.

    Another time our JS compressor crashed and output partial data (on perfectly valid Javascript input no less) and there was no quick way to diagnose the problem. Our entire app was effectively down, so I needed a quick fix. It’s at those times that you really appreciate being able to restore last night’s JS bundle with a single copy command.

    You need a simple backup system that works every time. Some people put /etc under version control, but this isn’t great because every server is at least somewhat unique (e.g. /etc/hosts, ip bindings). Nightly backups that allow you to simply diff and see what changed will never fail you. Many backup systems try to be too clever and you need to google command line options for basic operations. rdiff-backup gets almost everything right, although it breaks if you try to back up too much data.
  4. Learn how to boot from a rescue environment and chroot into your system. chroot is basically magic. You can use this to fix your boot partition, broken packages, broken firewall/network config, mangled kernel installs and more.

    We’ve only had to use this trick twice in the last 10 years. If a server doesn’t come back after a reboot and you start sweating, that’s when you need to keep your head cool and do some quick diagnostics. You can fail over, but failover is not without risks, and if you have no clue what happened you don’t know if the failover server(s) will collapse in the same way. Downtime sucks, but you always have 5 minutes to do some preliminary diagnostics and to think carefully about the next steps to take.

The lesson here is so simple and also one of the hardest ones for me personally to learn:

Slow down. Breathe. Don’t make things worse. Consider your options before acting.

Mistakes are unavoidable, but if you don’t let small mistakes cascade into something bigger you’ll notice you’ll spend very little of your time putting out fires.

Your app needs a USP

Diederik • @jdvhouten • • Part of Day 44

Your app need a good answer to the question: “Why not go with the established competitor’s product?”. That’s what your Unique Selling Proposition (USP) is for. This is especially important when you start out, because a new product has fewer features than the competition. You also don’t have brand recognition. People buy Apple products just because they’re made by Apple. A startup doesn’t have that advantage.

Your product is also likely to have some serious shortcomings. It takes years for software to get good, so you need something to compensate for your lack of features, lack of brand, and other ways in which your product isn’t great.

You can try to compete on price. Just charge less than everybody else, right? Except, this can backfire on you in two ways. One, you send a signal that your product belongs in the “budget” section and is therefore worse than other products. And two, the cheapest products get the most difficult-to-please and most demanding customers. I’m not entirely sure why this is, but my hunch is that people who look for the cheapest products have a negative disposition. Maybe they’re afraid they’re getting taken advantage of, or maybe they resent having to pay for software in the first place. When you charge more you get customers that look at the value provided by your software and decide based on that to go ahead with a purchase. For the first kind of customer any price is too high. The second kind of customer will be happy if you explain in plain language what your product costs and what they get in exchange. You don’t want to be the most expensive product in your market and you don’t want to be the cheapest. Just be somewhere in the middle. Maybe somewhere the lower 3rd, that way you can easily raise your prices as your product gets better without pricing yourself out of the market.

Can you distinguish yourself with some really cool feature? Probably not. It might work in the beginning, but once you get some success your competitors will notice. Established businesses have more resources and most features can be copied. Usually competitors will make a feature that’s equivalent and not exactly identical, but it amounts to the same thing. The one feature that made your product unique has been commoditized and you go back to square one.

You also can’t distinguish yourself based on “universal benefits”. You can say your product is “fast”, or “easy to use” but every competitor will say the same. You can say your product is “secure”, but even competitors with the worst security practices (and the record to prove it!) can put any number of logos on their website that prove their product meets every industry standard. Ever noticed how companies with terrible customer support have these Customer Service awards plaques on their website? You should absolutely work your hardest to provide great service, but to really distinguish yourself you need more.

So you can’t compete by having more features, you can’t compete on price, you can’t compete with cool unique features. If none of these approaches are good, what can you do?

To differentiate yourself in a way that lasts your product needs to be something the competitors explicitly are not. Good products drive a wedge in a market where customers have to choose whether they want a product of Type A or Type B, but they can’t choose both. If you segment the market intelligently you’re the first and therefore best product in this new subcategory. This puts you as a new startup in a terrific position.

If you’re going to compete with Gmail you don’t want to be the product that’s cheaper (Gmail is already free) or the product that’s faster (Gmail used to be fast, too) or the product that has the most features. You want to be the product that is about privacy. No ads. You fight newsletter spam and email open tracking and other bad practices. What is Gmail going to do? Nothing! Gmail is an ad-based product, after all.

With Thymer we’re boldly choosing a text-based UI. Many people, like us, use text-based IDEs all day. It’s the kind of software interface we like, so odds are, other people will like it too. It’s not a feature our competitors can just copy, because it defines what our app is. If we succeed I’m sure people will try to copy us outright. That’s inevitable. If our app is good enough clones won’t hurt us much.

There is a final reason why it’s important to have a clear USP for your app. You want to be the best product for this new segment you’ve created. Any time you think about adding a new feature or adding some text to your product page you can ask yourself if it fits with your core theme. This helps you focus on what matters. You don’t want to dilute your vision, you want to make it stronger. It will put off many people, but those who like it will end up really liking your product. When you start out, that’s exactly what you want.

Basic strategies for validating startup ideas

Diederik • @jdvhouten • • Part of Day 42

You know you want to start a startup, but you don’t know what to build. Maybe you have notebooks chock full of ideas but you don’t know which ones are good. Maybe you don’t have any ideas, because it feels like everything exists already.

You’ve googled for tips but discovered that most articles on the internet about validating startup ideas are aimed at startups that seek Venture Capital. Startups with Venture Capital can throw money at problems. They can buy growth, features, and positive press. When you’re a bootstrapped startup you won’t be able to outspend the competition which means you need to be scrappy and strategic. It’s a different game.

Bootstrapping refers to the (physically impossible) trick of pulling yourself up by your bootstraps (etymology). As a startup metaphor it means building a startup without any resources. It’s possible to start a web business from nothing because your only expenses are web hosting, a domain name, and a laptop. You need technical and business skills, but you can learn for free. There are no real barriers to entry. That’s great for society but it also means you’ll have a good deal of competition no matter what you decide you build. Bootstrapping is possible, but it’s not that easy!

We haven’t written much about idea validation so far because it’s so subjective. What do you want to get from the business? Do you primarily want to make your customers happy? Do you want to solve interesting technical challenges? Do you just want to make money? Do you want change the world for the better? Do you want the freedom of working for yourself? Do you just want the excitement? The satisfaction of forging your own path? Everybody has their own priorities and the only thing that matters is that you know what yours are.

Also, everybody has a different skills, abilities, and resources. Do you have a lot of time to get your business of the ground or do you need to launch ASAP? Can you self-fund to a degree, or are you broke? There is a large personal component to this. What may be a good idea for you might be an awful startup idea for me.

I say all this because I want to stress how subjective this all is. There are very few right and wrong answers and we certainly can’t tell you what to build.

That said, I’m not a total relativist. Some startup ideas are brilliant and original and I feel foolish for not having thought of them myself. Other startup ideas are objectively terrible and founders would spare themselves a lot of grief if they thought their business through before they invested a lot of time and money into it.

Essentially, you want to figure out if your startup idea has legs before you commit a ton of time to it. That’s all validation is.

You have two things to validate:
(1) if people want to use your product, and
(2) if it makes sense as a business.

PRODUCT VALIDATION

  • What makes your product unique (USP)?
    • Why should people buy your product and not somebody else’s?
    • You can’t just copy a product and add 1 cool feature.
    • Do users care about your USP?
  • Can you build a prototype in a reasonable amount of time?
    • Can you get away with building just a few core features?
    • The longer it takes to build your first prototype the more confident you need to be that your idea is good.
  • Is the product scope reasonable?
    • You can’t win by simply building more features. That’s a game for VC backed startups.
    • Small products are better suited for bootstrapping.
  • Do you have the necessary technical skills?
    • Do you know where the technical execution risks are?

BUSINESS VALIDATION

  • Can you find enough users?
    • Do these people exist?
    • Do you know how to reach them?
  • Will users buy your product?
    • Do you have free competitors?
    • Is the market very crowded?
    • Do you have successful bootstrapped competitors? If not, why not?
    • Can you explain to users how your product is better?
    • Can the competition easily copy your product?
  • Can you charge enough money?
    • Figure out how many customers you need for your revenue goal
    • What kind of churn can you expect? This twitter thread has some good data.
    • Do you need to compete on price or can you charge a premium?
  • Can you grow by self-funding?
    • Can you automate your sales/onboarding/growth process?
  • Does this business suit your personal strengths?
    • You want to put your strongest skills to good use
    • Will you stay passionate about it in the long run?
  • Any legal liabilities/risks?

Yep, this stuff is prosaic. You need a product and you need customers. Not exactly rocket science, right? And yet, making an effort to think your idea through is vital. The questions are basic but the implications of the questions are subtle and can easily trip you up.

You can figure out roughly how much you can charge for you product by looking up what your competitors charge. When you start out you want to be in the lower end of that range. You don’t want to to be the cheapest product because the cheapest products attract the most price sensitive and difficult to please customers.

Some people will tell you that you can validate a business by slapping a marketing page together and asking some people if they would pay for the product if you build it. It’s just a matter of cold emailing some people (or LinkedIn/Twitter message) and you’ve got your answers. I don’t think it’s that easy. If people are enthusiastic when you describe your startup to them that’s a good sign, but it doesn’t prove much. If you make something for an established market then you already know customers exist provided you make a good product. And if you make something really original you can’t explain what it is — you have to show people. But you have to build it first (or a prototype at least) in order to do that.

You won’t know for sure if people will actually pay for your app unless you put up a credit card form. Words are cheap, after all. But before you get to this point you have to build the product and find users. No real shortcuts here, either. If you’re good at sales maybe you can convince people to give you money for a product that doesn’t exist yet, but that’s not my style.

You can work out how long the first prototype will take by starting with the hard parts. If you can outclass your competitors by solving a hard or hairy (or schleppy) problem that will give you a big edge. You don’t want to spend weeks or months building scaffolding only to discover you don’t know how to make the thing that would make your product great.

You can figure out if your USP makes sense by working on your hero message. The hero message is the big text at the top of your product website that explains what your product does. Try to capture what makes your product unique. For the thymer.com splash page we explain how we’re different: it’s an editor, and we explain in two sentences what it does (tasks and planning) and what the target audience is (makers/creatives). We collect email addresses at the bottom. If we have a few hundred people waiting to try Thymer by the time we got a first version going then we’ll be able to get a ton of valuable feedback that will help us figure out if we’re on the right track.

The bottom line is, the more answers you get to the questions above the more you “de-risk” your startup. Some answers you can get by googling. Other answers you get by talking to prospective users/customers. At some point, though, if you’ve thought hard about your idea and if you haven’t found any critical flaws, just go ahead and commit. Take the time to build a solid prototype and discover if the app works as well in reality as it did in your head.

Randomness is a perspective

Diederik • @jdvhouten • • Part of Day 40

For many founders the whole process of building a product and bringing it to market is stressful and anxiety-inducing. Founders want confirmation that they’re on the right track. But is this a reasonable thing to want? And does it matter?

Startups are an unusual environment where you’ll succeed if you focus on the things that truly matter at the expense of everything else. Not many things in life are like that. In college, for instance, you can “cheat” by picking easy courses or by studying for the test instead of for comprehension. College is artificial and all the metrics can easily be gamed.

It’s not just college that’s artificial. Consider these occupations that never leave the make-believe world:

  • Tax advisors game metrics for a living. Government makes the tax laws and the tax industry figures out ways to skirt them. It’s an endless game of cat-and-mouse that provides no particular value to society. It’s basically a boring competitive video game where you get points according to some byzantine rule system and also the points are money.
  • Many, but not all, charities fall into this artificial category. They raise funds for causes that deserve our attention, but then what happens? Being good at fundraising and being good at making the world better are two entirely different skillsets. It shouldn’t surprise anybody that when charities optimize for their ability to tug on our heartstrings they get good at that, and only that. Charities don’t need to be good at improving outcomes in order to be considered successful, they just need to look like charities and play the part.
  • An employee in IT that works hard to keep everything running smoothly might get no credit because all the value provided is invisible. Another employee that works weekends to put out fires of his own creation gets lauded for his work ethic. You’d think that businesses, because they want to make money, actually get really good at making something people want to pay for. But you’d be wrong because a businesses isn’t a person and it doesn’t have a singular will. It’s an organization first and foremost, made up from individuals who have personal goals often at odds with the goals of the business. An established business can survive on momentum for decades, churning out low quality products with poor customer service.

Disconnects like these are the default everywhere, not the exception. I’m not making a point here about misaligned incentives. The point is that these artificial environments provide simple benchmarks for success for the people in them. At work and at school your boss/teacher will tell you what to do and whether you’re doing a good job. You might still be working on pointless projects (see Graeber’s bullshit jobs), but it doesn’t really matter, because you don’t get judged on efficacy.

From this perspective it makes total sense that people feel scared and anxious when they start a startup. They need to transition from an artificial environment to one where they have to sink or swim. The complete freedom and apparent randomness of startup life feels scary and unstructured. Nobody tells you what to do or if you’re doing a good job, and the outside feedback you get feels totally random.

Some weeks you’ll get many thank you messages. Other weeks people say mean things about you or your startup on twitter. You’ll never know what triggered the positive and negative reactions, but you can easily lose your mind trying to figure it out. We assume these fluctuations are just noise, unless we see clear contrary evidence. You want to focus on the things that matter at the expense of everything else, which means you can’t let outside forces hijack your day.

If you work every day to make your app better and if you work every day to get the word out there, what do you have to be anxious about? Traffic goes up, traffic goes down. Customers come, customers go. Competitors come and competitors go. Much of this is completely beyond your control. The process is what matters and the process is extremely simple:

Work on the product in chunks until you believe it’s good. First make the most basic solution that works. Then refine it until it’s good enough to ship. Start with the hard parts and work your way back to the easy stuff (registration, login, documentation, etc.). Work on marketing with an “all of the above strategy”. Finetuning is for later, after you have established your startup is viable.

What is under your control is the work you do, and there is nothing random about it. Luck still plays a major factor of course. You can’t know in advance how many people want the thing you’re making. Sometimes you swing and you miss. And if we keep swinging we’ll hit a homerun sooner or later. But hopefully sooner :).

PS: It’s day 40. We’re half way through! And we have 20% of an app! Oh boy.

Calculating SaaS pricing in reverse

Diederik • @jdvhouten • • Part of Day 38

Instead of asking what kind of revenue you can expect with your product, we like to turn the question around. Start with the kind of revenue you need to consider the product viable, and then work backwards.

These are the kind of exercises we do when noodling with possible pricing models for Thymer.com. 

Let’s keep it simple and pick a round number: one million in annual recurring revenue. What does it take to get to that revenue with different business models?

Option 1: free with pro upgrade

With consumer software you want people to try your product before asking for money, and the easiest way to accomplish that is by offering a generous free version. We can then sell a Pro upgrade for $50 (one-time fee).

1 million divided by $50 = 20.000 customers per year. 20 thousand. That sounds like a lot.

But not every user buys your software.

Maybe 10% of trial users do.

Trial users = customers * 10 = 200.000 trial users.

How many website visits do you need for that? Maybe another 5x, if the signup process is good.

Unique visits = trial users * 5 = 1 million visitors.

One million unique visitors per year, or 20k per week. Not an impossibly high amount, but it’s not peanuts either. Where is all that traffic going to come from?

Ads? That’s hard to pull off when you’re bootstrapping. Funded competitors will outbid you the moment you get traction. Besides, when you already have the expense of many free users you can’t afford to throw money at opaque ad systems.

Organic search traffic? Maybe, but it takes a long time to build traffic like that with articles and other content.

Twitter? Interviews? Youtube? Magazines?

You’ll need a massive marketing effort to get that kind of traffic going. And you need to do this every year, just to keep your revenue from collapsing. Let’s see if we have better options.

Option 2: prosumer subscription

Okay, let’s see what happens if instead of charging $50 once we use a subscription model.

The big challenge for consumer subscriptions is churn. Also, repeatedly charging a credit card will lead to a much higher customer support burden, plus you only get maybe 24 months of revenue on average from each customer. Maybe your product will have best in class net retention, but 5% monthly churn is typical for consumer subscriptions and at that rate you lose about half your customers every year.

That means you can charge $50 once or $25 per year for two years. The rest of the math will be the same as in scenario #1 which shows that subscriptions are no panacea. You still need a ton of traffic to get new trial users to get new customers to replace the ones that have churned.

Option 3: subscription for teams

The problem with scenario #1 is that you need a ton of traffic and awareness that we don’t know we’ll be able to get. The problem with scenario #2 is you add a bunch of complexity (subscriptions, renewals, cancelations, unpaid/overdue subs) but the LTV (lifetime value: the total amount paid by a customer for the duration of the subscription) of a customer won’t be much higher because of churn. The entire point of running a subscription service is that your customer base can accumulate over the years, but you won’t be able to do that when your customers leave in droves to chase the next cool thing. The harder you work at growing your business the more relentless churn will be in dragging you back down. 

We can kill two birds with one stone by introducing a subscription for teams: churn and volume. Team subscriptions are automatically more sticky than individual licenses because moving everybody to a different product is a hassle. This means dramatically lower churn. If your product is for small businesses or teams within larger businesses your churn will be maybe 2% per month (= 20% annually). And when you sell to a team you’re selling multiple licenses at once. That’s much faster than selling individual licenses!

If you charge $50 per month for a Teams subscription with 2% churn your LTV is 20x-30x that of individual prosumer subscriptions. That’s a huge difference!

Now, you only need find 400 customers for 1m in annual revenue. That’s 2 new customers a day and you’re golden. That sounds way easier than the 20.000 customers demanded by scenario 1.

You might have noticed that 400 customers @ 50 a month is only 200k and not 1m. That’s true, but you only need to compensate for churn, here. If you have 2000 customers at 50/mo and you lose 20% of those each year, then you need to find only 400 new customers to retain revenue. Any sales beyond that will grow your business. The math checks out!

—

Unless Thymer becomes a smash hit, our revenue will have to come from team subscriptions. Right now, I don’t see any other way. The bright side is that we should be able to keep the single user version of Thymer free or cheap because it won’t bring in much revenue anyway. The downside is that we might end up serving two different audiences, and that features that make sense for a Thymer Team make no sense for a single user version of Thymer (and the other way around). I’m sure this will turn into a careful balancing act. 

Some database design strategies

Diederik • @jdvhouten • • Part of Day 36

Friday Wim wrote about Data and Event Handling on the client side of Thymer.com. Today is about our approach to data storage on the server side. All storage, with very few exceptions, will go directly into a SQL database. MySQL is what we know best, so that’s what we’ll use.

Almost all of our database tables will have at least these columns:

  • “id” (unique auto increment primary key),
  • “guid” (unique, user facing),
  • “group-id” (integer foreign key),
  • “created-at” (datetime),
  • “updated-at” (datetime),
  • “deleted-at” (datetime, nullable),
  • “destroyed-at” (datetime, nullable),
  • “extra_json” (plain text)

I’ll explain briefly what our reasoning is for having these columns.

GUID with unique index + auto-increment int primary key

Having auto-increment primary keys for internal use is convenient, but exposing these keys to the outside world isn’t great. It’s better to use GUIDs in the (JSON) API. Integer primary keys are considerably faster, though, so you can use use both. That’s what we’re doing. A side benefit is that you can use the GUIDs for backup and export/import purposes. You can’t easily import data with auto-increment keys, because you’ll get key collisions, but you don’t have this difficulty when you use GUIDs. They don’t collide by design. If you ever have to shard your service, or want to move data between SQL servers for another reason having GUIDs to work with will save you a great deal of time.

It also makes for much friendlier APIs. You can prefix the GUIDs with the class table it corresponds to. If the wrong kind of GUID is passed to the API, for instance “user_56b2c555-1ccf-4b9a” instead of “project_854a3923-0be9-4106” you can generate a friendly error message that the API function expected a Project instance instead of a User. If you just pass a generic integer you have to return a confusing error message about insufficient permissions because you can’t determine what the API users is trying to do.

Redundant group-id columns

Suppose your product allows groups of users to collaborate and you don’t want to accidently mix up information between groups. You don’t want to put yourself in a situation where users see other people’s data just because you forget a “WHERE” clause in an SQL statement somewhere. Mistakes are unavoidable, but some mistakes are unacceptable and we plan to design our system so the worst kind of mistakes can’t happen. If all tables have a group column (or equivalent) then you can enforce through your ORM that all queries are automatically filtered for access. You can make all database queries go through a single point that ensures a user of Group A can only see data from Group A.

A side benefit of storing redundant group information is that it makes for very easy clean up when people want to close their account. You just go through all database tables and you drop all records for that user/group. If you have to infer which data belongs to which user through foreign keys you can create a difficult situation for yourself. Especially when you need to delete a lot of data you want to be able to do so incrementally. You don’t want your service to become slow (or crash altogether) because a background task locks your database tables.

Multiple datetime columns

Created-at and updated-at speak for themselves. These are timestamps you want to display to the user in some fashion, and they’re too useful not to have. Deleted-at is for soft deletions (tombstones). Sometimes deletion is expensive and you want to just flag many items as deleted first and clean up asynchronously. When the data has actually been destroyed you can use ‘destroyed-at‘. You could delete the actual row from the database, but foreign keys could prevent that. In addition, if you want to show good error messages to users (“this item has been deleted”) it’s helps to keep the row around. Of course, you only need the metadata for this. The columns that contain actual user data you can overwrite when the user wants the data deleted.

If you provide some kind of full text search it helps to have an indexed-at column. You can use this to trigger re-indexing. You can use datetime columns like this for any kind of asynchronous update you want to run. Just select all rows where ‘indexed_at < updated_at‘ and do the necessary processing. Want to re-index a specific group? Just backdate the updated_at column and a background daemon will jump into action. Want to re-index 10 rows that didn’t get processed correctly the first time? It’s trivial. This is much less fragile than having some kind of message queue where you enqueue rows for additional work. A message queue can restart or get out of sync with the main SQL database really easily, but a simple datetime column works every time.

Append-only database tables

Dealing with immutable (meaning: unchanging) state is easy, and gives you version information for free. Being able to show users how their data has changed over time is a useful feature to add and it doesn’t cost anything. It’s just a matter of selecting the most recent row to get the most current data.

Storage is cheap and database engines can deal with large tables just fine. You can always fix performance problems when (and if!) they happen.

A surprise benefit is that caches automatically get invalidated if you use append-only strategies. You can’t have stale cashes for data that doesn’t exist yet. That’s a nice thing to get for free. (Assuming your app is read-heavy. For write-heavy apps this might actually be a disadvantage, because your cashes get invalidated prematurely.)

Multiple queries instead instead of complex queries

When you use an ORM to build your database queries you’ll get pretty naive queries by default. The ORM will happily select much more data than needed, or do thousands of small queries when one will do (see: N+1 problem). When you save objects most ORMs save the entire object back into the database, and not just the single property you’ve changed. ORMs aren’t sophisticated tools by any stretch of the imagination.

ORMs are really great at generating the most basic SQL queries, and we’ve learned that if you can rework your problem as a sequence of very basic queries you don’t end up fighting your ORM. I’ll give an example.

Suppose you want to generate a list of the 20 most popular books written by any of the top 3 most most popular authors. In Django ORM pseudocode:

popular_authors_ids = Books.sort(most_popular).limit(3).author_ids_only()
books_by_popular_authors = Books.filter(author__id__in=popular_authors_ids).sort(most_popular).limit(20)

The example is a bit contrived, but I hope it illustrates the point. By first selecting a bunch of ids that you need for the second query you end up with two very simple (and fast) database queries. Modern SQL databases are extremely powerful, and you can accomplish almost anything in a single query. But by splitting the work up in multiple queries you get three main benefits:

1) you know your ORM is going to generate reasonable SQL
2) you often get extra opportunities for caching the intermediate results
3) you can do additional filtering in Python, Ruby, NodeJS before you make another round-trip to the database to get your final results. This is often faster and easier than doing the same in SQL. Unicode handling in your database and in your target language are always going to subtly different, same with regular expressions. You’re better off using the database as dumb storage.

Extra JSON column

There are plenty of situations where you want to tag a couple of rows for whatever reason. Maybe you’re experimenting with a new feature. Maybe you need temporary storage for a migration. Maybe you need some logging for a hard to replicate bug. Being able to quickly serialize some metadata without having to add an extra column to a database is a big time-saver. You don’t even need to use the JSON serialization functionality in your database for this to be useful. Just being able to store a BLOB of data is where the real value is at.

Insecure defaults considered harmful

Diederik • @jdvhouten • • Part of Day 34

Anything that involves input or output should not just be considered unsafe but actively hostile, much like the critters in Australia. The programming languages and libraries you have to use are not designed with security in mind. This means you have to be totally paranoid about everything.

Let’s go over a few places where you have to deal with insecure defaults.

Zip archives

Suppose you add some backup feature to your app where users can download their files as a .zip, or maybe you let users upload a theme as a zip file. What could possibly go wrong?

Let’s start with the zip-slip vulnerability, allowing attackers to write anywhere on the system or remotely execute programs by creating specially crafted zip files with filenames like "../../evil.sh". This kind of attack made a big splash on the internet a couple of years ago. Many archive libraries were affected, and with those many libraries probably thousands of websites.

Most programmers will just use a zip library and not think hard about all the ways it can blow up in their face. That’s why libraries should have safe defaults. Relative paths should not be allowed by default. Funky filenames should not be allowed (e.g. files with characters in them, like backslashes, that are forbidden on other platforms). Because the libraries don’t do these checks for you, it’s up to you to reject everything that looks sus. Use of unicode should be highly restricted as well by default, more about that in a bit.

Zip exploits have happened before of course. Take Zip bombs for instance. Zip bombs are small files when zipped but get huge when decompressed. Zip bombs are at least 20 years old, and yet I don’t know of a single zip library for any programming language that forces the programmer to even think about the possibility that unzipping a small file can fill up all disk space on their server thereby crash the whole thing.

It’s pretty strange, when you think about it. In most cases the programmer knows, within an order of magnitude, what a reasonable unzip size is. It might be 100mb, it might be a gigabyte or more. Why not force the programmer to specify what the maximum unzip size should be?

When your unzip library doesn’t enforce limits you have to get creative. You can unzip to a separate unzip partition that is small. That way any unzip bombs will detonate harmlessly. But really, is it reasonable to go through this trouble?

It’s not just about disk space. You want limits for all system resources. How can you limit how much memory can be allocated during the unzip process? How can you limit how much wall time you’re willing to allocate? You can also use ulimit or a whole virtual machine, but that introduces a great deal of extra complexity and complexity is another source of bugs and security vulnerabilities.

Unicode

Unicode is the default for anything, and in the coming years we are going to see many creative unicode exploits. In the zip example above all filenames and file paths are unicode that can contain, among many other things, funky zero-width characters:

Unicode characters can trip you up in many ways. Suppose you have a webapp where people log in with a username. What could go wrong when you allow zero-width spaces inside usernames? It can go very wrong when you strip whitespace inconsistently.

For example, during registration you only strip ascii whitespace (space, tab, newline, etc) when checking if a user with that username already exists, but you strip all unicode whitespace when saving the user to the database. An attacker can exploit this collision by registering a new user with zero-width space added to the username of the victim. Two users rows will be returned by a login query like this:

SELECT * FROM users WHERE username = 'bobbytables' AND pw_hash = 123

And databases typically return the oldest row first if no sort order is given, meaning the attacker has just logged on as the victim using his own password.

Layered paranoia helps here. First select the user row based on the username. If two rows are returned, bail. Only then validate whether that row matches the given password. You also want to use database uniqueness constraints so you can never end up with two rows in your user table with the same username.

XML – SVG

XML libraries support External Entities. Basically, you can upload an innocent looking XML file, and when parsed it includes a different file, like /etc/password, it can even allow full remote code execution.

A famous example here is ImageMagick, a popular graphics library used to create thumbnails for images. Upload a malicious image, BOOM, remote code execution (ImageTragick). This vulnerability existed for many years. ImageMagick was never intended to be used to process untrusted images passed through via web services. It’s just a product of a different era.

Any time you deal with XML files (or XML adjacent formats) you have to specifically check if the file format supports remote includes, and how the library deals with it. Even if remote includes just involve HTTP requests, and not access to your file system, you might still be in trouble. If you download the contents of a URL on behalf of a user the HTTP request is coming from inside your network. That means it’s behind your firewall, and if it’s a localhost request, it might be used to connect to internal diagnostics tools.

Maybe your http server runs a status package, like Apache Server Status. This page lists the most recent access log entries, and is accessible by default only from localhost. If a localhost access rule was your only layer of defense you’re now in trouble. Your access logs can contain sensitive info like single-use password-reset tokens.

User uploads malicious SVG file -> ImageMagick resolves External Entity and fetches Access Log via HTTP -> Renders to PNG and displays to user as thumbnail.

It’s hard to predict in advance how innocent features can be combined into critical security failures. Layers of defense help here. Limit what kind of image files can be uploaded. Google for best security practices for the libraries you use. Most foot-guns are well known years before the big exploits hit the mainstream.

Regular expressions

Regular expressions are great, but it’s still too easy to introduce serious denial of service vulnerabilities in your code by being slightly careless. We’ve been burned by this a few times. A simple regular expression that caused no trouble for years suddenly eats gigabytes of memory. Because of the memory pressure the linux OOM killer decides to kill memcached or the SQL server and then everything grinds to a halt.

What kind of regular expression can cause that kind of mayhem? Easy, one that looks like this: (a|aa)*c

A regular expression by default tries to find the longest match. This can result in exponential backtracking. For more reading see ReDoS on wikipedia. If you make it a habit to look for shortest matches, using *? instead of * you’re much less likely to write an exploitable regular expression. If you also validate input length and keep your regular expressions short and simple you should be fine.

Regular expressions are incredibly useful. Still, regular expression engines would be way more useful if you could give them a time and memory budget. If a regular expression is intended to take a few milliseconds I want an exception thrown when it takes 2 seconds instead. If a regular expression starts allocating memory I want to know about it, before it turns into a big problem.

Working towards good defaults

Take this gem of a comment from the golang zip library:

Instead of validating input and providing a safe API by default the library just pushes the responsibility onto the user (the programmer in this case) who is likely to be either too inexperienced or too pressured by deadlines to take all the necessary precautions.

The point here isn’t to suggest Go is bad, but that times have changed and most software written today has to survive in a hostile environment. The only way forward is to move to secure defaults everywhere, with flags for unsafe mode for those cases where you trust the input. It will take long time before we get there, and until then you need many layers of security.

Text-based user interfaces in 2022

Diederik • @jdvhouten • • Part of Day 32

Although we haven’t done much marketing for Thymer.com yet, we are getting a steady trickle of new email signups for the private beta. It doesn’t prove that we’re on the right track with Thymer, but it suggests that that there are at least some people who want to try an app that looks something like this:

The big trend in web design in the last decade has been towards simplicity: bigger fonts, more whitespace, more visuals, mobile first, and fewer buttons. With Thymer we are heading in the opposite direction. Thymer is keyboard focused, information dense, programmable, and feature rich.

I get why the trend is towards the simplest possible apps. Simple apps are easy to make and easy to support. And the world is huge so there is always demand for apps that get the basics right. Apps like these can look brilliant during the first 30 minutes, and then you run into the first limitation. Soon after into the second one. Then you wish that you could see more of your stuff, but you can’t. These apps are made primarily for mobile devices and on a big screen you just get more whitespace. You want to do some bulk actions, but that’s unwieldy or impossible. Even productivity apps have followed the trend where usability is sacrificed for visual appeal. It makes perfect sense, though, because it’s screenshots that sell your app, and dense apps look intimidating.

We’re taking a big risk by running counter to the trend. We know that there are many programmers who love apps that look like Thymer. The runaway successes of Sublime Text and VSCode are proof of that. Our hope, still unsubstantiated, is that there is a much larger group of people out there — not just programmers — who want a productivity/planning app that’s designed like an IDE. People who aren’t familiar yet with something like a Command Palette, but who are willing to try something new.

Design trends change. First skeuomorphism was cool, then flat design was the only game in town. Maybe, in the future people will demand apps that are simple on the surface but not at the expense of functionality. A new trend towards text-first interfaces for applications that are mainly about text, perhaps.

We’re deep into code now. A good deal of plumbing left to finish before we have something that resembles a buggy prototype. Then we finally have something substantial to show people and then we can ask people on the beta email list and here on 80daystartup what they think and what kind of features they would like to see. For me this is one of the most exciting parts of building a new product. Ideas that started as sketches on napkins and pizza boxes slowly start to take shape.

Single miracle startup

Diederik • @jdvhouten • • Part of Day 30

A common piece of startup wisdom is that you should go for a “single miracle startup”. If your startup requires no miracle at all to happen you’re probably aiming too low. If your startup requires multiple miracles to happen then failure is almost certain. A single miracle startup hits the sweet spot.

Why is it a bad idea to start a zero miracle startup? Suppose you decide to clone a simple but moderately successful B2B web product. This doesn’t require any miracles to happen. If the product is basic then cloning it shouldn’t even be difficult. Then it’s just a matter of finding customers and improving your product. As your product gets better you’ll get more customers, and as your marketing gets better you’ll get more leads. You can grind it out this way. It takes a long time, but you’ll get there eventually. A market that has a number of profitable businesses can always support one more, after all.

A big problem is that if your product and your business model are entirely derivative that any kind of runaway success is unlikely. It’s hard to get good press if your product is uninspired. Word of mouth can be one of your strongest assets, but the enthusiastic early adopters have already tried the original. They’ll pass on your clone. Those are pretty serious headwinds.

The odds that your clone will be more popular and more successful than the original are not good. Which means that your success has a ceiling. A pretty low one. You could clone a very successful product, but that’s way harder. If your product is just like Slack, but substantially worse and unfinished, who in their right mind will buy it?

You want to make your product different and therefore better at least in one dimension. You can make your product way faster, and distinguish yourself that way. Or you can make your product super easy to use. Or very pretty. Or integrate heavily with a popular platform and then your product will be the best X for users on platform Y. Any of these are better than straight-up cloning a product, but it’s still not optimal. You want to be so different that there’s a pretty good chance your product flops completely.

If you take some big product risks and it turns out there is an audience for it, then your product could create a new niche overnight. Your product will be the only one that meets a formerly unmet need, and that’s a big tailwind. This requires a miracle, but only a small one. It means that you don’t have to fight as hard to win unhappy customers from competitors. You’ll still need to do marketing, of course. People have to know your product exists.

We live in a huge and diverse world. Even wacky products end up with many devoted fans. By taking a seemingly riskier route, by making a product that is a little bit “out there”, you actually get better odds of success.

Typescript without Typescript

Diederik • @jdvhouten • • Part of Day 28

In some ways, Typescript is pretty awesome:

  • Typescript makes Javascript more strongly typed, which means you get to catch silly typos early. This is great.
  • Because of this typing information you can enjoy IDE advantages like autocompletion of object members, jumping to type definitions, or to function call sites. Nifty.
  • You can easily find unused code, and delete it confidently.
  • Refactoring code becomes way easier. Renaming identifiers is trivial, and when you really start to move code around Typescript is a real lifesaver. Those red squiggles turn an hour of tedious and maximum concentration refactoring work into a breezy 10 minute editing session.
  • Type inference means you don’t have to tediously define the type for every variable.

But Typescript is not without downsides:

  • You have to wait for compilation to finish every time you make a trivial change. Annoying on small projects, really annoying for bigger projects.
  • You get bloated Javascript code along with source maps for debugging. Extra layers of complexity that make debugging harder. If you want to debug an exception trace on a release server you now have to debug an ugly Typescript call stack.
  • Typescript includes Classes and Inheritance and other functionality that overlaps with modern ES6 Javascript. The additional features Typescript enables, like Class Interfaces and Enums, aren’t that compelling. The days of IE6 ECMAScript are long behind us. Modern Javascript is everywhere it’s a pretty good language that doesn’t need Typescript extensions.

What if you could get almost all the benefits of Typescript without Typescript? Turns out that you can. The trick? You just enable typescript mode in VSCode for plain ES6 Javascript projects. I just assumed that Typescript would only do typescript-y things for Typescript projects, but that’s not the case!

The trick is to add a jsconfig.json file to your project, that looks something like the one below. The “checkJS” property is the magic one that gets you 90% of Typescript in your ES6 projects. In my case, I’ve also added some extra type libraries like “ES2021” and “ESNext” (I’m not sure you actually need to declare both). Turns out Typescript otherwise doesn’t know functions like Array.at that are already pretty widely supported in browsers.

My jsconfig.json

{
    "compilerOptions": {
        "target": "ES6",
        "checkJs": true,
        // Tells TypeScript to read JS files, as
        // normally they are ignored as source files
        "allowJs": true,
        "lib": ["ES6", "DOM", "ES2021", "ESNext"],
        "alwaysStrict": true,
        // Generate d.ts files
        "declaration": true,
    }
}

The benefits of typescript without typescript, visualized:

A few limitations

  • Sometimes you have to nudge typescript into interpreting types correctly with JSDoc type annotations. You can define nullable types /** @type {MaybeMyClass?} */, type unions, for example if a variable is either a bool or a string: /** @type {boolean|string} */. Usually VSCode will suggest the correct annotation via call site type inference.
  • No template types or other advanced stuff. I don’t think they’re worth much anyway.
  • When Typescript gets in your way you can just ignore it, so really, you get almost all the upside with none of the downside.

In conclusion

Maybe none of this is news to you. In that case, sorry to disappoint. However, if you have never experienced Typescript error checking, refactoring, and navigation tooling in an ordinary and plain Javascript project you’ve got to try it. It doesn’t happen often that a piece of tech turns out to be way better than expected, but this is one of those cases. Typescript without Typescript is amazing and I’m sticking with it.