How do I quantify engineering investment?
Operating reporting for your engineering capability drives operating strategy and clarity beyond AI productivity measures
Before the dawn of LLMs. SaaS business growth has slowed, and as a response to this businesses had focused on metrics such as Net Retention Revenue.
Now the market is awash with how AI may make your teams more efficient, it will quickly shift to how are we measuring this, as organisations start to scrutinise the lack of ROI.
Traditionally most businesses with multiple customers will consider measures of productivity and margins with Revenue Per Head of employees. However for RnD intensive businesses or early stage companies this is inherently unfair.
Once your growth has started to level off and you’re limited to exploiting more from what you have your focus will shift and these metrics become very important. How you report on productivity and engineering efficiency is key to fair assessments by CEO’s, CFO’s and the boards that they report to.
There are multiple layers to measuring productivity. So how do we operationally approach this?
Reporting on Engineering Efficiency
On the team level, you have various metrics and reporting around team deliverables, code efficiency, support cases and bug tracking but this doesn’t translate well to something meaningful for execs and investors.
My suggestion is to split reporting as follows:
The Executive / Board Level Reporting View
Keep the lights on (KTLO)
The minimum tasks required to maintain the current level of service in the eyes of your customers.
For example:
• Maintaining current security posture
• Maintaining current levels of service uptime
• Service and ticket monitoring & troubleshooting
• Addressing functional defects reported by customers
• Regular or routine internal procedures
• Staying up to date with external dependencies
• Browsers, libraries, platforms, web services, partner changes, hardware, etc.
Then the rest is split into the following with the % benchmarks indicated.
Elective investments
Build new stuff (for customers) - 60%
Improve existing stuff (for customers) - 20%
Increase your productivity (not customer visible) - 20%
Elective investments comprise the following:
New Capabilities
• Adding a new product
• Adding a new feature or sub-feature
• Supporting a new platform or partner application
Quality Improvements
• Customer requested improvements
• Better performance and/or utilization
• Iterations to improve adoption, retention, and quality
• Improved product reliability or security
Internal Productivity
• Better developer tooling
• Testing automation
• Code restructuring and re-architecture
• Work to reduce the size of the KTLO bucket in the future
The ultimate goal would be to set the targets annually, and then implement monitoring of investment breakdowns and associated roadmaps across these categories quarterly. Metrics which are listed out in the assessment need to be aggregated to form reporting groups which input into the categories above. Whilst it won’t be possible to hit these benchmarks in the short term, demonstrating gradual improvements towards this will help support investment cases, and any additional headcount demands. Capturing this data must be phased over time. Metrics outlined below would need to be rolled up into reporting groups and would over time give a granular view of value generation and progress in different areas of the business.
The link below to a longer description and article on the approach for a reference.
Team and Mid-Management Reporting View
Team reporting
Forecast versus actuals - forecast of who is allocated to what across the fixed resource pool you have. How many hours or % is allocated by project, customer, other business or support demands? Tasks should be logged on how many hours were spent(regardless of how effort is forecast) and tracked tasks against projects and team allocation.
Product/customer delivery reporting
Burn down charts - for understanding if work committed to is delivered.
Velocity charts are used to understand the throughput of delivery.
Support Reporting
Numbers of support tickets per customer.
Number of support tickets by case type.
Number of support tickets by product/component area.
Speed of resolution.
This is to understand product, technical debt, and service priorities and whether you are meeting customer SLAs and contractual obligations.
Infrastructure
Bug tracking and resolution - number of issues per customer, product, time to resolution.
DR and failover - time to recovery, speed of resolution - is it within customer SLA’s?
Critical incident recovery and resolution time - how many incidents speed of response and customer impact? Is your business efficient where are the bottlenecks or specific areas of under investment? Are there significant numbers of issues all around one customer?
DORA - Releases
Deployment frequency: How often an organization releases to production.
Change failure rate: How often a change causes a failure in production.
Lead time for changes: How long it takes for committed code to enter production.
Mean time to recovery (MTTR): How long it takes to restore service after an unplanned outage or other incident.
Reliability: A holistic view of a software’s operational performance and stability, including factors like availability, latency, performance, and scalability.
Failed deployment recovery time: How long it takes to recover from a failed deployment.
Whether you meet your customer and contract SLAs is important but there is more to customer and business value than that. Which brings me onto..
Connecting productivity back to customer value
One area I see is constantly under considered is how support cases and DORA metrics and ultimately investment is connected back to customer value. How you might measure customer value, might be annual customer feedback there are various ways to gather this. LTV, is there a higher risk of customer churn if there is a higher level of support cases? Are those support cases coming from just one or two customers? Will investment in specific areas of engineering correlate to better retention? New customer acquisition or higher LTV? Are support cases connected with limited product offerings or clouded by new feature requests?
This story is important as its hard as an exec to argue with data, this story brings meaning for the whole business, context for the engineers on the coalface, context for the business execs trying to drive value and growth.
Operating reporting done properly it can actively inform key decisions on your technical and product roadmaps, and focus areas for investment.
AI may make you deliver faster but you won’t be measuring your operating mechanics very differently. How do you know what impact AI has on productivity if you’re not measuring it anyway?
I often get a lot of push back from early stage teams who don’t understand the value and effort involved of setting all this reporting up. But my consistent experience is without it the pressure from execs and boards is based on assumptions and undue pressure placed inappropriately without this data and insight to hand. Businesses that invest in basic operating reporting make money more efficiently than ones that don’t its as simple as that.