“ Agile doesn’t work for data science. Why? ”. Unlocking Agile for Data Science: A New Paradigm
It’s a common narrative in corporate circles: “Agile doesn’t work for data science.” Why?
Traditional software development managers often find themselves at loggerheads with data science teams, who frequently round off their sprint cycles with research and experimental results, yet no tangible software to showcase.
These leaders, conditioned to have clear-cut expectations, grow impatient and seek time estimates, only to hit the data science conundrum — uncertainty.
The data science problem space is often nebulous, and the solution landscape is vastly more expansive than standard software engineering, leading to no guaranteed outcomes.
When the demand for timeframes ensues, it triggers fear and frustration within the data science team, often pushing them to overestimate and retreat into their work silos. This dynamic breeds mistrust, exacerbating the problem further.
Another pitfall that organizations often stumble into is the inability or reluctance to commit to and prioritize potential projects.
With a shifting scope and changing priorities, data scientists are left questioning the relevance of their work, pushing them to adopt short-sighted development practices. The dream of building enduring products often dwindles into a reality of transient dashboards and fleeting notebook projects, leading to job dissatisfaction and high turnover rates.
In an attempt to manage the uncertainty, organizations may resort to timeboxing explorations. However, these arbitrary deadlines are rarely met and often stretch until budget constraints or executive impatience kicks in.
When the dust settles, data scientists realize they were never indispensable, they were simply ‘nice to have’. The fallout of these mismatches? Wasted careers and millions of dollars down the drain.
Despite these challenges, there’s a glimmer of hope on the horizon: Agile CAN work for data science, with a few strategic tweaks.
- Size your opportunities: Before diving headfirst into research, invite your data scientists and stakeholders to evaluate and size up potential projects. Aim for a wide range of opportunities, and quantify their return, whether in terms of revenue, cost reduction, time savings, or customer satisfaction. A ballpark figure works just fine.
- Anticipate Future Failures: Take timeboxing up a notch by introducing ‘failure projections.’ Ask your team hard-hitting questions. “If all we have after six months are failed experiments, will the opportunity still be significant enough?” This approach can help filter out projects that may not be worth the investment of time and resources in the long run.
- Track Research as a Graph: The reality of data science research is that one question often leads to several more, rendering traditional backlogs ineffective. Instead, visualize your research paths as a graph, keeping track of current insights and lingering questions. Regularly communicate these learnings and the graph to your stakeholders, thereby fostering transparency.
- Deploy with Agile: Once your research concludes, shift back to the conventional Agile framework for deployment. With a firm foundation in data and software projects, MLOps and Engineering can take the reins, driving the project to its finish line.
By embracing these practices, organizations can successfully adapt Agile methodologies for their data science needs, fostering a culture of transparency and collaboration.
