No matter how ambitious or moderate your plans for your Machine Learning product, you need to make sure that what you’re building works under the most minimally basic conditions for your application. This first prototyping step is the product’s proof-of-concept (PoC), and when executed thoughtfully and thoroughly, it paves the way for successful productization and minimizes unanticipated headaches.
Think of your PoC as the first step towards the Minimally Viable Product, your initial move from the whiteboard to the real world to ensure that the core components that might go into your MVP are even feasible. While your MVP might have a user-facing front end or be integrated within your larger application, your PoC could involve a standalone component no more complex than a command line interface (CLI). And PoCs can entail different steps in different contexts. If you plan on deploying a product using a novel deep learning algorithm you’ve implemented, your PoC could be a test to make sure that the algorithm produces acceptable results on simulated data. If you’re using a well known and validated approach, your PoC could apply the algorithm to a small subset of real-world examples specific to your product to make sure they work in their intended context.
However it might manifest, a PoC is above all a demonstration that the foundational ML elements to a solution can in fact be built and deployed in the way your intended product demands — and in the process, maps how to get there. By the time you’ve completed your PoC, how to build what you want should be totally unambiguous.
Before you jump into building your PoC, here are a few considerations you should reflect on to ensure that your project is as successful as it can be.
Understand why you're building
As engineers, it’s tempting to begin tinkering with abandon, guided by an impulse to innovate first and justify later. But it’s important to always keep in mind that your PoC isn’t there to exhibit a vague theoretical concept to satiate curiosity for its own sake. Rather, it’s one of the first of many investments of time, money and manpower towards a real thing used by real people.
In this sense, building a PoC in enterprise starkly contrasts with the purely conceptual research that comes out of labs and universities. Instead, organizations have to take a deeply human-centered approach from the beginning when designing ML. The illegible Jupyter Notebook that produced a state-of-the-art algorithm to increase a metric by 0.05 over the baseline usually should be avoided.
By never losing sight of why you’re building, you’ll avoid turning your PoC into a hobby horse that, while perhaps impressive, quickly hits a dead end.
Start simple
Once you turn from pure ideation (where things always seem to fit together so neatly) to the first lines of code, it’s tempting to feel slightly discouraged when confronting the distance between your PoC and your vision for the fully deployed product.
However, as you build out your PoC, keep in mind its goal is to demonstrate the feasibility and viability of your product or solution. Maybe your initial algorithm’s accuracy isn’t as high as it needs to be for production, or it runs more slowly than you’d like. Between your PoC and the end product, there's substantial effort in optimization, re-training, responding to edge cases, and more – all of which will substantially improve the product.
We’ve run this distance many times. One client wanted to implement a computer vision solution that could identify objects in 3D using a multi-channel video setup on vehicles in real time. Rather than focus on building the production solution, we started with a basic PoC that used historical data run offline at a few frames per second. What mattered in this scenario wasn’t polish and perfection, but rather to test and explore how the solution’s various algorithmic components could integrate in a coherent way. After proving out the PoC, we continued to build out the MVP and optimized our models for edge deployment which allowed us to achieve real-time performance.
In another recent project, we worked with a gaming client that wanted to build out a marketing engine to automatically send offers to players to effectively engage them at the right time. In that case, the PoC utilized a small subset of players in a real-world environment run in an A/B-test setting that allowed us to determine whether our approach would work. It did, and the PoC was ultimately fully productized and deployed. Starting small allowed us to prove out the concept, minimize risk, and lay the groundwork for successful productionization.
Determine requirements and constraints early
Both before and during the PoC development process, it’s critically important to keep in mind the business and user requirements. Some of those you’ll foresee before you even begin, and some of them will reveal themselves as the PoC is built.
What data will you need, and is there sufficient historical data collected? How will you access these data in the real application? What are the budgetary constraints you’re working within? Do you know how fast the application will need to run? Who are the end users, and how will this improve their experience? What timelines and constraints are you working with?
On a somewhat optimistically cautionary note, although you should be realistically estimating what you’ll need, your worst case scenario often isn’t the most accurate. The common misconception that companies need to be the size of Amazon to utilize Machine Learning is just that: a misconception. A great deal can be achieved in low data and resource environments. Even cost-wise, your early expectations may be grimmer than reality. With one client, we were able to reduce cloud costs, which had reached over $1 million per year, in half with optimization of their models and inference pipelines.
Finally, and more intangibly, make sure you’re aware of any organizational and human constraints. Do you have buy-in from key stakeholders in the project? Is this aligned with the organization’s overarching purpose? These sorts of considerations might not be as easily quantified, and discovering them certainly requires different methods than cost rundowns, but they’re no less important.
Determine who will build it
Once you’ve settled on building your PoC, you have to figure out who is going to actually do the building. Having the proper resources in place to execute an idea can make the difference between years of toil with no results and deploying v2 based on user feedback in a few months. Personnel-wise, there are four potential paths to building a PoC:
Buy off the shelf
This usually involves taking an existing SaaS platform and implementing it to your uses. If an off-the-shelf solution turns out to be close to perfect for you without much tweaking, that’s great. But many of our clients have turned to us after trying and failing to make an existing solution bend to their wills. Off-the-shelf options are inherently not customizable, often pose integration difficulties, and can be acquired by larger companies that render them unusable for you, leaving you scrambling.
Hire for internal roles
While it’s tempting to hire full-time staff, building a team from the ground up is often incredibly challenging. Additionally, you may discover in the course of developing your PoC that your original product is less viable than you’d originally imagine, either demanding more personnel to complete or rendering it infeasible altogether and leaving you with newly onboarded data scientists with nothing to do, or having to rapidly re-downsize after making fresh hires, which can be demoralizing within the organization.
Hire an independent contractor
If you’re building a PoC with an existing team in place and you need to augment it with one or two additional seats, using independent contractors can make a lot of sense. But relying on a lone contractor to build out your entire PoC is usually too tall a task for any one individual, especially if they lack deep ML experience. And most contractors don’t have the range of expertise to not only build the algorithmic components, integrate the components with an existing service, perform data engineering, and so on, all of which are best performed by various individuals capable of doing each.
Hire a team
Bringing on an independent team strikes the right balance between all the other three options. By relying on an outside team, organizations get the contractual flexibility of working with a freelancer, the diverse expertise of a cohesive group, and the turnkey convenience associated with off-the-shelf options. We’ve seen firsthand how companies that struggled with any of those other options have watched the skies clear when bringing on external teams.
As Strong, we’ve built PoC applications that incorporate computer vision, deep learning, natural language processing, reinforcement learning, and other state-of-the-art approaches for Fortune 500s, seed-stage startups, and everyone in between. Along the way, we’ve developed an acute sense of what it takes to build an ML PoC that can successfully pave the way for a full-fledged data product.
Contact us to find out how we can help you take your ideas from the whiteboard to the real world.
