A model is at its best just before being deployed to production. Same with your new model.įrom the moment you deploy your model to production, it begins to degrade in terms of performance. The value of a new car goes down 10% the moment they’re driven out of a dealership-the moment they’re “deployed” to the real world. They’re dynamic and sensitive to real changes in the real world.īut models are kind of like cars. Modified by the author and adapted from the source.Īs you can see from the image above, machine learning models degrade over time. But now, who’s taking care of what you deployed? The software might have a DevOps team that monitors and maintains the system in production but with machine learning applications, you can’t just hand your deployed models off to the Ops team-in your case, it has to be a shared responsibility. But now, enter the machine learning world:ĭeploying your model was likely a hassle in itself. Your application scales to a lot of users, and it works as intended and solves the problem it was built to solve. In terms of monitoring your application, you might be worried about system metrics, error rates, traffic volume, app loading times, infrastructure (server counts, load, CPU/GPU usage), things like that. In fact, your team may decide on a steady and periodic release of new versions as you mostly upgrade to meet new system requirements or new business needs. Based on the software development lifecycle, it should work as expected because you have rigorously tested it and deployed it. Great, right? You almost don’t have to worry about anything. Take a look at your traditional software application after deployment: Congratulations … But it does not stop at deployment Challenges and best practices for monitoring your models in production,īy the end of this article, you should know exactly what to do after deploying your model, including how to monitor your models in production, how to spot problems, how to troubleshoot, and how to approach the “life” of your model beyond monitoring.Different monitoring and observability platforms and how to choose them,.What you should monitor in production and how to monitor it,.Why you need to also own and monitor models in production,.Why deployment is not the final step for you,.I’m still trying to recover from the bruises that my boss left on me, and the least I can do is help you not end up in a hospital bed after “successful model deployment”, like me. I’m not going to bore you with the cliché reasons why the typical way of deploying working software just doesn’t cut it with machine learning applications. Well, not quite, which we got to realize in a relatively dramatic fashion. Our model was serving requests in real-time and returning results in batches-good stuff! Surely that was enough, right? Right? “Congratulations!” I told myself and my colleague, “Our hard work has surely paid off, hasn’t it?”. A few months ago, we finally deployed to production after months of perfecting our model.
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |