Release criterion for systems based on machine learning




Friday, February 21, 2014

9:58 AM

Release criterion plays a critical role in releasing software. Good release criterion includes setting up the right quality bars that product has to pass before it's deemed ready. Traditional software testing methods focused on various test design approaches on coming up with the right set of test cases that assures some measure of quality.



How does this work in a world where the software is largely machine learned, how does one come up with a good release criterion.? When shipping initial set of models where models are only trained with limited set of real world data it's hard to estimate how well they do once the models are live and taking the real traffic.

Comments

Post a Comment

Popular posts from this blog

math curriculum

I enjoy helping my kids with their math. This interest led me to think about the process of teaching math. What are the goals? What concepts to teach? How to order the topics while teaching? Best ways to teach a topic and related topics? How to balance theory, intuition, and practice? How to verify if the student understood what was being taught? Every match teacher must have these and may have already thought about it. If I can access that research and understand how to teach, then teaching may become easier.  These questions led me to discover and understand the process of curriculum design. In USA common core is the Common Core State Standards establish clear, consistent guidelines for what every student should know and be able to do in math and English language arts from kindergarten through 12th grade.The new standards also provide a way for teachers to measure student progress throughout the school year and ensure that students are on the pathway t...
Read more

Shipping machine learned systems

For the last 3 years I have been working on shipping a large distributed system where many subsystems are powered by machine learned models. Before this I used to work in building cloud services, developing operating system protocols and services. Here are some lessons from this transition so far. Feature begins after you ship the code: Earlier feature design and development stage is where majority of the energy is spent. You write your functional specification, do detailed design , implement the feature, unit test, functional test, end2 end test, stress test and after the feature passes release criterion you release the feature. In machine learned system you need to all of the above, but that's just stage 1 of the feature. Many of these features to work well need large amount of data, for your v1 feature you will not have all the data you need, feature is not yet available in real world so you cannot fully anticipate what will be the real usage of your feature is. 
Read more

Seeings things, simple things

How do you see? How do experience? How do you learn? By changing what you are. Literally changing. We are physical beings first and meta-physical later. The way you grow is to changing some things physically. Your brain, how it morphs to what it receives from its signals will change to suit its needs of signals. What you see really is what counts, what you really hear and understand truly is what counts. Laws of nature has to be simple, because it started simple and basic rules needed to be simple for simple things. Principal of real change comes from real perception. So what we truly see , truly hear , truly believe is what changes. It implies these things practically every day to us. Seek  true beauty. Express your thoughts clearly. Positive, soul sustaining activities. Experiences that enrich. Experiences is all that matter. What also matters is orchestrating these. What a simple rule again. A simple tug of war between signal-change-signa...
Read more