Nasscom Community

AutoML : bridge demand – supply gap for AI solutions

2 Mins read

The rise of digital, technology advances, changing customer preferences made ‘data driven decision making’ an imperative for organizations to compete in the marketplace. Across industries, as organizations undertake the journey of embedding data led insights and intelligence into business processes, the demand

  • supply gap for AI talent is widening. As the wise saying goes – “it’s not about humans vs AI but it’s about humans with AI vs humans without AI”. A manifestation of this saying is ‘Data scientists with AutoML (the process of automating machine learning tasks) vs Data scientists without AutoML’. AutoML bridges the demand – supply gap for AI talent.

AutoML brings productivity gains either through one or a combination of the following ways –

    • Accelerates AI development
    • Enhances bandwidth of Data Scientists:

      • AutoML creates benchmark models or identifies interesting features which can be used by a skilled data scientist
      • Enables substituting low proficiency bandwidth (Citizen data scientists or Junior data scientists)

 

However not all AutoML tools are the same. Tools vary in their sophistication and strength across areas – AI life cycle phases, data handling, technology integration capabilities. To elaborate further on these areas:

  • AI life cycle phases -from data preparation, exploration feature design, model design & development, iterations & history, ranking & selecting models against success metrics, hyper parameter tuning and optimization, validation, testing, deployment to model monitoring
  • Cloud: seamless functioning if it is not from one of the leading Cloud providers
  • Integration/hand-off capabilities & interfaces with other technology stack
  • Depth and breadth of algorithms and techniques
  • Embedded domain knowledge. E.g: Industry/business function specific AutoML tools incorporate certain scenarios & business sense such as seasonality, cannibalization, decay and second order effects, industry accepted analytical approaches to variable treatment and normalization – Campaign design and effectiveness measurement, Fraud detection, Market mix modeling, Pricing, SKU rationalization, Retail Shelf Space Optimization, Route optimization et al

 

Therefore, before selecting any tool- it’s important to assess & examine the following areas and determine the suitability of the AutoML to a particular context. Such assessment also informs right allocation mix of data science & engineering talent and AutoML to solve the problem.

  • Problem statement, business stakeholders, users, business impact, sensitivity & criticality, other interdependent parties
  • Desired skill proficiency for key roles such as Data engineering, Data science, ML engineering
  • Data sets leveraged for solving the problem: internal data, external data, structured, unstructured, video, image, size, newness et al
  • Fitment to enterprise technology stack
  • Nature/phase of AI solution: Proof of concept, prototype, enterprise grade solution

Significant investments, innovation, more consumption, and demand of AI solutions are leading to better AutoML tools that include new capabilities such as synthetic data generation and address ethics and privacy in AI. Enterprise grade AI deployment requires confidence, trust in the model if the model is performing as expected and not introducing any unintended consequences and biases. Typically, such deployments require a collaborative effort among business and data science teams to explain, interpret, debate & review the models & the process. While AutoML tools help scale AI talent either by reducing ‘human development’

 

bandwidth or by substituting with relatively low proficiency bandwidth, humans still need to review and certify readiness of outputs for decision making.

 

*Disclaimer – the views and opinions expressed in this article are completely personal and do not reflect the opinions or beliefs of any current or past institute, the author was/is associated with.