Deploying AI systems using the conventional approach requires a complex infrastructure setup, skilled and expensive professionals, and a significant amount of time. This level of investment, both in terms of finances and time, is feasible for only a select few enterprises that can commit to constructing an entire AI system from start to finish. Yet, the aspiration to harness the advantages inherent in their data and information repositories is universal among all companies.

‍MindsDB is an AI database that offers a comprehensive solution for deploying and managing AI models. It abstracts AI models as virtual tables, bringing AI closer to data sources. It goes further by automating and optimizing the complete AI workflow, starting with data source integration and ML framework configuration, advancing through AI model deployment, and concluding with automation of AI workflows. The outcome? Real-time predictions are at your disposal whenever fresh data arrives.

Employing Jobs for AI Workflow Automation

Let’s start from the beginning by connecting the data source to MindsDB and configuring the ML framework of choice.

‍The CREATE DATABASE statement enables you to connect virtually any data source, including databases, data warehouses, data lakehouses, and even applications.

CREATE DATABASE data_source
WITH
     ENGINE = engine,
     PARAMETERS = {
       "key": "value",
       ...
     };

‍You can choose one of over 100 data integrations and follow instructions on how to connect it to MindsDB with the CREATE DATABASE statement.

‍MindsDB integrates over 10 ML frameworks that can be configured with the CREATE ML_ENGINE statement. For example, if you opt for the OpenAI framework, you may need to provide your OpenAI API key, as below.

CREATE ML_ENGINE openai_engine
FROM openai
USING
     api_key = “your_api_key”;

‍Now that we connected the data source and configured the ML framework, we can go ahead and create a model. The CREATE MODEL statement creates, trains, and deploys models. It abstracts models as virtual tables so you can fetch predictions and forecasts by joining the model with the input data table.

CREATE MODEL ai_model
FROM training_dataset
     (SELECT predict_column, feature_column1, feature_column2, …
      FROM training_table)
PREDICT predict_column
USING
     engine = engine_name;

‍Check out how to create regression models, classification models, and time series models, and how to use large language models (LLMs), such as OpenAI or Hugging Face.

‍Here is how to make batch predictions by joining the model with the input data table:

SELECT input.feature_column1, input.feature_column2, output.predict_column
FROM data_source.data_table AS input
JOIN ai_model AS output;

‍The feature columns from the input data table are fed into the model, which then processes all input data using the underlying algorithm and outputs the predictions for each data row.

‍It is common that in time you may have more training data available and want to enhance the model’s capabilities. MindsDB provides the FINETUNE command that finetunes the model with defined data.

FINETUNE ai_model
FROM data_source
     (SELECT predict_column, feature_column1, feature_column2, …
      FROM training_table
      WHERE created_at > “2023-08-04”);

‍All training data created after Aug 4th, 2023 is going to be utilized for finetuning the model.

‍Alternatively, you can opt for using the RETRAIN command that retrains the model with the entire training dataset, including old and new data. This process takes at least as much time as the initial training of the model, as there is more data available now.

‍Now, if we want to ensure the most accurate predictions, we should automate the process of finetuning the model and saving predictions into a database table that could then be used to visualize results, for example, with Tableau. Let’s see how to do that with jobs.

‍We use the CREATE JOB statement to create and schedule a job.

CREATE JOB ai_workflow_automation (

         -- finetuning a model with new training data
         FINETUNE ai_model
         FROM data_source
              (SELECT predict_column, feature_column1, feature_column2, …
               FROM training_table
               WHERE created_at > “2023-08-04 00:00:00”
               AND created_at > “{{PREVIOUS_START_DATETIME}}”)
         USING
              join_learn_process = true;

         -- creating or replacing an existing table with predictions made by the finetuned model
         CREATE OR REPLACE TABLE predictions (
              SELECT input.feature_column1, input.feature_column2, output.predict_column
              FROM data_source.data_table AS input
              JOIN ai_model AS output;
         )
)
START '2023-09-01 00:00:00'
END '2023-12-01 00:00:00'
EVERY 7 days;

‍The job is scheduled to run once a week starting from September till December. Every time the job runs, it is going to finetune the model with the new training data, as defined by created_at > “{{PREVIOUS_START_DATETIME}}”. We use the {{PREVIOUS_START_DATETIME}} variable that will be replaced by the timestamp of the previous job run to ensure that only the new training data is used to finetune the model. The join_learn_process parameter ensures that the finetuning process must complete before we query for predictions.

‍The next part of the job is to create a new data table or replace an existing one and save predictions made by the newly fine-tuned model. This ensures that we utilize all the resources (here, all the historical data) to make sure the predictions are accurate.

‍Check out more examples of automating AI workflows by following our tutorials like Twitter chatbot or Slack chatbot.

Elevate Productivity through AI Workflow Automation

MindsDB simplifies AI workflows, making them accessible and intuitive for all developers. Furthermore, it introduces AI automation, allowing you to relax while your data works to yield advantageous outcomes.

‍Take a hands-on approach to exploring MindsDB by creating a demo account on MindsDB Cloud or installing MindsDB locally (via pip or Docker). And if you plan to utilize MindsDB for production systems, we recommend considering MindsDB Starter which provides managed instances, ensuring greater security and scalability for your projects.

‍Whichever option you choose, MindsDB provides flexibility and ensures a smooth experience for all your AI projects.