In this article, you will see how MindsDB fully automates AI workflows, connecting any source of data with any AI/ML model, and enabling the flow of real-time data and predictions between them.

‍One of the core components of MindsDB is a Job, a command that allows anything in MindsDB to be run either on a timer (e.g. every day) or based on an event (e.g. when a new row is added to my DB).

‍Read along to explore how using Jobs radically simplifies building Applied AI use cases:

‍JOBS 101

MindsDB can connect to any source of data - a database, warehouse, stream or app. Here, we’ll connect to Binance (a crypto exchange) to get a feed of real-time price information:.

CREATE DATABASE my_binance
WITH ENGINE = 'binance';

That’s it, you now have access to the Binance API! Now you can take a look at the data for the crypto pair that will be ultimately used::

SELECT *
FROM my_binance.aggregated_trade_data
WHERE symbol = 'BTCUSDT'
LIMIT 10;

But what if you wanted to automate pulling the data every day? You guessed it, this is where a JOB can come in handy to run this over and over. You just wrap the query with the previous CREATE JOB syntax:

CREATE JOB binance_data (
SELECT *
FROM my_binance.aggregated_trade_data
WHERE symbol = 'BTCUSDT'
LIMIT 10
)
END '2024-02-01 00:00:00'
EVERY 1 day;

There you have it - you’ve learned what a JOB is and how they can work by automating getting the data from Binance every minute. This is just a super-simple toy example; in practice you might want to save SELECT output into a VIEW for this functionality as you move to building a fully automated end-to-end Crypto forecasting solution.‍

Real-Time Trading Data Forecasts

MindsDB enables users to automate every element of a production AI workflow by scheduling Jobs. A Job can be run on either a time-based or data-based trigger, depending on what your workflow calls for.

‍In this tutorial, you will walk through automating real-time forecasts like a Cryptocurrency trader for BTC/USDT trading pair. You can then automate the sending of these forecasts to Slack.

‍Every time the job runs, it will retrain a time-series model using the latest trade data from Binance. Then, it will insert real-time predictions into your Slack every 5 minutes to get you up to date with the latest developments of the BTC/USDT trading pair.

CREATE JOB btcusdt_forecasts_to_slack (

  -- step 1: retrain the model with new data to improve its accuracy
  RETRAIN cryptocurrency_forecast_model
  FROM my_binance
    (
      SELECT *
      FROM aggregated_trade_data
      WHERE symbol = 'BTCUSDT'
    )
  USING
      join_learn_process = true;

  -- step 2: make fresh forecasts for the following 10 minutes and insert it into slack
  INSERT INTO btcusdt_slack_app.channels (channel, text)
  VALUES("btcusdt-real-time-forecasts", "Here are the BTCUSDT forecasts for the next 10 minutes:");

  INSERT INTO btcusdt_slack_app.channels (channel, text)
    SELECT "btcusdt-real-time-forecasts" AS channel, concat('timestamp: ', cast(to_timestamp(cast(m.open_time as bigint)) as string), ' -> open price: ', m.open_price) AS text
    FROM btcusdt_recent AS d
    JOIN cryptocurrency_forecast_model AS m
    WHERE d.open_time > LATEST;

)
EVERY 5 minutes;

‍That is how you can automate the entire AI workflow - use the CREATE JOB statement to start a job that executes queries defined within it.

Here is the Slack output of one job run:

Follow along to find out more about Binance and Slack connectors and the time-series model used to forecast the open price values.

‍Alternatively, you can watch the video tutorials ([1] and [2]) that walk you through the process of setting up data and a model, connecting Slack, and automating real-time forecasts as Slack notifications.

Job Components

Read along to learn more about all the components used in the job, including the connector to Binance data, the time-series model, and the connector to Slack.

Binance Data Connector

The training data is required to train a time-series model. This tutorial uses data from Binance, a company that operates a cryptocurrency exchange.

‍MindsDB integrates with Binance and enables users to fetch real-time data. Here is how to connect Binance to your MindsDB account.

CREATE DATABASE my_binance
WITH ENGINE = 'binance';

It doesn’t require any additional parameters, so now you can query for Binance data.

SELECT *
FROM my_binance.aggregated_trade_data
WHERE symbol = 'BTCUSDT';

This tutorial focuses on the BTC/USDT trading pair.

Time-Series Model

The Binance trade data is updated every minute. Therefore, the time-series model is going to predict the open prices of the BTC/USDT trading pair for the next 10 minutes.

CREATE MODEL cryptocurrency_forecast_model
FROM my_binance
  (
    SELECT *
    FROM aggregated_trade_data
    WHERE symbol = 'BTCUSDT'
  )
PREDICT open_price
ORDER BY open_time
WINDOW 100
HORIZON 10;

‍Use the CREATE MODEL statement to create, train, and deploy a model. The FROM clause defines the training data used to train the model - here, the latest Binance data is used. The PREDICT clause specifies the column to be predicted - here, the open price of the BTC/USDT trading pair is to be forecasted. As it is a time-series model, you should order the data by a date column - here, it is the open time when the open price takes effect. Finally, the WINDOW clause defines the window the model looks back at while making forecasts - here, the model looks back at sets of 100 rows (intervals of 100 minutes). The HORIZON clause defines how many rows into the future the model will forecast - here, it forecasts the next 10 rows (the next 10 minutes).

‍After executing the CREATE MODEL statement as above, you can check the progress status using this query:

DESCRIBE cryptocurrency_forecast_model;

Once the status reads complete, you can proceed to make forecasts.

‍First, save the Binance data into a view, which will be joined with the model to provide the latest Binance trade data.

CREATE VIEW btcusdt_recent AS (
  SELECT *
  FROM my_binance.aggregated_trade_data
  WHERE symbol = 'BTCUSDT'
);

‍Next, query for forecasts by joining the model with the Binance input data table.

SELECT to_timestamp(cast(m.open_time as bigint)) AS open_time,
       m.open_price,
       m.open_price_explain
FROM btcusdt_recent AS d
JOIN cryptocurrency_forecast_model AS m
WHERE d.open_time > LATEST;

Here is a sample output:

Please note that the Binance data is updated every minute, so every time you query the model, you will get forecasts for the following 10 minutes.

Slack Connector

This tutorial automates not only getting real-time forecasts but also notifying you about it on Slack. So, let’s connect your Slack workspace to MindsDB.

‍Follow this instruction to set up your Slack app and generate the Slack bot token. Once you get the Slack bot token and integrate your Slack app into one of the Slack channels, you can connect it to MindsDB.

CREATE DATABASE btcusdt_slack_app
WITH
  ENGINE = 'slack',
  PARAMETERS = {
      "token": "xoxb-..."
    };

Here is how to send messages to a Slack channel:

INSERT INTO btcusdt_slack_app.channels (channel, text)
VALUES("btcusdt-real-time-forecasts", "BTCUSDT forecasts coming soon.");

Let’s put it all together again.

Automate Real-Time Forecasts with MindsDB

Now that you connected Binance to MindsDB and used its trade data to train the time-series model, you can set up a job that will be retraining this time-series model periodically using the latest Binance data. It is to keep improving the model’s accuracy and performance.

‍In the second step, the job will insert real-time forecasts of the BTC/USDT trading pair for the next 10 minutes and notify you on Slack.

‍The job will be scheduled to execute every 5 minutes. Thus, the forecasts for the last 5 rows of the current job execution will overlap with the forecasts for the first 5 rows of the next job execution.

CREATE JOB btcusdt_forecasts_to_slack (

  -- step 1: retrain the model with new data to improve its accuracy
  RETRAIN cryptocurrency_forecast_model
  FROM my_binance
    (
      SELECT *
      FROM aggregated_trade_data
      WHERE symbol = 'BTCUSDT'
    )
  USING
      join_learn_process = true;

  -- step 2: make fresh forecasts for the following 10 minutes and insert it into slack
  INSERT INTO btcusdt_slack_app.channels (channel, text)
  VALUES("btcusdt-real-time-forecasts", "Here are the BTCUSDT forecasts for the next 10 minutes:");

  INSERT INTO btcusdt_slack_app.channels (channel, text)
    SELECT "btcusdt-real-time-forecasts" AS channel, concat('timestamp: ', cast(to_timestamp(cast(m.open_time as bigint)) as string), ' -> open price: ', m.open_price) AS text
    FROM btcusdt_recent AS d
    JOIN cryptocurrency_forecast_model AS m
    WHERE d.open_time > LATEST;

)
EVERY 5 minutes;

Let’s look at the Slack output after a couple of job executions.

The job sends messages to the Slack channel where the Slack app was integrated. There you have it - a fully automated end-to-end alert system!

Develop and Automate AI Workflows with Ease

MindsDB enables users to develop and automate AI workflows easily. You can integrate data from over 100 data sources, including popular databases, like PostgreSQL, MySQL, MS SQL Server, and MongoDB, and applications, such as Slack, Twitter, Shopify, YouTube, and more. Moreover, you can choose from over 15 ML frameworks, including OpenAI, Nixtla, LangChain, LlamaIndex, and more, to create and deploy AI models with a single command.

‍Take a hands-on approach to exploring MindsDB by creating a demo account on MindsDB Cloud or installing MindsDB locally (via pip or Docker). And if you plan to use MindsDB for production systems, we recommend considering MindsDB Starter which provides managed instances, ensuring greater security and scalability for your projects.

‍Whichever option you choose, MindsDB provides flexibility and ensures a smooth experience for all your AI projects.