Qubole General Help

Expand all | Collapse all

Qubole Event

  • 1.  Qubole Event

    Posted 21 days ago
    Hello,
    Can we do Event monitor in Qubole? If yes then how can we achieve this?

    Can someone help on this? I need to monitor file creation in AWS s3 location. If that file creates i need to trigger the Qubole workflows.

    Thanks in Advance

    ------------------------------
    Gururaj Raykar
    ------------------------------


  • 2.  RE: Qubole Event

    Quboler
    Posted 21 days ago
    Edited by Minesh Patel 21 days ago
    Hi Gururaj,

    One option is to use AWS Lambda + Qubole API / SDK. Here's a blog which talks about that approach: https://www.qubole.com/blog/on-demand-etl-pipeline-with-amazon-lambda-and-qubole/

    Another option is to use Airflow+File Sensor: https://docs.qubole.com/en/latest/rest-api/sensor_api/file-sensor-api.html

    The airflow option might be a bit involved.

    -minesh

    ------------------------------
    Minesh Patel
    VP, Solutions Architecture
    Qubole
    CA
    ------------------------------