![]() For instance - this is an entirely valid Operator that will use the put_your_image_here image as the base. However - you’re not forced to use the same image as your scheduler and webserver - you can inject whatever image you want - as long as airflow is installed on the image. You can have this be the same image as you run your scheduler with - in this case, the default docker image you run is the same between the workers (that run tasks), the scheduler, and the webserver. My airflow service runs as a kubernetes deployment, and has two containers, one for the webserver and one for the scheduler. When the worker pod runs, a new Airflow instance is instantiated and the code that is being asked (whether shell, python, sql, etc.) is run on the worker Airflow Webserver instance as a LocalExecutor. Before the Kubernetes Executors, all previous Airflow solutions like Celery Executor or Sequential Executor required static workers. With Kubernetes Executors, the workers are dynamic resource allocation. I.e., if the task is listed as running, but no Kubernetes pod is running, killing the task and rerunning a new kubernetes pod. Airflow architecture details (photo by me) Kubernetes Executors. This alleviates the pressure on the main scheduler / webserver instance since the main scheduler is only required to check that a pod was created and sanity checks. The task pod will communicate directly with the SQL backend for your Airflow instance and manage updating the status of the SQL backend’s results. your DAGs in a Kubernetes PersistentVolume, which is mounted to all scheduler/web/worker Pods. With this - each and every task that gets generated by the Executor creates a new Kubernetes pod with the image you specify running a seperate Airflow instance with the same SQL backend for results - the task is then ran in the Webserver of the spawned pod. port-forward the airflow webserver kubectl port-forward. Worker_container_repository = /your_image
0 Comments
Leave a Reply. |
AuthorWrite something about yourself. No need to be fancy, just an overview. ArchivesCategories |