google cloud platform - How to process dataflow two batch files simultaneously on GCP

Question

I want to process two files from gcp to dataflow at the same time simultaneously.

I think it will be possible if one more file comes in side-input.

However, in this case, I think it will be processed every time, not just once.

e.g) How to read and process file1 and file2 at the same time (do I have to put two files in one file and just follow the path?)

I'd appreciate it if you could give me a good example or advice.

Thank you.

question from:https://stackoverflow.com/questions/65911634/how-to-process-dataflow-two-batch-files-simultaneously-on-gcp

Answer 1

If you know the 2 files from the beginning you can simply have a pipeline with 2 entry (fileIO)

I don't know your language, but by design you can do this

     PCollection1                        PCollection2
          |                                   |
   FileIO(readFile1)                   FileIO(readFile2)
          |                                   |
    Transform file                       Transform file
          |                                   |
    WriteIO(sink)                        WriteIO(sink)

You can imagine side input, flatten, group by,... all depends on your needs.